From 775507bad433d1bb1503bc078f74556fc56df622 Mon Sep 17 00:00:00 2001 From: Paco Date: Fri, 28 Aug 2026 18:04:17 -0700 Subject: [PATCH 01/15] docs: RFC for carrying sysext/confext images in COSI files Proposes an optional `extensions` section in COSI metadata (revision 1.3) holding `sysexts` and `confexts` arrays. Each entry reuses the existing `ImageFile` object to reference a ZSTD-compressed DDI under `images/extensions/`, plus an optional destination path mirroring the Host Configuration `Extension` object. Covers the metadata schema, tar layout, versioning and forward-compatibility, Trident-side consumption (effective extension set across the extensions, osconfig and selinux subsystems), A/B update and rollback semantics, the Host Configuration interaction, and validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0000-cosi-extension-images.md | 774 ++++++++++++++++++ 1 file changed, 774 insertions(+) create mode 100644 docs/Development/RFCs/0000-cosi-extension-images.md diff --git a/docs/Development/RFCs/0000-cosi-extension-images.md b/docs/Development/RFCs/0000-cosi-extension-images.md new file mode 100644 index 0000000000..186cca3988 --- /dev/null +++ b/docs/Development/RFCs/0000-cosi-extension-images.md @@ -0,0 +1,774 @@ +# 0000 COSI Extension Images + +- Date: 2026-08-28 +- RFC PR: [microsoft/trident#0000](https://github.com/microsoft/trident/pull/0000) +- Issue: [microsoft/trident#0000](https://github.com/microsoft/trident/issues/0000) + +## Summary + +This RFC proposes carrying systemd system extension (sysext) and configuration +extension (confext) images inside a COSI file. Extension images become ZSTD +compressed members of the COSI tar, exactly like partition images, and a new +optional `extensions` section in `metadata.json` describes them. Each entry +reuses the existing [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) +object to point at the payload in the tar, and carries the destination path on +the target OS, mirroring the existing +[`Extension`](../../Reference/Host-Configuration/API-Reference/Extension.md) +object in the Host Configuration. Trident then deploys those extensions as part +of the image, in the same operation and the same reboot as the rest of the OS, +instead of fetching them from a separate location at deploy time. + +## Motivation and Goals + +Today the only way to get an extension onto a Trident-managed host is +`os.sysexts` / `os.confexts` in the Host Configuration, where each entry is a +URL plus a SHA-384. That works, but it makes the extension a second artefact +with a second lifecycle: + +- **A second thing to host.** The operator must publish the `.raw` DDI + somewhere Trident can reach (`http://`, `https://`, `file://` or `oci://`) + and keep it there for as long as any host might re-run the flow. +- **A second download at deploy time.** The COSI is streamed sparsely from one + endpoint; each extension is fetched in full from another. Two endpoints means + two sets of network, proxy, TLS and authentication failure modes on the + provisioning path. +- **A second integrity story.** The COSI's integrity is anchored by the + `metadata.json` hash in `os.image.sha384`, which transitively covers every + partition image. An extension's hash is anchored by whoever wrote the Host + Configuration. Bundling puts the extension under the same root of trust as + the rest of the OS. +- **A second update.** This is the important one. Extension changes are + currently a Runtime Update, distinct from the A/B update that ships the OS. + Delivering operator content that must land alongside a new OS therefore means + two operations, two rollout gates and potentially two disruptions. + +### A/B Updates Are the Strategic Point + +If extensions ride the COSI, then operator content is delivered and activated +in the *same* A/B update and the *same* reboot as the OS. It reuses the +rollout, [health-gating](../../Reference/Host-Configuration/API-Reference/Health.md) +and rollback machinery that already exists for images, rather than needing a +parallel channel with its own staging, verification and failure semantics. + +Rollback becomes trivially correct: extension images are files inside the +target slot's own filesystem, so the previous slot still holds the previous +extension set, untouched. Rolling back to the previous root volume rolls back +the extensions with it. See [Rollback](#rollback). + +### Why Not Just Bake the Files Into the Root Filesystem Image? + +An obvious objection: the image builder controls the root filesystem image, so +it could simply place `my-tool.raw` in `/var/lib/extensions/` before the COSI +is built, with no spec change at all. That is a genuine alternative and it +works for some configurations. It loses on four counts: + +1. **Verity.** With root or usr verity, any change to the protected filesystem + changes the root hash and requires re-signing. Extensions live outside the + protected tree precisely so they can change independently; the mechanism + that makes them attractive is defeated if adding one means rebuilding and + re-signing the base image. +2. **Destination volumes that are not image-provisioned.** `/var/lib/extensions/` + is commonly on a separate volume, which may be newly created rather than + written from a partition image. Files baked into the image's `/var` never + arrive in that case. +3. **Opacity.** Bytes inside a filesystem image are invisible to anything that + reads COSI metadata. An `extensions` section makes the extension set + inspectable, hashable and enumerable without mounting anything, which + matters for inventory and supply-chain tooling. +4. **Trident does not know.** Trident enables `systemd-sysext.service` and + `systemd-confext.service`, validates placement, and reads `extension-release` + only for extensions it knows about. Files that appear inside a filesystem + image bypass all of that, and the merge services are never enabled. + +### Goals + +- Extension images can be carried in a COSI file and deployed by Trident with + no second artefact, no second endpoint and no second update. +- The metadata addition is small, reuses existing objects, and looks familiar + to anyone who already knows `os.sysexts` / `os.confexts`. +- The Host Configuration public API does not change. + +## Scope + +### Requirements + +- Add an optional `extensions` object to the COSI metadata root, containing + `sysexts` and `confexts` arrays. +- Each entry MUST identify its payload in the tar via an `ImageFile` object and + MAY specify a destination path on the target OS. +- Extension payloads are ordinary ZSTD-compressed tar members, covered by the + existing compression and integrity rules. +- Trident deploys COSI-borne extensions during Clean Install and A/B Update, + reusing the existing extensions subsystem. +- COSI-borne extensions and Host Configuration extensions coexist; conflicts + are a hard error. +- New COSI validation errors for the extension section. + +### Out of Scope + +- Changing the Host Configuration `Extension` object. It stays exactly as it + is. +- Runtime Update of COSI-borne extensions. A COSI-borne extension can only + change when the image changes, which is an A/B update. +- Signing or attestation of individual extension images beyond the SHA-384 + chain already provided by COSI. +- Making extensions compatible with SELinux. That limitation is orthogonal and + unchanged; see [SELinux](#selinux). +- Building the DDIs. This RFC specifies how a COSI carries an extension image, + not how the extension image is produced. + +### Exit Criteria + +- COSI revision 1.3 is published with the `extensions` section, a + `cosi-metadata-v1.3.schema.json`, and valid/invalid samples under + `tests/cosi/metadata_samples/v1.3/`. +- Trident deploys a COSI carrying at least one sysext and one confext on Clean + Install and on A/B Update, and the extensions are merged after reboot. +- A/B rollback returns the host to the previous extension set with no extra + work. +- Conflicts between COSI-borne and Host-Configuration-borne extensions produce + a structured error. + +## Dependencies + +- A COSI writer able to emit the section. [Image Customizer](https://github.com/microsoft/azure-linux-image-tools) + is the reference implementation of the COSI writer side. +- No incomplete Trident features are required. + +## Implementation + +### Tar Layout + +Extension payloads are ZSTD-compressed DDI files stored under +`images/extensions/`, for example `images/extensions/my-tool.rawzst`. + +They stay under `images/` deliberately. The COSI specification already states +that region images "MAY be placed in subdirectories of `images/` to organize +them" and that "Readers MUST be able to handle images in subdirectories", so no +new tar-layout rule is needed. More importantly, `ImageFile.path` is +constrained by `"pattern": "^images/.+"` in every published schema. Relaxing +that pattern to also permit `extensions/` would mean a v1.3 `ImageFile` that +fails validation against the v1.0–1.2 schemas, breaking the one object this +proposal is supposed to reuse unchanged, and breaking any generic COSI +validator that has an `ImageFile` definition compiled in. There is no +compensating benefit: a subdirectory conveys the same organisation. + +Two ordering constraints from the existing spec must be respected: + +- Since revision 1.2, the primary GPT image MUST be the file immediately after + `metadata.json`. Extension payloads MUST NOT be placed between them. +- Region images MUST appear in the tar in the physical order of the regions on + the source disk. Extension payloads are not regions, so they do not + participate in that ordering, but interleaving them would make the region + ordering harder to verify and would hurt sparse-read locality. Extension + payloads SHOULD therefore be written after all region images. + +Extension payloads are compressed with ZSTD like everything else, and their +window log MUST be accounted for in the existing root `compression.maxWindowLog` +field. No change is needed to that object; writers simply must not forget the +new images when computing the maximum. + +Older readers are unaffected by the extra tar members. The spec already says +"the tar file MAY contain other files, but Trident MUST ignore them", and +Trident's orphan-image check +(`V1_2ImageFileHasNoCorrespondingPartition`) is driven by the `images[]` and +`disk.gptRegions[]` metadata arrays, not by walking tar entries, so extra +members under `images/` do not trip it. + +### Metadata Schema + +A new optional root field `extensions`: + +| Field | Type | Added in | Required | Description | +| ------------ | ------------------------------------- | -------- | -------- | ----------------------------------------------- | +| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images carried by this COSI file. | + +#### `Extensions` Object + +| Field | Type | Added in | Required | Description | +| ---------- | ------------------------------------------ | -------- | -------- | ---------------------------------- | +| `sysexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | System extension images. | +| `confexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | Configuration extension images. | + +#### `ExtensionImage` Object + +| Field | Type | Added in | Required | Description | +| ------- | ------------------------------ | -------- | --------------- | -------------------------------------------------------- | +| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | Details of the compressed extension image in the tar. | +| `path` | string | 1.3 | No | Absolute destination path of the extension on the target OS. | + +JSON Schema fragment: + +```json +{ + "properties": { + "extensions": { + "description": "Extension images carried by this COSI file.", + "$ref": "#/$defs/Extensions" + } + }, + "$defs": { + "Extensions": { + "type": "object", + "properties": { + "sysexts": { + "description": "System extension images to place on the target OS.", + "type": "array", + "items": { "$ref": "#/$defs/ExtensionImage" } + }, + "confexts": { + "description": "Configuration extension images to place on the target OS.", + "type": "array", + "items": { "$ref": "#/$defs/ExtensionImage" } + } + } + }, + "ExtensionImage": { + "type": "object", + "required": ["image"], + "properties": { + "image": { + "description": "Details of the compressed extension image file in the tar file.", + "$ref": "#/$defs/ImageFile" + }, + "path": { + "description": "Absolute path of the extension image on the target OS. The file name MUST be `{name}.raw`, where `{name}` matches the suffix of the `extension-release.{name}` file inside the image. When omitted, the reader places the image in its default directory for the extension kind.", + "type": "string", + "pattern": "^/.+\\.raw$" + } + } + } + } +} +``` + +Worked example: + +```json +{ + "version": "1.3", + "osArch": "x86_64", + "osRelease": "NAME=\"Microsoft Azure Linux\"\nID=azurelinux\nVERSION_ID=\"3.0\"\n", + "images": [ + // Filesystem objects, unchanged. + ], + "disk": { + // Disk object, unchanged. + }, + "osPackages": [ + // OsPackage objects, unchanged. + ], + "bootloader": { + // Bootloader object, unchanged. + }, + "compression": { "maxWindowLog": 22 }, + "extensions": { + "sysexts": [ + { + "image": { + "path": "images/extensions/gpu-driver.rawzst", + "compressedSize": 41943040, + "uncompressedSize": 134217728, + "sha384": "3a1f9c0d4e5b6a7c8d9e0f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e" + }, + "path": "/var/lib/extensions/gpu-driver.raw" + }, + { + "image": { + "path": "images/extensions/debug-tools.rawzst", + "compressedSize": 8388608, + "uncompressedSize": 33554432, + "sha384": "b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1" + } + } + ], + "confexts": [ + { + "image": { + "path": "images/extensions/fleet-config.rawzst", + "compressedSize": 262144, + "uncompressedSize": 1048576, + "sha384": "c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2" + }, + "path": "/var/lib/confexts/fleet-config.raw" + } + ] + } +} +``` + +#### Why Two Arrays Instead of One Array With a `kind` + +Two arrays, mirroring `os.sysexts` / `os.confexts`. + +- The distinction is not cosmetic. Sysexts and confexts have disjoint sets of + permitted destination directories (`VALID_SYSEXT_DIRECTORIES` vs + `VALID_CONFEXT_DIRECTORIES`), disjoint defaults, different + `extension-release` locations (`/usr/lib/extension-release.d/` vs + `/etc/extension-release.d/`), different identity fields (`SYSEXT_ID` vs + `CONFEXT_ID`) and different activation units. Every rule downstream branches + on the kind. +- With two arrays the kind is structural, so no `kind` field is needed and it + cannot be omitted or wrong. +- It is the shape the Host Configuration already uses, which is the stated + design constraint. A reader that already has code for `os.sysexts` has code + for this. + +The one-array-plus-discriminator form would be preferable if the two kinds +shared their rules and the set of kinds were open-ended. Neither is true here. + +#### What Each Entry Carries, and What It Does Not + +`image` and an optional `path`. That is the whole object. Everything else that +was considered is derivable, and every derivable field is a new opportunity for +two sources of truth to disagree. + +- **`path` (kept, optional).** Not derivable: it is the one genuinely new piece + of information, the image author's intent about where the file goes. Optional + because the Host Configuration's `Extension.path` is optional and defaults to + `/var/lib/extensions/{name}.raw` or `/var/lib/confexts/{name}.raw`; making it + required here would be a gratuitous divergence, and the default is what most + writers want anyway. When present it MUST be absolute, MUST end in `.raw`, + MUST sit in a permitted directory for its kind, and its file name MUST match + the `extension-release` suffix — all rules that already exist. +- **Extension name (rejected).** Trident derives the name by mounting the DDI + and reading the `extension-release.{name}` file name, and already enforces + that the destination file name matches it. A `name` field would be a third + copy of the same string (DDI, `path`, `name`) and a third thing to + cross-check. +- **`kind` (rejected).** Structural, see above. +- **Extension ID (`SYSEXT_ID` / `CONFEXT_ID`) (rejected).** Read from + `extension-release`. It is the identity Trident already keys A/B state on; + duplicating it in metadata invites drift with the payload. +- **Enable-on-first-boot (rejected).** systemd merges everything it finds in + the extension directories; there is no per-extension enable switch in the + Host Configuration today, and inventing one here would create COSI-only + semantics that the Host Configuration cannot express. If a per-extension + toggle is wanted, it should be designed once for both surfaces. +- **Version / OS-compatibility metadata (rejected).** `ID`, `VERSION_ID`, + `SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE` and `SYSEXT_SCOPE` are + already inside the DDI's `extension-release`, and systemd — not Trident — is + the authority that enforces them at merge time. Copying them into metadata + would let a COSI claim compatibility the payload does not have. +- **A separate hash of the uncompressed DDI (rejected, with a caveat).** See + below. + +#### A Note on `sha384` Semantics + +`ImageFile.sha384` is the hash of the **compressed** image. The Host +Configuration's `Extension.sha384` is the hash of the **raw** extension image +file. Reusing `ImageFile` therefore changes what the recorded hash covers. + +For integrity that is fine, and it is exactly what already happens for +partition images: Trident hashes the compressed stream as it decompresses, so a +match proves the payload is intact. Adding a second hash of the uncompressed +DDI would buy nothing for integrity. + +There is one caveat worth recording. Trident's `ExtensionData.sha384` is also +used as a change detector: for an extension whose ID exists in both the old and +the new configuration, a differing hash means "content changed, replace it". +Compressed hashes are not stable across recompressions, so two COSIs built from +byte-identical DDIs at different ZSTD levels would look "changed". The +consequence is a redundant copy of an identical file, never a missed update, so +this is a cosmetic inefficiency rather than a correctness problem. It is called +out here so it is not rediscovered as a bug. + +### Versioning + +This lands in **COSI revision 1.3**. `extensions` is optional; absent means the +same as empty. + +**Newer Trident, older COSI (1.0–1.2).** No `extensions` field, no extensions +from the image, current behaviour exactly. Nothing to do. + +**Older Trident, newer COSI (1.3 with extensions).** Trident's version check +only rejects `major != 1`, and both the spec and the metadata parser require +unknown fields to be ignored. An already-shipped Trident will therefore accept +a 1.3 COSI, deploy the OS correctly, and **silently omit the extensions**. + +That is the honest and slightly uncomfortable answer, and it cannot be fixed +in-band for readers that already exist: any tripwire field we add is, by +definition, a field they are required to ignore. Three mitigations, none of +which is a fix: + +1. **Document it** as a property of the format: consuming extensions requires a + reader that understands revision 1.3. +2. **Warn from now on.** `validate_cosi_metadata_version` should log a warning + when `minor` exceeds the highest revision the binary knows about. This does + not help binaries already in the field, but it makes every future minor bump + diagnosable from the log. +3. **Gate the rollout.** An operator who cannot control the Trident version on + the target should add a + [health check](../../Reference/Host-Configuration/API-Reference/Health.md) + asserting the extension is merged. An A/B update that lands on a too-old + Trident then fails its health gate and rolls back, instead of quietly + producing a host missing its extensions. + +### Trident-Side Consumption + +The core idea is that the extensions subsystem gains a second *source* for +extensions, and nothing else about it changes. Concretely, Trident computes an +**effective extension set** = Host Configuration entries ∪ COSI entries, and +the places that today read `ctx.spec.os.sysexts` / `ctx.spec.os.confexts` read +the effective set instead. + +```mermaid +flowchart LR + HC["Host Configuration
os.sysexts / os.confexts"] --> Eff + COSI["COSI metadata
extensions.sysexts / .confexts"] --> Eff + Eff["Effective extension set"] --> Ext["ExtensionsSubsystem
stage, mount, read extension-release, place"] + Eff --> Osc["osconfig subsystem
enable systemd-sysext / systemd-confext"] + Eff --> Sel["selinux subsystem
enforcing-mode rejection"] +``` + +The three consumers of the effective set are worth naming individually, +because two of them are easy to miss: + +- **`ExtensionsSubsystem`.** `populate_extensions` currently downloads each + Host Configuration entry with a `FileReader` over `Extension.url` into the + staging directory, verifies the hash, then mounts the DDI to read + `extension-release`. For a COSI-borne entry the only difference is where the + bytes come from: the image streaming pipeline decompresses the tar member + into the same staging directory and verifies `ImageFile.sha384` over the + compressed stream. Everything downstream — the `extension-release` read, the + `{name}.raw` file-name check, the default-path logic, directory creation, + and the placement/removal logic in `set_up_extensions` — is unchanged. +- **`osconfig`.** `systemd-sysext.service` and `systemd-confext.service` are + enabled only when `ctx.spec.os.sysexts` / `confexts` are non-empty. If a host + gets its extensions solely from the COSI and its Host Configuration lists + none, the merge services would never be enabled and the extensions would sit + on disk doing nothing. This check **must** move to the effective set. It is + the single most likely way to ship this feature broken. +- **`selinux`.** The dynamic validation that rejects `enforcing` mode when + extensions are configured has the same shape and the same problem. It must + also move to the effective set, otherwise a COSI carrying extensions plus an + enforcing-mode Host Configuration passes validation and produces a host with + a mislabelled `/usr`, `/opt` or `/etc`. + +`derive_host_configuration` (used by [disk streaming](../../Explanation/Disk-Streaming.md)) +does not need to synthesise extension entries; the effective set is computed +from the COSI directly, so the derived-Host-Configuration path gets extensions +for free. + +#### Where Images Are Written + +- **Clean Install and A/B Update.** `provision()` runs with the target root + mounted at `mount_path`. Extensions are staged under + `{mount_path}/var/lib/extensions/.staging` and then moved to their + destination inside the target root, which is exactly what happens today. For + A/B, that destination is on the *inactive* slot, so the running system is + untouched until the reboot. +- **Runtime Update.** COSI-borne extensions do not participate. A COSI-borne + extension can only change if the COSI changes, and a COSI change is an image + change, which forces an A/B update. Host Configuration extensions continue to + be runtime-updatable exactly as today. + +This gives a useful invariant: **Trident never needs to read the previous +COSI.** If `spec.os.image` equals `spec_old.os.image`, the old COSI's extension +set is the new COSI's extension set. If they differ, the servicing type is an +A/B update, and `set_up_extensions` already skips removal of the old set on +Clean Install and A/B Update because the old files live in the other slot. + +#### Rollback + +Nothing needs retaining, because nothing is shared. + +Extension images are ordinary files inside the slot's filesystem, and Trident +already requires that when A/B volumes are configured, every extension +destination is on an A/B volume and not on a shared or read-only one. So the +previous slot's `/var/lib/extensions/` still contains the previous extension +set, byte for byte, after the update. An A/B rollback boots the previous root +volume, `systemd-sysext` merges what it finds there, and the extension set +reverts with the OS. No extra bookkeeping, no staging area to preserve, no +"previous extension" directory. + +This is the same property that makes A/B updates work for the OS itself, and it +is the main reason bundling is attractive: the extension inherits the rollback +semantics of the image instead of needing its own. + +#### Interaction With Host Configuration `sysexts` / `confexts` + +**Merge, with a hard error on collision.** + +The two lists express different intents and both are legitimate at the same +time. COSI extensions are content the *image author* considers part of this OS +— a GPU driver, a kernel-module extension. Host Configuration extensions are +content the *operator* adds at deploy time — a monitoring agent, a site-specific +confext. Forbidding both would be gratuitous: the natural case is an image that +ships one sysext and an operator who adds another. + +Silent precedence is the option to avoid. If a Host Configuration entry +silently overrode a bundled one, the host would run software the image author +never validated, and the only symptom would be a version number in a log +somewhere. Failing loudly is better than being quietly wrong. + +A collision is defined two ways, both already meaningful in the Host +Configuration today: + +1. **Same destination path.** Detectable statically whenever both sides specify + `path`. This extends the existing `DuplicateExtensionImagePath` rule across + the merged set. +2. **Same extension ID** (`SYSEXT_ID` / `CONFEXT_ID` within a kind). Only + detectable after the DDIs are mounted, which the subsystem does anyway. Note + that ID uniqueness is documented for the Host Configuration today but not + enforced in code; the merged set makes collisions more likely, so this + should become an enforced check. + +Two entries that are identical in every respect are still a collision. Making +"identical is fine" an exception invites hash-comparison subtleties for no real +benefit — the operator can simply drop their entry. + +If a per-extension override is later shown to be necessary, it should be an +explicit opt-in on the Host Configuration side, not a default. That is listed +as an [open question](#open-questions). + +### Validation + +#### COSI Metadata Validation + +New `CosiMetadataErrorKind` variants, following the existing `V1_` +prefix convention: + +| Variant | Condition | +| -------------------------------------------------- | -------------------------------------------------------------------------------------------- | +| `V1_3ExtensionDestinationPathNotAbsolute` | `path` is present and is not absolute. | +| `V1_3ExtensionDestinationPathInvalidFileExtension` | `path` is present and does not end in `.raw`. | +| `V1_3ExtensionDestinationPathInvalidDirectory` | `path`'s parent is not in `VALID_SYSEXT_DIRECTORIES` / `VALID_CONFEXT_DIRECTORIES` for the kind. | +| `V1_3DuplicateExtensionDestinationPath` | Two entries resolve to the same destination path. | +| `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | +| `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | + +The existing "every image referenced by the metadata is present in the tar" +check must be extended to walk `extensions.sysexts[]` and +`extensions.confexts[]` as well, so a metadata entry pointing at a missing tar +member fails at load time rather than mid-provision. + +Directory and file-extension rules deliberately reuse the constants and logic +behind `Extension::validate_sysext` / `validate_confext` rather than +re-implementing them, so the two surfaces cannot drift. + +#### Deploy-Time Validation + +- **File name matches the `extension-release` suffix.** Already enforced: + `read_extension_release` requires `path` to end with `{name}.raw` where + `{name}` is the suffix of the `extension-release.{name}` file found inside + the DDI. Applies unchanged to COSI-borne entries. +- **Exactly one `extension-release` file, and `SYSEXT_ID` / `CONFEXT_ID` + present.** Already enforced, applies unchanged. +- **SHA-384 agreement.** `ImageFile.sha384` is verified over the compressed + stream as the payload is decompressed into the staging directory, using the + same hashing reader used for partition images. Mismatch aborts servicing. +- **Extension ID uniqueness across the merged set**, per kind. New check. +- **`extension-release` OS compatibility.** Warn, do not fail, when a bundled + extension declares `ID=` and that `ID`/`VERSION_ID` does not match + the COSI's own `osRelease`. This is a build-time mistake worth surfacing + early — systemd will refuse to merge such an extension at boot and the host + will come up silently missing it — but systemd's matching rules + (`SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE`, `_any`) are subtle enough + that Trident should not be the authority. Warning early and letting systemd + decide is the right split. + +#### On `ID=_any` + +`extension-release` already declares which cadence case an extension is in: +`ID=` plus `VERSION_ID` / `SYSEXT_LEVEL` means "this extension is bound +to this OS build", while `ID=_any` means "this extension is portable across OS +versions". + +Trident should **not** warn or refuse when an `ID=_any` extension is bundled. +`_any` describes what the extension is *compatible with*, not how it *should be +delivered*. Bundling a portable extension is a perfectly reasonable packaging +choice — it is how you get an air-gapped or single-artefact deployment — and +refusing it would break legitimate use for no safety benefit. The genuinely +useful check is the opposite one described above: a bundled `ID=` +extension that does not match the COSI's own `osRelease` is almost certainly a +build error. + +### SELinux + +Unchanged. Extensions remain incompatible with SELinux in enforcing mode on +systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. +Carrying the image in a COSI does not touch the labelling problem in any way. + +The one thing that must change is *where the check looks*: as noted above, the +dynamic validation that raises `ExtensionImagesAndSelinuxUnsupported` currently +inspects the Host Configuration lists only, and must inspect the effective set, +or a COSI-only extension configuration would bypass it entirely. + +## Public API Design + +The COSI metadata format is a public contract with an independent +[specification](../../Reference/Composable-OS-Image.md) and published JSON +Schemas, so the additions above constitute the public API change. They are +additive and optional. + +The Host Configuration API does **not** change. `os.sysexts` and `os.confexts` +keep their meaning and their shape. The observable behavioural difference for +an existing user is that a host may now end up with more extensions than its +Host Configuration lists, and that specifying an extension that the image +already carries is now an error rather than a duplicate. + +## Testing and Metrics + +- **Schema tests.** Add `docs/Reference/Composable-OS-Image/cosi-metadata-v1.3.schema.json` + and samples under `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`. The + existing schema-validation workflow picks up a new revision automatically + given the schema file and a matching samples directory. Invalid samples + should cover: `image.path` outside `images/`, non-absolute destination, + destination not ending in `.raw`, destination in a disallowed directory, and + a duplicated destination path. +- **Unit tests.** Metadata parse and validate for each new error variant; + effective-set computation, including the collision cases; default-path + resolution for a COSI entry with no `path`. +- **Functional tests.** Build a COSI carrying one sysext and one confext; + assert Clean Install places both, enables the merge units, and that both are + merged after boot. +- **Servicing tests.** A/B update from a COSI with extension set A to a COSI + with extension set B; assert the new slot has B, the old slot still has A, + and that a rollback restores A with no additional servicing. +- **Negative tests.** Same extension in both the Host Configuration and the + COSI produces the structured error; COSI extension plus SELinux `enforcing` + is rejected. + +## Servicing + +The change is additive at every layer. + +- A revision 1.2 COSI is still valid and behaves identically. +- A revision 1.3 COSI with no `extensions` section behaves identically. +- A revision 1.3 COSI *with* extensions, read by a Trident that predates this + work, deploys the OS correctly and omits the extensions silently. This is the + one forward-compatibility hazard; mitigations are discussed under + [Versioning](#versioning). +- Hosts already using `os.sysexts` / `os.confexts` are unaffected unless they + move to an image that carries the same extension, which is a deliberate + action that now produces an explicit error. + +## Implementation Plan + +1. Specification: COSI revision 1.3, `extensions` section, v1.3 JSON Schema and + samples. +2. Trident reader: parse `extensions`, metadata validation and the new error + variants, and the minor-version warning. +3. Effective extension set, plumbed into the extensions, `osconfig` and + `selinux` subsystems, including the collision errors. +4. Streaming a tar member into the extension staging directory, replacing the + URL fetch for COSI-borne entries. +5. Tests, then user-facing documentation updates to + [Sysexts](../../Explanation/Sysexts.md), + [Confexts](../../Explanation/Confexts.md) and + [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). + +Steps 1 and 2 are independently useful: a reader that parses and validates the +section but ignores it is a safe intermediate state, and it makes the +minor-version warning available sooner. + +## Counter-Arguments + +### Drawbacks + +- **Cadence coupling.** This is the real cost. Bundling ties the extension's + release cadence to the image's: shipping a new version of the extension means + shipping a new COSI and taking an A/B update and a reboot. For content that + must match the OS anyway — kernel modules, GPU drivers, anything built + against a specific kernel — this is free, because that content could never + have moved independently. For content deliberately versioned independently of + the OS, it is a genuine regression in agility, and the Host Configuration + route remains the right answer. The proposal adds an option; it does not + remove one. +- **COSI files get bigger,** and every host downloading the image pays for + every bundled extension, including ones it will not use. There is no + per-host selection mechanism. +- **No way to opt out.** A host that wants the image but not one of its bundled + extensions has no way to say so. See [Open Questions](#open-questions). +- **A second place extensions can come from,** which means a merge, which means + collision rules, which is genuinely more complexity in the extensions + subsystem than exists today. +- **The forward-compatibility hazard** described under Versioning: an older + reader silently produces a host without the extensions. + +### Alternatives + +**Keep everything in the Host Configuration (status quo).** Maximum decoupling: +the extension is versioned, hosted and updated entirely independently, and a +change is a Runtime Update with no reboot. This stays fully supported. It is +the wrong answer only when the extension must land in lockstep with a new OS, +where it forces two operations and two rollout gates. + +**Bake the `.raw` files into the root filesystem image.** No spec change at +all. Discussed in [Motivation](#why-not-just-bake-the-files-into-the-root-filesystem-image); +loses on verity, on destinations that are not image-provisioned, on +inspectability, and on Trident not knowing to enable the merge units. + +**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** Rejected +in [Tar Layout](#tar-layout): it would fork the definition of the one object +this design reuses, and buys only cosmetics. + +**One array with a `kind` discriminator.** Rejected in +[Why Two Arrays](#why-two-arrays-instead-of-one-array-with-a-kind): the kinds +share almost no rules, and the Host Configuration already uses two lists. + +#### Prior Art + +The decoupled alternative is well established. Flatcar's sysext-bakery pattern +combines pre-built sysext DDIs with +[`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html): +the operator hosts artefacts over HTTP alongside a `SHA256SUMS` manifest, +`sysupdate` discovers and stages new versions against a transfer definition, +and activation requires a `systemd-sysext refresh` or a reboot. It is a good +design for content with its own cadence, and it is close in spirit to Trident's +existing `os.sysexts` — with the notable difference that `systemd-sysupdate` +has no OCI transport, whereas Trident's `Extension.url` already accepts +`oci://` and can therefore reuse a registry the operator is likely to have. + +This proposal is a third option rather than a replacement for either. Flatcar's +model optimises for extensions that move independently of the OS; Trident's +current model does the same with a better transport story; carrying the +extension in the COSI optimises for extensions that must move *with* the OS, +and pays for it in cadence. All three should coexist, and the +`extension-release` fields (`ID=_any` versus `ID=`) are the honest +signal for which one a given extension belongs in. + +## Open Questions + +- **Should there be an override escape hatch?** The proposal makes a + Host-Configuration-versus-COSI collision an error. The alternative is an + explicit opt-in on the Host Configuration side meaning "yes, I know the image + ships this, use mine instead". Useful for pinning a hotfixed extension + without rebuilding the image; also a way to run a combination the image + author never validated. Recommendation is to ship strict and add the hatch + only if a real need appears, but this is a product decision. +- **Should there be a way to suppress a bundled extension?** Same question from + the other direction: a "deny" list would let an operator take an image + without one of its extensions. It has no analogue in the current API and no + concrete requester yet. +- **Should `path` be required rather than optional?** Optional-with-defaults is + proposed for symmetry with the Host Configuration. Requiring it would make + every COSI explicit about placement at the cost of diverging from the object + it is meant to resemble. +- **Naming.** `extensions` containing `sysexts` / `confexts` versus two + top-level `sysexts` / `confexts` arrays. The nested form keeps the root + object tidy and groups the feature; the flat form is a shorter path + expression and an even closer match to `os.sysexts`. +- **Should ID uniqueness be enforced for Host Configuration extensions + independently of this work?** It is documented but unenforced today. This + proposal needs it for the merged set; it is arguably a pre-existing gap that + deserves its own fix. +- **Should the writer be required to place extension payloads after all region + images,** or merely encouraged? The proposal says SHOULD. + +## Future Possibilities + +- **Portable service images.** They are DDIs with the same shape and the same + placement problem. If Trident ever manages them, `extensions` generalises + naturally or gains a sibling. +- **Per-extension selection at deploy time.** If a real need appears for a + single image serving hosts with different extension sets, a selection + mechanism keyed on extension ID would build on this section. +- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` extensions are already + parsed by `ExtensionRelease` but not acted on. A bundled extension is a + natural fit for initrd scope, since the payload is available before the root + filesystem is. +- **Extension inventory in Host Status.** Reporting the merged extension set, + with IDs and hashes, would make "what is actually running on this host" + answerable without inspecting the filesystem. From 386b6e78bbbec3233f2b6f68ba744cca88b2b4ea Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 8 Sep 2026 18:23:40 -0700 Subject: [PATCH 02/15] docs: tighten COSI extension images RFC Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0000-cosi-extension-images.md | 932 ++++++++---------- 1 file changed, 412 insertions(+), 520 deletions(-) diff --git a/docs/Development/RFCs/0000-cosi-extension-images.md b/docs/Development/RFCs/0000-cosi-extension-images.md index 186cca3988..793b9c1ca9 100644 --- a/docs/Development/RFCs/0000-cosi-extension-images.md +++ b/docs/Development/RFCs/0000-cosi-extension-images.md @@ -1,203 +1,162 @@ # 0000 COSI Extension Images -- Date: 2026-08-28 +- Date: 2026-09-08 - RFC PR: [microsoft/trident#0000](https://github.com/microsoft/trident/pull/0000) - Issue: [microsoft/trident#0000](https://github.com/microsoft/trident/issues/0000) ## Summary -This RFC proposes carrying systemd system extension (sysext) and configuration -extension (confext) images inside a COSI file. Extension images become ZSTD -compressed members of the COSI tar, exactly like partition images, and a new -optional `extensions` section in `metadata.json` describes them. Each entry -reuses the existing [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) -object to point at the payload in the tar, and carries the destination path on -the target OS, mirroring the existing +COSI files cannot carry systemd system extension (sysext) or configuration +extension (confext) images. This RFC stores extension images as ZSTD-compressed +members of the COSI tar, alongside partition images, and adds an optional +`extensions` section to the COSI metadata to describe them. Each entry reuses +the existing [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) +object and carries an optional destination path, mirroring the [`Extension`](../../Reference/Host-Configuration/API-Reference/Extension.md) -object in the Host Configuration. Trident then deploys those extensions as part -of the image, in the same operation and the same reboot as the rest of the OS, -instead of fetching them from a separate location at deploy time. +object in the Host Configuration. Trident deploys bundled extensions as part of +the image, in the same operation as the rest of the OS. ## Motivation and Goals -Today the only way to get an extension onto a Trident-managed host is -`os.sysexts` / `os.confexts` in the Host Configuration, where each entry is a -URL plus a SHA-384. That works, but it makes the extension a second artefact -with a second lifecycle: - -- **A second thing to host.** The operator must publish the `.raw` DDI - somewhere Trident can reach (`http://`, `https://`, `file://` or `oci://`) - and keep it there for as long as any host might re-run the flow. -- **A second download at deploy time.** The COSI is streamed sparsely from one - endpoint; each extension is fetched in full from another. Two endpoints means - two sets of network, proxy, TLS and authentication failure modes on the - provisioning path. -- **A second integrity story.** The COSI's integrity is anchored by the - `metadata.json` hash in `os.image.sha384`, which transitively covers every - partition image. An extension's hash is anchored by whoever wrote the Host - Configuration. Bundling puts the extension under the same root of trust as - the rest of the OS. -- **A second update.** This is the important one. Extension changes are - currently a Runtime Update, distinct from the A/B update that ships the OS. - Delivering operator content that must land alongside a new OS therefore means - two operations, two rollout gates and potentially two disruptions. - -### A/B Updates Are the Strategic Point - -If extensions ride the COSI, then operator content is delivered and activated -in the *same* A/B update and the *same* reboot as the OS. It reuses the -rollout, [health-gating](../../Reference/Host-Configuration/API-Reference/Health.md) -and rollback machinery that already exists for images, rather than needing a -parallel channel with its own staging, verification and failure semantics. - -Rollback becomes trivially correct: extension images are files inside the -target slot's own filesystem, so the previous slot still holds the previous -extension set, untouched. Rolling back to the previous root volume rolls back -the extensions with it. See [Rollback](#rollback). - -### Why Not Just Bake the Files Into the Root Filesystem Image? - -An obvious objection: the image builder controls the root filesystem image, so -it could simply place `my-tool.raw` in `/var/lib/extensions/` before the COSI -is built, with no spec change at all. That is a genuine alternative and it -works for some configurations. It loses on four counts: - -1. **Verity.** With root or usr verity, any change to the protected filesystem - changes the root hash and requires re-signing. Extensions live outside the - protected tree precisely so they can change independently; the mechanism - that makes them attractive is defeated if adding one means rebuilding and - re-signing the base image. -2. **Destination volumes that are not image-provisioned.** `/var/lib/extensions/` - is commonly on a separate volume, which may be newly created rather than - written from a partition image. Files baked into the image's `/var` never - arrive in that case. -3. **Opacity.** Bytes inside a filesystem image are invisible to anything that - reads COSI metadata. An `extensions` section makes the extension set - inspectable, hashable and enumerable without mounting anything, which - matters for inventory and supply-chain tooling. -4. **Trident does not know.** Trident enables `systemd-sysext.service` and - `systemd-confext.service`, validates placement, and reads `extension-release` - only for extensions it knows about. Files that appear inside a filesystem - image bypass all of that, and the merge services are never enabled. +Extensions are configured today through `os.sysexts` and `os.confexts` in the +Host Configuration, where each entry is a URL and a SHA-384. This makes an +extension a second artefact with a separate lifecycle: + +- It must be hosted separately, for as long as any host may re-run the flow. +- It is fetched in full from a second endpoint at deploy time, with its own + network, proxy and authentication failure modes. +- Its integrity is anchored by whoever wrote the Host Configuration, not by + `os.image.sha384`. +- Changing it is a Runtime Update, distinct from the A/B update that ships the + OS. Content that must land with a new OS requires two operations. + +### A/B Updates + +A bundled extension is delivered and activated in the same A/B update and the +same reboot as the OS, and reuses the existing rollout, health-gating and +rollback machinery. + +Rollback requires no additional work: extension images are files in the target +slot's filesystem, so the previous slot retains the previous set. See +[Rollback](#rollback). + +### Why Not Place the Files in the Root Filesystem Image + +The image builder could place `my-tool.raw` in `/var/lib/extensions/` before the +COSI is built. This works in some configurations, but: + +1. With root or usr verity, modifying the protected filesystem changes the root + hash and requires re-signing. +2. Extension destinations are commonly on a separate volume, which may be + created empty rather than written from a partition image. +3. Contents of a filesystem image are not visible to tooling that reads COSI + metadata. +4. Trident enables `systemd-sysext.service` and `systemd-confext.service`, + validates placement, and reads `extension-release` only for extensions it + knows about. ### Goals -- Extension images can be carried in a COSI file and deployed by Trident with - no second artefact, no second endpoint and no second update. -- The metadata addition is small, reuses existing objects, and looks familiar - to anyone who already knows `os.sysexts` / `os.confexts`. -- The Host Configuration public API does not change. +- Carry extension images in a COSI file and deploy them without a second + artefact, endpoint or update. +- Reuse existing metadata objects and match the shape of the Host Configuration + API. +- Leave the Host Configuration API unchanged. ## Scope ### Requirements -- Add an optional `extensions` object to the COSI metadata root, containing +- An optional `extensions` object in the COSI metadata root, containing `sysexts` and `confexts` arrays. -- Each entry MUST identify its payload in the tar via an `ImageFile` object and - MAY specify a destination path on the target OS. -- Extension payloads are ordinary ZSTD-compressed tar members, covered by the - existing compression and integrity rules. -- Trident deploys COSI-borne extensions during Clean Install and A/B Update, - reusing the existing extensions subsystem. -- COSI-borne extensions and Host Configuration extensions coexist; conflicts - are a hard error. -- New COSI validation errors for the extension section. +- Each entry identifies its payload with an `ImageFile` object and may specify a + destination path on the target OS. +- Extension payloads are ZSTD-compressed tar members, subject to the existing + compression and integrity rules. +- Trident deploys bundled extensions on Clean Install and A/B Update, reusing + the extensions subsystem. +- Bundled and Host Configuration extensions coexist; conflicts are an error. +- COSI metadata validation covers the new section. ### Out of Scope -- Changing the Host Configuration `Extension` object. It stays exactly as it - is. -- Runtime Update of COSI-borne extensions. A COSI-borne extension can only - change when the image changes, which is an A/B update. -- Signing or attestation of individual extension images beyond the SHA-384 - chain already provided by COSI. -- Making extensions compatible with SELinux. That limitation is orthogonal and - unchanged; see [SELinux](#selinux). -- Building the DDIs. This RFC specifies how a COSI carries an extension image, - not how the extension image is produced. +- Changes to the Host Configuration `Extension` object. +- Runtime Update of bundled extensions. A bundled extension can only change when + the image changes, which is an A/B update. +- Signing or attestation beyond the SHA-384 chain COSI already provides. +- SELinux compatibility. That limitation is unchanged; see [SELinux](#selinux). +- Producing the DDIs. This RFC specifies how a COSI carries an extension image, + not how the image is built. ### Exit Criteria -- COSI revision 1.3 is published with the `extensions` section, a - `cosi-metadata-v1.3.schema.json`, and valid/invalid samples under +- COSI revision 1.3 published with the `extensions` section, + `cosi-metadata-v1.3.schema.json`, and samples under `tests/cosi/metadata_samples/v1.3/`. -- Trident deploys a COSI carrying at least one sysext and one confext on Clean - Install and on A/B Update, and the extensions are merged after reboot. -- A/B rollback returns the host to the previous extension set with no extra - work. -- Conflicts between COSI-borne and Host-Configuration-borne extensions produce - a structured error. +- Trident deploys a COSI carrying a sysext and a confext on Clean Install and on + A/B Update, and both are merged after reboot. +- A/B rollback restores the previous extension set with no additional servicing. +- Conflicts between bundled and Host Configuration extensions produce a + structured error. ## Dependencies -- A COSI writer able to emit the section. [Image Customizer](https://github.com/microsoft/azure-linux-image-tools) - is the reference implementation of the COSI writer side. -- No incomplete Trident features are required. +A COSI writer that emits the section. +[Image Customizer](https://github.com/microsoft/azure-linux-image-tools) is the +reference writer. No incomplete Trident features are required. ## Implementation ### Tar Layout -Extension payloads are ZSTD-compressed DDI files stored under -`images/extensions/`, for example `images/extensions/my-tool.rawzst`. - -They stay under `images/` deliberately. The COSI specification already states -that region images "MAY be placed in subdirectories of `images/` to organize -them" and that "Readers MUST be able to handle images in subdirectories", so no -new tar-layout rule is needed. More importantly, `ImageFile.path` is -constrained by `"pattern": "^images/.+"` in every published schema. Relaxing -that pattern to also permit `extensions/` would mean a v1.3 `ImageFile` that -fails validation against the v1.0–1.2 schemas, breaking the one object this -proposal is supposed to reuse unchanged, and breaking any generic COSI -validator that has an `ImageFile` definition compiled in. There is no -compensating benefit: a subdirectory conveys the same organisation. - -Two ordering constraints from the existing spec must be respected: - -- Since revision 1.2, the primary GPT image MUST be the file immediately after - `metadata.json`. Extension payloads MUST NOT be placed between them. -- Region images MUST appear in the tar in the physical order of the regions on - the source disk. Extension payloads are not regions, so they do not - participate in that ordering, but interleaving them would make the region - ordering harder to verify and would hurt sparse-read locality. Extension - payloads SHOULD therefore be written after all region images. - -Extension payloads are compressed with ZSTD like everything else, and their -window log MUST be accounted for in the existing root `compression.maxWindowLog` -field. No change is needed to that object; writers simply must not forget the -new images when computing the maximum. - -Older readers are unaffected by the extra tar members. The spec already says -"the tar file MAY contain other files, but Trident MUST ignore them", and -Trident's orphan-image check +Extension payloads are ZSTD-compressed DDI files under `images/extensions/`, +for example `images/extensions/my-tool.rawzst`. + +They remain under `images/`. The specification already permits subdirectories of +`images/` and requires readers to handle them, so no new tar-layout rule is +needed. `ImageFile.path` is constrained by `"pattern": "^images/.+"` in every +published schema; relaxing it would produce a revision 1.3 `ImageFile` that +fails validation against the 1.0–1.2 schemas and break any validator with an +`ImageFile` definition compiled in, for no benefit beyond a shorter path. + +Two existing ordering constraints apply: + +- Since revision 1.2 the primary GPT image must immediately follow + `metadata.json`. Extension payloads must not be placed between them. +- Region images must appear in the physical order of the regions on the source + disk. Extension payloads are not regions, but interleaving them complicates + verification of that ordering and hurts sparse-read locality. Extension + payloads should be written after all region images. + +Writers must account for extension payloads when computing the existing root +`compression.maxWindowLog`. + +Older readers are unaffected by the additional tar members. The specification +requires readers to ignore unknown files, and Trident's orphan-image check (`V1_2ImageFileHasNoCorrespondingPartition`) is driven by the `images[]` and -`disk.gptRegions[]` metadata arrays, not by walking tar entries, so extra -members under `images/` do not trip it. +`disk.gptRegions[]` arrays rather than by walking tar entries. ### Metadata Schema -A new optional root field `extensions`: +A new optional root field: -| Field | Type | Added in | Required | Description | -| ------------ | ------------------------------------- | -------- | -------- | ----------------------------------------------- | -| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images carried by this COSI file. | +| Field | Type | Added in | Required | Description | +| ------------ | -------------------------------- | -------- | -------- | ------------------------------------------- | +| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images carried by this COSI file. | #### `Extensions` Object -| Field | Type | Added in | Required | Description | -| ---------- | ------------------------------------------ | -------- | -------- | ---------------------------------- | -| `sysexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | System extension images. | -| `confexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | Configuration extension images. | +| Field | Type | Added in | Required | Description | +| ---------- | ------------------------------------------ | -------- | -------- | ------------------------------- | +| `sysexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | System extension images. | +| `confexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | Configuration extension images. | #### `ExtensionImage` Object -| Field | Type | Added in | Required | Description | -| ------- | ------------------------------ | -------- | --------------- | -------------------------------------------------------- | -| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | Details of the compressed extension image in the tar. | -| `path` | string | 1.3 | No | Absolute destination path of the extension on the target OS. | - -JSON Schema fragment: +| Field | Type | Added in | Required | Description | +| ------- | -------------------------------------------------------------------- | -------- | --------------- | ------------------------------------------------------------ | +| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | Details of the compressed extension image in the tar file. | +| `path` | string | 1.3 | No | Absolute destination path of the extension on the target OS. | ```json { @@ -242,7 +201,7 @@ JSON Schema fragment: } ``` -Worked example: +Example: ```json { @@ -297,120 +256,97 @@ Worked example: } ``` -#### Why Two Arrays Instead of One Array With a `kind` - -Two arrays, mirroring `os.sysexts` / `os.confexts`. - -- The distinction is not cosmetic. Sysexts and confexts have disjoint sets of - permitted destination directories (`VALID_SYSEXT_DIRECTORIES` vs - `VALID_CONFEXT_DIRECTORIES`), disjoint defaults, different - `extension-release` locations (`/usr/lib/extension-release.d/` vs - `/etc/extension-release.d/`), different identity fields (`SYSEXT_ID` vs - `CONFEXT_ID`) and different activation units. Every rule downstream branches - on the kind. -- With two arrays the kind is structural, so no `kind` field is needed and it - cannot be omitted or wrong. -- It is the shape the Host Configuration already uses, which is the stated - design constraint. A reader that already has code for `os.sysexts` has code - for this. - -The one-array-plus-discriminator form would be preferable if the two kinds -shared their rules and the set of kinds were open-ended. Neither is true here. - -#### What Each Entry Carries, and What It Does Not - -`image` and an optional `path`. That is the whole object. Everything else that -was considered is derivable, and every derivable field is a new opportunity for -two sources of truth to disagree. - -- **`path` (kept, optional).** Not derivable: it is the one genuinely new piece - of information, the image author's intent about where the file goes. Optional - because the Host Configuration's `Extension.path` is optional and defaults to - `/var/lib/extensions/{name}.raw` or `/var/lib/confexts/{name}.raw`; making it - required here would be a gratuitous divergence, and the default is what most - writers want anyway. When present it MUST be absolute, MUST end in `.raw`, - MUST sit in a permitted directory for its kind, and its file name MUST match - the `extension-release` suffix — all rules that already exist. -- **Extension name (rejected).** Trident derives the name by mounting the DDI - and reading the `extension-release.{name}` file name, and already enforces - that the destination file name matches it. A `name` field would be a third - copy of the same string (DDI, `path`, `name`) and a third thing to - cross-check. -- **`kind` (rejected).** Structural, see above. -- **Extension ID (`SYSEXT_ID` / `CONFEXT_ID`) (rejected).** Read from - `extension-release`. It is the identity Trident already keys A/B state on; - duplicating it in metadata invites drift with the payload. -- **Enable-on-first-boot (rejected).** systemd merges everything it finds in - the extension directories; there is no per-extension enable switch in the - Host Configuration today, and inventing one here would create COSI-only - semantics that the Host Configuration cannot express. If a per-extension - toggle is wanted, it should be designed once for both surfaces. -- **Version / OS-compatibility metadata (rejected).** `ID`, `VERSION_ID`, - `SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE` and `SYSEXT_SCOPE` are - already inside the DDI's `extension-release`, and systemd — not Trident — is - the authority that enforces them at merge time. Copying them into metadata - would let a COSI claim compatibility the payload does not have. -- **A separate hash of the uncompressed DDI (rejected, with a caveat).** See - below. - -#### A Note on `sha384` Semantics - -`ImageFile.sha384` is the hash of the **compressed** image. The Host -Configuration's `Extension.sha384` is the hash of the **raw** extension image -file. Reusing `ImageFile` therefore changes what the recorded hash covers. - -For integrity that is fine, and it is exactly what already happens for -partition images: Trident hashes the compressed stream as it decompresses, so a -match proves the payload is intact. Adding a second hash of the uncompressed -DDI would buy nothing for integrity. - -There is one caveat worth recording. Trident's `ExtensionData.sha384` is also -used as a change detector: for an extension whose ID exists in both the old and -the new configuration, a differing hash means "content changed, replace it". -Compressed hashes are not stable across recompressions, so two COSIs built from -byte-identical DDIs at different ZSTD levels would look "changed". The -consequence is a redundant copy of an identical file, never a missed update, so -this is a cosmetic inefficiency rather than a correctness problem. It is called -out here so it is not rediscovered as a bug. +#### Two Arrays Rather Than One With a `kind` Field + +Sysexts and confexts have disjoint sets of permitted destination directories +(`VALID_SYSEXT_DIRECTORIES` and `VALID_CONFEXT_DIRECTORIES`), different +defaults, different `extension-release` locations +(`/usr/lib/extension-release.d/` and `/etc/extension-release.d/`), different +identity fields (`SYSEXT_ID` and `CONFEXT_ID`) and different activation units. +Every downstream rule branches on the kind. + +With two arrays the kind is structural, so no `kind` field is required and it +cannot be omitted or incorrect. It is also the shape the Host Configuration +already uses. + +#### Entry Contents + +An entry carries `image` and an optional `path`. Everything else considered is +derivable from the payload, and each derivable field adds a second source of +truth. + +- **`path`** is retained. It is the destination on the target OS and is not + derivable. It is optional because `Extension.path` is optional and defaults to + `/var/lib/extensions/{name}.raw` or `/var/lib/confexts/{name}.raw`. When + present it must be absolute, must end in `.raw`, must be in a permitted + directory for its kind, and its file name must match the `extension-release` + suffix. These rules already exist. +- **Extension name** is rejected. Trident derives it from the + `extension-release.{name}` file name inside the DDI and already requires the + destination file name to match. A `name` field would be a third copy of the + same string. +- **`kind`** is rejected; it is structural. +- **Extension ID** (`SYSEXT_ID`, `CONFEXT_ID`) is rejected. It is read from + `extension-release` and is the identity Trident keys A/B state on. +- **Enable-on-first-boot** is rejected. systemd merges everything in the + extension directories, and the Host Configuration has no per-extension switch. + Adding one here would create semantics the Host Configuration cannot express. +- **Version and compatibility metadata** is rejected. `ID`, `VERSION_ID`, + `SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE` and `SYSEXT_SCOPE` are in the + DDI's `extension-release`, and systemd enforces them at merge time. Copying + them into metadata would allow a COSI to claim compatibility the payload does + not have. + +#### `sha384` Semantics + +`ImageFile.sha384` covers the compressed image; `Extension.sha384` covers the +raw extension image file. Reusing `ImageFile` therefore changes what the hash +covers. + +For integrity this is equivalent to partition images: Trident hashes the +compressed stream as it decompresses, and a match proves the payload is intact. +A second hash of the uncompressed DDI would add nothing. + +One consequence should be recorded. `ExtensionData.sha384` is also used as a +change detector: for an extension whose ID is present in both the old and new +configuration, a differing hash means the content changed. Compressed hashes are +not stable across recompressions, so two COSIs built from identical DDIs at +different ZSTD levels appear to differ. The result is a redundant copy of an +identical file, never a missed update. ### Versioning -This lands in **COSI revision 1.3**. `extensions` is optional; absent means the -same as empty. +This lands in COSI revision 1.3. `extensions` is optional; absent is equivalent +to empty. -**Newer Trident, older COSI (1.0–1.2).** No `extensions` field, no extensions -from the image, current behaviour exactly. Nothing to do. +**Newer Trident, older COSI (1.0–1.2).** No `extensions` field, no bundled +extensions, current behaviour. **Older Trident, newer COSI (1.3 with extensions).** Trident's version check -only rejects `major != 1`, and both the spec and the metadata parser require -unknown fields to be ignored. An already-shipped Trident will therefore accept -a 1.3 COSI, deploy the OS correctly, and **silently omit the extensions**. - -That is the honest and slightly uncomfortable answer, and it cannot be fixed -in-band for readers that already exist: any tripwire field we add is, by -definition, a field they are required to ignore. Three mitigations, none of -which is a fix: - -1. **Document it** as a property of the format: consuming extensions requires a - reader that understands revision 1.3. -2. **Warn from now on.** `validate_cosi_metadata_version` should log a warning - when `minor` exceeds the highest revision the binary knows about. This does - not help binaries already in the field, but it makes every future minor bump - diagnosable from the log. -3. **Gate the rollout.** An operator who cannot control the Trident version on - the target should add a +rejects only `major != 1`, and both the specification and the metadata parser +require unknown fields to be ignored. An existing Trident will accept a 1.3 +COSI, deploy the OS correctly, and omit the extensions without reporting +anything. + +This cannot be corrected in-band for readers that already exist: any tripwire +field is a field they are required to ignore. Three partial mitigations: + +1. Document that consuming bundled extensions requires a reader that understands + revision 1.3. +2. Have `validate_cosi_metadata_version` warn when `minor` exceeds the highest + revision the binary knows. This does not help existing binaries, but makes + future minor bumps diagnosable from the log. +3. Where the Trident version on the target cannot be controlled, add a [health check](../../Reference/Host-Configuration/API-Reference/Health.md) - asserting the extension is merged. An A/B update that lands on a too-old - Trident then fails its health gate and rolls back, instead of quietly - producing a host missing its extensions. + asserting the extension is merged, so an A/B update onto a too-old Trident + fails its gate and rolls back. ### Trident-Side Consumption -The core idea is that the extensions subsystem gains a second *source* for -extensions, and nothing else about it changes. Concretely, Trident computes an -**effective extension set** = Host Configuration entries ∪ COSI entries, and -the places that today read `ctx.spec.os.sysexts` / `ctx.spec.os.confexts` read -the effective set instead. +The extensions subsystem gains a second source. Trident computes an **effective +extension set**, the union of the Host Configuration entries and the COSI +entries, and the code that reads `ctx.spec.os.sysexts` and +`ctx.spec.os.confexts` reads the effective set instead. ```mermaid flowchart LR @@ -421,354 +357,310 @@ flowchart LR Eff --> Sel["selinux subsystem
enforcing-mode rejection"] ``` -The three consumers of the effective set are worth naming individually, -because two of them are easy to miss: +There are three consumers: - **`ExtensionsSubsystem`.** `populate_extensions` currently downloads each - Host Configuration entry with a `FileReader` over `Extension.url` into the - staging directory, verifies the hash, then mounts the DDI to read - `extension-release`. For a COSI-borne entry the only difference is where the - bytes come from: the image streaming pipeline decompresses the tar member - into the same staging directory and verifies `ImageFile.sha384` over the - compressed stream. Everything downstream — the `extension-release` read, the - `{name}.raw` file-name check, the default-path logic, directory creation, - and the placement/removal logic in `set_up_extensions` — is unchanged. + entry with a `FileReader` over `Extension.url` into the staging directory, + verifies the hash, and mounts the DDI to read `extension-release`. For a + bundled entry only the source changes: the image streaming pipeline + decompresses the tar member into the same staging directory and verifies + `ImageFile.sha384` over the compressed stream. The `extension-release` read, + the `{name}.raw` check, default-path resolution, directory creation and + `set_up_extensions` are unchanged. - **`osconfig`.** `systemd-sysext.service` and `systemd-confext.service` are - enabled only when `ctx.spec.os.sysexts` / `confexts` are non-empty. If a host - gets its extensions solely from the COSI and its Host Configuration lists - none, the merge services would never be enabled and the extensions would sit - on disk doing nothing. This check **must** move to the effective set. It is - the single most likely way to ship this feature broken. -- **`selinux`.** The dynamic validation that rejects `enforcing` mode when - extensions are configured has the same shape and the same problem. It must - also move to the effective set, otherwise a COSI carrying extensions plus an - enforcing-mode Host Configuration passes validation and produces a host with - a mislabelled `/usr`, `/opt` or `/etc`. - -`derive_host_configuration` (used by [disk streaming](../../Explanation/Disk-Streaming.md)) -does not need to synthesise extension entries; the effective set is computed -from the COSI directly, so the derived-Host-Configuration path gets extensions -for free. + enabled only when `ctx.spec.os.sysexts` or `confexts` are non-empty. A host + whose extensions come only from the COSI would never have the merge units + enabled, and the extensions would remain on disk unused. This check must read + the effective set. +- **`selinux`.** The dynamic validation raising + `ExtensionImagesAndSelinuxUnsupported` has the same problem. Without the + change, a COSI carrying extensions combined with an enforcing-mode Host + Configuration passes validation and produces a host with a mislabelled `/usr`, + `/opt` or `/etc`. + +`derive_host_configuration`, used by +[disk streaming](../../Explanation/Disk-Streaming.md), does not need to +synthesise extension entries; the effective set is computed from the COSI +directly. #### Where Images Are Written - **Clean Install and A/B Update.** `provision()` runs with the target root mounted at `mount_path`. Extensions are staged under - `{mount_path}/var/lib/extensions/.staging` and then moved to their - destination inside the target root, which is exactly what happens today. For - A/B, that destination is on the *inactive* slot, so the running system is - untouched until the reboot. -- **Runtime Update.** COSI-borne extensions do not participate. A COSI-borne - extension can only change if the COSI changes, and a COSI change is an image - change, which forces an A/B update. Host Configuration extensions continue to - be runtime-updatable exactly as today. - -This gives a useful invariant: **Trident never needs to read the previous -COSI.** If `spec.os.image` equals `spec_old.os.image`, the old COSI's extension -set is the new COSI's extension set. If they differ, the servicing type is an -A/B update, and `set_up_extensions` already skips removal of the old set on -Clean Install and A/B Update because the old files live in the other slot. + `{mount_path}/var/lib/extensions/.staging` and moved to their destination + inside the target root, as today. For A/B the destination is on the inactive + slot, so the running system is untouched until reboot. +- **Runtime Update.** Bundled extensions do not participate. A bundled extension + can only change if the COSI changes, which is an image change and forces an + A/B update. Host Configuration extensions remain runtime-updatable. + +This yields an invariant: Trident never needs to read the previous COSI. If +`spec.os.image` equals `spec_old.os.image`, the previous bundled set is the +current one. If they differ, the servicing type is an A/B update, and +`set_up_extensions` already skips removal of the old set on Clean Install and +A/B Update. #### Rollback -Nothing needs retaining, because nothing is shared. - -Extension images are ordinary files inside the slot's filesystem, and Trident -already requires that when A/B volumes are configured, every extension -destination is on an A/B volume and not on a shared or read-only one. So the -previous slot's `/var/lib/extensions/` still contains the previous extension -set, byte for byte, after the update. An A/B rollback boots the previous root -volume, `systemd-sysext` merges what it finds there, and the extension set -reverts with the OS. No extra bookkeeping, no staging area to preserve, no -"previous extension" directory. +Extension images are files in the slot's own filesystem, and Trident already +requires every extension destination to be on an A/B volume, not shared and not +read-only, when A/B volumes are configured. The previous slot therefore retains +the previous extension set after an update. An A/B rollback boots the previous +root volume, `systemd-sysext` merges what it finds there, and the extension set +reverts with the OS. No additional bookkeeping is required. -This is the same property that makes A/B updates work for the OS itself, and it -is the main reason bundling is attractive: the extension inherits the rollback -semantics of the image instead of needing its own. +#### Interaction With Host Configuration `sysexts` and `confexts` -#### Interaction With Host Configuration `sysexts` / `confexts` +The two sets are merged, and a collision is an error. -**Merge, with a hard error on collision.** +The lists express different intents and are both valid at once. Bundled +extensions are content the image author considers part of the OS, such as a GPU +driver. Host Configuration extensions are content the operator adds at deploy +time, such as a monitoring agent. An image that ships one sysext and an operator +who adds another is the expected case. -The two lists express different intents and both are legitimate at the same -time. COSI extensions are content the *image author* considers part of this OS -— a GPU driver, a kernel-module extension. Host Configuration extensions are -content the *operator* adds at deploy time — a monitoring agent, a site-specific -confext. Forbidding both would be gratuitous: the natural case is an image that -ships one sysext and an operator who adds another. - -Silent precedence is the option to avoid. If a Host Configuration entry -silently overrode a bundled one, the host would run software the image author -never validated, and the only symptom would be a version number in a log -somewhere. Failing loudly is better than being quietly wrong. +Silent precedence is rejected: if a Host Configuration entry overrode a bundled +one, the host would run software the image author did not validate, with no +visible symptom. A collision is defined two ways, both already meaningful in the Host -Configuration today: +Configuration: -1. **Same destination path.** Detectable statically whenever both sides specify +1. **Same destination path.** Detectable statically when both sides specify `path`. This extends the existing `DuplicateExtensionImagePath` rule across the merged set. -2. **Same extension ID** (`SYSEXT_ID` / `CONFEXT_ID` within a kind). Only - detectable after the DDIs are mounted, which the subsystem does anyway. Note - that ID uniqueness is documented for the Host Configuration today but not - enforced in code; the merged set makes collisions more likely, so this - should become an enforced check. +2. **Same extension ID** within a kind. Detectable only after the DDIs are + mounted, which the subsystem does regardless. ID uniqueness is documented for + the Host Configuration today but not enforced in code; the merged set makes + collisions more likely, so it should become an enforced check. -Two entries that are identical in every respect are still a collision. Making -"identical is fine" an exception invites hash-comparison subtleties for no real -benefit — the operator can simply drop their entry. +Two entries that are identical in every respect are still a collision. An +"identical is acceptable" exception would introduce hash-comparison subtleties +for no benefit, since the operator can remove their entry. -If a per-extension override is later shown to be necessary, it should be an -explicit opt-in on the Host Configuration side, not a default. That is listed -as an [open question](#open-questions). +An explicit per-extension override, should one prove necessary, belongs on the +Host Configuration side as an opt-in. See [Open Questions](#open-questions). ### Validation #### COSI Metadata Validation New `CosiMetadataErrorKind` variants, following the existing `V1_` -prefix convention: - -| Variant | Condition | -| -------------------------------------------------- | -------------------------------------------------------------------------------------------- | -| `V1_3ExtensionDestinationPathNotAbsolute` | `path` is present and is not absolute. | -| `V1_3ExtensionDestinationPathInvalidFileExtension` | `path` is present and does not end in `.raw`. | -| `V1_3ExtensionDestinationPathInvalidDirectory` | `path`'s parent is not in `VALID_SYSEXT_DIRECTORIES` / `VALID_CONFEXT_DIRECTORIES` for the kind. | -| `V1_3DuplicateExtensionDestinationPath` | Two entries resolve to the same destination path. | -| `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | -| `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | - -The existing "every image referenced by the metadata is present in the tar" -check must be extended to walk `extensions.sysexts[]` and -`extensions.confexts[]` as well, so a metadata entry pointing at a missing tar -member fails at load time rather than mid-provision. - -Directory and file-extension rules deliberately reuse the constants and logic -behind `Extension::validate_sysext` / `validate_confext` rather than -re-implementing them, so the two surfaces cannot drift. +prefix: + +| Variant | Condition | +| ------------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| `V1_3ExtensionDestinationPathNotAbsolute` | `path` is present and not absolute. | +| `V1_3ExtensionDestinationPathInvalidFileExtension` | `path` is present and does not end in `.raw`. | +| `V1_3ExtensionDestinationPathInvalidDirectory` | `path`'s parent is not in `VALID_SYSEXT_DIRECTORIES` or `VALID_CONFEXT_DIRECTORIES` for the kind. | +| `V1_3DuplicateExtensionDestinationPath` | Two entries resolve to the same destination path. | +| `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | +| `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | + +The existing check that every image referenced by the metadata is present in the +tar must also walk `extensions.sysexts[]` and `extensions.confexts[]`, so a +missing tar member fails at load rather than mid-provision. + +Directory and file-extension rules reuse the constants and logic behind +`Extension::validate_sysext` and `validate_confext` rather than reimplementing +them. #### Deploy-Time Validation -- **File name matches the `extension-release` suffix.** Already enforced: - `read_extension_release` requires `path` to end with `{name}.raw` where - `{name}` is the suffix of the `extension-release.{name}` file found inside - the DDI. Applies unchanged to COSI-borne entries. -- **Exactly one `extension-release` file, and `SYSEXT_ID` / `CONFEXT_ID` - present.** Already enforced, applies unchanged. +- **File name matches the `extension-release` suffix.** Already enforced by + `read_extension_release`; applies unchanged. +- **Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID` + present.** Already enforced; applies unchanged. - **SHA-384 agreement.** `ImageFile.sha384` is verified over the compressed stream as the payload is decompressed into the staging directory, using the - same hashing reader used for partition images. Mismatch aborts servicing. + hashing reader used for partition images. A mismatch aborts servicing. - **Extension ID uniqueness across the merged set**, per kind. New check. - **`extension-release` OS compatibility.** Warn, do not fail, when a bundled - extension declares `ID=` and that `ID`/`VERSION_ID` does not match - the COSI's own `osRelease`. This is a build-time mistake worth surfacing - early — systemd will refuse to merge such an extension at boot and the host - will come up silently missing it — but systemd's matching rules - (`SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE`, `_any`) are subtle enough - that Trident should not be the authority. Warning early and letting systemd - decide is the right split. - -#### On `ID=_any` - -`extension-release` already declares which cadence case an extension is in: -`ID=` plus `VERSION_ID` / `SYSEXT_LEVEL` means "this extension is bound -to this OS build", while `ID=_any` means "this extension is portable across OS -versions". - -Trident should **not** warn or refuse when an `ID=_any` extension is bundled. -`_any` describes what the extension is *compatible with*, not how it *should be -delivered*. Bundling a portable extension is a perfectly reasonable packaging -choice — it is how you get an air-gapped or single-artefact deployment — and -refusing it would break legitimate use for no safety benefit. The genuinely -useful check is the opposite one described above: a bundled `ID=` -extension that does not match the COSI's own `osRelease` is almost certainly a -build error. + extension declares `ID=` and that `ID` or `VERSION_ID` does not match + the COSI's own `osRelease`. This is a build error worth surfacing early, since + systemd will refuse to merge the extension at boot and the host will come up + without it. systemd remains the authority: its matching rules + (`SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE`, `_any`) are intricate enough + that Trident should not reimplement them. + +#### `ID=_any` + +`extension-release` already declares the cadence case: `ID=` with +`VERSION_ID` or `SYSEXT_LEVEL` binds the extension to an OS build, while +`ID=_any` marks it portable across OS versions. + +Trident should not warn or refuse when an `ID=_any` extension is bundled. `_any` +describes what the extension is compatible with, not how it should be delivered. +Bundling a portable extension is a legitimate packaging choice, and is how an +air-gapped or single-artefact deployment is achieved. The useful check is the +opposite one above: a bundled `ID=` extension that does not match the +COSI's `osRelease` is almost certainly a build error. ### SELinux Unchanged. Extensions remain incompatible with SELinux in enforcing mode on systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. -Carrying the image in a COSI does not touch the labelling problem in any way. +Carrying the image in a COSI does not affect labelling. -The one thing that must change is *where the check looks*: as noted above, the -dynamic validation that raises `ExtensionImagesAndSelinuxUnsupported` currently -inspects the Host Configuration lists only, and must inspect the effective set, -or a COSI-only extension configuration would bypass it entirely. +What must change is where the check looks. The dynamic validation raising +`ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration lists +only, and must inspect the effective set; otherwise a COSI-only extension +configuration bypasses it. ## Public API Design -The COSI metadata format is a public contract with an independent +The COSI metadata format is a public contract with its own [specification](../../Reference/Composable-OS-Image.md) and published JSON -Schemas, so the additions above constitute the public API change. They are -additive and optional. +schemas, so the additions above are the public API change. They are additive and +optional. -The Host Configuration API does **not** change. `os.sysexts` and `os.confexts` -keep their meaning and their shape. The observable behavioural difference for -an existing user is that a host may now end up with more extensions than its -Host Configuration lists, and that specifying an extension that the image -already carries is now an error rather than a duplicate. +The Host Configuration API does not change. The observable difference for an +existing user is that a host may end up with more extensions than its Host +Configuration lists, and that specifying an extension the image already carries +is now an error rather than a duplicate. ## Testing and Metrics -- **Schema tests.** Add `docs/Reference/Composable-OS-Image/cosi-metadata-v1.3.schema.json` - and samples under `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`. The - existing schema-validation workflow picks up a new revision automatically - given the schema file and a matching samples directory. Invalid samples - should cover: `image.path` outside `images/`, non-absolute destination, - destination not ending in `.raw`, destination in a disallowed directory, and - a duplicated destination path. -- **Unit tests.** Metadata parse and validate for each new error variant; - effective-set computation, including the collision cases; default-path - resolution for a COSI entry with no `path`. -- **Functional tests.** Build a COSI carrying one sysext and one confext; - assert Clean Install places both, enables the merge units, and that both are - merged after boot. -- **Servicing tests.** A/B update from a COSI with extension set A to a COSI - with extension set B; assert the new slot has B, the old slot still has A, - and that a rollback restores A with no additional servicing. -- **Negative tests.** Same extension in both the Host Configuration and the - COSI produces the structured error; COSI extension plus SELinux `enforcing` - is rejected. +- **Schema.** Add `cosi-metadata-v1.3.schema.json` and samples under + `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`. The existing + schema-validation workflow picks up a new revision given the schema file and a + matching samples directory. Invalid samples should cover `image.path` outside + `images/`, a non-absolute destination, a destination not ending in `.raw`, a + destination in a disallowed directory, and a duplicated destination path. +- **Unit.** Metadata parse and validation for each new error variant; effective + set computation, including collisions; default-path resolution for an entry + with no `path`. +- **Functional.** Build a COSI carrying a sysext and a confext; assert Clean + Install places both, enables the merge units, and that both merge after boot. +- **Servicing.** A/B update from a COSI with extension set A to one with set B; + assert the new slot has B, the old slot retains A, and that rollback restores + A without additional servicing. +- **Negative.** The same extension in both the Host Configuration and the COSI + produces the structured error; a bundled extension with SELinux `enforcing` is + rejected. ## Servicing The change is additive at every layer. -- A revision 1.2 COSI is still valid and behaves identically. -- A revision 1.3 COSI with no `extensions` section behaves identically. -- A revision 1.3 COSI *with* extensions, read by a Trident that predates this - work, deploys the OS correctly and omits the extensions silently. This is the - one forward-compatibility hazard; mitigations are discussed under +- A revision 1.2 COSI remains valid and behaves identically. +- A revision 1.3 COSI without `extensions` behaves identically. +- A revision 1.3 COSI with extensions, read by a Trident predating this work, + deploys the OS correctly and omits the extensions silently. Mitigations are in [Versioning](#versioning). -- Hosts already using `os.sysexts` / `os.confexts` are unaffected unless they - move to an image that carries the same extension, which is a deliberate - action that now produces an explicit error. +- Hosts using `os.sysexts` or `os.confexts` are unaffected unless they move to an + image carrying the same extension, which now produces an explicit error. ## Implementation Plan -1. Specification: COSI revision 1.3, `extensions` section, v1.3 JSON Schema and - samples. +1. Specification: COSI revision 1.3, the `extensions` section, the v1.3 JSON + schema and samples. 2. Trident reader: parse `extensions`, metadata validation and the new error variants, and the minor-version warning. 3. Effective extension set, plumbed into the extensions, `osconfig` and `selinux` subsystems, including the collision errors. -4. Streaming a tar member into the extension staging directory, replacing the - URL fetch for COSI-borne entries. -5. Tests, then user-facing documentation updates to +4. Stream a tar member into the extension staging directory, replacing the URL + fetch for bundled entries. +5. Tests, then documentation updates to [Sysexts](../../Explanation/Sysexts.md), [Confexts](../../Explanation/Confexts.md) and [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). Steps 1 and 2 are independently useful: a reader that parses and validates the -section but ignores it is a safe intermediate state, and it makes the -minor-version warning available sooner. +section but ignores it is a safe intermediate state and makes the minor-version +warning available sooner. ## Counter-Arguments ### Drawbacks -- **Cadence coupling.** This is the real cost. Bundling ties the extension's - release cadence to the image's: shipping a new version of the extension means - shipping a new COSI and taking an A/B update and a reboot. For content that - must match the OS anyway — kernel modules, GPU drivers, anything built - against a specific kernel — this is free, because that content could never - have moved independently. For content deliberately versioned independently of - the OS, it is a genuine regression in agility, and the Host Configuration - route remains the right answer. The proposal adds an option; it does not - remove one. -- **COSI files get bigger,** and every host downloading the image pays for - every bundled extension, including ones it will not use. There is no - per-host selection mechanism. -- **No way to opt out.** A host that wants the image but not one of its bundled - extensions has no way to say so. See [Open Questions](#open-questions). -- **A second place extensions can come from,** which means a merge, which means - collision rules, which is genuinely more complexity in the extensions - subsystem than exists today. -- **The forward-compatibility hazard** described under Versioning: an older - reader silently produces a host without the extensions. +- **Cadence coupling.** Bundling ties the extension's release cadence to the + image's: a new version of the extension requires a new COSI, an A/B update and + a reboot. For content that must match the OS, such as kernel modules or GPU + drivers, this costs nothing, because that content could not move independently + in any case. For content deliberately versioned independently of the OS it is a + real loss of agility, and the Host Configuration route remains correct. This + RFC adds an option; it removes none. +- **Size.** COSI files grow, and every host downloading the image pays for every + bundled extension, including those it does not use. There is no per-host + selection mechanism. +- **No opt-out.** A host that wants the image but not one of its bundled + extensions cannot express that. See [Open Questions](#open-questions). +- **Two sources.** Merging, and the collision rules it requires, is more + complexity in the extensions subsystem than exists today. +- **Forward compatibility.** An older reader produces a host without the + extensions and reports nothing. ### Alternatives -**Keep everything in the Host Configuration (status quo).** Maximum decoupling: -the extension is versioned, hosted and updated entirely independently, and a -change is a Runtime Update with no reboot. This stays fully supported. It is -the wrong answer only when the extension must land in lockstep with a new OS, -where it forces two operations and two rollout gates. +**Host Configuration only (status quo).** Maximum decoupling: the extension is +versioned, hosted and updated independently, and a change is a Runtime Update +with no reboot. This remains fully supported, and is the wrong answer only when +the extension must land in lockstep with a new OS. -**Bake the `.raw` files into the root filesystem image.** No spec change at -all. Discussed in [Motivation](#why-not-just-bake-the-files-into-the-root-filesystem-image); -loses on verity, on destinations that are not image-provisioned, on -inspectability, and on Trident not knowing to enable the merge units. +**Place the `.raw` files in the root filesystem image.** No specification change. +Discussed in +[Why Not Place the Files in the Root Filesystem Image](#why-not-place-the-files-in-the-root-filesystem-image). -**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** Rejected -in [Tar Layout](#tar-layout): it would fork the definition of the one object -this design reuses, and buys only cosmetics. +**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** Rejected in +[Tar Layout](#tar-layout). **One array with a `kind` discriminator.** Rejected in -[Why Two Arrays](#why-two-arrays-instead-of-one-array-with-a-kind): the kinds -share almost no rules, and the Host Configuration already uses two lists. +[Two Arrays Rather Than One With a `kind` Field](#two-arrays-rather-than-one-with-a-kind-field). #### Prior Art -The decoupled alternative is well established. Flatcar's sysext-bakery pattern -combines pre-built sysext DDIs with +The decoupled approach is established. Flatcar's sysext-bakery pattern combines +pre-built sysext DDIs with [`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html): -the operator hosts artefacts over HTTP alongside a `SHA256SUMS` manifest, -`sysupdate` discovers and stages new versions against a transfer definition, -and activation requires a `systemd-sysext refresh` or a reboot. It is a good -design for content with its own cadence, and it is close in spirit to Trident's -existing `os.sysexts` — with the notable difference that `systemd-sysupdate` -has no OCI transport, whereas Trident's `Extension.url` already accepts -`oci://` and can therefore reuse a registry the operator is likely to have. - -This proposal is a third option rather than a replacement for either. Flatcar's -model optimises for extensions that move independently of the OS; Trident's -current model does the same with a better transport story; carrying the -extension in the COSI optimises for extensions that must move *with* the OS, -and pays for it in cadence. All three should coexist, and the -`extension-release` fields (`ID=_any` versus `ID=`) are the honest -signal for which one a given extension belongs in. +the operator hosts artefacts over HTTP with a `SHA256SUMS` manifest, sysupdate +stages new versions against a transfer definition, and activation requires a +`systemd-sysext refresh` or a reboot. It suits content with its own cadence and +is close in spirit to `os.sysexts`, with the difference that +`systemd-sysupdate` has no OCI transport, whereas `Extension.url` already +accepts `oci://`. + +This RFC is a third option rather than a replacement for either. Both existing +approaches optimise for extensions that move independently of the OS; carrying +the extension in the COSI optimises for extensions that must move with the OS, +at the cost of cadence. The `extension-release` fields, `ID=_any` against +`ID=`, indicate which approach a given extension belongs in. ## Open Questions -- **Should there be an override escape hatch?** The proposal makes a - Host-Configuration-versus-COSI collision an error. The alternative is an - explicit opt-in on the Host Configuration side meaning "yes, I know the image - ships this, use mine instead". Useful for pinning a hotfixed extension - without rebuilding the image; also a way to run a combination the image - author never validated. Recommendation is to ship strict and add the hatch - only if a real need appears, but this is a product decision. -- **Should there be a way to suppress a bundled extension?** Same question from - the other direction: a "deny" list would let an operator take an image - without one of its extensions. It has no analogue in the current API and no - concrete requester yet. -- **Should `path` be required rather than optional?** Optional-with-defaults is - proposed for symmetry with the Host Configuration. Requiring it would make - every COSI explicit about placement at the cost of diverging from the object - it is meant to resemble. -- **Naming.** `extensions` containing `sysexts` / `confexts` versus two - top-level `sysexts` / `confexts` arrays. The nested form keeps the root - object tidy and groups the feature; the flat form is a shorter path - expression and an even closer match to `os.sysexts`. -- **Should ID uniqueness be enforced for Host Configuration extensions - independently of this work?** It is documented but unenforced today. This - proposal needs it for the merged set; it is arguably a pre-existing gap that - deserves its own fix. -- **Should the writer be required to place extension payloads after all region - images,** or merely encouraged? The proposal says SHOULD. +- **Should an override escape hatch exist?** This RFC makes a collision between + the Host Configuration and the COSI an error. The alternative is an explicit + opt-in on the Host Configuration side meaning "the image ships this, use mine + instead". It permits pinning a hotfixed extension without rebuilding the image, + and equally permits running a combination the image author did not validate. + The recommendation is to ship strict and add the hatch if a need appears. +- **Should a bundled extension be suppressible?** A deny list would let an + operator take an image without one of its extensions. There is no analogue in + the current API and no concrete requester. +- **Should `path` be required?** Optional with defaults is proposed for symmetry + with the Host Configuration. Requiring it would make every COSI explicit about + placement, at the cost of diverging from the object it mirrors. +- **Naming.** `extensions` containing `sysexts` and `confexts`, against two + top-level `sysexts` and `confexts` arrays. The nested form groups the feature + and keeps the root object small; the flat form is a closer match to + `os.sysexts`. +- **Should Host Configuration extension ID uniqueness be enforced independently + of this work?** It is documented but unenforced. This RFC requires it for the + merged set, but it is arguably a pre-existing gap deserving its own fix. +- **Should writers be required to place extension payloads after all region + images, or merely encouraged?** This RFC says SHOULD. ## Future Possibilities - **Portable service images.** They are DDIs with the same shape and the same - placement problem. If Trident ever manages them, `extensions` generalises - naturally or gains a sibling. -- **Per-extension selection at deploy time.** If a real need appears for a - single image serving hosts with different extension sets, a selection - mechanism keyed on extension ID would build on this section. -- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` extensions are already - parsed by `ExtensionRelease` but not acted on. A bundled extension is a - natural fit for initrd scope, since the payload is available before the root - filesystem is. -- **Extension inventory in Host Status.** Reporting the merged extension set, - with IDs and hashes, would make "what is actually running on this host" - answerable without inspecting the filesystem. + placement problem. If Trident manages them, `extensions` generalises or gains a + sibling. +- **Per-extension selection at deploy time.** If a single image must serve hosts + with different extension sets, a selection mechanism keyed on extension ID + would build on this section. +- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed by + `ExtensionRelease` but not acted on. A bundled extension suits initrd scope, + since the payload is available before the root filesystem is. +- **Extension inventory in Host Status.** Reporting the merged set, with IDs and + hashes, would make the set of running extensions answerable without inspecting + the filesystem. From ddab67ab33fb7407085adf612868655048b2212c Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 8 Sep 2026 18:24:18 -0700 Subject: [PATCH 03/15] docs: rename RFC to 0789 per PR number Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...cosi-extension-images.md => 0789-cosi-extension-images.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename docs/Development/RFCs/{0000-cosi-extension-images.md => 0789-cosi-extension-images.md} (99%) diff --git a/docs/Development/RFCs/0000-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md similarity index 99% rename from docs/Development/RFCs/0000-cosi-extension-images.md rename to docs/Development/RFCs/0789-cosi-extension-images.md index 793b9c1ca9..d3d3ffea20 100644 --- a/docs/Development/RFCs/0000-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -1,7 +1,7 @@ -# 0000 COSI Extension Images +# 0789 COSI Extension Images - Date: 2026-09-08 -- RFC PR: [microsoft/trident#0000](https://github.com/microsoft/trident/pull/0000) +- RFC PR: [microsoft/trident#789](https://github.com/microsoft/trident/pull/789) - Issue: [microsoft/trident#0000](https://github.com/microsoft/trident/issues/0000) ## Summary From 21228ad061ab98ebb1ed1757df88e9e388edaea5 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 9 Sep 2026 16:33:41 -0700 Subject: [PATCH 04/15] docs: add extension-only update path to COSI extensions RFC An extension-only change to a COSI changes the metadata hash, which forces an A/B update via ab_update_required() before any subsystem is consulted. Adds an OS content digest recorded in Host Status, and a requested-then-verified Runtime Update path gated on digest equality, with refusal as the only failure mode. Rejects an extension-only COSI variant and records the weaker rollback guarantee of the runtime path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 118 +++++++++++++++--- 1 file changed, 98 insertions(+), 20 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index d3d3ffea20..16c7a2ec15 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -75,14 +75,17 @@ COSI is built. This works in some configurations, but: compression and integrity rules. - Trident deploys bundled extensions on Clean Install and A/B Update, reusing the extensions subsystem. +- A change confined to the extension set is applicable as a Runtime Update, + without rewriting partitions or rebooting. See + [Extension-Only Updates](#extension-only-updates). - Bundled and Host Configuration extensions coexist; conflicts are an error. - COSI metadata validation covers the new section. ### Out of Scope - Changes to the Host Configuration `Extension` object. -- Runtime Update of bundled extensions. A bundled extension can only change when - the image changes, which is an A/B update. +- Extension-only COSI files, meaning a COSI carrying extensions and no partition + images. See [Extension-Only Updates](#extension-only-updates). - Signing or attestation beyond the SHA-384 chain COSI already provides. - SELinux compatibility. That limitation is unchanged; see [SELinux](#selinux). - Producing the DDIs. This RFC specifies how a COSI carries an extension image, @@ -390,15 +393,9 @@ directly. `{mount_path}/var/lib/extensions/.staging` and moved to their destination inside the target root, as today. For A/B the destination is on the inactive slot, so the running system is untouched until reboot. -- **Runtime Update.** Bundled extensions do not participate. A bundled extension - can only change if the COSI changes, which is an image change and forces an - A/B update. Host Configuration extensions remain runtime-updatable. - -This yields an invariant: Trident never needs to read the previous COSI. If -`spec.os.image` equals `spec_old.os.image`, the previous bundled set is the -current one. If they differ, the servicing type is an A/B update, and -`set_up_extensions` already skips removal of the old set on Clean Install and -A/B Update. +- **Runtime Update.** `provision()` is not called, so partitions are never + touched. Extensions are staged and placed in the running root. See + [Extension-Only Updates](#extension-only-updates). #### Rollback @@ -409,6 +406,68 @@ the previous extension set after an update. An A/B rollback boots the previous root volume, `systemd-sysext` merges what it finds there, and the extension set reverts with the OS. No additional bookkeeping is required. +This applies to the A/B path. Rollback of an extension-only Runtime Update is +weaker; see [Extension-Only Updates](#extension-only-updates). + +#### Extension-Only Updates + +`select_servicing_type` short-circuits: `ab_update_required()` returns true +whenever `os.image.sha384` differs, before any subsystem is consulted. The +metadata hash covers the `extensions` section, so changing only a bundled +extension changes the hash and forces a full A/B update and a reboot. Without +further work, bundling would make an extension update strictly more disruptive +than it is today. + +An extension-only COSI is the wrong shape for this. The metadata root requires +`images`, `disk` with at least one `gptRegions` entry, `bootloader` and +`osPackages`, so a COSI without partition images is not representable, and +making it representable would require those fields to become optional. Every +consumer of `os.image` — filesystem source resolution, +`derive_host_configuration`, ESP detection, verity setup — assumes the image +describes a complete OS. The distinction should be drawn on content equality, +not on absent content. + +**OS content digest.** Define a digest over everything in the metadata that +determines the deployed OS: the `sha384` of every entry in `images[]` and +`disk.gptRegions[]`, the `disk` geometry, `bootloader`, `osArch` and +`osRelease`, excluding `extensions`. Trident records the digest of the deployed +image in the Host Status. Two COSIs with equal digests deploy an identical OS +and differ only in their extensions. + +Trident cannot determine this otherwise. It retains the URL and metadata hash of +the applied image, not the metadata itself, and re-fetching the previous COSI to +diff it is not dependable. + +**The path is requested and then verified, never inferred.** An extension-only +update proceeds when all of the following hold: + +1. A Runtime Update is explicitly requested. +2. The new COSI's OS content digest equals the digest recorded for the deployed + image. +3. The recorded image is the deployed one: servicing completed, no A/B update + pending, and the active volume matches the Host Status. + +If a Runtime Update is requested and condition 2 or 3 does not hold, Trident +fails with a structured error naming what differs. It must not promote +the operation to an A/B update, and must not apply the extension change while +leaving other changes unapplied. Refusal is the only acceptable failure. + +There is no surface for requesting a servicing type today; `forceAbUpdate` is +the nearest precedent. The surface is an [open question](#open-questions). + +**Rollback is weaker on this path.** An A/B extension update is rolled back by +booting the other slot. A Runtime Update replaces files in the running slot, +because `set_up_extensions` skips removal of the superseded image only on Clean +Install and A/B Update. Rollback therefore re-runs the subsystem with the specs +reversed and re-acquires the previous extension image from its source. For a +bundled extension that source is the previous COSI, so a runtime rollback +requires the previous COSI to remain reachable; an A/B rollback requires +nothing. + +This is the reason the path is not automatic. Selecting it trades a rollback +guarantee for the absence of a reboot, and that trade should be made by the +operator rather than inferred from a property of two images. + #### Interaction With Host Configuration `sysexts` and `confexts` The two sets are merged, and a collision is an error. @@ -534,10 +593,13 @@ is now an error rather than a duplicate. Install places both, enables the merge units, and that both merge after boot. - **Servicing.** A/B update from a COSI with extension set A to one with set B; assert the new slot has B, the old slot retains A, and that rollback restores - A without additional servicing. + A without additional servicing. Separately, an extension-only Runtime Update + between two COSIs with equal OS content digests; assert partitions are + untouched and the extension set changes without a reboot. - **Negative.** The same extension in both the Host Configuration and the COSI produces the structured error; a bundled extension with SELinux `enforcing` is - rejected. + rejected; a Runtime Update requested between two COSIs with differing OS + content digests is refused rather than promoted or partially applied. ## Servicing @@ -561,7 +623,9 @@ The change is additive at every layer. `selinux` subsystems, including the collision errors. 4. Stream a tar member into the extension staging directory, replacing the URL fetch for bundled entries. -5. Tests, then documentation updates to +5. OS content digest, its record in the Host Status, and the extension-only + Runtime Update path with its verification and refusal behaviour. +6. Tests, then documentation updates to [Sysexts](../../Explanation/Sysexts.md), [Confexts](../../Explanation/Confexts.md) and [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). @@ -575,12 +639,13 @@ warning available sooner. ### Drawbacks - **Cadence coupling.** Bundling ties the extension's release cadence to the - image's: a new version of the extension requires a new COSI, an A/B update and - a reboot. For content that must match the OS, such as kernel modules or GPU - drivers, this costs nothing, because that content could not move independently - in any case. For content deliberately versioned independently of the OS it is a - real loss of agility, and the Host Configuration route remains correct. This - RFC adds an option; it removes none. + image's: a new version of the extension requires a new COSI. + [Extension-Only Updates](#extension-only-updates) removes the reboot, but not + the image rebuild and republication. For content that must match the OS, such + as kernel modules or GPU drivers, this costs nothing, because that content + could not move independently in any case. For content deliberately versioned + independently of the OS it remains a loss of agility, and the Host + Configuration route stays correct. This RFC adds an option; it removes none. - **Size.** COSI files grow, and every host downloading the image pays for every bundled extension, including those it does not use. There is no per-host selection mechanism. @@ -628,6 +693,19 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against ## Open Questions +- **What surface requests an extension-only Runtime Update?** A `trident update` + flag, an internal parameter alongside `forceAbUpdate`, or a Host Configuration + field. This RFC requires the request to be explicit but does not choose. +- **What does the OS content digest cover?** The proposal includes `images[]`, + `disk.gptRegions[]`, `disk` geometry, `bootloader`, `osArch` and `osRelease`. + `osPackages` and `id` are excluded, since neither affects the deployed bytes. + Excluding too much permits a Runtime Update between images that differ in ways + Trident does not model; including too much rejects legitimate cases. +- **Should the digest be published in the COSI metadata rather than computed?** + Computing it in Trident requires no writer cooperation and cannot be + misdeclared. Publishing it would let a writer state intent directly, at the + cost of trusting the writer on a decision that governs whether partitions are + rewritten. - **Should an override escape hatch exist?** This RFC makes a collision between the Host Configuration and the COSI an error. The alternative is an explicit opt-in on the Host Configuration side meaning "the image ships this, use mine From 8771b1077fee2e74942597e59c39aded7bc016a0 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 9 Sep 2026 16:38:05 -0700 Subject: [PATCH 05/15] docs: record a deployed image summary and define extension-only construction Replaces the OS content digest with a structured summary of the deployed image's metadata, recorded in the Host Status, so Trident can distinguish a partition change from an extension-only change from a no-op, and report which partition differs. Falls back to the current os.image.sha384 comparison where no summary is recorded. Adds the construction requirement: filesystem images are not reproducible, so an extension-only COSI must be built by copying an existing COSI's region images verbatim rather than by rebuilding. An unreproducible build degrades to a redundant A/B update, never a skipped one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 164 +++++++++++++----- 1 file changed, 119 insertions(+), 45 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 16c7a2ec15..ee8b98d660 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -106,7 +106,14 @@ COSI is built. This works in some configurations, but: A COSI writer that emits the section. [Image Customizer](https://github.com/microsoft/azure-linux-image-tools) is the -reference writer. No incomplete Trident features are required. +reference writer. + +[Extension-Only Updates](#extension-only-updates) additionally require a writer +mode that copies an existing COSI's region images verbatim while replacing its +extension set. Without it, bundled extensions still work, but every extension +change is an A/B update. + +No incomplete Trident features are required. ## Implementation @@ -427,42 +434,102 @@ consumer of `os.image` — filesystem source resolution, describes a complete OS. The distinction should be drawn on content equality, not on absent content. -**OS content digest.** Define a digest over everything in the metadata that -determines the deployed OS: the `sha384` of every entry in `images[]` and -`disk.gptRegions[]`, the `disk` geometry, `bootloader`, `osArch` and -`osRelease`, excluding `extensions`. Trident records the digest of the deployed -image in the Host Status. Two COSIs with equal digests deploy an identical OS -and differ only in their extensions. +##### Deployed Image Summary + +Trident retains the URL and metadata hash of the applied image, not the metadata +itself, and re-fetching the previous COSI to diff it is not dependable. It must +therefore record what it deployed. + +Trident records a summary of the deployed image's metadata in the Host Status: +for each entry in `images[]` and `disk.gptRegions[]`, the `image.sha384`, +`uncompressedSize` and the entry's identity (partition number, mount point, +`fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, the `disk` +geometry and `bootloader`. Excluded: + +- `extensions`, which is the subject of the comparison. +- `osPackages`, which Trident validates but never acts on. +- `compression.maxWindowLog`, which governs decompression rather than deployed + content, and which a new extension may legitimately raise. +- `id`, which identifies the file rather than its content. + +A structured summary rather than a single digest: the digest is derivable from +it, and the summary additionally allows Trident to report which partition +differs rather than only that something does. + +The summary carries its own schema version. Where the recorded summary is absent +or of an unrecognised version, Trident falls back to comparing +`os.image.sha384`, which is current behaviour. This is the upgrade path for +existing hosts and the escape hatch if the summary's definition changes. + +Comparing the summary against the new COSI's metadata gives three outcomes: + +| Summary | Extension set | Result | +| -------- | ------------- | -------------------------------------------------- | +| Equal | Equal | No-op. | +| Equal | Differs | Extension-only change; a Runtime Update suffices. | +| Differs | Any | A/B update. | -Trident cannot determine this otherwise. It retains the URL and metadata hash of -the applied image, not the metadata itself, and re-fetching the previous COSI to -diff it is not dependable. +The no-op case is new. `os.image.sha384` changes whenever any part of the +metadata changes, including `id` and the ordering of `osPackages`, so a +materially identical image currently forces an A/B update. -**The path is requested and then verified, never inferred.** An extension-only -update proceeds when all of the following hold: +This modifies `ab_update_required()`, which governs a core decision. The +fallback above confines the change to hosts that have a recorded summary, so +behaviour for existing hosts is unaltered until they are next deployed. + +##### Constructing an Extension-Only Update + +Equality of the summary requires the region images to be byte-identical, and +filesystem images are not naturally reproducible. Filesystem UUIDs, inode +timestamps, superblock creation and mount times, and allocation order all vary +between builds, and ZSTD output varies with compression level and library +version. Rebuilding from the same source will not generally produce the same +images. + +Requiring reproducible rebuilds is therefore the wrong approach. The build +operation is not "rebuild the image identically with a different extension set", +it is "copy an existing COSI, replacing its extension set". Region images and +their metadata entries are copied verbatim from the input COSI; no filesystem is +rebuilt and no region image is recompressed. Only the extension tar members, the +`extensions` section, `compression.maxWindowLog` where the new extensions +require a larger window, and `id` change. + +Equality is then guaranteed by construction rather than hoped for through +reproducibility, and the operation is a tar rewrite rather than an image build. +This is the mode a COSI writer would need to provide. + +Where a COSI is instead rebuilt from source, its region images will differ, the +summary will differ, and Trident will select an A/B update. This is the safe +outcome: the mechanism degrades to current behaviour rather than misclassifying +a changed OS as unchanged. The failure mode of an unreproducible build is a +redundant A/B update, never a skipped one. + +##### Requested and Verified, Never Inferred + +An extension-only update proceeds when all of the following hold: 1. A Runtime Update is explicitly requested. -2. The new COSI's OS content digest equals the digest recorded for the deployed - image. +2. The recorded summary equals the summary computed from the new COSI. 3. The recorded image is the deployed one: servicing completed, no A/B update pending, and the active volume matches the Host Status. If a Runtime Update is requested and condition 2 or 3 does not hold, Trident -fails with a structured error naming what differs. It must not promote -the operation to an A/B update, and must not apply the extension change while +fails with a structured error naming what differs. It must not promote the +operation to an A/B update, and must not apply the extension change while leaving other changes unapplied. Refusal is the only acceptable failure. There is no surface for requesting a servicing type today; `forceAbUpdate` is the nearest precedent. The surface is an [open question](#open-questions). -**Rollback is weaker on this path.** An A/B extension update is rolled back by -booting the other slot. A Runtime Update replaces files in the running slot, -because `set_up_extensions` skips removal of the superseded image only on Clean -Install and A/B Update. Rollback therefore re-runs the subsystem with the specs -reversed and re-acquires the previous extension image from its source. For a -bundled extension that source is the previous COSI, so a runtime rollback -requires the previous COSI to remain reachable; an A/B rollback requires -nothing. +##### Rollback on This Path + +An A/B extension update is rolled back by booting the other slot. A Runtime +Update replaces files in the running slot, because `set_up_extensions` skips +removal of the superseded image only on Clean Install and A/B Update. Rollback +therefore re-runs the subsystem with the specs reversed and re-acquires the +previous extension image from its source. For a bundled extension that source is +the previous COSI, so a runtime rollback requires the previous COSI to remain +reachable; an A/B rollback requires nothing. This is the reason the path is not automatic. Selecting it trades a rollback guarantee for the absence of a reboot, and that trade should be made by the @@ -594,12 +661,12 @@ is now an error rather than a duplicate. - **Servicing.** A/B update from a COSI with extension set A to one with set B; assert the new slot has B, the old slot retains A, and that rollback restores A without additional servicing. Separately, an extension-only Runtime Update - between two COSIs with equal OS content digests; assert partitions are - untouched and the extension set changes without a reboot. + between a COSI and a copy of it with a different extension set; assert + partitions are untouched and the extension set changes without a reboot. - **Negative.** The same extension in both the Host Configuration and the COSI produces the structured error; a bundled extension with SELinux `enforcing` is - rejected; a Runtime Update requested between two COSIs with differing OS - content digests is refused rather than promoted or partially applied. + rejected; a Runtime Update requested between two COSIs with differing image + summaries is refused rather than promoted or partially applied. ## Servicing @@ -612,6 +679,9 @@ The change is additive at every layer. [Versioning](#versioning). - Hosts using `os.sysexts` or `os.confexts` are unaffected unless they move to an image carrying the same extension, which now produces an explicit error. +- Existing hosts have no recorded image summary, so servicing type selection is + unchanged for them until their next deployment. See + [Deployed Image Summary](#deployed-image-summary). ## Implementation Plan @@ -623,7 +693,7 @@ The change is additive at every layer. `selinux` subsystems, including the collision errors. 4. Stream a tar member into the extension staging directory, replacing the URL fetch for bundled entries. -5. OS content digest, its record in the Host Status, and the extension-only +5. Deployed image summary recorded in the Host Status, and the extension-only Runtime Update path with its verification and refusal behaviour. 6. Tests, then documentation updates to [Sysexts](../../Explanation/Sysexts.md), @@ -640,12 +710,13 @@ warning available sooner. - **Cadence coupling.** Bundling ties the extension's release cadence to the image's: a new version of the extension requires a new COSI. - [Extension-Only Updates](#extension-only-updates) removes the reboot, but not - the image rebuild and republication. For content that must match the OS, such - as kernel modules or GPU drivers, this costs nothing, because that content - could not move independently in any case. For content deliberately versioned - independently of the OS it remains a loss of agility, and the Host - Configuration route stays correct. This RFC adds an option; it removes none. + [Extension-Only Updates](#extension-only-updates) removes the reboot, and + reduces the build to a copy, but a new COSI must still be published and + distributed. For content that must match the OS, such as kernel modules or GPU + drivers, this costs nothing, because that content could not move independently + in any case. For content deliberately versioned independently of the OS it + remains a loss of agility, and the Host Configuration route stays correct. This + RFC adds an option; it removes none. - **Size.** COSI files grow, and every host downloading the image pays for every bundled extension, including those it does not use. There is no per-host selection mechanism. @@ -696,16 +767,19 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against - **What surface requests an extension-only Runtime Update?** A `trident update` flag, an internal parameter alongside `forceAbUpdate`, or a Host Configuration field. This RFC requires the request to be explicit but does not choose. -- **What does the OS content digest cover?** The proposal includes `images[]`, - `disk.gptRegions[]`, `disk` geometry, `bootloader`, `osArch` and `osRelease`. - `osPackages` and `id` are excluded, since neither affects the deployed bytes. - Excluding too much permits a Runtime Update between images that differ in ways - Trident does not model; including too much rejects legitimate cases. -- **Should the digest be published in the COSI metadata rather than computed?** - Computing it in Trident requires no writer cooperation and cannot be - misdeclared. Publishing it would let a writer state intent directly, at the - cost of trusting the writer on a decision that governs whether partitions are - rewritten. +- **What does the deployed image summary cover?** The proposal includes the + region and filesystem image hashes and identities, `disk` geometry, + `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, + `osPackages`, `compression` and `id`. Excluding too much permits a Runtime + Update between images differing in ways Trident does not model; including too + much rejects legitimate cases. `version` is currently excluded, on the basis + that a revision bump with identical images deploys identical bytes. +- **Should the summary be a structured record or a digest?** A record is + proposed, so that Trident can report which partition differs. A digest is + smaller and simpler, at the cost of the diagnosis. +- **Is the no-op outcome desirable?** Recognising a materially identical image + and doing nothing is a behavioural change independent of extensions, and + arguably belongs in its own RFC. - **Should an override escape hatch exist?** This RFC makes a collision between the Host Configuration and the COSI an error. The alternative is an explicit opt-in on the Host Configuration side meaning "the image ships this, use mine From c09e46cf6fdcbf9905f85edf72fc3d76b5ec872c Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:28:25 -0700 Subject: [PATCH 06/15] docs: address review findings in COSI extensions RFC - Add a deployed extension inventory in Host Status; the Host Configuration cannot represent bundled extensions, so spec_old does not describe what is deployed. - Give effective entries a source discriminator; update_host_configuration errors when a processed extension has no matching Host Configuration entry, and compressed and raw sha384 values must never be compared against one another. - Record three prerequisite defects: in-place replacement deletes the newly installed image when ID and destination are unchanged; runtime finalize and rollback build their context with image: None so staged payloads must be durable; extension placement is validated against the Host Configuration only and before the COSI is read. - Correct the A/B rollback claim, which held only for Host Configuration entries, and drop the unsupported read-only assertion. - SELinux check must move to a hook running on every servicing type; it sits in configure(), which returns early for runtime updates. - Split metadata-time from payload-time validation; omitted paths cannot be collision-checked statically, and there is no load-time member existence check since tar entries are discovered lazily. - Warn on extension-release ID mismatch only; systemd accepts a VERSION_ID mismatch when SYSEXT_LEVEL matches. - Note that an omission by an older reader does not self-heal, and that a no-op must still persist the accepted image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 251 +++++++++++++----- 1 file changed, 189 insertions(+), 62 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index ee8b98d660..19cb2ba07f 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -108,12 +108,35 @@ A COSI writer that emits the section. [Image Customizer](https://github.com/microsoft/azure-linux-image-tools) is the reference writer. -[Extension-Only Updates](#extension-only-updates) additionally require a writer -mode that copies an existing COSI's region images verbatim while replacing its -extension set. Without it, bundled extensions still work, but every extension -change is an A/B update. - -No incomplete Trident features are required. +[Extension-Only Updates](#extension-only-updates) additionally require: + +- A writer mode that copies an existing COSI's region images verbatim while + replacing its extension set. Without it, bundled extensions still work, but + every extension change is an A/B update. +- A surface for requesting a servicing type. None exists today; + `forceAbUpdate` is the nearest precedent. This is a prerequisite, not an + optional refinement, because the path is never inferred. + +Three defects in the existing extensions subsystem must be fixed first. All are +reachable today with Host Configuration extensions; bundling makes the first two +the common case rather than the exception. + +1. **In-place replacement deletes the new image.** When an extension keeps its + ID and destination but changes content, `set_up_extensions` schedules the ID + for both addition and removal. The addition renames the new image over the + destination; the removal then deletes that same path, because the old + entry's `temp_path` is its destination. The guard assumes the two paths + differ. A bundled extension updated in place hits this on every Runtime + Update. +2. **Staged payloads do not survive the operation.** Runtime finalize and + rollback construct their `EngineContext` with `image: None`, so the COSI is + unavailable after stage. Bundled payloads must be staged durably, and the + superseded image must be retained until the operation completes. +3. **Extension placement is validated against the Host Configuration only, and + before the COSI is read.** `validate_extension_images_locations` is static + validation over `os.sysexts` and `os.confexts`. Bundled destinations need + the same check after the COSI is loaded. See + [Destination Validation](#destination-validation). ## Implementation @@ -317,9 +340,15 @@ For integrity this is equivalent to partition images: Trident hashes the compressed stream as it decompresses, and a match proves the payload is intact. A second hash of the uncompressed DDI would add nothing. -One consequence should be recorded. `ExtensionData.sha384` is also used as a -change detector: for an extension whose ID is present in both the old and new -configuration, a differing hash means the content changed. Compressed hashes are +One consequence must be handled. `ExtensionData.sha384` is a single field used +both as the change detector and as the key matching a processed extension back +to its Host Configuration entry. It cannot hold two different meanings at once, +so the effective entry must record the hash together with its source, and +comparisons must only be made between hashes of the same kind. Comparing a +compressed hash against a raw-file hash is always unequal and would report +every extension as changed. + +Within one source the comparison is sound but imprecise: compressed hashes are not stable across recompressions, so two COSIs built from identical DDIs at different ZSTD levels appear to differ. The result is a redundant copy of an identical file, never a missed update. @@ -351,6 +380,13 @@ field is a field they are required to ignore. Three partial mitigations: asserting the extension is merged, so an A/B update onto a too-old Trident fails its gate and rolls back. +The omission does not self-heal. An old reader records the new COSI's URL and +hash as deployed, so after Trident is upgraded the same Host Configuration no +longer represents an image change and the skipped extensions are never +installed. Recovery requires a forced redeployment or a republished COSI. This +is the strongest argument for the health check above rather than a documented +caveat alone. + ### Trident-Side Consumption The extensions subsystem gains a second source. Trident computes an **effective @@ -367,6 +403,10 @@ flowchart LR Eff --> Sel["selinux subsystem
enforcing-mode rejection"] ``` +Each effective entry carries its source, Host Configuration or COSI. The source +determines where a resolved path is written back and whether the entry is +subject to Host Configuration validation. + There are three consumers: - **`ExtensionsSubsystem`.** `populate_extensions` currently downloads each @@ -375,24 +415,58 @@ There are three consumers: bundled entry only the source changes: the image streaming pipeline decompresses the tar member into the same staging directory and verifies `ImageFile.sha384` over the compressed stream. The `extension-release` read, - the `{name}.raw` check, default-path resolution, directory creation and - `set_up_extensions` are unchanged. + the `{name}.raw` check and default-path resolution are unchanged. + `update_host_configuration` currently fails if a processed extension has no + Host Configuration entry matching by hash, so it must write back resolved + paths for Host-Configuration-sourced entries only. - **`osconfig`.** `systemd-sysext.service` and `systemd-confext.service` are enabled only when `ctx.spec.os.sysexts` or `confexts` are non-empty. A host whose extensions come only from the COSI would never have the merge units enabled, and the extensions would remain on disk unused. This check must read the effective set. - **`selinux`.** The dynamic validation raising - `ExtensionImagesAndSelinuxUnsupported` has the same problem. Without the - change, a COSI carrying extensions combined with an enforcing-mode Host - Configuration passes validation and produces a host with a mislabelled `/usr`, - `/opt` or `/etc`. + `ExtensionImagesAndSelinuxUnsupported` must also read the effective set, or a + COSI carrying extensions combined with an enforcing-mode Host Configuration + passes validation and produces a host with a mislabelled `/usr`, `/opt` or + `/etc`. Reading the effective set is not sufficient on its own: the check runs + inside `configure`, which returns early for Runtime Updates, and + `SelinuxSubsystem` inherits `runs_on = REQUIRES_REBOOT`. It must move to a + validation hook that runs on every servicing type, otherwise + [Extension-Only Updates](#extension-only-updates) bypass it. `derive_host_configuration`, used by [disk streaming](../../Explanation/Disk-Streaming.md), does not need to synthesise extension entries; the effective set is computed from the COSI directly. +#### Deployed Extension Inventory + +The Host Configuration records the extensions an operator asked for. It cannot +record bundled ones, so `ctx.spec_old.os.sysexts` and `confexts` do not describe +what is actually deployed. Without a second record, Trident cannot determine +which bundled extensions to remove, detect collisions against the deployed set, +compare extension sets for servicing selection, or reconstruct its state when +finalize runs separately from stage. + +Trident therefore records a deployed extension inventory in the Host Status, +alongside the [deployed image summary](#deployed-image-summary). Per entry: +kind, extension ID, name, resolved destination, source, and the hash under that +source's semantics. The subsystem populates `extensions_old` from the inventory +rather than from `ctx.spec_old` alone. + +#### Destination Validation + +`validate_extension_images_locations` rejects extension destinations that are +not on an A/B volume when A/B is configured. It is static validation over the +Host Configuration lists, and runs before the COSI is read, so it cannot see +bundled destinations. + +The equivalent check for the effective set must run after the COSI is loaded, as +dynamic validation against the storage graph. It must reject a destination that +is shared when A/B is configured, and should additionally reject a destination +that is read-only or verity-protected — a case the existing check does not +cover for Host Configuration entries either. + #### Where Images Are Written - **Clean Install and A/B Update.** `provision()` runs with the target root @@ -406,15 +480,19 @@ directly. #### Rollback -Extension images are files in the slot's own filesystem, and Trident already -requires every extension destination to be on an A/B volume, not shared and not -read-only, when A/B volumes are configured. The previous slot therefore retains -the previous extension set after an update. An A/B rollback boots the previous -root volume, `systemd-sysext` merges what it finds there, and the extension set -reverts with the OS. No additional bookkeeping is required. +Extension images are files in the slot's own filesystem. When every destination +is on an A/B volume, the previous slot retains the previous extension set after +an update, an A/B rollback boots the previous root volume, `systemd-sysext` +merges what it finds there, and the extension set reverts with the OS. No +additional bookkeeping is required. + +That guarantee is conditional on the placement check covering bundled +destinations; see [Destination Validation](#destination-validation). A bundled +extension written to a shared volume changes the running slot immediately and +cannot be rolled back. -This applies to the A/B path. Rollback of an extension-only Runtime Update is -weaker; see [Extension-Only Updates](#extension-only-updates). +Rollback of an extension-only Runtime Update is weaker; see +[Extension-Only Updates](#extension-only-updates). #### Extension-Only Updates @@ -519,21 +597,33 @@ operation to an A/B update, and must not apply the extension change while leaving other changes unapplied. Refusal is the only acceptable failure. There is no surface for requesting a servicing type today; `forceAbUpdate` is -the nearest precedent. The surface is an [open question](#open-questions). +the nearest precedent. See [Dependencies](#dependencies). + +A no-op must still record the accepted image and summary in the Host Status. +`select_servicing_type` returning `NoActiveServicing` causes `update` to return +immediately without persisting anything, so without a metadata-only status +transition every subsequent invocation would repeat the comparison. ##### Rollback on This Path An A/B extension update is rolled back by booting the other slot. A Runtime Update replaces files in the running slot, because `set_up_extensions` skips removal of the superseded image only on Clean Install and A/B Update. Rollback -therefore re-runs the subsystem with the specs reversed and re-acquires the -previous extension image from its source. For a bundled extension that source is -the previous COSI, so a runtime rollback requires the previous COSI to remain -reachable; an A/B rollback requires nothing. - -This is the reason the path is not automatic. Selecting it trades a rollback -guarantee for the absence of a reboot, and that trade should be made by the -operator rather than inferred from a property of two images. +re-runs the subsystem with the specs reversed and must restore the previous +image. + +It cannot re-fetch it. Runtime finalize and rollback build their +`EngineContext` with `image: None`, so the COSI is not available after stage, +and auto-rollback runs unconditionally after a finalize failure. The superseded +image must therefore be retained in the staging directory for the duration of +the operation, and removed only once it completes. This is the second +prerequisite in [Dependencies](#dependencies). + +Rollback is consequently bounded by the staging directory surviving, rather than +by booting a slot that was never touched. This is the reason the path is not +automatic: selecting it trades a rollback guarantee for the absence of a reboot, +and that trade should be made by the operator rather than inferred from a +property of two images. #### Interaction With Host Configuration `sysexts` and `confexts` @@ -571,28 +661,37 @@ Host Configuration side as an opt-in. See [Open Questions](#open-questions). #### COSI Metadata Validation -New `CosiMetadataErrorKind` variants, following the existing `V1_` -prefix: +Checks that need only the metadata. New `CosiMetadataErrorKind` variants, +following the existing `V1_` prefix: | Variant | Condition | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `V1_3ExtensionDestinationPathNotAbsolute` | `path` is present and not absolute. | | `V1_3ExtensionDestinationPathInvalidFileExtension` | `path` is present and does not end in `.raw`. | | `V1_3ExtensionDestinationPathInvalidDirectory` | `path`'s parent is not in `VALID_SYSEXT_DIRECTORIES` or `VALID_CONFEXT_DIRECTORIES` for the kind. | -| `V1_3DuplicateExtensionDestinationPath` | Two entries resolve to the same destination path. | +| `V1_3DuplicateExtensionDestinationPath` | Two entries both specify `path` and specify the same one. | | `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | | `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | -The existing check that every image referenced by the metadata is present in the -tar must also walk `extensions.sysexts[]` and `extensions.confexts[]`, so a -missing tar member fails at load rather than mid-provision. - Directory and file-extension rules reuse the constants and logic behind `Extension::validate_sysext` and `validate_confext` rather than reimplementing them. +Destination collisions are only partly detectable here. An entry that omits +`path` resolves to `{default directory}/{name}.raw`, and `{name}` comes from the +`extension-release` file inside the DDI, so two omitted paths may collide +without the metadata showing it. That case is caught after mounting. + +Member existence is not checked at load. `Cosi::new` scans tar entries only as +far as `metadata.json` and discovers the rest on demand, so a reference to a +missing member surfaces when the member is read. Changing this would require +scanning the whole archive up front, which conflicts with sparse reads. The +`extensions` section inherits the existing behaviour. + #### Deploy-Time Validation +Checks that require the payload, performed after the DDI is mounted. + - **File name matches the `extension-release` suffix.** Already enforced by `read_extension_release`; applies unchanged. - **Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID` @@ -600,14 +699,20 @@ them. - **SHA-384 agreement.** `ImageFile.sha384` is verified over the compressed stream as the payload is decompressed into the staging directory, using the hashing reader used for partition images. A mismatch aborts servicing. -- **Extension ID uniqueness across the merged set**, per kind. New check. +- **Resolved destination collisions** across the effective set, covering the + omitted-`path` cases the metadata check cannot see. New check. +- **Extension ID uniqueness** across the effective set, per kind. New check. +- **Destination placement** against the storage graph. See + [Destination Validation](#destination-validation). - **`extension-release` OS compatibility.** Warn, do not fail, when a bundled - extension declares `ID=` and that `ID` or `VERSION_ID` does not match - the COSI's own `osRelease`. This is a build error worth surfacing early, since - systemd will refuse to merge the extension at boot and the host will come up - without it. systemd remains the authority: its matching rules - (`SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE`, `_any`) are intricate enough - that Trident should not reimplement them. + extension declares `ID=` and that `ID` does not match the COSI's own + `osRelease`. This is a build error worth surfacing early, since systemd will + refuse to merge the extension at boot and the host will come up without it. + + The warning is on `ID` alone. systemd accepts a `VERSION_ID` mismatch when the + applicable `SYSEXT_LEVEL` or `CONFEXT_LEVEL` matches instead, so warning on + version would flag valid extensions. Trident should not reproduce that + precedence; systemd remains the authority. #### `ID=_any` @@ -628,10 +733,12 @@ Unchanged. Extensions remain incompatible with SELinux in enforcing mode on systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. Carrying the image in a COSI does not affect labelling. -What must change is where the check looks. The dynamic validation raising -`ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration lists -only, and must inspect the effective set; otherwise a COSI-only extension -configuration bypasses it. +What must change is where the check looks and when it runs. The validation +raising `ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration +lists only, and runs inside `configure`, which returns early for anything other +than Clean Install and A/B Update. It must read the effective set and run on +every servicing type, or both a COSI-only configuration and an +[extension-only Runtime Update](#extension-only-updates) bypass it. ## Public API Design @@ -654,8 +761,9 @@ is now an error rather than a duplicate. `images/`, a non-absolute destination, a destination not ending in `.raw`, a destination in a disallowed directory, and a duplicated destination path. - **Unit.** Metadata parse and validation for each new error variant; effective - set computation, including collisions; default-path resolution for an entry - with no `path`. + set computation, including collisions and the source discriminator; + default-path resolution for an entry with no `path`; replacement of an + extension that keeps its ID and destination but changes content. - **Functional.** Build a COSI carrying a sysext and a confext; assert Clean Install places both, enables the merge units, and that both merge after boot. - **Servicing.** A/B update from a COSI with extension set A to one with set B; @@ -663,10 +771,15 @@ is now an error rather than a duplicate. A without additional servicing. Separately, an extension-only Runtime Update between a COSI and a copy of it with a different extension set; assert partitions are untouched and the extension set changes without a reboot. + Separately again, finalize and rollback of an extension-only Runtime Update + invoked as distinct processes, asserting the previous image is restored + without access to either COSI. - **Negative.** The same extension in both the Host Configuration and the COSI produces the structured error; a bundled extension with SELinux `enforcing` is - rejected; a Runtime Update requested between two COSIs with differing image - summaries is refused rather than promoted or partially applied. + rejected on A/B update and on extension-only Runtime Update; a bundled + destination on a shared volume is rejected when A/B is configured; a Runtime + Update requested between two COSIs with differing image summaries is refused + rather than promoted or partially applied. ## Servicing @@ -685,14 +798,20 @@ The change is additive at every layer. ## Implementation Plan +0. Prerequisites from [Dependencies](#dependencies): the in-place replacement + fix, durable staged payloads, and the request surface. The first is a + standalone bug fix and should land independently. 1. Specification: COSI revision 1.3, the `extensions` section, the v1.3 JSON schema and samples. 2. Trident reader: parse `extensions`, metadata validation and the new error variants, and the minor-version warning. -3. Effective extension set, plumbed into the extensions, `osconfig` and - `selinux` subsystems, including the collision errors. +3. Effective extension set with its source discriminator, and the deployed + extension inventory in the Host Status; plumbed into the extensions, + `osconfig` and `selinux` subsystems, including the collision errors and the + move of the SELinux check to a hook that runs on every servicing type. 4. Stream a tar member into the extension staging directory, replacing the URL - fetch for bundled entries. + fetch for bundled entries. Dynamic destination validation against the storage + graph. 5. Deployed image summary recorded in the Host Status, and the extension-only Runtime Update path with its verification and refusal behaviour. 6. Tests, then documentation updates to @@ -702,7 +821,8 @@ The change is additive at every layer. Steps 1 and 2 are independently useful: a reader that parses and validates the section but ignores it is a safe intermediate state and makes the minor-version -warning available sooner. +warning available sooner. Steps 3 and 4 deliver bundled extensions on Clean +Install and A/B Update; step 5 is separable and could be deferred. ## Counter-Arguments @@ -766,7 +886,9 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against - **What surface requests an extension-only Runtime Update?** A `trident update` flag, an internal parameter alongside `forceAbUpdate`, or a Host Configuration - field. This RFC requires the request to be explicit but does not choose. + field. The request must be explicit; the surface is undecided. It must also + define precedence against `forceAbUpdate` and whether the request persists + across separate stage and finalize invocations. - **What does the deployed image summary cover?** The proposal includes the region and filesystem image hashes and identities, `disk` geometry, `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, @@ -777,9 +899,14 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against - **Should the summary be a structured record or a digest?** A record is proposed, so that Trident can report which partition differs. A digest is smaller and simpler, at the cost of the diagnosis. -- **Is the no-op outcome desirable?** Recognising a materially identical image - and doing nothing is a behavioural change independent of extensions, and - arguably belongs in its own RFC. +- **Does the no-op outcome belong in this RFC?** It requires a metadata-only + status transition that records an accepted image without creating a + rollbackable operation. That is a behavioural change independent of + extensions and may deserve its own RFC. +- **Is retaining the superseded image in the staging directory sufficient for + runtime rollback,** or should the extension subsystem keep a durable + per-extension backup outside the staging directory? The former is simpler; the + latter survives a staging-directory cleanup. - **Should an override escape hatch exist?** This RFC makes a collision between the Host Configuration and the COSI an error. The alternative is an explicit opt-in on the Host Configuration side meaning "the image ships this, use mine From 0da0c8451b61c74ae60cde501fe5755c530298db Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:35:05 -0700 Subject: [PATCH 07/15] docs: address second-pass review findings in COSI extensions RFC - Inventory and image summary need current/pending records, mirroring spec/spec_old, or finalize cannot reconstruct the target set and rollback cannot reconstruct the previous one. - Scope the summary comparison to the image's servicing requirement; select_servicing_type still takes the maximum across subsystems, so an extension-only runtime update also requires no other subsystem to need A/B. - Disk streaming places extensions but osconfig returns early on is_stream_image, so the merge units are never enabled. - Replace the blanket verity rejection: the root-verity /etc overlay is backed by var/lib/trident-overlay on an A/B volume, so /etc/extensions is a supported destination on clean install and A/B update. - Stage on the destination filesystem; the fixed /var/lib/extensions/.staging assumes /var is writable and makes cross-filesystem placement non-atomic. - The permitted-directory allow-list is Trident policy, not a format property; keep it out of the schema and cover it in Rust validation tests. - Move the servicing-type request surface out of Open Questions, where it contradicted its listing as a prerequisite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 97 +++++++++++++------ 1 file changed, 65 insertions(+), 32 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 19cb2ba07f..a65fc404a8 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -113,9 +113,12 @@ reference writer. - A writer mode that copies an existing COSI's region images verbatim while replacing its extension set. Without it, bundled extensions still work, but every extension change is an A/B update. -- A surface for requesting a servicing type. None exists today; - `forceAbUpdate` is the nearest precedent. This is a prerequisite, not an - optional refinement, because the path is never inferred. +- A surface for requesting a servicing type: a `trident update` flag, an + internal parameter alongside `forceAbUpdate`, or a Host Configuration field. + None exists today. This is a prerequisite rather than an optional refinement, + because the path is never inferred, and it must define precedence against + `forceAbUpdate` and whether the request persists across separate stage and + finalize invocations. Three defects in the existing extensions subsystem must be fixed first. All are reachable today with Host Configuration extensions; bundling makes the first two @@ -437,7 +440,11 @@ There are three consumers: `derive_host_configuration`, used by [disk streaming](../../Explanation/Disk-Streaming.md), does not need to synthesise extension entries; the effective set is computed from the COSI -directly. +directly. Streaming is not complete on its own, however: `osconfig` returns +early when `is_stream_image` is set, so the merge units are never enabled and +the extensions would be placed but not merged. Either streaming must enable the +units specifically, or a streamed image must ship with them enabled and the +proposal must say so. #### Deployed Extension Inventory @@ -454,6 +461,13 @@ kind, extension ID, name, resolved destination, source, and the hash under that source's semantics. The subsystem populates `extensions_old` from the inventory rather than from `ctx.spec_old` alone. +Both the inventory and the image summary follow the discipline already applied +to `spec` and `spec_old`: a staged operation records a pending record while the +deployed one is retained, finalize promotes it, rollback discards it, and +`update` resetting a previously staged operation reverts to the deployed one. +Without that, finalize cannot reconstruct the target set, since it runs with +`image: None`, and rollback cannot reconstruct the previous one. + #### Destination Validation `validate_extension_images_locations` rejects extension destinations that are @@ -463,17 +477,30 @@ bundled destinations. The equivalent check for the effective set must run after the COSI is loaded, as dynamic validation against the storage graph. It must reject a destination that -is shared when A/B is configured, and should additionally reject a destination -that is read-only or verity-protected — a case the existing check does not -cover for Host Configuration entries either. +is shared when A/B is configured. + +It must validate the filesystem that actually backs the destination for the +servicing type in question, not the image's nominal layout. Under root-verity +the engine mounts a writable `/etc` overlay during provision, backed by +`/var/lib/trident-overlay`, which is expected to be on an A/B volume, so +`/etc/extensions/` is writable and rolls back with the slot on Clean Install and +A/B Update. Rejecting verity-backed destinations outright would reject that +supported case. The same destination has no such backing on a Runtime Update and +must be rejected there. #### Where Images Are Written - **Clean Install and A/B Update.** `provision()` runs with the target root - mounted at `mount_path`. Extensions are staged under - `{mount_path}/var/lib/extensions/.staging` and moved to their destination - inside the target root, as today. For A/B the destination is on the inactive - slot, so the running system is untouched until reboot. + mounted at `mount_path`. Extensions are staged and then moved to their + destination inside the target root. For A/B the destination is on the + inactive slot, so the running system is untouched until reboot. + + Staging currently uses a fixed `/var/lib/extensions/.staging`, with a + non-atomic copy as fallback when the rename crosses a filesystem boundary. + That assumes `/var` is writable and makes placement non-atomic for + destinations such as `/etc/extensions/` or `/usr/lib/confexts/`. Staging + should instead use a temporary file on the destination's own filesystem, so + the rename is always atomic and no unrelated volume is required. - **Runtime Update.** `provision()` is not called, so partitions are never touched. Extensions are staged and placed in the running root. See [Extension-Only Updates](#extension-only-updates). @@ -541,18 +568,24 @@ existing hosts and the escape hatch if the summary's definition changes. Comparing the summary against the new COSI's metadata gives three outcomes: -| Summary | Extension set | Result | -| -------- | ------------- | -------------------------------------------------- | -| Equal | Equal | No-op. | -| Equal | Differs | Extension-only change; a Runtime Update suffices. | -| Differs | Any | A/B update. | +| Summary | Extension set | Image requires | +| ------- | ------------- | --------------------------------------- | +| Equal | Equal | Nothing. | +| Equal | Differs | A Runtime Update. | +| Differs | Any | An A/B update. | + +This replaces `ab_update_required()`'s contribution only. It is the image's +requirement, not the outcome: `select_servicing_type` still takes the maximum +across subsystems, so a concurrent change to users, modules or the kernel +command line can independently require an A/B update. An extension-only Runtime +Update additionally requires that no other subsystem asks for more; where one +does, the request is refused rather than silently widened. The no-op case is new. `os.image.sha384` changes whenever any part of the metadata changes, including `id` and the ordering of `osPackages`, so a materially identical image currently forces an A/B update. -This modifies `ab_update_required()`, which governs a core decision. The -fallback above confines the change to hosts that have a recorded summary, so +The fallback above confines the change to hosts that have a recorded summary, so behaviour for existing hosts is unaltered until they are next deployed. ##### Constructing an Extension-Only Update @@ -590,9 +623,10 @@ An extension-only update proceeds when all of the following hold: 2. The recorded summary equals the summary computed from the new COSI. 3. The recorded image is the deployed one: servicing completed, no A/B update pending, and the active volume matches the Host Status. +4. No other subsystem requires an A/B update. -If a Runtime Update is requested and condition 2 or 3 does not hold, Trident -fails with a structured error naming what differs. It must not promote the +If a Runtime Update is requested and any of conditions 2 to 4 does not hold, +Trident fails with a structured error naming what differs. It must not promote the operation to an A/B update, and must not apply the extension change while leaving other changes unapplied. Refusal is the only acceptable failure. @@ -757,9 +791,14 @@ is now an error rather than a duplicate. - **Schema.** Add `cosi-metadata-v1.3.schema.json` and samples under `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`. The existing schema-validation workflow picks up a new revision given the schema file and a - matching samples directory. Invalid samples should cover `image.path` outside - `images/`, a non-absolute destination, a destination not ending in `.raw`, a - destination in a disallowed directory, and a duplicated destination path. + matching samples directory. Invalid samples cover what the schema can express: + `image.path` outside `images/`, a non-absolute destination, and a destination + not ending in `.raw`. + + The permitted-directory allow-list is Trident policy rather than a property of + the format — systemd searches directories Trident does not accept — so it is + deliberately absent from the schema, and the kind-specific directory and + duplicate-destination rules are covered by Rust validation tests instead. - **Unit.** Metadata parse and validation for each new error variant; effective set computation, including collisions and the source discriminator; default-path resolution for an entry with no `path`; replacement of an @@ -884,11 +923,6 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against ## Open Questions -- **What surface requests an extension-only Runtime Update?** A `trident update` - flag, an internal parameter alongside `forceAbUpdate`, or a Host Configuration - field. The request must be explicit; the surface is undecided. It must also - define precedence against `forceAbUpdate` and whether the request persists - across separate stage and finalize invocations. - **What does the deployed image summary cover?** The proposal includes the region and filesystem image hashes and identities, `disk` geometry, `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, @@ -903,10 +937,9 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against status transition that records an accepted image without creating a rollbackable operation. That is a behavioural change independent of extensions and may deserve its own RFC. -- **Is retaining the superseded image in the staging directory sufficient for - runtime rollback,** or should the extension subsystem keep a durable - per-extension backup outside the staging directory? The former is simpler; the - latter survives a staging-directory cleanup. +- **Where does the retained superseded image live** for runtime rollback? The + staging directory is simplest; a separately validated persistent location + survives a staging cleanup and does not assume `/var` is writable. - **Should an override escape hatch exist?** This RFC makes a collision between the Host Configuration and the COSI an error. The alternative is an explicit opt-in on the Host Configuration side meaning "the image ships this, use mine From eb068686085149bf90f7fdfacb753e5150d21f0c Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:38:59 -0700 Subject: [PATCH 08/15] docs: address third-pass review findings in COSI extensions RFC - Include fsType in the deployed image summary; it drives resize decisions, so two images differing only in fsType must not compare equal. - Promotion of pending records follows the servicing type: rebooting servicing reaches *Finalized before reboot and commits only after booting the expected root, so the pending record must survive the reboot and be discarded when the boot check fails. - ManualRollbackChainItem carries only kind, spec, active volume and install index, so it must be extended or manual rollback cannot restore the bundled extension set. - Remove the stale Future Possibility that restated the now-required inventory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 39 +++++++++++++------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index a65fc404a8..0aac0238ff 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -463,10 +463,22 @@ rather than from `ctx.spec_old` alone. Both the inventory and the image summary follow the discipline already applied to `spec` and `spec_old`: a staged operation records a pending record while the -deployed one is retained, finalize promotes it, rollback discards it, and -`update` resetting a previously staged operation reverts to the deployed one. -Without that, finalize cannot reconstruct the target set, since it runs with -`image: None`, and rollback cannot reconstruct the previous one. +deployed one is retained, and `update` resetting a previously staged operation +reverts to the deployed one. Without that, finalize cannot reconstruct the +target set, since it runs with `image: None`, and rollback cannot reconstruct +the previous one. + +Promotion follows the servicing type, not the finalize step. A Runtime Update +promotes on completion. Clean Install, A/B Update and manual A/B rollback only +reach a `*Finalized` state before rebooting, and are committed after the host +boots from the expected root; the pending record must survive that reboot and be +promoted at commit, and be discarded when the boot check fails and `spec_old` is +restored. + +`ManualRollbackChainItem` carries only the kind, `spec`, active volume and +install index, so a historical inventory and summary cannot be recovered from +it. It must carry both, or a manual rollback cannot restore the bundled +extension set — particularly a runtime rollback, where no slot reconstructs it. #### Destination Validation @@ -548,8 +560,8 @@ therefore record what it deployed. Trident records a summary of the deployed image's metadata in the Host Status: for each entry in `images[]` and `disk.gptRegions[]`, the `image.sha384`, `uncompressedSize` and the entry's identity (partition number, mount point, -`fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, the `disk` -geometry and `bootloader`. Excluded: +`fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, +the `disk` geometry and `bootloader`. Excluded: - `extensions`, which is the subject of the comparison. - `osPackages`, which Trident validates but never acts on. @@ -812,7 +824,9 @@ is now an error rather than a duplicate. partitions are untouched and the extension set changes without a reboot. Separately again, finalize and rollback of an extension-only Runtime Update invoked as distinct processes, asserting the previous image is restored - without access to either COSI. + without access to either COSI. Assert that a pending inventory survives the + reboot of an A/B update and is promoted at commit, and discarded when the + boot check fails. - **Negative.** The same extension in both the Host Configuration and the COSI produces the structured error; a bundled extension with SELinux `enforcing` is rejected on A/B update and on extension-only Runtime Update; a bundled @@ -924,8 +938,8 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against ## Open Questions - **What does the deployed image summary cover?** The proposal includes the - region and filesystem image hashes and identities, `disk` geometry, - `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, + region and filesystem image hashes and identities including `fsType`, `disk` + geometry, `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, `osPackages`, `compression` and `id`. Excluding too much permits a Runtime Update between images differing in ways Trident does not model; including too much rejects legitimate cases. `version` is currently excluded, on the basis @@ -973,6 +987,7 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against - **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed by `ExtensionRelease` but not acted on. A bundled extension suits initrd scope, since the payload is available before the root filesystem is. -- **Extension inventory in Host Status.** Reporting the merged set, with IDs and - hashes, would make the set of running extensions answerable without inspecting - the filesystem. +- **Reporting the extension inventory.** The + [deployed extension inventory](#deployed-extension-inventory) is internal + state. Surfacing it through the CLI and the gRPC API would make the set of + running extensions answerable without inspecting the filesystem. From d69bdc70f1d9a451a06fe66f8930fa9734286a42 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:41:55 -0700 Subject: [PATCH 09/15] docs: address fourth-pass review findings in COSI extensions RFC - A manual runtime rollback has neither the target COSI nor the superseded payload, so inventory metadata alone cannot restore a bundled extension. Either retain payloads for the depth of the rollback chain, or refuse the operation; restoring nothing silently is not acceptable. - ManualRollbackRuntime promotes its pending record on completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 0aac0238ff..09f38f4fbc 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -475,11 +475,24 @@ boots from the expected root; the pending record must survive that reboot and be promoted at commit, and be discarded when the boot check fails and `spec_old` is restored. +A successful `ManualRollbackRuntime` promotes its pending record on completion, +in the same way as a Runtime Update; it transitions straight to `Provisioned` +and never reboots. + `ManualRollbackChainItem` carries only the kind, `spec`, active volume and install index, so a historical inventory and summary cannot be recovered from it. It must carry both, or a manual rollback cannot restore the bundled extension set — particularly a runtime rollback, where no slot reconstructs it. +Metadata alone is not enough for a manual runtime rollback. The target COSI is +not available, since the rollback context is built with `image: None`, and the +superseded payload is retained only for the duration of the operation that +replaced it. Either superseded bundled payloads are kept in a content-addressed +store for as long as the rollback chain references them, with the inventory +holding the reference, or a manual runtime rollback across a bundled extension +change must be refused. Refusing is the safe default; restoring nothing silently +is not acceptable. + #### Destination Validation `validate_extension_images_locations` rejects extension destinations that are @@ -951,9 +964,11 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against status transition that records an accepted image without creating a rollbackable operation. That is a behavioural change independent of extensions and may deserve its own RFC. -- **Where does the retained superseded image live** for runtime rollback? The - staging directory is simplest; a separately validated persistent location - survives a staging cleanup and does not assume `/var` is writable. +- **Where does the retained superseded image live** for runtime rollback, and + for how long? The staging directory is simplest but only covers the operation + itself; a content-addressed store bounded by the rollback chain also covers + manual runtime rollback, at the cost of disk. The alternative is to refuse + manual runtime rollback across a bundled extension change. - **Should an override escape hatch exist?** This RFC makes a collision between the Host Configuration and the COSI an error. The alternative is an explicit opt-in on the Host Configuration side meaning "the image ships this, use mine From e1f1bf42641dbfb14c56160f480b4dbae21020cb Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:44:41 -0700 Subject: [PATCH 10/15] docs: resolve scope contradiction for extension-only updates The runtime path was a stated requirement but absent from the exit criteria and marked deferrable in the plan. Add it to the exit criteria and state that step 5 may land separately but is required for the RFC to be complete. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/Development/RFCs/0789-cosi-extension-images.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 09f38f4fbc..97062e8b2f 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -99,6 +99,9 @@ COSI is built. This works in some configurations, but: - Trident deploys a COSI carrying a sysext and a confext on Clean Install and on A/B Update, and both are merged after reboot. - A/B rollback restores the previous extension set with no additional servicing. +- An extension-only change between a COSI and a copy of it with a different + extension set is applied as a Runtime Update, without rewriting partitions or + rebooting, and is refused when anything else differs. - Conflicts between bundled and Host Configuration extensions produce a structured error. @@ -888,7 +891,10 @@ The change is additive at every layer. Steps 1 and 2 are independently useful: a reader that parses and validates the section but ignores it is a safe intermediate state and makes the minor-version warning available sooner. Steps 3 and 4 deliver bundled extensions on Clean -Install and A/B Update; step 5 is separable and could be deferred. +Install and A/B Update, and are shippable without step 5. Step 5 may land +separately but is required for this RFC to be complete, since without it every +extension change is an A/B update and the cadence cost the proposal sets out to +bound is not bounded. ## Counter-Arguments From e86ea65551a93d32ac019894a7a6505516771ec1 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 10:47:03 -0700 Subject: [PATCH 11/15] docs: exclude disk streaming from bundled extensions Streaming places extension files but osconfig returns early on is_stream_image, so the merge units are never enabled. Refuse to stream a COSI carrying extensions rather than leaving files that never merge; record enabling the units on a streamed install as follow-up work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 27 ++++++++++++------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 97062e8b2f..1d13766314 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -86,6 +86,9 @@ COSI is built. This works in some configurations, but: - Changes to the Host Configuration `Extension` object. - Extension-only COSI files, meaning a COSI carrying extensions and no partition images. See [Extension-Only Updates](#extension-only-updates). +- Bundled extensions under [disk streaming](../../Explanation/Disk-Streaming.md), + which cannot enable the merge units. Streaming a COSI that carries extensions + is refused. - Signing or attestation beyond the SHA-384 chain COSI already provides. - SELinux compatibility. That limitation is unchanged; see [SELinux](#selinux). - Producing the DDIs. This RFC specifies how a COSI carries an extension image, @@ -97,7 +100,8 @@ COSI is built. This works in some configurations, but: `cosi-metadata-v1.3.schema.json`, and samples under `tests/cosi/metadata_samples/v1.3/`. - Trident deploys a COSI carrying a sysext and a confext on Clean Install and on - A/B Update, and both are merged after reboot. + A/B Update, and both are merged after reboot. Streaming such a COSI is + refused. - A/B rollback restores the previous extension set with no additional servicing. - An extension-only change between a COSI and a copy of it with a different extension set is applied as a Runtime Update, without rewriting partitions or @@ -440,14 +444,13 @@ There are three consumers: validation hook that runs on every servicing type, otherwise [Extension-Only Updates](#extension-only-updates) bypass it. -`derive_host_configuration`, used by -[disk streaming](../../Explanation/Disk-Streaming.md), does not need to -synthesise extension entries; the effective set is computed from the COSI -directly. Streaming is not complete on its own, however: `osconfig` returns -early when `is_stream_image` is set, so the merge units are never enabled and -the extensions would be placed but not merged. Either streaming must enable the -units specifically, or a streamed image must ship with them enabled and the -proposal must say so. +[Disk streaming](../../Explanation/Disk-Streaming.md) is excluded. The effective +set would be computed from the COSI without `derive_host_configuration` needing +to synthesise anything, but `osconfig` returns early when `is_stream_image` is +set, so the merge units are never enabled and the extensions would be placed and +never merged. Trident refuses to stream a COSI that carries extensions rather +than producing that state. Lifting the exclusion means giving streaming a way to +enable the two units, which is separable work. #### Deployed Extension Inventory @@ -843,7 +846,8 @@ is now an error rather than a duplicate. without access to either COSI. Assert that a pending inventory survives the reboot of an A/B update and is promoted at commit, and discarded when the boot check fails. -- **Negative.** The same extension in both the Host Configuration and the COSI +- **Negative.** Streaming a COSI that carries extensions is refused; the same + extension in both the Host Configuration and the COSI produces the structured error; a bundled extension with SELinux `enforcing` is rejected on A/B update and on extension-only Runtime Update; a bundled destination on a shared volume is rejected when A/B is configured; a Runtime @@ -1008,6 +1012,9 @@ at the cost of cadence. The `extension-release` fields, `ID=_any` against - **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed by `ExtensionRelease` but not acted on. A bundled extension suits initrd scope, since the payload is available before the root filesystem is. +- **Bundled extensions under disk streaming.** Requires a way to enable + `systemd-sysext.service` and `systemd-confext.service` on a streamed install, + which `osconfig` currently skips entirely. - **Reporting the extension inventory.** The [deployed extension inventory](#deployed-extension-inventory) is internal state. Surfacing it through the CLI and the gRPC API would make the set of From 9b0fa220976859fdeb58f14ec6bb1788f1c0471b Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 11:44:54 -0700 Subject: [PATCH 12/15] docs: condense COSI extensions RFC Review findings had been folded in as prose, leaving most points stated once and then re-justified. Cuts about a quarter of the text with no change to any decision, constraint or cited behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 1057 +++++++---------- 1 file changed, 432 insertions(+), 625 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 1d13766314..9471af6a5f 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -1,82 +1,69 @@ # 0789 COSI Extension Images -- Date: 2026-09-08 +- Date: 2026-10-08 - RFC PR: [microsoft/trident#789](https://github.com/microsoft/trident/pull/789) - Issue: [microsoft/trident#0000](https://github.com/microsoft/trident/issues/0000) ## Summary COSI files cannot carry systemd system extension (sysext) or configuration -extension (confext) images. This RFC stores extension images as ZSTD-compressed -members of the COSI tar, alongside partition images, and adds an optional -`extensions` section to the COSI metadata to describe them. Each entry reuses -the existing [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) -object and carries an optional destination path, mirroring the +extension (confext) images. This RFC stores them as ZSTD-compressed members of +the COSI tar, alongside partition images, and adds an optional `extensions` +section to the metadata. Each entry reuses the existing +[`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) object and +carries an optional destination path, mirroring the [`Extension`](../../Reference/Host-Configuration/API-Reference/Extension.md) -object in the Host Configuration. Trident deploys bundled extensions as part of -the image, in the same operation as the rest of the OS. +object in the Host Configuration. ## Motivation and Goals -Extensions are configured today through `os.sysexts` and `os.confexts` in the -Host Configuration, where each entry is a URL and a SHA-384. This makes an -extension a second artefact with a separate lifecycle: +Extensions are configured today through `os.sysexts` and `os.confexts`, each +entry a URL and a SHA-384. That makes an extension a second artefact with a +separate lifecycle: -- It must be hosted separately, for as long as any host may re-run the flow. -- It is fetched in full from a second endpoint at deploy time, with its own - network, proxy and authentication failure modes. -- Its integrity is anchored by whoever wrote the Host Configuration, not by +- Hosted separately, for as long as any host may re-run the flow. +- Fetched from a second endpoint at deploy time, with its own network, proxy and + authentication failure modes. +- Integrity anchored by whoever wrote the Host Configuration, not by `os.image.sha384`. -- Changing it is a Runtime Update, distinct from the A/B update that ships the - OS. Content that must land with a new OS requires two operations. +- Changed by a Runtime Update, distinct from the A/B update that ships the OS, + so content that must land with a new OS requires two operations. -### A/B Updates - -A bundled extension is delivered and activated in the same A/B update and the -same reboot as the OS, and reuses the existing rollout, health-gating and -rollback machinery. - -Rollback requires no additional work: extension images are files in the target -slot's filesystem, so the previous slot retains the previous set. See -[Rollback](#rollback). +Bundled, an extension is delivered and activated in the same A/B update and the +same reboot as the OS, reusing the existing rollout, health-gating and rollback +machinery. Rollback needs no extra work; see [Rollback](#rollback). ### Why Not Place the Files in the Root Filesystem Image -The image builder could place `my-tool.raw` in `/var/lib/extensions/` before the -COSI is built. This works in some configurations, but: +The builder could place `my-tool.raw` in `/var/lib/extensions/` before the COSI +is built. That works in some configurations, but: -1. With root or usr verity, modifying the protected filesystem changes the root +1. Under root or usr verity, modifying the protected filesystem changes the root hash and requires re-signing. 2. Extension destinations are commonly on a separate volume, which may be created empty rather than written from a partition image. -3. Contents of a filesystem image are not visible to tooling that reads COSI - metadata. -4. Trident enables `systemd-sysext.service` and `systemd-confext.service`, - validates placement, and reads `extension-release` only for extensions it - knows about. +3. Filesystem image contents are invisible to tooling that reads COSI metadata. +4. Trident enables the merge units, validates placement and reads + `extension-release` only for extensions it knows about. ### Goals -- Carry extension images in a COSI file and deploy them without a second - artefact, endpoint or update. -- Reuse existing metadata objects and match the shape of the Host Configuration - API. +- Deploy extensions from a COSI without a second artefact, endpoint or update. +- Reuse existing metadata objects and the shape of the Host Configuration API. - Leave the Host Configuration API unchanged. ## Scope ### Requirements -- An optional `extensions` object in the COSI metadata root, containing - `sysexts` and `confexts` arrays. -- Each entry identifies its payload with an `ImageFile` object and may specify a - destination path on the target OS. -- Extension payloads are ZSTD-compressed tar members, subject to the existing - compression and integrity rules. -- Trident deploys bundled extensions on Clean Install and A/B Update, reusing - the extensions subsystem. -- A change confined to the extension set is applicable as a Runtime Update, - without rewriting partitions or rebooting. See +- An optional `extensions` object in the metadata root, holding `sysexts` and + `confexts` arrays. +- Each entry identifies its payload with an `ImageFile` and may specify a + destination path. +- Payloads are ZSTD-compressed tar members under the existing compression and + integrity rules. +- Trident deploys bundled extensions on Clean Install and A/B Update. +- A change confined to the extension set is applicable as a Runtime Update. See [Extension-Only Updates](#extension-only-updates). - Bundled and Host Configuration extensions coexist; conflicts are an error. - COSI metadata validation covers the new section. @@ -84,30 +71,25 @@ COSI is built. This works in some configurations, but: ### Out of Scope - Changes to the Host Configuration `Extension` object. -- Extension-only COSI files, meaning a COSI carrying extensions and no partition - images. See [Extension-Only Updates](#extension-only-updates). +- Extension-only COSI files, carrying extensions and no partition images. See + [Extension-Only Updates](#extension-only-updates). - Bundled extensions under [disk streaming](../../Explanation/Disk-Streaming.md), - which cannot enable the merge units. Streaming a COSI that carries extensions - is refused. + which cannot enable the merge units. Streaming such a COSI is refused. - Signing or attestation beyond the SHA-384 chain COSI already provides. -- SELinux compatibility. That limitation is unchanged; see [SELinux](#selinux). -- Producing the DDIs. This RFC specifies how a COSI carries an extension image, - not how the image is built. +- SELinux compatibility, which is unchanged. See [SELinux](#selinux). +- Producing the DDIs. ### Exit Criteria -- COSI revision 1.3 published with the `extensions` section, - `cosi-metadata-v1.3.schema.json`, and samples under - `tests/cosi/metadata_samples/v1.3/`. -- Trident deploys a COSI carrying a sysext and a confext on Clean Install and on - A/B Update, and both are merged after reboot. Streaming such a COSI is - refused. -- A/B rollback restores the previous extension set with no additional servicing. -- An extension-only change between a COSI and a copy of it with a different - extension set is applied as a Runtime Update, without rewriting partitions or - rebooting, and is refused when anything else differs. -- Conflicts between bundled and Host Configuration extensions produce a - structured error. +- COSI revision 1.3 published with the section, `cosi-metadata-v1.3.schema.json` + and samples under `tests/cosi/metadata_samples/v1.3/`. +- Trident deploys a COSI carrying a sysext and a confext on Clean Install and + A/B Update, both merged after reboot. Streaming such a COSI is refused. +- A/B rollback restores the previous extension set with no extra servicing. +- An extension-only change is applied as a Runtime Update without rewriting + partitions or rebooting, and refused when anything else differs. +- Conflicts between bundled and Host Configuration extensions are a structured + error. ## Dependencies @@ -122,60 +104,49 @@ reference writer. every extension change is an A/B update. - A surface for requesting a servicing type: a `trident update` flag, an internal parameter alongside `forceAbUpdate`, or a Host Configuration field. - None exists today. This is a prerequisite rather than an optional refinement, - because the path is never inferred, and it must define precedence against - `forceAbUpdate` and whether the request persists across separate stage and - finalize invocations. + None exists. It must define precedence against `forceAbUpdate` and whether the + request survives separate stage and finalize invocations. -Three defects in the existing extensions subsystem must be fixed first. All are -reachable today with Host Configuration extensions; bundling makes the first two -the common case rather than the exception. +Three existing defects must be fixed first. All are reachable today with Host +Configuration extensions; bundling makes the first two routine. 1. **In-place replacement deletes the new image.** When an extension keeps its ID and destination but changes content, `set_up_extensions` schedules the ID for both addition and removal. The addition renames the new image over the - destination; the removal then deletes that same path, because the old - entry's `temp_path` is its destination. The guard assumes the two paths - differ. A bundled extension updated in place hits this on every Runtime - Update. + destination; the removal then deletes that path, because the old entry's + `temp_path` is its destination. The guard assumes the paths differ. 2. **Staged payloads do not survive the operation.** Runtime finalize and rollback construct their `EngineContext` with `image: None`, so the COSI is - unavailable after stage. Bundled payloads must be staged durably, and the - superseded image must be retained until the operation completes. -3. **Extension placement is validated against the Host Configuration only, and - before the COSI is read.** `validate_extension_images_locations` is static - validation over `os.sysexts` and `os.confexts`. Bundled destinations need - the same check after the COSI is loaded. See + gone after stage. Payloads must be staged durably, and the superseded image + retained until the operation completes. +3. **Placement is validated against the Host Configuration only, before the COSI + is read.** `validate_extension_images_locations` is static validation over + `os.sysexts` and `os.confexts`. See [Destination Validation](#destination-validation). ## Implementation ### Tar Layout -Extension payloads are ZSTD-compressed DDI files under `images/extensions/`, -for example `images/extensions/my-tool.rawzst`. +Payloads are ZSTD-compressed DDIs under `images/extensions/`, for example +`images/extensions/my-tool.rawzst`. -They remain under `images/`. The specification already permits subdirectories of -`images/` and requires readers to handle them, so no new tar-layout rule is -needed. `ImageFile.path` is constrained by `"pattern": "^images/.+"` in every -published schema; relaxing it would produce a revision 1.3 `ImageFile` that -fails validation against the 1.0–1.2 schemas and break any validator with an -`ImageFile` definition compiled in, for no benefit beyond a shorter path. +They stay under `images/`: the specification already permits subdirectories +there and requires readers to handle them. Relaxing `ImageFile.path`'s +`^images/.+` pattern would produce a 1.3 `ImageFile` that fails the 1.0–1.2 +schemas and break any validator with that definition compiled in, buying only a +shorter path. -Two existing ordering constraints apply: +Two existing ordering rules apply. The primary GPT image must immediately follow +`metadata.json` since revision 1.2, so payloads must not sit between them. +Region images must appear in the physical order of the regions on disk; +payloads are not regions, but interleaving them obscures that ordering and hurts +sparse-read locality, so they should be written after all region images. -- Since revision 1.2 the primary GPT image must immediately follow - `metadata.json`. Extension payloads must not be placed between them. -- Region images must appear in the physical order of the regions on the source - disk. Extension payloads are not regions, but interleaving them complicates - verification of that ordering and hurts sparse-read locality. Extension - payloads should be written after all region images. +Writers must account for payloads in the root `compression.maxWindowLog`. -Writers must account for extension payloads when computing the existing root -`compression.maxWindowLog`. - -Older readers are unaffected by the additional tar members. The specification -requires readers to ignore unknown files, and Trident's orphan-image check +Older readers are unaffected: the specification requires them to ignore unknown +files, and Trident's orphan-image check (`V1_2ImageFileHasNoCorrespondingPartition`) is driven by the `images[]` and `disk.gptRegions[]` arrays rather than by walking tar entries. @@ -301,108 +272,83 @@ Example: #### Two Arrays Rather Than One With a `kind` Field -Sysexts and confexts have disjoint sets of permitted destination directories -(`VALID_SYSEXT_DIRECTORIES` and `VALID_CONFEXT_DIRECTORIES`), different -defaults, different `extension-release` locations -(`/usr/lib/extension-release.d/` and `/etc/extension-release.d/`), different -identity fields (`SYSEXT_ID` and `CONFEXT_ID`) and different activation units. -Every downstream rule branches on the kind. - -With two arrays the kind is structural, so no `kind` field is required and it -cannot be omitted or incorrect. It is also the shape the Host Configuration -already uses. +Sysexts and confexts have disjoint permitted directories +(`VALID_SYSEXT_DIRECTORIES`, `VALID_CONFEXT_DIRECTORIES`), different defaults, +different `extension-release` locations, different identity fields (`SYSEXT_ID`, +`CONFEXT_ID`) and different activation units. Every downstream rule branches on +the kind. Two arrays make it structural, so it cannot be omitted or wrong, and +match the Host Configuration. #### Entry Contents -An entry carries `image` and an optional `path`. Everything else considered is -derivable from the payload, and each derivable field adds a second source of -truth. - -- **`path`** is retained. It is the destination on the target OS and is not - derivable. It is optional because `Extension.path` is optional and defaults to - `/var/lib/extensions/{name}.raw` or `/var/lib/confexts/{name}.raw`. When - present it must be absolute, must end in `.raw`, must be in a permitted - directory for its kind, and its file name must match the `extension-release` - suffix. These rules already exist. -- **Extension name** is rejected. Trident derives it from the - `extension-release.{name}` file name inside the DDI and already requires the - destination file name to match. A `name` field would be a third copy of the - same string. -- **`kind`** is rejected; it is structural. -- **Extension ID** (`SYSEXT_ID`, `CONFEXT_ID`) is rejected. It is read from - `extension-release` and is the identity Trident keys A/B state on. -- **Enable-on-first-boot** is rejected. systemd merges everything in the - extension directories, and the Host Configuration has no per-extension switch. - Adding one here would create semantics the Host Configuration cannot express. -- **Version and compatibility metadata** is rejected. `ID`, `VERSION_ID`, +`image` and an optional `path`. Everything else is derivable from the payload, +and each derivable field is a second source of truth. + +- **`path`** is kept: the destination is not derivable. Optional, because + `Extension.path` is optional and defaults to `/var/lib/extensions/{name}.raw` + or `/var/lib/confexts/{name}.raw`. The existing rules apply — absolute, + ending in `.raw`, in a permitted directory for its kind, file name matching the + `extension-release` suffix. +- **Extension name** is rejected: derived from the `extension-release.{name}` + file name, and the destination file name must already match it. +- **`kind`** is rejected: structural. +- **Extension ID** is rejected: read from `extension-release`, and already the + identity Trident keys A/B state on. +- **Enable-on-first-boot** is rejected: systemd merges everything in the + extension directories, and the Host Configuration has no equivalent switch. +- **Version and compatibility metadata** is rejected: `ID`, `VERSION_ID`, `SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE` and `SYSEXT_SCOPE` are in the - DDI's `extension-release`, and systemd enforces them at merge time. Copying - them into metadata would allow a COSI to claim compatibility the payload does - not have. + DDI and enforced by systemd. Copying them would let a COSI claim compatibility + the payload lacks. #### `sha384` Semantics `ImageFile.sha384` covers the compressed image; `Extension.sha384` covers the -raw extension image file. Reusing `ImageFile` therefore changes what the hash -covers. - -For integrity this is equivalent to partition images: Trident hashes the -compressed stream as it decompresses, and a match proves the payload is intact. -A second hash of the uncompressed DDI would add nothing. - -One consequence must be handled. `ExtensionData.sha384` is a single field used -both as the change detector and as the key matching a processed extension back -to its Host Configuration entry. It cannot hold two different meanings at once, -so the effective entry must record the hash together with its source, and -comparisons must only be made between hashes of the same kind. Comparing a -compressed hash against a raw-file hash is always unequal and would report +raw file. For integrity this matches partition images — Trident hashes the +compressed stream as it decompresses — so a second hash of the DDI would add +nothing. + +`ExtensionData.sha384` is a single field serving both as change detector and as +the key matching a processed extension to its Host Configuration entry, so the +effective entry must record the hash with its source and compare only within a +source. A compressed hash against a raw hash is always unequal and would report every extension as changed. Within one source the comparison is sound but imprecise: compressed hashes are -not stable across recompressions, so two COSIs built from identical DDIs at -different ZSTD levels appear to differ. The result is a redundant copy of an -identical file, never a missed update. +unstable across recompressions, so identical DDIs at different ZSTD levels +appear to differ. The result is a redundant copy, never a missed update. ### Versioning -This lands in COSI revision 1.3. `extensions` is optional; absent is equivalent -to empty. +COSI revision 1.3. `extensions` is optional; absent equals empty. -**Newer Trident, older COSI (1.0–1.2).** No `extensions` field, no bundled -extensions, current behaviour. +A newer Trident reading a 1.0–1.2 COSI sees no bundled extensions and behaves as +today. -**Older Trident, newer COSI (1.3 with extensions).** Trident's version check -rejects only `major != 1`, and both the specification and the metadata parser -require unknown fields to be ignored. An existing Trident will accept a 1.3 -COSI, deploy the OS correctly, and omit the extensions without reporting -anything. +An older Trident reading a 1.3 COSI accepts it — the version check rejects only +`major != 1`, and unknown fields must be ignored — deploys the OS correctly, and +omits the extensions silently. This cannot be corrected in-band, since any +tripwire field is one such a reader must ignore. Nor does it self-heal: the old +reader records the new COSI as deployed, so after upgrading Trident the same +Host Configuration is no longer an image change and the skipped extensions are +never installed. Recovery needs a forced redeployment or a republished COSI. -This cannot be corrected in-band for readers that already exist: any tripwire -field is a field they are required to ignore. Three partial mitigations: +Mitigations: -1. Document that consuming bundled extensions requires a reader that understands - revision 1.3. -2. Have `validate_cosi_metadata_version` warn when `minor` exceeds the highest - revision the binary knows. This does not help existing binaries, but makes - future minor bumps diagnosable from the log. -3. Where the Trident version on the target cannot be controlled, add a +1. Document that bundled extensions require a reader that understands 1.3. +2. Warn in `validate_cosi_metadata_version` when `minor` exceeds the highest + known revision. This makes future minor bumps diagnosable from the log. +3. Where the target's Trident version cannot be controlled, gate the update on a [health check](../../Reference/Host-Configuration/API-Reference/Health.md) - asserting the extension is merged, so an A/B update onto a too-old Trident - fails its gate and rolls back. - -The omission does not self-heal. An old reader records the new COSI's URL and -hash as deployed, so after Trident is upgraded the same Host Configuration no -longer represents an image change and the skipped extensions are never -installed. Recovery requires a forced redeployment or a republished COSI. This -is the strongest argument for the health check above rather than a documented -caveat alone. + asserting the extension is merged, so the A/B update fails and rolls back. + Given that the omission does not self-heal, this is the only real safeguard. ### Trident-Side Consumption -The extensions subsystem gains a second source. Trident computes an **effective -extension set**, the union of the Host Configuration entries and the COSI -entries, and the code that reads `ctx.spec.os.sysexts` and -`ctx.spec.os.confexts` reads the effective set instead. +Trident computes an **effective extension set**, the union of the Host +Configuration and COSI entries, and the code reading `ctx.spec.os.sysexts` and +`ctx.spec.os.confexts` reads that instead. Each entry carries its source, which +determines where a resolved path is written back. ```mermaid flowchart LR @@ -413,321 +359,245 @@ flowchart LR Eff --> Sel["selinux subsystem
enforcing-mode rejection"] ``` -Each effective entry carries its source, Host Configuration or COSI. The source -determines where a resolved path is written back and whether the entry is -subject to Host Configuration validation. +Three consumers: -There are three consumers: - -- **`ExtensionsSubsystem`.** `populate_extensions` currently downloads each - entry with a `FileReader` over `Extension.url` into the staging directory, - verifies the hash, and mounts the DDI to read `extension-release`. For a - bundled entry only the source changes: the image streaming pipeline - decompresses the tar member into the same staging directory and verifies - `ImageFile.sha384` over the compressed stream. The `extension-release` read, - the `{name}.raw` check and default-path resolution are unchanged. - `update_host_configuration` currently fails if a processed extension has no +- **`ExtensionsSubsystem`.** `populate_extensions` downloads each entry over + `Extension.url`, verifies the hash and mounts the DDI to read + `extension-release`. For a bundled entry only the source changes: the image + streaming pipeline decompresses the tar member into the same staging directory + and verifies `ImageFile.sha384` over the compressed stream. The rest is + unchanged. `update_host_configuration` fails when a processed extension has no Host Configuration entry matching by hash, so it must write back resolved paths for Host-Configuration-sourced entries only. -- **`osconfig`.** `systemd-sysext.service` and `systemd-confext.service` are - enabled only when `ctx.spec.os.sysexts` or `confexts` are non-empty. A host - whose extensions come only from the COSI would never have the merge units - enabled, and the extensions would remain on disk unused. This check must read - the effective set. -- **`selinux`.** The dynamic validation raising - `ExtensionImagesAndSelinuxUnsupported` must also read the effective set, or a - COSI carrying extensions combined with an enforcing-mode Host Configuration - passes validation and produces a host with a mislabelled `/usr`, `/opt` or - `/etc`. Reading the effective set is not sufficient on its own: the check runs - inside `configure`, which returns early for Runtime Updates, and - `SelinuxSubsystem` inherits `runs_on = REQUIRES_REBOOT`. It must move to a - validation hook that runs on every servicing type, otherwise - [Extension-Only Updates](#extension-only-updates) bypass it. +- **`osconfig`.** The merge units are enabled only when `ctx.spec.os.sysexts` or + `confexts` are non-empty, so a COSI-only host would never have them enabled + and its extensions would sit unused. This check must read the effective set. +- **`selinux`.** The validation raising `ExtensionImagesAndSelinuxUnsupported` + must read the effective set too, or a bundled extension with an enforcing-mode + Host Configuration passes and produces a mislabelled `/usr`, `/opt` or `/etc`. + That alone is insufficient: the check runs inside `configure`, which returns + early for Runtime Updates, and `SelinuxSubsystem` inherits + `runs_on = REQUIRES_REBOOT`. It must move to a hook running on every servicing + type, or [Extension-Only Updates](#extension-only-updates) bypass it. [Disk streaming](../../Explanation/Disk-Streaming.md) is excluded. The effective -set would be computed from the COSI without `derive_host_configuration` needing -to synthesise anything, but `osconfig` returns early when `is_stream_image` is -set, so the merge units are never enabled and the extensions would be placed and -never merged. Trident refuses to stream a COSI that carries extensions rather -than producing that state. Lifting the exclusion means giving streaming a way to -enable the two units, which is separable work. +set needs no synthesis from `derive_host_configuration`, but `osconfig` returns +early when `is_stream_image` is set, so the units are never enabled and the +extensions would be placed and never merged. Trident refuses to stream a COSI +carrying extensions rather than produce that state. Lifting the exclusion means +giving streaming a way to enable the units, which is separable work. #### Deployed Extension Inventory -The Host Configuration records the extensions an operator asked for. It cannot -record bundled ones, so `ctx.spec_old.os.sysexts` and `confexts` do not describe -what is actually deployed. Without a second record, Trident cannot determine -which bundled extensions to remove, detect collisions against the deployed set, -compare extension sets for servicing selection, or reconstruct its state when +The Host Configuration records what an operator asked for, so `ctx.spec_old` +does not describe what is deployed. Without a second record Trident cannot +determine which bundled extensions to remove, detect collisions against the +deployed set, compare sets for servicing selection, or rebuild its state when finalize runs separately from stage. -Trident therefore records a deployed extension inventory in the Host Status, -alongside the [deployed image summary](#deployed-image-summary). Per entry: -kind, extension ID, name, resolved destination, source, and the hash under that -source's semantics. The subsystem populates `extensions_old` from the inventory -rather than from `ctx.spec_old` alone. - -Both the inventory and the image summary follow the discipline already applied -to `spec` and `spec_old`: a staged operation records a pending record while the -deployed one is retained, and `update` resetting a previously staged operation -reverts to the deployed one. Without that, finalize cannot reconstruct the -target set, since it runs with `image: None`, and rollback cannot reconstruct -the previous one. - -Promotion follows the servicing type, not the finalize step. A Runtime Update -promotes on completion. Clean Install, A/B Update and manual A/B rollback only -reach a `*Finalized` state before rebooting, and are committed after the host -boots from the expected root; the pending record must survive that reboot and be -promoted at commit, and be discarded when the boot check fails and `spec_old` is -restored. - -A successful `ManualRollbackRuntime` promotes its pending record on completion, -in the same way as a Runtime Update; it transitions straight to `Provisioned` -and never reboots. - -`ManualRollbackChainItem` carries only the kind, `spec`, active volume and -install index, so a historical inventory and summary cannot be recovered from -it. It must carry both, or a manual rollback cannot restore the bundled -extension set — particularly a runtime rollback, where no slot reconstructs it. - -Metadata alone is not enough for a manual runtime rollback. The target COSI is -not available, since the rollback context is built with `image: None`, and the -superseded payload is retained only for the duration of the operation that -replaced it. Either superseded bundled payloads are kept in a content-addressed +Trident records a deployed extension inventory in the Host Status, alongside the +[deployed image summary](#deployed-image-summary): per entry, kind, extension +ID, name, resolved destination, source, and the hash under that source's +semantics. `extensions_old` is populated from it rather than from +`ctx.spec_old`. + +Inventory and summary follow the discipline already applied to `spec` and +`spec_old`: a staged operation records a pending record while the deployed one +is retained, and resetting a staged operation reverts to the deployed one. +Promotion follows the servicing type rather than the finalize step. Runtime +Update and `ManualRollbackRuntime` promote on completion, transitioning straight +to `Provisioned`. Clean Install, A/B Update and manual A/B rollback only reach a +`*Finalized` state before rebooting and commit after booting the expected root, +so their pending record must survive the reboot, be promoted at commit, and be +discarded when the boot check fails and `spec_old` is restored. + +`ManualRollbackChainItem` carries only kind, `spec`, active volume and install +index, so it must carry the historical inventory and summary as well. Even then, +metadata alone is insufficient for a manual runtime rollback: the context is +built with `image: None` and the superseded payload is retained only for the +operation that replaced it. Either payloads are kept in a content-addressed store for as long as the rollback chain references them, with the inventory -holding the reference, or a manual runtime rollback across a bundled extension -change must be refused. Refusing is the safe default; restoring nothing silently -is not acceptable. +holding the reference, or manual runtime rollback across a bundled extension +change is refused. Refusing is the safe default; silently restoring nothing is +not. #### Destination Validation -`validate_extension_images_locations` rejects extension destinations that are -not on an A/B volume when A/B is configured. It is static validation over the -Host Configuration lists, and runs before the COSI is read, so it cannot see -bundled destinations. - -The equivalent check for the effective set must run after the COSI is loaded, as -dynamic validation against the storage graph. It must reject a destination that -is shared when A/B is configured. +`validate_extension_images_locations` rejects destinations not on an A/B volume +when A/B is configured, but it is static validation over the Host Configuration +lists and runs before the COSI is read. -It must validate the filesystem that actually backs the destination for the -servicing type in question, not the image's nominal layout. Under root-verity -the engine mounts a writable `/etc` overlay during provision, backed by -`/var/lib/trident-overlay`, which is expected to be on an A/B volume, so -`/etc/extensions/` is writable and rolls back with the slot on Clean Install and -A/B Update. Rejecting verity-backed destinations outright would reject that -supported case. The same destination has no such backing on a Runtime Update and -must be rejected there. +The equivalent check for the effective set runs after the COSI loads, as dynamic +validation against the storage graph, rejecting a destination that is shared +when A/B is configured. It must resolve the filesystem that actually backs the +destination for the servicing type, not the image's nominal layout: under +root-verity the engine mounts a writable `/etc` overlay during provision, backed +by `/var/lib/trident-overlay` on an A/B volume, so `/etc/extensions/` is +writable and rolls back with the slot. Rejecting verity-backed destinations +outright would reject that supported case. The same destination has no such +backing on a Runtime Update and must be rejected there. #### Where Images Are Written -- **Clean Install and A/B Update.** `provision()` runs with the target root - mounted at `mount_path`. Extensions are staged and then moved to their - destination inside the target root. For A/B the destination is on the - inactive slot, so the running system is untouched until reboot. - - Staging currently uses a fixed `/var/lib/extensions/.staging`, with a - non-atomic copy as fallback when the rename crosses a filesystem boundary. - That assumes `/var` is writable and makes placement non-atomic for - destinations such as `/etc/extensions/` or `/usr/lib/confexts/`. Staging - should instead use a temporary file on the destination's own filesystem, so - the rename is always atomic and no unrelated volume is required. -- **Runtime Update.** `provision()` is not called, so partitions are never - touched. Extensions are staged and placed in the running root. See - [Extension-Only Updates](#extension-only-updates). +On Clean Install and A/B Update, `provision()` runs with the target root at +`mount_path`, and extensions are staged and moved to their destination inside +it. For A/B that is the inactive slot, so the running system is untouched until +reboot. Staging currently uses a fixed `/var/lib/extensions/.staging` with a +non-atomic copy when the rename crosses a filesystem boundary, which assumes +`/var` is writable and makes placement non-atomic for `/etc/extensions/` or +`/usr/lib/confexts/`. It should use a temporary file on the destination's own +filesystem. + +On Runtime Update `provision()` is not called, so partitions are never touched +and extensions are placed in the running root. See +[Extension-Only Updates](#extension-only-updates). #### Rollback -Extension images are files in the slot's own filesystem. When every destination -is on an A/B volume, the previous slot retains the previous extension set after -an update, an A/B rollback boots the previous root volume, `systemd-sysext` -merges what it finds there, and the extension set reverts with the OS. No -additional bookkeeping is required. - -That guarantee is conditional on the placement check covering bundled -destinations; see [Destination Validation](#destination-validation). A bundled -extension written to a shared volume changes the running slot immediately and -cannot be rolled back. +Extension images are files in the slot's own filesystem. Where every destination +is on an A/B volume, the previous slot retains the previous set, an A/B rollback +boots it, `systemd-sysext` merges what it finds, and the extension set reverts +with the OS. -Rollback of an extension-only Runtime Update is weaker; see -[Extension-Only Updates](#extension-only-updates). +That guarantee depends on the placement check covering bundled destinations; see +[Destination Validation](#destination-validation). A bundled extension on a +shared volume changes the running slot immediately and cannot be rolled back. #### Extension-Only Updates -`select_servicing_type` short-circuits: `ab_update_required()` returns true -whenever `os.image.sha384` differs, before any subsystem is consulted. The -metadata hash covers the `extensions` section, so changing only a bundled -extension changes the hash and forces a full A/B update and a reboot. Without -further work, bundling would make an extension update strictly more disruptive -than it is today. +`ab_update_required()` returns true whenever `os.image.sha384` differs, before +any subsystem is consulted, and the metadata hash covers the `extensions` +section. Changing only a bundled extension would therefore force a full A/B +update, making bundling strictly more disruptive than the status quo. An extension-only COSI is the wrong shape for this. The metadata root requires `images`, `disk` with at least one `gptRegions` entry, `bootloader` and -`osPackages`, so a COSI without partition images is not representable, and -making it representable would require those fields to become optional. Every -consumer of `os.image` — filesystem source resolution, -`derive_host_configuration`, ESP detection, verity setup — assumes the image -describes a complete OS. The distinction should be drawn on content equality, -not on absent content. +`osPackages`, and every consumer of `os.image` — filesystem source resolution, +`derive_host_configuration`, ESP detection, verity setup — assumes a complete +OS. The distinction is drawn on content equality, not absent content. ##### Deployed Image Summary -Trident retains the URL and metadata hash of the applied image, not the metadata -itself, and re-fetching the previous COSI to diff it is not dependable. It must -therefore record what it deployed. +Trident retains only the URL and metadata hash of the applied image, and +re-fetching the previous COSI to diff it is not dependable, so it must record +what it deployed. -Trident records a summary of the deployed image's metadata in the Host Status: -for each entry in `images[]` and `disk.gptRegions[]`, the `image.sha384`, -`uncompressedSize` and the entry's identity (partition number, mount point, -`fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, -the `disk` geometry and `bootloader`. Excluded: +The summary holds, for each entry in `images[]` and `disk.gptRegions[]`, the +`image.sha384`, `uncompressedSize` and the entry's identity (partition number, +mount point, `fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, +`osRelease`, `disk` geometry and `bootloader`. Excluded: -- `extensions`, which is the subject of the comparison. +- `extensions`, the subject of the comparison. - `osPackages`, which Trident validates but never acts on. -- `compression.maxWindowLog`, which governs decompression rather than deployed - content, and which a new extension may legitimately raise. +- `compression.maxWindowLog`, which governs decompression rather than content + and which a new extension may legitimately raise. - `id`, which identifies the file rather than its content. -A structured summary rather than a single digest: the digest is derivable from -it, and the summary additionally allows Trident to report which partition -differs rather than only that something does. - -The summary carries its own schema version. Where the recorded summary is absent -or of an unrecognised version, Trident falls back to comparing -`os.image.sha384`, which is current behaviour. This is the upgrade path for -existing hosts and the escape hatch if the summary's definition changes. +A record rather than a digest: the digest is derivable from it, and the record +also lets Trident report which partition differs. -Comparing the summary against the new COSI's metadata gives three outcomes: +The summary is versioned. Where it is absent or unrecognised, Trident falls back +to comparing `os.image.sha384`, so existing hosts are unaffected until their +next deployment. -| Summary | Extension set | Image requires | -| ------- | ------------- | --------------------------------------- | -| Equal | Equal | Nothing. | -| Equal | Differs | A Runtime Update. | -| Differs | Any | An A/B update. | +| Summary | Extension set | Image requires | +| ------- | ------------- | ----------------- | +| Equal | Equal | Nothing. | +| Equal | Differs | A Runtime Update. | +| Differs | Any | An A/B update. | -This replaces `ab_update_required()`'s contribution only. It is the image's -requirement, not the outcome: `select_servicing_type` still takes the maximum -across subsystems, so a concurrent change to users, modules or the kernel -command line can independently require an A/B update. An extension-only Runtime -Update additionally requires that no other subsystem asks for more; where one -does, the request is refused rather than silently widened. +This is the image's requirement, not the outcome. `select_servicing_type` still +takes the maximum across subsystems, so a concurrent change to users, modules or +the kernel command line can independently require an A/B update. -The no-op case is new. `os.image.sha384` changes whenever any part of the +The no-op row is new: `os.image.sha384` changes whenever any part of the metadata changes, including `id` and the ordering of `osPackages`, so a materially identical image currently forces an A/B update. -The fallback above confines the change to hosts that have a recorded summary, so -behaviour for existing hosts is unaltered until they are next deployed. - ##### Constructing an Extension-Only Update -Equality of the summary requires the region images to be byte-identical, and -filesystem images are not naturally reproducible. Filesystem UUIDs, inode -timestamps, superblock creation and mount times, and allocation order all vary -between builds, and ZSTD output varies with compression level and library -version. Rebuilding from the same source will not generally produce the same -images. - -Requiring reproducible rebuilds is therefore the wrong approach. The build -operation is not "rebuild the image identically with a different extension set", -it is "copy an existing COSI, replacing its extension set". Region images and -their metadata entries are copied verbatim from the input COSI; no filesystem is -rebuilt and no region image is recompressed. Only the extension tar members, the -`extensions` section, `compression.maxWindowLog` where the new extensions -require a larger window, and `id` change. - -Equality is then guaranteed by construction rather than hoped for through -reproducibility, and the operation is a tar rewrite rather than an image build. -This is the mode a COSI writer would need to provide. - -Where a COSI is instead rebuilt from source, its region images will differ, the -summary will differ, and Trident will select an A/B update. This is the safe -outcome: the mechanism degrades to current behaviour rather than misclassifying -a changed OS as unchanged. The failure mode of an unreproducible build is a -redundant A/B update, never a skipped one. +Summary equality requires byte-identical region images, and filesystem images +are not reproducible: filesystem UUIDs, inode timestamps, superblock creation +and mount times and allocation order vary between builds, and ZSTD output varies +with level and library version. + +So the build operation is not "rebuild the image identically with a different +extension set", it is "copy an existing COSI, replacing its extension set". +Region images and their metadata entries are copied verbatim; nothing is rebuilt +or recompressed. Only the extension tar members, the `extensions` section, +`compression.maxWindowLog` where a larger window is needed, and `id` change. +Equality then holds by construction, and the operation is a tar rewrite rather +than an image build. + +A COSI rebuilt from source instead will differ, and Trident will select an A/B +update. The failure mode of an unreproducible build is a redundant A/B update, +never a skipped one. ##### Requested and Verified, Never Inferred -An extension-only update proceeds when all of the following hold: +An extension-only update proceeds when all of: 1. A Runtime Update is explicitly requested. 2. The recorded summary equals the summary computed from the new COSI. 3. The recorded image is the deployed one: servicing completed, no A/B update - pending, and the active volume matches the Host Status. + pending, active volume matching the Host Status. 4. No other subsystem requires an A/B update. -If a Runtime Update is requested and any of conditions 2 to 4 does not hold, -Trident fails with a structured error naming what differs. It must not promote the -operation to an A/B update, and must not apply the extension change while -leaving other changes unapplied. Refusal is the only acceptable failure. - -There is no surface for requesting a servicing type today; `forceAbUpdate` is -the nearest precedent. See [Dependencies](#dependencies). +If a Runtime Update is requested and any of 2 to 4 fails, Trident fails with a +structured error naming what differs. It must not promote the operation to an +A/B update, and must not apply the extension change while leaving other changes +unapplied. -A no-op must still record the accepted image and summary in the Host Status. +A no-op must still record the accepted image and summary. `select_servicing_type` returning `NoActiveServicing` causes `update` to return -immediately without persisting anything, so without a metadata-only status -transition every subsequent invocation would repeat the comparison. +without persisting anything, so a metadata-only status transition is needed or +every subsequent invocation repeats the comparison. ##### Rollback on This Path -An A/B extension update is rolled back by booting the other slot. A Runtime -Update replaces files in the running slot, because `set_up_extensions` skips -removal of the superseded image only on Clean Install and A/B Update. Rollback -re-runs the subsystem with the specs reversed and must restore the previous -image. - -It cannot re-fetch it. Runtime finalize and rollback build their -`EngineContext` with `image: None`, so the COSI is not available after stage, -and auto-rollback runs unconditionally after a finalize failure. The superseded -image must therefore be retained in the staging directory for the duration of -the operation, and removed only once it completes. This is the second -prerequisite in [Dependencies](#dependencies). - -Rollback is consequently bounded by the staging directory surviving, rather than -by booting a slot that was never touched. This is the reason the path is not -automatic: selecting it trades a rollback guarantee for the absence of a reboot, -and that trade should be made by the operator rather than inferred from a -property of two images. +A Runtime Update replaces files in the running slot, because +`set_up_extensions` skips removal of the superseded image only on Clean Install +and A/B Update. Rollback re-runs the subsystem with the specs reversed and must +restore the previous image, but cannot re-fetch it: finalize and rollback build +their `EngineContext` with `image: None`, and auto-rollback runs unconditionally +after a finalize failure. The superseded image must be retained for the duration +of the operation. -#### Interaction With Host Configuration `sysexts` and `confexts` +Rollback is therefore bounded by the retained payload surviving, rather than by +booting an untouched slot. That is why the path is requested rather than +inferred: it trades a rollback guarantee for the absence of a reboot, and the +operator should make that trade. -The two sets are merged, and a collision is an error. +#### Interaction With Host Configuration `sysexts` and `confexts` -The lists express different intents and are both valid at once. Bundled -extensions are content the image author considers part of the OS, such as a GPU -driver. Host Configuration extensions are content the operator adds at deploy -time, such as a monitoring agent. An image that ships one sysext and an operator -who adds another is the expected case. +The sets are merged, and a collision is an error. -Silent precedence is rejected: if a Host Configuration entry overrode a bundled -one, the host would run software the image author did not validate, with no +Both are valid at once and express different intents: bundled extensions are +content the image author considers part of the OS, Host Configuration entries +are content the operator adds at deploy time. An image shipping one sysext and +an operator adding another is the expected case. Silent precedence is rejected, +since an override would run software the image author never validated with no visible symptom. -A collision is defined two ways, both already meaningful in the Host -Configuration: - -1. **Same destination path.** Detectable statically when both sides specify - `path`. This extends the existing `DuplicateExtensionImagePath` rule across - the merged set. -2. **Same extension ID** within a kind. Detectable only after the DDIs are - mounted, which the subsystem does regardless. ID uniqueness is documented for - the Host Configuration today but not enforced in code; the merged set makes - collisions more likely, so it should become an enforced check. +A collision is either: -Two entries that are identical in every respect are still a collision. An -"identical is acceptable" exception would introduce hash-comparison subtleties -for no benefit, since the operator can remove their entry. +1. **The same destination path**, detectable from metadata when both sides + specify `path`, extending the existing `DuplicateExtensionImagePath` rule. +2. **The same extension ID** within a kind, detectable only after mounting. ID + uniqueness is documented but unenforced today; the merged set makes + collisions likelier, so it should become an enforced check. -An explicit per-extension override, should one prove necessary, belongs on the -Host Configuration side as an opt-in. See [Open Questions](#open-questions). +Two entries identical in every respect are still a collision; an exception would +introduce hash-comparison subtleties when the operator can simply drop their +entry. A per-extension override, if needed, belongs on the Host Configuration +side as an opt-in. See [Open Questions](#open-questions). ### Validation #### COSI Metadata Validation -Checks that need only the metadata. New `CosiMetadataErrorKind` variants, -following the existing `V1_` prefix: +New `CosiMetadataErrorKind` variants, following the existing `V1_` +prefix: | Variant | Condition | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------- | @@ -738,188 +608,149 @@ following the existing `V1_` prefix: | `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | | `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | -Directory and file-extension rules reuse the constants and logic behind -`Extension::validate_sysext` and `validate_confext` rather than reimplementing -them. +Directory and file-extension rules reuse the logic behind +`Extension::validate_sysext` and `validate_confext`. -Destination collisions are only partly detectable here. An entry that omits -`path` resolves to `{default directory}/{name}.raw`, and `{name}` comes from the -`extension-release` file inside the DDI, so two omitted paths may collide -without the metadata showing it. That case is caught after mounting. +Destination collisions are only partly detectable here: an entry omitting `path` +resolves using the `{name}` inside the DDI, so two omitted paths may collide +invisibly. That case is caught after mounting. Member existence is not checked at load. `Cosi::new` scans tar entries only as -far as `metadata.json` and discovers the rest on demand, so a reference to a -missing member surfaces when the member is read. Changing this would require -scanning the whole archive up front, which conflicts with sparse reads. The -`extensions` section inherits the existing behaviour. +far as `metadata.json` and discovers the rest on demand, so a missing member +surfaces when it is read. Checking up front would require scanning the whole +archive, which conflicts with sparse reads. #### Deploy-Time Validation -Checks that require the payload, performed after the DDI is mounted. +After the DDI is mounted: - **File name matches the `extension-release` suffix.** Already enforced by - `read_extension_release`; applies unchanged. -- **Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID` - present.** Already enforced; applies unchanged. -- **SHA-384 agreement.** `ImageFile.sha384` is verified over the compressed - stream as the payload is decompressed into the staging directory, using the - hashing reader used for partition images. A mismatch aborts servicing. + `read_extension_release`. +- **Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID`.** + Already enforced. +- **SHA-384 agreement**, verified over the compressed stream as the payload is + decompressed. A mismatch aborts servicing. - **Resolved destination collisions** across the effective set, covering the - omitted-`path` cases the metadata check cannot see. New check. -- **Extension ID uniqueness** across the effective set, per kind. New check. + omitted-`path` cases. +- **Extension ID uniqueness** across the effective set, per kind. - **Destination placement** against the storage graph. See [Destination Validation](#destination-validation). -- **`extension-release` OS compatibility.** Warn, do not fail, when a bundled - extension declares `ID=` and that `ID` does not match the COSI's own - `osRelease`. This is a build error worth surfacing early, since systemd will - refuse to merge the extension at boot and the host will come up without it. - - The warning is on `ID` alone. systemd accepts a `VERSION_ID` mismatch when the - applicable `SYSEXT_LEVEL` or `CONFEXT_LEVEL` matches instead, so warning on - version would flag valid extensions. Trident should not reproduce that - precedence; systemd remains the authority. +- **`extension-release` OS compatibility.** Warn when a bundled extension + declares `ID=` that does not match the COSI's `osRelease`: systemd will + refuse to merge it and the host will come up without it. Warn on `ID` alone — + systemd accepts a `VERSION_ID` mismatch when `SYSEXT_LEVEL` or `CONFEXT_LEVEL` + matches, so warning on version would flag valid extensions. #### `ID=_any` -`extension-release` already declares the cadence case: `ID=` with -`VERSION_ID` or `SYSEXT_LEVEL` binds the extension to an OS build, while -`ID=_any` marks it portable across OS versions. - Trident should not warn or refuse when an `ID=_any` extension is bundled. `_any` -describes what the extension is compatible with, not how it should be delivered. -Bundling a portable extension is a legitimate packaging choice, and is how an -air-gapped or single-artefact deployment is achieved. The useful check is the -opposite one above: a bundled `ID=` extension that does not match the -COSI's `osRelease` is almost certainly a build error. +describes what the extension is compatible with, not how it should be delivered, +and bundling a portable extension is how an air-gapped or single-artefact +deployment is achieved. The useful check is the inverse one above. ### SELinux Unchanged. Extensions remain incompatible with SELinux in enforcing mode on -systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. -Carrying the image in a COSI does not affect labelling. +systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`, +and carrying the image in a COSI does not affect labelling. -What must change is where the check looks and when it runs. The validation -raising `ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration -lists only, and runs inside `configure`, which returns early for anything other -than Clean Install and A/B Update. It must read the effective set and run on -every servicing type, or both a COSI-only configuration and an -[extension-only Runtime Update](#extension-only-updates) bypass it. +What changes is where the check looks and when it runs: the validation raising +`ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration lists only +and runs inside `configure`, which returns early for anything but Clean Install +and A/B Update. It must read the effective set and run on every servicing type. ## Public API Design The COSI metadata format is a public contract with its own -[specification](../../Reference/Composable-OS-Image.md) and published JSON -schemas, so the additions above are the public API change. They are additive and -optional. +[specification](../../Reference/Composable-OS-Image.md) and published schemas, so +the additions above are the public API change. They are additive and optional. -The Host Configuration API does not change. The observable difference for an -existing user is that a host may end up with more extensions than its Host -Configuration lists, and that specifying an extension the image already carries -is now an error rather than a duplicate. +The Host Configuration API does not change. The observable difference is that a +host may end up with more extensions than its Host Configuration lists, and that +specifying an extension the image already carries is now an error. ## Testing and Metrics - **Schema.** Add `cosi-metadata-v1.3.schema.json` and samples under - `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`. The existing - schema-validation workflow picks up a new revision given the schema file and a - matching samples directory. Invalid samples cover what the schema can express: - `image.path` outside `images/`, a non-absolute destination, and a destination - not ending in `.raw`. - - The permitted-directory allow-list is Trident policy rather than a property of - the format — systemd searches directories Trident does not accept — so it is - deliberately absent from the schema, and the kind-specific directory and - duplicate-destination rules are covered by Rust validation tests instead. -- **Unit.** Metadata parse and validation for each new error variant; effective - set computation, including collisions and the source discriminator; - default-path resolution for an entry with no `path`; replacement of an - extension that keeps its ID and destination but changes content. -- **Functional.** Build a COSI carrying a sysext and a confext; assert Clean - Install places both, enables the merge units, and that both merge after boot. -- **Servicing.** A/B update from a COSI with extension set A to one with set B; - assert the new slot has B, the old slot retains A, and that rollback restores - A without additional servicing. Separately, an extension-only Runtime Update - between a COSI and a copy of it with a different extension set; assert - partitions are untouched and the extension set changes without a reboot. - Separately again, finalize and rollback of an extension-only Runtime Update - invoked as distinct processes, asserting the previous image is restored - without access to either COSI. Assert that a pending inventory survives the - reboot of an A/B update and is promoted at commit, and discarded when the - boot check fails. -- **Negative.** Streaming a COSI that carries extensions is refused; the same - extension in both the Host Configuration and the COSI - produces the structured error; a bundled extension with SELinux `enforcing` is - rejected on A/B update and on extension-only Runtime Update; a bundled - destination on a shared volume is rejected when A/B is configured; a Runtime - Update requested between two COSIs with differing image summaries is refused + `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`; the existing workflow + picks up a new revision automatically. Invalid samples cover what the schema + can express: `image.path` outside `images/`, a non-absolute destination, and a + destination not ending in `.raw`. The permitted-directory allow-list is Trident + policy rather than a property of the format — systemd searches directories + Trident does not accept — so it stays out of the schema, and the + kind-specific directory and duplicate-destination rules are covered by Rust + tests. +- **Unit.** Each new error variant; effective set computation including + collisions and the source discriminator; default-path resolution; replacement + of an extension keeping its ID and destination but changing content. +- **Functional.** A COSI carrying a sysext and a confext: Clean Install places + both, enables the merge units, and both merge after boot. +- **Servicing.** A/B update from extension set A to B, asserting the new slot has + B, the old retains A, and rollback restores A. An extension-only Runtime Update + between a COSI and a copy with a different extension set, asserting partitions + are untouched. Finalize and rollback of that update as distinct processes, + asserting the previous image is restored without access to either COSI. A + pending inventory surviving an A/B reboot, promoted at commit and discarded + when the boot check fails. +- **Negative.** Streaming a COSI carrying extensions is refused; the same + extension in both sources is a structured error; a bundled extension with + SELinux `enforcing` is rejected on A/B update and on extension-only Runtime + Update; a bundled destination on a shared volume is rejected when A/B is + configured; a Runtime Update between differing image summaries is refused rather than promoted or partially applied. ## Servicing -The change is additive at every layer. +Additive at every layer. -- A revision 1.2 COSI remains valid and behaves identically. -- A revision 1.3 COSI without `extensions` behaves identically. -- A revision 1.3 COSI with extensions, read by a Trident predating this work, - deploys the OS correctly and omits the extensions silently. Mitigations are in - [Versioning](#versioning). +- A 1.2 COSI, and a 1.3 COSI without `extensions`, behave identically. +- A 1.3 COSI with extensions read by an older Trident deploys the OS and omits + the extensions. See [Versioning](#versioning). - Hosts using `os.sysexts` or `os.confexts` are unaffected unless they move to an - image carrying the same extension, which now produces an explicit error. + image carrying the same extension, which is now an explicit error. - Existing hosts have no recorded image summary, so servicing type selection is - unchanged for them until their next deployment. See - [Deployed Image Summary](#deployed-image-summary). + unchanged until their next deployment. ## Implementation Plan -0. Prerequisites from [Dependencies](#dependencies): the in-place replacement - fix, durable staged payloads, and the request surface. The first is a - standalone bug fix and should land independently. -1. Specification: COSI revision 1.3, the `extensions` section, the v1.3 JSON - schema and samples. +0. Prerequisites from [Dependencies](#dependencies). The in-place replacement fix + is a standalone bug and should land independently. +1. COSI revision 1.3: the `extensions` section, the schema and samples. 2. Trident reader: parse `extensions`, metadata validation and the new error variants, and the minor-version warning. -3. Effective extension set with its source discriminator, and the deployed - extension inventory in the Host Status; plumbed into the extensions, - `osconfig` and `selinux` subsystems, including the collision errors and the - move of the SELinux check to a hook that runs on every servicing type. -4. Stream a tar member into the extension staging directory, replacing the URL - fetch for bundled entries. Dynamic destination validation against the storage - graph. -5. Deployed image summary recorded in the Host Status, and the extension-only - Runtime Update path with its verification and refusal behaviour. +3. Effective extension set with its source discriminator and the deployed + extension inventory; plumbed into the extensions, `osconfig` and `selinux` + subsystems, including the collision errors and the move of the SELinux check. +4. Stream a tar member into the staging directory, replacing the URL fetch for + bundled entries. Dynamic destination validation. +5. Deployed image summary and the extension-only Runtime Update path. 6. Tests, then documentation updates to [Sysexts](../../Explanation/Sysexts.md), [Confexts](../../Explanation/Confexts.md) and [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). Steps 1 and 2 are independently useful: a reader that parses and validates the -section but ignores it is a safe intermediate state and makes the minor-version -warning available sooner. Steps 3 and 4 deliver bundled extensions on Clean -Install and A/B Update, and are shippable without step 5. Step 5 may land -separately but is required for this RFC to be complete, since without it every -extension change is an A/B update and the cadence cost the proposal sets out to -bound is not bounded. +section but ignores it is a safe intermediate state. Steps 3 and 4 deliver +bundled extensions on Clean Install and A/B Update and are shippable without +step 5, but step 5 is required for this RFC to be complete — without it every +extension change is an A/B update. ## Counter-Arguments ### Drawbacks -- **Cadence coupling.** Bundling ties the extension's release cadence to the - image's: a new version of the extension requires a new COSI. - [Extension-Only Updates](#extension-only-updates) removes the reboot, and - reduces the build to a copy, but a new COSI must still be published and - distributed. For content that must match the OS, such as kernel modules or GPU - drivers, this costs nothing, because that content could not move independently - in any case. For content deliberately versioned independently of the OS it - remains a loss of agility, and the Host Configuration route stays correct. This - RFC adds an option; it removes none. -- **Size.** COSI files grow, and every host downloading the image pays for every - bundled extension, including those it does not use. There is no per-host - selection mechanism. -- **No opt-out.** A host that wants the image but not one of its bundled - extensions cannot express that. See [Open Questions](#open-questions). -- **Two sources.** Merging, and the collision rules it requires, is more - complexity in the extensions subsystem than exists today. +- **Cadence coupling.** A new version of the extension requires a new COSI. + [Extension-Only Updates](#extension-only-updates) remove the reboot and reduce + the build to a copy, but a new COSI must still be published. For content that + must match the OS, such as kernel modules or GPU drivers, this costs nothing. + For content versioned independently it is a real loss of agility, and the Host + Configuration route stays correct. +- **Size.** COSI files grow, and every host pays for every bundled extension, + including those it does not use. +- **No opt-out.** A host wanting the image but not one of its extensions cannot + say so. See [Open Questions](#open-questions). +- **Two sources.** Merging and its collision rules are more complexity in the + extensions subsystem than exists today. - **Forward compatibility.** An older reader produces a host without the extensions and reports nothing. @@ -927,11 +758,11 @@ bound is not bounded. **Host Configuration only (status quo).** Maximum decoupling: the extension is versioned, hosted and updated independently, and a change is a Runtime Update -with no reboot. This remains fully supported, and is the wrong answer only when -the extension must land in lockstep with a new OS. +with no reboot. Remains fully supported, and is wrong only when the extension +must land in lockstep with a new OS. **Place the `.raw` files in the root filesystem image.** No specification change. -Discussed in +See [Why Not Place the Files in the Root Filesystem Image](#why-not-place-the-files-in-the-root-filesystem-image). **Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** Rejected in @@ -942,80 +773,56 @@ Discussed in #### Prior Art -The decoupled approach is established. Flatcar's sysext-bakery pattern combines -pre-built sysext DDIs with +Flatcar's sysext-bakery pattern pairs pre-built DDIs with [`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html): -the operator hosts artefacts over HTTP with a `SHA256SUMS` manifest, sysupdate -stages new versions against a transfer definition, and activation requires a -`systemd-sysext refresh` or a reboot. It suits content with its own cadence and -is close in spirit to `os.sysexts`, with the difference that -`systemd-sysupdate` has no OCI transport, whereas `Extension.url` already -accepts `oci://`. - -This RFC is a third option rather than a replacement for either. Both existing -approaches optimise for extensions that move independently of the OS; carrying -the extension in the COSI optimises for extensions that must move with the OS, -at the cost of cadence. The `extension-release` fields, `ID=_any` against -`ID=`, indicate which approach a given extension belongs in. +artefacts hosted over HTTP with a `SHA256SUMS` manifest, staged against a +transfer definition, activated by a refresh or reboot. It suits content with its +own cadence and is close in spirit to `os.sysexts`, except that +`systemd-sysupdate` has no OCI transport whereas `Extension.url` accepts +`oci://`. Both optimise for extensions that move independently of the OS; +bundling optimises for extensions that must move with it, at the cost of cadence. +`ID=_any` against `ID=` indicates which a given extension belongs in. ## Open Questions -- **What does the deployed image summary cover?** The proposal includes the - region and filesystem image hashes and identities including `fsType`, `disk` - geometry, `bootloader`, `osArch` and `osRelease`, and excludes `extensions`, - `osPackages`, `compression` and `id`. Excluding too much permits a Runtime - Update between images differing in ways Trident does not model; including too - much rejects legitimate cases. `version` is currently excluded, on the basis - that a revision bump with identical images deploys identical bytes. -- **Should the summary be a structured record or a digest?** A record is - proposed, so that Trident can report which partition differs. A digest is - smaller and simpler, at the cost of the diagnosis. -- **Does the no-op outcome belong in this RFC?** It requires a metadata-only - status transition that records an accepted image without creating a - rollbackable operation. That is a behavioural change independent of - extensions and may deserve its own RFC. -- **Where does the retained superseded image live** for runtime rollback, and - for how long? The staging directory is simplest but only covers the operation - itself; a content-addressed store bounded by the rollback chain also covers - manual runtime rollback, at the cost of disk. The alternative is to refuse - manual runtime rollback across a bundled extension change. -- **Should an override escape hatch exist?** This RFC makes a collision between - the Host Configuration and the COSI an error. The alternative is an explicit - opt-in on the Host Configuration side meaning "the image ships this, use mine - instead". It permits pinning a hotfixed extension without rebuilding the image, - and equally permits running a combination the image author did not validate. - The recommendation is to ship strict and add the hatch if a need appears. -- **Should a bundled extension be suppressible?** A deny list would let an - operator take an image without one of its extensions. There is no analogue in +- **What does the deployed image summary cover?** Excluding too much permits a + Runtime Update between images differing in ways Trident does not model; + including too much rejects legitimate cases. `version` is currently excluded, + on the basis that a revision bump with identical images deploys identical + bytes. +- **A structured record or a digest?** A record is proposed, so Trident can + report which partition differs. A digest is smaller and simpler. +- **Does the no-op outcome belong here?** It needs a metadata-only status + transition recording an accepted image without a rollbackable operation — a + behavioural change independent of extensions. +- **Where does the retained superseded image live, and for how long?** The + staging directory covers only the operation itself; a content-addressed store + bounded by the rollback chain also covers manual runtime rollback, at the cost + of disk. The alternative is refusing that rollback. +- **Should an override escape hatch exist?** An opt-in on the Host Configuration + side would permit pinning a hotfixed extension without rebuilding the image, + and equally permit running a combination the image author did not validate. + Recommendation: ship strict, add the hatch if a need appears. +- **Should a bundled extension be suppressible?** A deny list has no analogue in the current API and no concrete requester. - **Should `path` be required?** Optional with defaults is proposed for symmetry - with the Host Configuration. Requiring it would make every COSI explicit about - placement, at the cost of diverging from the object it mirrors. + with the Host Configuration. - **Naming.** `extensions` containing `sysexts` and `confexts`, against two - top-level `sysexts` and `confexts` arrays. The nested form groups the feature - and keeps the root object small; the flat form is a closer match to - `os.sysexts`. + top-level arrays. The nested form keeps the root object small; the flat form + is a closer match to `os.sysexts`. - **Should Host Configuration extension ID uniqueness be enforced independently - of this work?** It is documented but unenforced. This RFC requires it for the - merged set, but it is arguably a pre-existing gap deserving its own fix. -- **Should writers be required to place extension payloads after all region - images, or merely encouraged?** This RFC says SHOULD. + of this work?** Documented but unenforced; arguably a pre-existing gap. +- **Must writers place payloads after all region images, or is SHOULD enough?** ## Future Possibilities -- **Portable service images.** They are DDIs with the same shape and the same - placement problem. If Trident manages them, `extensions` generalises or gains a - sibling. -- **Per-extension selection at deploy time.** If a single image must serve hosts - with different extension sets, a selection mechanism keyed on extension ID - would build on this section. +- **Portable service images.** DDIs with the same shape and placement problem. +- **Per-extension selection at deploy time**, keyed on extension ID, if one + image must serve hosts with different extension sets. - **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed by - `ExtensionRelease` but not acted on. A bundled extension suits initrd scope, - since the payload is available before the root filesystem is. -- **Bundled extensions under disk streaming.** Requires a way to enable - `systemd-sysext.service` and `systemd-confext.service` on a streamed install, - which `osconfig` currently skips entirely. -- **Reporting the extension inventory.** The - [deployed extension inventory](#deployed-extension-inventory) is internal - state. Surfacing it through the CLI and the gRPC API would make the set of - running extensions answerable without inspecting the filesystem. + `ExtensionRelease` but not acted on. A bundled payload suits initrd scope, + being available before the root filesystem. +- **Bundled extensions under disk streaming**, which needs a way to enable the + merge units on a streamed install. +- **Reporting the extension inventory** through the CLI and gRPC API, making the + running extension set answerable without inspecting the filesystem. From 6ba18a280659657086172cc8d82cb98601ee42b8 Mon Sep 17 00:00:00 2001 From: Paco Date: Thu, 8 Oct 2026 12:09:23 -0700 Subject: [PATCH 13/15] docs: cut COSI extensions RFC to one claim per sentence Prose down from 6691 to 3907 words across two passes. Drops the redundant consumer diagram and trims the example to one entry per array. No decision, constraint or cited behaviour changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 822 +++++++----------- 1 file changed, 331 insertions(+), 491 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 9471af6a5f..72e292ae43 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -6,32 +6,25 @@ ## Summary -COSI files cannot carry systemd system extension (sysext) or configuration -extension (confext) images. This RFC stores them as ZSTD-compressed members of -the COSI tar, alongside partition images, and adds an optional `extensions` +COSI files cannot carry sysext or confext images. This RFC stores them as +ZSTD-compressed members of the COSI tar and adds an optional `extensions` section to the metadata. Each entry reuses the existing [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) object and -carries an optional destination path, mirroring the +an optional destination path, mirroring the [`Extension`](../../Reference/Host-Configuration/API-Reference/Extension.md) object in the Host Configuration. ## Motivation and Goals -Extensions are configured today through `os.sysexts` and `os.confexts`, each -entry a URL and a SHA-384. That makes an extension a second artefact with a -separate lifecycle: +Extensions are configured through `os.sysexts` and `os.confexts`, each entry a +URL and a SHA-384. That makes an extension a second artefact: separately hosted, +fetched from a second endpoint at deploy time, with integrity anchored outside +`os.image.sha384`, and updated by a Runtime Update rather than the A/B update +that ships the OS. Content that must land with a new OS needs two operations. -- Hosted separately, for as long as any host may re-run the flow. -- Fetched from a second endpoint at deploy time, with its own network, proxy and - authentication failure modes. -- Integrity anchored by whoever wrote the Host Configuration, not by - `os.image.sha384`. -- Changed by a Runtime Update, distinct from the A/B update that ships the OS, - so content that must land with a new OS requires two operations. - -Bundled, an extension is delivered and activated in the same A/B update and the -same reboot as the OS, reusing the existing rollout, health-gating and rollback -machinery. Rollback needs no extra work; see [Rollback](#rollback). +Bundled, an extension is delivered and activated in the same A/B update and +reboot as the OS, reusing the existing rollout, health-gating and rollback +machinery. See [Rollback](#rollback). ### Why Not Place the Files in the Root Filesystem Image @@ -42,13 +35,13 @@ is built. That works in some configurations, but: hash and requires re-signing. 2. Extension destinations are commonly on a separate volume, which may be created empty rather than written from a partition image. -3. Filesystem image contents are invisible to tooling that reads COSI metadata. +3. Filesystem image contents are invisible to tooling reading COSI metadata. 4. Trident enables the merge units, validates placement and reads `extension-release` only for extensions it knows about. ### Goals -- Deploy extensions from a COSI without a second artefact, endpoint or update. +- Deploy extensions from a COSI with no second artefact, endpoint or update. - Reuse existing metadata objects and the shape of the Host Configuration API. - Leave the Host Configuration API unchanged. @@ -59,35 +52,32 @@ is built. That works in some configurations, but: - An optional `extensions` object in the metadata root, holding `sysexts` and `confexts` arrays. - Each entry identifies its payload with an `ImageFile` and may specify a - destination path. + destination. - Payloads are ZSTD-compressed tar members under the existing compression and integrity rules. - Trident deploys bundled extensions on Clean Install and A/B Update. - A change confined to the extension set is applicable as a Runtime Update. See [Extension-Only Updates](#extension-only-updates). - Bundled and Host Configuration extensions coexist; conflicts are an error. -- COSI metadata validation covers the new section. ### Out of Scope - Changes to the Host Configuration `Extension` object. -- Extension-only COSI files, carrying extensions and no partition images. See - [Extension-Only Updates](#extension-only-updates). -- Bundled extensions under [disk streaming](../../Explanation/Disk-Streaming.md), - which cannot enable the merge units. Streaming such a COSI is refused. -- Signing or attestation beyond the SHA-384 chain COSI already provides. -- SELinux compatibility, which is unchanged. See [SELinux](#selinux). +- Extension-only COSI files, carrying extensions and no partition images. +- Bundled extensions under [disk streaming](../../Explanation/Disk-Streaming.md). + Streaming such a COSI is refused. +- Signing beyond the SHA-384 chain COSI already provides. +- SELinux compatibility, unchanged. See [SELinux](#selinux). - Producing the DDIs. ### Exit Criteria -- COSI revision 1.3 published with the section, `cosi-metadata-v1.3.schema.json` - and samples under `tests/cosi/metadata_samples/v1.3/`. -- Trident deploys a COSI carrying a sysext and a confext on Clean Install and - A/B Update, both merged after reboot. Streaming such a COSI is refused. +- COSI revision 1.3 published, with schema and samples. +- A COSI carrying a sysext and a confext deploys on Clean Install and A/B + Update, both merged after reboot; streaming it is refused. - A/B rollback restores the previous extension set with no extra servicing. -- An extension-only change is applied as a Runtime Update without rewriting - partitions or rebooting, and refused when anything else differs. +- An extension-only change applies as a Runtime Update, and is refused when + anything else differs. - Conflicts between bundled and Host Configuration extensions are a structured error. @@ -97,66 +87,54 @@ A COSI writer that emits the section. [Image Customizer](https://github.com/microsoft/azure-linux-image-tools) is the reference writer. -[Extension-Only Updates](#extension-only-updates) additionally require: +[Extension-Only Updates](#extension-only-updates) also require: -- A writer mode that copies an existing COSI's region images verbatim while - replacing its extension set. Without it, bundled extensions still work, but - every extension change is an A/B update. -- A surface for requesting a servicing type: a `trident update` flag, an - internal parameter alongside `forceAbUpdate`, or a Host Configuration field. - None exists. It must define precedence against `forceAbUpdate` and whether the - request survives separate stage and finalize invocations. +- A writer mode copying an existing COSI's region images verbatim while + replacing its extension set. Without it, every extension change is an A/B + update. +- A surface for requesting a servicing type. None exists; `forceAbUpdate` is the + nearest precedent. It must define precedence against `forceAbUpdate` and + whether the request survives separate stage and finalize invocations. -Three existing defects must be fixed first. All are reachable today with Host -Configuration extensions; bundling makes the first two routine. +Three existing defects must be fixed first. All are reachable today; bundling +makes the first two routine. 1. **In-place replacement deletes the new image.** When an extension keeps its ID and destination but changes content, `set_up_extensions` schedules the ID - for both addition and removal. The addition renames the new image over the - destination; the removal then deletes that path, because the old entry's - `temp_path` is its destination. The guard assumes the paths differ. + for both addition and removal. The addition renames over the destination; the + removal then deletes that path, because the old entry's `temp_path` is its + destination. The guard assumes the paths differ. 2. **Staged payloads do not survive the operation.** Runtime finalize and - rollback construct their `EngineContext` with `image: None`, so the COSI is - gone after stage. Payloads must be staged durably, and the superseded image - retained until the operation completes. + rollback build their `EngineContext` with `image: None`, so the COSI is gone + after stage. 3. **Placement is validated against the Host Configuration only, before the COSI - is read.** `validate_extension_images_locations` is static validation over - `os.sysexts` and `os.confexts`. See - [Destination Validation](#destination-validation). + is read.** See [Destination Validation](#destination-validation). ## Implementation ### Tar Layout -Payloads are ZSTD-compressed DDIs under `images/extensions/`, for example -`images/extensions/my-tool.rawzst`. - -They stay under `images/`: the specification already permits subdirectories -there and requires readers to handle them. Relaxing `ImageFile.path`'s -`^images/.+` pattern would produce a 1.3 `ImageFile` that fails the 1.0–1.2 -schemas and break any validator with that definition compiled in, buying only a -shorter path. +Payloads are ZSTD-compressed DDIs under `images/extensions/`. -Two existing ordering rules apply. The primary GPT image must immediately follow -`metadata.json` since revision 1.2, so payloads must not sit between them. -Region images must appear in the physical order of the regions on disk; -payloads are not regions, but interleaving them obscures that ordering and hurts -sparse-read locality, so they should be written after all region images. +They stay under `images/`, which the specification already permits and requires +readers to handle. Relaxing `ImageFile.path`'s `^images/.+` pattern would +produce a 1.3 `ImageFile` that fails the 1.0–1.2 schemas, for a shorter path. -Writers must account for payloads in the root `compression.maxWindowLog`. +The primary GPT image must immediately follow `metadata.json` since revision +1.2, so payloads must not sit between them. Payloads are not regions and do not +participate in region ordering, but should be written after all region images to +keep that ordering verifiable and preserve sparse-read locality. Writers must +account for them in `compression.maxWindowLog`. -Older readers are unaffected: the specification requires them to ignore unknown -files, and Trident's orphan-image check -(`V1_2ImageFileHasNoCorrespondingPartition`) is driven by the `images[]` and -`disk.gptRegions[]` arrays rather than by walking tar entries. +Older readers are unaffected: unknown files must be ignored, and Trident's +orphan-image check (`V1_2ImageFileHasNoCorrespondingPartition`) reads the +`images[]` and `disk.gptRegions[]` arrays rather than walking tar entries. ### Metadata Schema -A new optional root field: - -| Field | Type | Added in | Required | Description | -| ------------ | -------------------------------- | -------- | -------- | ------------------------------------------- | -| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images carried by this COSI file. | +| Field | Type | Added in | Required | Description | +| ------------ | -------------------------------- | -------- | -------- | ------------------------------- | +| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images in this COSI. | #### `Extensions` Object @@ -167,10 +145,10 @@ A new optional root field: #### `ExtensionImage` Object -| Field | Type | Added in | Required | Description | -| ------- | -------------------------------------------------------------------- | -------- | --------------- | ------------------------------------------------------------ | -| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | Details of the compressed extension image in the tar file. | -| `path` | string | 1.3 | No | Absolute destination path of the extension on the target OS. | +| Field | Type | Added in | Required | Description | +| ------- | -------------------------------------------------------------------- | -------- | --------------- | ----------------------------------------------- | +| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | The compressed extension image in the tar file. | +| `path` | string | 1.3 | No | Absolute destination path on the target OS. | ```json { @@ -218,208 +196,148 @@ A new optional root field: Example: ```json -{ - "version": "1.3", - "osArch": "x86_64", - "osRelease": "NAME=\"Microsoft Azure Linux\"\nID=azurelinux\nVERSION_ID=\"3.0\"\n", - "images": [ - // Filesystem objects, unchanged. - ], - "disk": { - // Disk object, unchanged. - }, - "osPackages": [ - // OsPackage objects, unchanged. - ], - "bootloader": { - // Bootloader object, unchanged. - }, - "compression": { "maxWindowLog": 22 }, - "extensions": { - "sysexts": [ - { - "image": { - "path": "images/extensions/gpu-driver.rawzst", - "compressedSize": 41943040, - "uncompressedSize": 134217728, - "sha384": "3a1f9c0d4e5b6a7c8d9e0f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e" - }, - "path": "/var/lib/extensions/gpu-driver.raw" +"extensions": { + "sysexts": [ + { + "image": { + "path": "images/extensions/gpu-driver.rawzst", + "compressedSize": 41943040, + "uncompressedSize": 134217728, + "sha384": "3a1f9c0d4e5b6a7c8d9e0f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e" }, - { - "image": { - "path": "images/extensions/debug-tools.rawzst", - "compressedSize": 8388608, - "uncompressedSize": 33554432, - "sha384": "b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1" - } - } - ], - "confexts": [ - { - "image": { - "path": "images/extensions/fleet-config.rawzst", - "compressedSize": 262144, - "uncompressedSize": 1048576, - "sha384": "c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2" - }, - "path": "/var/lib/confexts/fleet-config.raw" + "path": "/var/lib/extensions/gpu-driver.raw" + } + ], + "confexts": [ + { + "image": { + "path": "images/extensions/fleet-config.rawzst", + "compressedSize": 262144, + "uncompressedSize": 1048576, + "sha384": "c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2" } - ] - } + } + ] } ``` #### Two Arrays Rather Than One With a `kind` Field -Sysexts and confexts have disjoint permitted directories -(`VALID_SYSEXT_DIRECTORIES`, `VALID_CONFEXT_DIRECTORIES`), different defaults, -different `extension-release` locations, different identity fields (`SYSEXT_ID`, -`CONFEXT_ID`) and different activation units. Every downstream rule branches on -the kind. Two arrays make it structural, so it cannot be omitted or wrong, and -match the Host Configuration. +Sysexts and confexts have disjoint permitted directories, different defaults, +different `extension-release` locations, different identity fields and different +activation units. Every downstream rule branches on the kind. Two arrays make it +structural, and match the Host Configuration. #### Entry Contents `image` and an optional `path`. Everything else is derivable from the payload, and each derivable field is a second source of truth. -- **`path`** is kept: the destination is not derivable. Optional, because - `Extension.path` is optional and defaults to `/var/lib/extensions/{name}.raw` - or `/var/lib/confexts/{name}.raw`. The existing rules apply — absolute, - ending in `.raw`, in a permitted directory for its kind, file name matching the - `extension-release` suffix. -- **Extension name** is rejected: derived from the `extension-release.{name}` - file name, and the destination file name must already match it. +- **`path`** is kept: the destination is not derivable. Optional, defaulting as + `Extension.path` does. Existing rules apply — absolute, ending in `.raw`, in a + permitted directory for its kind, file name matching the `extension-release` + suffix. +- **Name** is rejected: derived from `extension-release.{name}`, which the + destination file name must already match. - **`kind`** is rejected: structural. - **Extension ID** is rejected: read from `extension-release`, and already the identity Trident keys A/B state on. - **Enable-on-first-boot** is rejected: systemd merges everything in the extension directories, and the Host Configuration has no equivalent switch. -- **Version and compatibility metadata** is rejected: `ID`, `VERSION_ID`, - `SYSEXT_LEVEL`, `CONFEXT_LEVEL`, `ARCHITECTURE` and `SYSEXT_SCOPE` are in the - DDI and enforced by systemd. Copying them would let a COSI claim compatibility - the payload lacks. +- **Version and compatibility metadata** is rejected: already in the DDI and + enforced by systemd. Copying it would let a COSI claim compatibility the + payload lacks. #### `sha384` Semantics `ImageFile.sha384` covers the compressed image; `Extension.sha384` covers the -raw file. For integrity this matches partition images — Trident hashes the -compressed stream as it decompresses — so a second hash of the DDI would add -nothing. +raw file. For integrity this matches partition images, since Trident hashes the +compressed stream as it decompresses. -`ExtensionData.sha384` is a single field serving both as change detector and as -the key matching a processed extension to its Host Configuration entry, so the -effective entry must record the hash with its source and compare only within a -source. A compressed hash against a raw hash is always unequal and would report -every extension as changed. +`ExtensionData.sha384` serves both as change detector and as the key matching a +processed extension to its Host Configuration entry, so entries must record the +hash with its source and compare only within a source. Across sources the +comparison is always unequal and would report every extension as changed. -Within one source the comparison is sound but imprecise: compressed hashes are -unstable across recompressions, so identical DDIs at different ZSTD levels -appear to differ. The result is a redundant copy, never a missed update. +Within a source it is sound but imprecise: compressed hashes are unstable across +recompressions, so identical DDIs at different ZSTD levels appear to differ. The +result is a redundant copy, never a missed update. ### Versioning -COSI revision 1.3. `extensions` is optional; absent equals empty. - -A newer Trident reading a 1.0–1.2 COSI sees no bundled extensions and behaves as -today. - -An older Trident reading a 1.3 COSI accepts it — the version check rejects only -`major != 1`, and unknown fields must be ignored — deploys the OS correctly, and -omits the extensions silently. This cannot be corrected in-band, since any -tripwire field is one such a reader must ignore. Nor does it self-heal: the old -reader records the new COSI as deployed, so after upgrading Trident the same -Host Configuration is no longer an image change and the skipped extensions are -never installed. Recovery needs a forced redeployment or a republished COSI. - -Mitigations: - -1. Document that bundled extensions require a reader that understands 1.3. -2. Warn in `validate_cosi_metadata_version` when `minor` exceeds the highest - known revision. This makes future minor bumps diagnosable from the log. -3. Where the target's Trident version cannot be controlled, gate the update on a - [health check](../../Reference/Host-Configuration/API-Reference/Health.md) - asserting the extension is merged, so the A/B update fails and rolls back. - Given that the omission does not self-heal, this is the only real safeguard. +COSI revision 1.3. `extensions` is optional; absent equals empty. A newer +Trident reading a 1.0–1.2 COSI behaves as today. + +An older Trident reading a 1.3 COSI accepts it, since the version check rejects +only `major != 1` and unknown fields must be ignored. It deploys the OS and +omits the extensions silently. This cannot be corrected in-band: any tripwire +field is one such a reader must ignore. Nor does it self-heal — the old reader +records the new COSI as deployed, so after upgrading Trident the same Host +Configuration is no longer an image change. Recovery needs a forced +redeployment. + +Mitigations: document the requirement; warn in +`validate_cosi_metadata_version` when `minor` exceeds the highest known +revision; and, where the target's Trident version cannot be controlled, gate the +update on a +[health check](../../Reference/Host-Configuration/API-Reference/Health.md) +asserting the extension is merged. Given that the omission does not self-heal, +the health check is the only real safeguard. ### Trident-Side Consumption Trident computes an **effective extension set**, the union of the Host Configuration and COSI entries, and the code reading `ctx.spec.os.sysexts` and -`ctx.spec.os.confexts` reads that instead. Each entry carries its source, which -determines where a resolved path is written back. - -```mermaid -flowchart LR - HC["Host Configuration
os.sysexts / os.confexts"] --> Eff - COSI["COSI metadata
extensions.sysexts / .confexts"] --> Eff - Eff["Effective extension set"] --> Ext["ExtensionsSubsystem
stage, mount, read extension-release, place"] - Eff --> Osc["osconfig subsystem
enable systemd-sysext / systemd-confext"] - Eff --> Sel["selinux subsystem
enforcing-mode rejection"] -``` - -Three consumers: - -- **`ExtensionsSubsystem`.** `populate_extensions` downloads each entry over - `Extension.url`, verifies the hash and mounts the DDI to read - `extension-release`. For a bundled entry only the source changes: the image - streaming pipeline decompresses the tar member into the same staging directory - and verifies `ImageFile.sha384` over the compressed stream. The rest is - unchanged. `update_host_configuration` fails when a processed extension has no - Host Configuration entry matching by hash, so it must write back resolved - paths for Host-Configuration-sourced entries only. +`confexts` reads that instead. Each entry carries its source. Three consumers: + +- **`ExtensionsSubsystem`.** For a bundled entry only the source changes: the + image streaming pipeline decompresses the tar member into the staging + directory and verifies `ImageFile.sha384` over the compressed stream, in place + of the `FileReader` download. `update_host_configuration` fails when a + processed extension has no Host Configuration entry matching by hash, so it + must write back resolved paths for Host-Configuration-sourced entries only. - **`osconfig`.** The merge units are enabled only when `ctx.spec.os.sysexts` or - `confexts` are non-empty, so a COSI-only host would never have them enabled - and its extensions would sit unused. This check must read the effective set. + `confexts` are non-empty, so a COSI-only host would never have them enabled. - **`selinux`.** The validation raising `ExtensionImagesAndSelinuxUnsupported` - must read the effective set too, or a bundled extension with an enforcing-mode - Host Configuration passes and produces a mislabelled `/usr`, `/opt` or `/etc`. - That alone is insufficient: the check runs inside `configure`, which returns - early for Runtime Updates, and `SelinuxSubsystem` inherits - `runs_on = REQUIRES_REBOOT`. It must move to a hook running on every servicing - type, or [Extension-Only Updates](#extension-only-updates) bypass it. - -[Disk streaming](../../Explanation/Disk-Streaming.md) is excluded. The effective -set needs no synthesis from `derive_host_configuration`, but `osconfig` returns -early when `is_stream_image` is set, so the units are never enabled and the -extensions would be placed and never merged. Trident refuses to stream a COSI -carrying extensions rather than produce that state. Lifting the exclusion means -giving streaming a way to enable the units, which is separable work. + must also read the effective set, but that alone is insufficient: it runs + inside `configure`, which returns early for Runtime Updates, and + `SelinuxSubsystem` inherits `runs_on = REQUIRES_REBOOT`. It must move to a + hook running on every servicing type, or + [Extension-Only Updates](#extension-only-updates) bypass it. + +[Disk streaming](../../Explanation/Disk-Streaming.md) is excluded: `osconfig` +returns early when `is_stream_image` is set, so the units are never enabled and +the extensions would be placed and never merged. Trident refuses to stream a +COSI carrying extensions rather than produce that state. #### Deployed Extension Inventory The Host Configuration records what an operator asked for, so `ctx.spec_old` -does not describe what is deployed. Without a second record Trident cannot -determine which bundled extensions to remove, detect collisions against the -deployed set, compare sets for servicing selection, or rebuild its state when -finalize runs separately from stage. +does not describe what is deployed. Without a second record Trident cannot tell +which bundled extensions to remove, detect collisions against the deployed set, +compare sets for servicing selection, or rebuild its state when finalize runs +separately from stage. Trident records a deployed extension inventory in the Host Status, alongside the -[deployed image summary](#deployed-image-summary): per entry, kind, extension -ID, name, resolved destination, source, and the hash under that source's -semantics. `extensions_old` is populated from it rather than from -`ctx.spec_old`. - -Inventory and summary follow the discipline already applied to `spec` and -`spec_old`: a staged operation records a pending record while the deployed one -is retained, and resetting a staged operation reverts to the deployed one. -Promotion follows the servicing type rather than the finalize step. Runtime -Update and `ManualRollbackRuntime` promote on completion, transitioning straight -to `Provisioned`. Clean Install, A/B Update and manual A/B rollback only reach a -`*Finalized` state before rebooting and commit after booting the expected root, -so their pending record must survive the reboot, be promoted at commit, and be -discarded when the boot check fails and `spec_old` is restored. +[deployed image summary](#deployed-image-summary): kind, extension ID, name, +resolved destination, source, and the hash under that source's semantics. +`extensions_old` is populated from it. + +Inventory and summary follow the discipline applied to `spec` and `spec_old`: a +staged operation records a pending record while the deployed one is retained. +Promotion follows the servicing type, not the finalize step. Runtime Update and +`ManualRollbackRuntime` promote on completion. Clean Install, A/B Update and +manual A/B rollback reach `*Finalized` before rebooting and commit after booting +the expected root, so their pending record must survive the reboot and be +discarded when the boot check fails. `ManualRollbackChainItem` carries only kind, `spec`, active volume and install -index, so it must carry the historical inventory and summary as well. Even then, -metadata alone is insufficient for a manual runtime rollback: the context is -built with `image: None` and the superseded payload is retained only for the -operation that replaced it. Either payloads are kept in a content-addressed -store for as long as the rollback chain references them, with the inventory -holding the reference, or manual runtime rollback across a bundled extension -change is refused. Refusing is the safe default; silently restoring nothing is -not. +index, so it must carry the historical inventory and summary too. Even then a +manual runtime rollback has no payload: its context is built with `image: None` +and the superseded image is retained only for the operation that replaced it. +Either payloads are kept in a content-addressed store for as long as the +rollback chain references them, or that rollback is refused. Refusing is the +safe default. #### Destination Validation @@ -427,39 +345,31 @@ not. when A/B is configured, but it is static validation over the Host Configuration lists and runs before the COSI is read. -The equivalent check for the effective set runs after the COSI loads, as dynamic -validation against the storage graph, rejecting a destination that is shared -when A/B is configured. It must resolve the filesystem that actually backs the -destination for the servicing type, not the image's nominal layout: under -root-verity the engine mounts a writable `/etc` overlay during provision, backed -by `/var/lib/trident-overlay` on an A/B volume, so `/etc/extensions/` is -writable and rolls back with the slot. Rejecting verity-backed destinations -outright would reject that supported case. The same destination has no such -backing on a Runtime Update and must be rejected there. +The equivalent check for the effective set runs after the COSI loads, against +the storage graph. It must resolve the filesystem that actually backs the +destination for the servicing type: under root-verity the engine mounts a +writable `/etc` overlay during provision, backed by `/var/lib/trident-overlay` +on an A/B volume, so `/etc/extensions/` is writable and rolls back with the +slot. Rejecting verity-backed destinations outright would reject that case. The +same destination has no such backing on a Runtime Update. #### Where Images Are Written On Clean Install and A/B Update, `provision()` runs with the target root at -`mount_path`, and extensions are staged and moved to their destination inside -it. For A/B that is the inactive slot, so the running system is untouched until -reboot. Staging currently uses a fixed `/var/lib/extensions/.staging` with a -non-atomic copy when the rename crosses a filesystem boundary, which assumes -`/var` is writable and makes placement non-atomic for `/etc/extensions/` or -`/usr/lib/confexts/`. It should use a temporary file on the destination's own -filesystem. - -On Runtime Update `provision()` is not called, so partitions are never touched -and extensions are placed in the running root. See -[Extension-Only Updates](#extension-only-updates). +`mount_path` and extensions are placed inside it — for A/B, the inactive slot. +Staging currently uses a fixed `/var/lib/extensions/.staging` with a non-atomic +copy when the rename crosses a filesystem boundary; it should use a temporary +file on the destination's own filesystem. + +On Runtime Update `provision()` is not called, so partitions are never touched. #### Rollback -Extension images are files in the slot's own filesystem. Where every destination -is on an A/B volume, the previous slot retains the previous set, an A/B rollback -boots it, `systemd-sysext` merges what it finds, and the extension set reverts -with the OS. +Extension images are files in the slot's own filesystem, so where every +destination is on an A/B volume the previous slot retains the previous set and +an A/B rollback reverts it with the OS. -That guarantee depends on the placement check covering bundled destinations; see +That depends on the placement check covering bundled destinations; see [Destination Validation](#destination-validation). A bundled extension on a shared volume changes the running slot immediately and cannot be rolled back. @@ -467,14 +377,13 @@ shared volume changes the running slot immediately and cannot be rolled back. `ab_update_required()` returns true whenever `os.image.sha384` differs, before any subsystem is consulted, and the metadata hash covers the `extensions` -section. Changing only a bundled extension would therefore force a full A/B -update, making bundling strictly more disruptive than the status quo. +section. Changing only a bundled extension would force a full A/B update, +making bundling strictly more disruptive than the status quo. An extension-only COSI is the wrong shape for this. The metadata root requires `images`, `disk` with at least one `gptRegions` entry, `bootloader` and -`osPackages`, and every consumer of `os.image` — filesystem source resolution, -`derive_host_configuration`, ESP detection, verity setup — assumes a complete -OS. The distinction is drawn on content equality, not absent content. +`osPackages`, and every consumer of `os.image` assumes a complete OS. The +distinction is drawn on content equality, not absent content. ##### Deployed Image Summary @@ -482,23 +391,19 @@ Trident retains only the URL and metadata hash of the applied image, and re-fetching the previous COSI to diff it is not dependable, so it must record what it deployed. -The summary holds, for each entry in `images[]` and `disk.gptRegions[]`, the -`image.sha384`, `uncompressedSize` and the entry's identity (partition number, -mount point, `fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, -`osRelease`, `disk` geometry and `bootloader`. Excluded: - -- `extensions`, the subject of the comparison. -- `osPackages`, which Trident validates but never acts on. -- `compression.maxWindowLog`, which governs decompression rather than content - and which a new extension may legitimately raise. -- `id`, which identifies the file rather than its content. +The summary holds, per entry in `images[]` and `disk.gptRegions[]`, the +`image.sha384`, `uncompressedSize` and identity (partition number, mount point, +`fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, +`disk` geometry and `bootloader`. It excludes `extensions`, the subject of the +comparison; `osPackages`, which Trident validates but never acts on; +`compression.maxWindowLog`, which a new extension may legitimately raise; and +`id`, which identifies the file rather than its content. -A record rather than a digest: the digest is derivable from it, and the record -also lets Trident report which partition differs. +A record rather than a digest, so Trident can report which partition differs. -The summary is versioned. Where it is absent or unrecognised, Trident falls back -to comparing `os.image.sha384`, so existing hosts are unaffected until their -next deployment. +The summary is versioned. Where absent or unrecognised, Trident falls back to +comparing `os.image.sha384`, so existing hosts are unaffected until their next +deployment. | Summary | Extension set | Image requires | | ------- | ------------- | ----------------- | @@ -506,52 +411,44 @@ next deployment. | Equal | Differs | A Runtime Update. | | Differs | Any | An A/B update. | -This is the image's requirement, not the outcome. `select_servicing_type` still -takes the maximum across subsystems, so a concurrent change to users, modules or -the kernel command line can independently require an A/B update. - -The no-op row is new: `os.image.sha384` changes whenever any part of the -metadata changes, including `id` and the ordering of `osPackages`, so a -materially identical image currently forces an A/B update. +This is the image's requirement, not the outcome; `select_servicing_type` still +takes the maximum across subsystems. The no-op row is new, since +`os.image.sha384` changes whenever any part of the metadata does. ##### Constructing an Extension-Only Update Summary equality requires byte-identical region images, and filesystem images -are not reproducible: filesystem UUIDs, inode timestamps, superblock creation -and mount times and allocation order vary between builds, and ZSTD output varies -with level and library version. +are not reproducible: UUIDs, inode timestamps, superblock times and allocation +order vary between builds, and ZSTD output varies with level and library +version. So the build operation is not "rebuild the image identically with a different extension set", it is "copy an existing COSI, replacing its extension set". -Region images and their metadata entries are copied verbatim; nothing is rebuilt -or recompressed. Only the extension tar members, the `extensions` section, -`compression.maxWindowLog` where a larger window is needed, and `id` change. -Equality then holds by construction, and the operation is a tar rewrite rather -than an image build. +Region images and their metadata entries are copied verbatim; only the extension +members, the `extensions` section, `compression.maxWindowLog` and `id` change. +Equality then holds by construction, and the operation is a tar rewrite. -A COSI rebuilt from source instead will differ, and Trident will select an A/B -update. The failure mode of an unreproducible build is a redundant A/B update, -never a skipped one. +A COSI rebuilt from source will differ and Trident will select an A/B update. +The failure mode of an unreproducible build is a redundant A/B update, never a +skipped one. ##### Requested and Verified, Never Inferred An extension-only update proceeds when all of: 1. A Runtime Update is explicitly requested. -2. The recorded summary equals the summary computed from the new COSI. +2. The recorded summary equals the one computed from the new COSI. 3. The recorded image is the deployed one: servicing completed, no A/B update pending, active volume matching the Host Status. 4. No other subsystem requires an A/B update. -If a Runtime Update is requested and any of 2 to 4 fails, Trident fails with a -structured error naming what differs. It must not promote the operation to an -A/B update, and must not apply the extension change while leaving other changes -unapplied. +If 2, 3 or 4 fails, Trident fails with a structured error naming what differs. +It must not promote the operation to an A/B update, nor apply the extension +change while leaving other changes unapplied. A no-op must still record the accepted image and summary. -`select_servicing_type` returning `NoActiveServicing` causes `update` to return -without persisting anything, so a metadata-only status transition is needed or -every subsequent invocation repeats the comparison. +`select_servicing_type` returning `NoActiveServicing` makes `update` return +without persisting anything, so a metadata-only status transition is needed. ##### Rollback on This Path @@ -559,38 +456,29 @@ A Runtime Update replaces files in the running slot, because `set_up_extensions` skips removal of the superseded image only on Clean Install and A/B Update. Rollback re-runs the subsystem with the specs reversed and must restore the previous image, but cannot re-fetch it: finalize and rollback build -their `EngineContext` with `image: None`, and auto-rollback runs unconditionally -after a finalize failure. The superseded image must be retained for the duration -of the operation. +their context with `image: None`, and auto-rollback runs unconditionally after a +finalize failure. The superseded image must be retained for the operation. -Rollback is therefore bounded by the retained payload surviving, rather than by -booting an untouched slot. That is why the path is requested rather than -inferred: it trades a rollback guarantee for the absence of a reboot, and the -operator should make that trade. +Rollback is therefore bounded by that retained payload rather than by booting an +untouched slot, which is why the path is requested rather than inferred. #### Interaction With Host Configuration `sysexts` and `confexts` The sets are merged, and a collision is an error. -Both are valid at once and express different intents: bundled extensions are -content the image author considers part of the OS, Host Configuration entries -are content the operator adds at deploy time. An image shipping one sysext and -an operator adding another is the expected case. Silent precedence is rejected, -since an override would run software the image author never validated with no -visible symptom. +Both are valid at once: bundled extensions are content the image author +considers part of the OS, Host Configuration entries are content the operator +adds. Silent precedence is rejected, since an override would run software the +image author never validated with no visible symptom. -A collision is either: +A collision is either the same destination path, detectable from metadata when +both sides specify `path`, or the same extension ID within a kind, detectable +only after mounting. ID uniqueness is documented but unenforced today, so it +should become an enforced check. -1. **The same destination path**, detectable from metadata when both sides - specify `path`, extending the existing `DuplicateExtensionImagePath` rule. -2. **The same extension ID** within a kind, detectable only after mounting. ID - uniqueness is documented but unenforced today; the merged set makes - collisions likelier, so it should become an enforced check. - -Two entries identical in every respect are still a collision; an exception would -introduce hash-comparison subtleties when the operator can simply drop their -entry. A per-extension override, if needed, belongs on the Host Configuration -side as an opt-in. See [Open Questions](#open-questions). +Two entries identical in every respect are still a collision; the operator can +drop theirs. A per-extension override, if needed, belongs on the Host +Configuration side as an opt-in. See [Open Questions](#open-questions). ### Validation @@ -608,56 +496,43 @@ prefix: | `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | | `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | -Directory and file-extension rules reuse the logic behind -`Extension::validate_sysext` and `validate_confext`. - -Destination collisions are only partly detectable here: an entry omitting `path` -resolves using the `{name}` inside the DDI, so two omitted paths may collide -invisibly. That case is caught after mounting. +These reuse the logic behind `Extension::validate_sysext` and +`validate_confext`. -Member existence is not checked at load. `Cosi::new` scans tar entries only as -far as `metadata.json` and discovers the rest on demand, so a missing member -surfaces when it is read. Checking up front would require scanning the whole -archive, which conflicts with sparse reads. +Destination collisions are only partly detectable here, since an entry omitting +`path` resolves using the `{name}` inside the DDI. Member existence is not +checked at load either: `Cosi::new` scans only as far as `metadata.json` and +discovers the rest on demand, and checking up front would conflict with sparse +reads. #### Deploy-Time Validation -After the DDI is mounted: - -- **File name matches the `extension-release` suffix.** Already enforced by - `read_extension_release`. -- **Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID`.** - Already enforced. -- **SHA-384 agreement**, verified over the compressed stream as the payload is - decompressed. A mismatch aborts servicing. -- **Resolved destination collisions** across the effective set, covering the - omitted-`path` cases. -- **Extension ID uniqueness** across the effective set, per kind. -- **Destination placement** against the storage graph. See - [Destination Validation](#destination-validation). -- **`extension-release` OS compatibility.** Warn when a bundled extension - declares `ID=` that does not match the COSI's `osRelease`: systemd will - refuse to merge it and the host will come up without it. Warn on `ID` alone — - systemd accepts a `VERSION_ID` mismatch when `SYSEXT_LEVEL` or `CONFEXT_LEVEL` - matches, so warning on version would flag valid extensions. - -#### `ID=_any` - -Trident should not warn or refuse when an `ID=_any` extension is bundled. `_any` -describes what the extension is compatible with, not how it should be delivered, -and bundling a portable extension is how an air-gapped or single-artefact -deployment is achieved. The useful check is the inverse one above. +After the DDI is mounted. The first two are already enforced by +`read_extension_release` and apply unchanged. + +- File name matches the `extension-release` suffix. +- Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID`. +- SHA-384 agreement, verified over the compressed stream during decompression. +- Resolved destination collisions across the effective set. +- Extension ID uniqueness across the effective set, per kind. +- Destination placement. See [Destination Validation](#destination-validation). +- `extension-release` OS compatibility: warn when a bundled extension declares + an `ID=` not matching the COSI's `osRelease`, since systemd will refuse + to merge it. Warn on `ID` alone — systemd accepts a `VERSION_ID` mismatch when + `SYSEXT_LEVEL` matches, so warning on version would flag valid extensions. + +Trident should not warn or refuse on `ID=_any`. It describes what the extension +is compatible with, not how it should be delivered, and bundling a portable +extension is how an air-gapped deployment is achieved. ### SELinux Unchanged. Extensions remain incompatible with SELinux in enforcing mode on -systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`, -and carrying the image in a COSI does not affect labelling. +systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. -What changes is where the check looks and when it runs: the validation raising -`ExtensionImagesAndSelinuxUnsupported` inspects the Host Configuration lists only -and runs inside `configure`, which returns early for anything but Clean Install -and A/B Update. It must read the effective set and run on every servicing type. +What changes is where the check looks and when it runs: it inspects the Host +Configuration lists only, and runs inside `configure`, which returns early for +anything but Clean Install and A/B Update. ## Public API Design @@ -665,75 +540,61 @@ The COSI metadata format is a public contract with its own [specification](../../Reference/Composable-OS-Image.md) and published schemas, so the additions above are the public API change. They are additive and optional. -The Host Configuration API does not change. The observable difference is that a -host may end up with more extensions than its Host Configuration lists, and that -specifying an extension the image already carries is now an error. +The Host Configuration API does not change. A host may now end up with more +extensions than it lists, and specifying one the image already carries is an +error. ## Testing and Metrics - **Schema.** Add `cosi-metadata-v1.3.schema.json` and samples under - `tests/cosi/metadata_samples/v1.3/{valid,invalid}/`; the existing workflow - picks up a new revision automatically. Invalid samples cover what the schema - can express: `image.path` outside `images/`, a non-absolute destination, and a - destination not ending in `.raw`. The permitted-directory allow-list is Trident - policy rather than a property of the format — systemd searches directories - Trident does not accept — so it stays out of the schema, and the - kind-specific directory and duplicate-destination rules are covered by Rust - tests. + `tests/cosi/metadata_samples/v1.3/`; the existing workflow picks up a new + revision automatically. Invalid samples cover what the schema can express: + `image.path` outside `images/`, a non-absolute destination, one not ending in + `.raw`. The permitted-directory allow-list is Trident policy, not a property + of the format, so it stays out of the schema and is tested in Rust. - **Unit.** Each new error variant; effective set computation including collisions and the source discriminator; default-path resolution; replacement of an extension keeping its ID and destination but changing content. -- **Functional.** A COSI carrying a sysext and a confext: Clean Install places +- **Functional.** Clean Install of a COSI carrying a sysext and a confext places both, enables the merge units, and both merge after boot. -- **Servicing.** A/B update from extension set A to B, asserting the new slot has - B, the old retains A, and rollback restores A. An extension-only Runtime Update - between a COSI and a copy with a different extension set, asserting partitions - are untouched. Finalize and rollback of that update as distinct processes, - asserting the previous image is restored without access to either COSI. A - pending inventory surviving an A/B reboot, promoted at commit and discarded - when the boot check fails. -- **Negative.** Streaming a COSI carrying extensions is refused; the same - extension in both sources is a structured error; a bundled extension with - SELinux `enforcing` is rejected on A/B update and on extension-only Runtime - Update; a bundled destination on a shared volume is rejected when A/B is - configured; a Runtime Update between differing image summaries is refused - rather than promoted or partially applied. +- **Servicing.** A/B update from extension set A to B, with rollback restoring + A. An extension-only Runtime Update, asserting partitions are untouched. Its + finalize and rollback as distinct processes, asserting the previous image is + restored without access to either COSI. A pending inventory surviving an A/B + reboot and discarded when the boot check fails. +- **Negative.** Streaming a COSI carrying extensions; the same extension in both + sources; a bundled extension with SELinux `enforcing`, on both A/B and + extension-only Runtime Update; a bundled destination on a shared volume; a + Runtime Update between differing image summaries. ## Servicing -Additive at every layer. - -- A 1.2 COSI, and a 1.3 COSI without `extensions`, behave identically. -- A 1.3 COSI with extensions read by an older Trident deploys the OS and omits - the extensions. See [Versioning](#versioning). -- Hosts using `os.sysexts` or `os.confexts` are unaffected unless they move to an - image carrying the same extension, which is now an explicit error. -- Existing hosts have no recorded image summary, so servicing type selection is - unchanged until their next deployment. +Additive at every layer. A 1.2 COSI, and a 1.3 COSI without `extensions`, behave +identically. A 1.3 COSI with extensions read by an older Trident deploys the OS +and omits them; see [Versioning](#versioning). Existing hosts have no recorded +image summary, so servicing type selection is unchanged until their next +deployment. ## Implementation Plan -0. Prerequisites from [Dependencies](#dependencies). The in-place replacement fix - is a standalone bug and should land independently. -1. COSI revision 1.3: the `extensions` section, the schema and samples. -2. Trident reader: parse `extensions`, metadata validation and the new error - variants, and the minor-version warning. -3. Effective extension set with its source discriminator and the deployed - extension inventory; plumbed into the extensions, `osconfig` and `selinux` - subsystems, including the collision errors and the move of the SELinux check. -4. Stream a tar member into the staging directory, replacing the URL fetch for - bundled entries. Dynamic destination validation. +0. Prerequisites from [Dependencies](#dependencies). The in-place replacement + fix is a standalone bug and should land independently. +1. COSI revision 1.3: the `extensions` section, schema and samples. +2. Trident reader: parse and validate `extensions`, plus the minor-version + warning. +3. Effective extension set and deployed extension inventory, plumbed into the + extensions, `osconfig` and `selinux` subsystems. +4. Stream a tar member into the staging directory; dynamic destination + validation. 5. Deployed image summary and the extension-only Runtime Update path. 6. Tests, then documentation updates to [Sysexts](../../Explanation/Sysexts.md), [Confexts](../../Explanation/Confexts.md) and [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). -Steps 1 and 2 are independently useful: a reader that parses and validates the -section but ignores it is a safe intermediate state. Steps 3 and 4 deliver -bundled extensions on Clean Install and A/B Update and are shippable without -step 5, but step 5 is required for this RFC to be complete — without it every -extension change is an A/B update. +Steps 1 and 2 are independently useful, and steps 3 and 4 are shippable without +step 5. Step 5 is still required for this RFC to be complete, since without it +every extension change is an A/B update. ## Counter-Arguments @@ -741,88 +602,67 @@ extension change is an A/B update. - **Cadence coupling.** A new version of the extension requires a new COSI. [Extension-Only Updates](#extension-only-updates) remove the reboot and reduce - the build to a copy, but a new COSI must still be published. For content that - must match the OS, such as kernel modules or GPU drivers, this costs nothing. - For content versioned independently it is a real loss of agility, and the Host - Configuration route stays correct. -- **Size.** COSI files grow, and every host pays for every bundled extension, - including those it does not use. + the build to a copy, but it must still be published. Free for content that + must match the OS; a real loss of agility for content versioned independently, + where the Host Configuration route stays correct. +- **Size.** Every host pays for every bundled extension, including unused ones. - **No opt-out.** A host wanting the image but not one of its extensions cannot - say so. See [Open Questions](#open-questions). -- **Two sources.** Merging and its collision rules are more complexity in the - extensions subsystem than exists today. -- **Forward compatibility.** An older reader produces a host without the - extensions and reports nothing. + say so. +- **Two sources.** Merging and its collision rules add complexity. +- **Forward compatibility.** An older reader omits the extensions silently. ### Alternatives -**Host Configuration only (status quo).** Maximum decoupling: the extension is -versioned, hosted and updated independently, and a change is a Runtime Update -with no reboot. Remains fully supported, and is wrong only when the extension -must land in lockstep with a new OS. +**Host Configuration only (status quo).** Maximum decoupling, and a change is a +Runtime Update with no reboot. Remains supported, and is wrong only when the +extension must land in lockstep with a new OS. -**Place the `.raw` files in the root filesystem image.** No specification change. -See -[Why Not Place the Files in the Root Filesystem Image](#why-not-place-the-files-in-the-root-filesystem-image). +**Place the `.raw` files in the root filesystem image.** See +[above](#why-not-place-the-files-in-the-root-filesystem-image). -**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** Rejected in +**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** See [Tar Layout](#tar-layout). -**One array with a `kind` discriminator.** Rejected in -[Two Arrays Rather Than One With a `kind` Field](#two-arrays-rather-than-one-with-a-kind-field). - -#### Prior Art +**One array with a `kind` discriminator.** See +[above](#two-arrays-rather-than-one-with-a-kind-field). -Flatcar's sysext-bakery pattern pairs pre-built DDIs with -[`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html): -artefacts hosted over HTTP with a `SHA256SUMS` manifest, staged against a -transfer definition, activated by a refresh or reboot. It suits content with its -own cadence and is close in spirit to `os.sysexts`, except that -`systemd-sysupdate` has no OCI transport whereas `Extension.url` accepts -`oci://`. Both optimise for extensions that move independently of the OS; -bundling optimises for extensions that must move with it, at the cost of cadence. -`ID=_any` against `ID=` indicates which a given extension belongs in. +**Prior art.** Flatcar's sysext-bakery pairs pre-built DDIs with +[`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html), +hosting artefacts over HTTP with a `SHA256SUMS` manifest. Like `os.sysexts`, it +optimises for extensions that move independently of the OS, though without an +OCI transport. Bundling optimises for extensions that must move with it. ## Open Questions - **What does the deployed image summary cover?** Excluding too much permits a - Runtime Update between images differing in ways Trident does not model; - including too much rejects legitimate cases. `version` is currently excluded, - on the basis that a revision bump with identical images deploys identical - bytes. -- **A structured record or a digest?** A record is proposed, so Trident can - report which partition differs. A digest is smaller and simpler. + Runtime Update between images differing in ways Trident does not model. + `version` is currently excluded. +- **A structured record or a digest?** A record costs more but names the + partition that differs. - **Does the no-op outcome belong here?** It needs a metadata-only status - transition recording an accepted image without a rollbackable operation — a - behavioural change independent of extensions. + transition, which is a change independent of extensions. - **Where does the retained superseded image live, and for how long?** The - staging directory covers only the operation itself; a content-addressed store - bounded by the rollback chain also covers manual runtime rollback, at the cost - of disk. The alternative is refusing that rollback. -- **Should an override escape hatch exist?** An opt-in on the Host Configuration - side would permit pinning a hotfixed extension without rebuilding the image, - and equally permit running a combination the image author did not validate. - Recommendation: ship strict, add the hatch if a need appears. -- **Should a bundled extension be suppressible?** A deny list has no analogue in - the current API and no concrete requester. -- **Should `path` be required?** Optional with defaults is proposed for symmetry - with the Host Configuration. -- **Naming.** `extensions` containing `sysexts` and `confexts`, against two - top-level arrays. The nested form keeps the root object small; the flat form - is a closer match to `os.sysexts`. -- **Should Host Configuration extension ID uniqueness be enforced independently - of this work?** Documented but unenforced; arguably a pre-existing gap. + staging directory covers only the operation; a content-addressed store bounded + by the rollback chain also covers manual runtime rollback, at the cost of + disk. +- **Should an override escape hatch exist?** It would permit pinning a hotfixed + extension, and equally permit a combination the image author did not validate. + Recommendation: ship strict. +- **Should a bundled extension be suppressible?** No analogue in the current API + and no concrete requester. +- **Should `path` be required?** Optional is proposed for symmetry with + `Extension.path`. +- **Naming.** `extensions` containing both arrays, against two top-level arrays. +- **Should Host Configuration extension ID uniqueness be enforced separately?** + It is a pre-existing gap this RFC depends on. - **Must writers place payloads after all region images, or is SHOULD enough?** ## Future Possibilities -- **Portable service images.** DDIs with the same shape and placement problem. -- **Per-extension selection at deploy time**, keyed on extension ID, if one - image must serve hosts with different extension sets. -- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed by - `ExtensionRelease` but not acted on. A bundled payload suits initrd scope, - being available before the root filesystem. -- **Bundled extensions under disk streaming**, which needs a way to enable the - merge units on a streamed install. -- **Reporting the extension inventory** through the CLI and gRPC API, making the - running extension set answerable without inspecting the filesystem. +- **Portable service images**, DDIs with the same shape and placement problem. +- **Per-extension selection at deploy time**, keyed on extension ID. +- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed but not acted + on, and a bundled payload is available before the root filesystem. +- **Bundled extensions under disk streaming**, needing a way to enable the merge + units on a streamed install. +- **Reporting the extension inventory** through the CLI and gRPC API. From 3985a8f32c0229e7fe97bf40142fa15c425730f9 Mon Sep 17 00:00:00 2001 From: Paco Date: Tue, 13 Oct 2026 16:44:08 -0700 Subject: [PATCH 14/15] docs: clarify managed placement in COSI extensions RFC Keep path optional for Trident-managed placement, describe current defaults and the unresolved slotted-policy selection from RFC 0838. Remove repeated rationale while retaining schema, update, rollback and validation rules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 746 +++++------------- 1 file changed, 202 insertions(+), 544 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 72e292ae43..31f7b5f422 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -1,632 +1,313 @@ # 0789 COSI Extension Images -- Date: 2026-10-08 +- Date: 2026-09-08 - RFC PR: [microsoft/trident#789](https://github.com/microsoft/trident/pull/789) -- Issue: [microsoft/trident#0000](https://github.com/microsoft/trident/issues/0000) ## Summary -COSI files cannot carry sysext or confext images. This RFC stores them as -ZSTD-compressed members of the COSI tar and adds an optional `extensions` -section to the metadata. Each entry reuses the existing -[`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) object and -an optional destination path, mirroring the -[`Extension`](../../Reference/Host-Configuration/API-Reference/Extension.md) -object in the Host Configuration. +Carry sysext and confext images as compressed members of a COSI tar. An optional +`extensions` section lists each [`ImageFile`](../../Reference/Composable-OS-Image.md#imagefile-object) +and, when placement is prescribed, its destination `path`. Trident can deploy +extensions with the OS image instead of fetching them separately. ## Motivation and Goals -Extensions are configured through `os.sysexts` and `os.confexts`, each entry a -URL and a SHA-384. That makes an extension a second artefact: separately hosted, -fetched from a second endpoint at deploy time, with integrity anchored outside -`os.image.sha384`, and updated by a Runtime Update rather than the A/B update -that ships the OS. Content that must land with a new OS needs two operations. - -Bundled, an extension is delivered and activated in the same A/B update and -reboot as the OS, reusing the existing rollout, health-gating and rollback -machinery. See [Rollback](#rollback). - -### Why Not Place the Files in the Root Filesystem Image - -The builder could place `my-tool.raw` in `/var/lib/extensions/` before the COSI -is built. That works in some configurations, but: - -1. Under root or usr verity, modifying the protected filesystem changes the root - hash and requires re-signing. -2. Extension destinations are commonly on a separate volume, which may be - created empty rather than written from a partition image. -3. Filesystem image contents are invisible to tooling reading COSI metadata. -4. Trident enables the merge units, validates placement and reads - `extension-release` only for extensions it knows about. - -### Goals - -- Deploy extensions from a COSI with no second artefact, endpoint or update. -- Reuse existing metadata objects and the shape of the Host Configuration API. -- Leave the Host Configuration API unchanged. +Host Configuration extensions need separate hosting, downloads and updates. +Bundling gives OS-coupled extensions the COSI's integrity chain and the same +A/B rollout, reboot and rollback as the OS. Host Configuration extensions remain +available for content released independently. ## Scope ### Requirements -- An optional `extensions` object in the metadata root, holding `sysexts` and - `confexts` arrays. -- Each entry identifies its payload with an `ImageFile` and may specify a - destination. -- Payloads are ZSTD-compressed tar members under the existing compression and - integrity rules. -- Trident deploys bundled extensions on Clean Install and A/B Update. -- A change confined to the extension set is applicable as a Runtime Update. See - [Extension-Only Updates](#extension-only-updates). -- Bundled and Host Configuration extensions coexist; conflicts are an error. +- COSI revision 1.3 adds optional `extensions.sysexts` and + `extensions.confexts` arrays, each entry `{ image, path? }`. +- Clean Install and A/B Update deploy the effective set from COSI and Host + Configuration; collisions fail. +- An extension-only change can use Runtime Update without rewriting partitions + or rebooting. ### Out of Scope -- Changes to the Host Configuration `Extension` object. -- Extension-only COSI files, carrying extensions and no partition images. -- Bundled extensions under [disk streaming](../../Explanation/Disk-Streaming.md). - Streaming such a COSI is refused. -- Signing beyond the SHA-384 chain COSI already provides. -- SELinux compatibility, unchanged. See [SELinux](#selinux). -- Producing the DDIs. +Changing the Host Configuration API, DDIs, SELinux support or signing; +partition-less COSI files; and bundled extensions under +[disk streaming](../../Explanation/Disk-Streaming.md), which must be refused +until streaming can enable the merge units. ### Exit Criteria -- COSI revision 1.3 published, with schema and samples. -- A COSI carrying a sysext and a confext deploys on Clean Install and A/B - Update, both merged after reboot; streaming it is refused. -- A/B rollback restores the previous extension set with no extra servicing. -- An extension-only change applies as a Runtime Update, and is refused when - anything else differs. -- Conflicts between bundled and Host Configuration extensions are a structured - error. +Publish the 1.3 schema and samples. Deploy sysexts and confexts on Clean Install +and A/B Update with slot-correct rollback; apply a verified extension-only +change by Runtime Update. Reject collisions and unsupported disk streaming. ## Dependencies -A COSI writer that emits the section. -[Image Customizer](https://github.com/microsoft/azure-linux-image-tools) is the -reference writer. - -[Extension-Only Updates](#extension-only-updates) also require: - -- A writer mode copying an existing COSI's region images verbatim while - replacing its extension set. Without it, every extension change is an A/B - update. -- A surface for requesting a servicing type. None exists; `forceAbUpdate` is the - nearest precedent. It must define precedence against `forceAbUpdate` and - whether the request survives separate stage and finalize invocations. - -Three existing defects must be fixed first. All are reachable today; bundling -makes the first two routine. - -1. **In-place replacement deletes the new image.** When an extension keeps its - ID and destination but changes content, `set_up_extensions` schedules the ID - for both addition and removal. The addition renames over the destination; the - removal then deletes that path, because the old entry's `temp_path` is its - destination. The guard assumes the paths differ. -2. **Staged payloads do not survive the operation.** Runtime finalize and - rollback build their `EngineContext` with `image: None`, so the COSI is gone - after stage. -3. **Placement is validated against the Host Configuration only, before the COSI - is read.** See [Destination Validation](#destination-validation). +[Image Customizer](https://github.com/microsoft/azure-linux-image-tools), or +another writer, must emit the section. Extension-only updates also need a +COSI-to-COSI writer that copies region images verbatim and an explicit +Runtime Update request (including its precedence over `forceAbUpdate`). + +Fix the existing `set_up_extensions` replacement bug first: with the same ID +and destination but different content, it installs the new file then deletes +that path as the old file. Runtime stage/finalize also needs durable payloads; +finalize and rollback construct their context with `image: None`. ## Implementation ### Tar Layout -Payloads are ZSTD-compressed DDIs under `images/extensions/`. - -They stay under `images/`, which the specification already permits and requires -readers to handle. Relaxing `ImageFile.path`'s `^images/.+` pattern would -produce a 1.3 `ImageFile` that fails the 1.0–1.2 schemas, for a shorter path. - -The primary GPT image must immediately follow `metadata.json` since revision -1.2, so payloads must not sit between them. Payloads are not regions and do not -participate in region ordering, but should be written after all region images to -keep that ordering verifiable and preserve sparse-read locality. Writers must -account for them in `compression.maxWindowLog`. - -Older readers are unaffected: unknown files must be ignored, and Trident's -orphan-image check (`V1_2ImageFileHasNoCorrespondingPartition`) reads the -`images[]` and `disk.gptRegions[]` arrays rather than walking tar entries. +Payloads are ZSTD-compressed DDIs under `images/extensions/`; this preserves +`ImageFile.path`'s `^images/.+` constraint. The primary GPT image must +immediately follow `metadata.json`. Write extension members after the +disk-region images and include their window sizes in `compression.maxWindowLog`. +Older readers ignore the additional tar members. ### Metadata Schema -| Field | Type | Added in | Required | Description | -| ------------ | -------------------------------- | -------- | -------- | ------------------------------- | -| `extensions` | [Extensions](#extensions-object) | 1.3 | No | Extension images in this COSI. | - -#### `Extensions` Object - -| Field | Type | Added in | Required | Description | -| ---------- | ------------------------------------------ | -------- | -------- | ------------------------------- | -| `sysexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | System extension images. | -| `confexts` | [ExtensionImage](#extensionimage-object)[] | 1.3 | No | Configuration extension images. | - -#### `ExtensionImage` Object - -| Field | Type | Added in | Required | Description | -| ------- | -------------------------------------------------------------------- | -------- | --------------- | ----------------------------------------------- | -| `image` | [ImageFile](../../Reference/Composable-OS-Image.md#imagefile-object) | 1.3 | Yes (since 1.3) | The compressed extension image in the tar file. | -| `path` | string | 1.3 | No | Absolute destination path on the target OS. | +The two arrays mirror Host Configuration and determine the kind; each entry +contains only an `image` and optional `path`. This is the proposed addition to +the [1.2 schema](../../Reference/Composable-OS-Image/cosi-metadata-v1.2.schema.json): ```json { "properties": { - "extensions": { - "description": "Extension images carried by this COSI file.", - "$ref": "#/$defs/Extensions" - } + "extensions": { "$ref": "#/$defs/Extensions" } }, "$defs": { "Extensions": { "type": "object", "properties": { - "sysexts": { - "description": "System extension images to place on the target OS.", - "type": "array", - "items": { "$ref": "#/$defs/ExtensionImage" } - }, - "confexts": { - "description": "Configuration extension images to place on the target OS.", - "type": "array", - "items": { "$ref": "#/$defs/ExtensionImage" } - } + "sysexts": { "type": "array", "items": { "$ref": "#/$defs/ExtensionImage" } }, + "confexts": { "type": "array", "items": { "$ref": "#/$defs/ExtensionImage" } } } }, "ExtensionImage": { "type": "object", "required": ["image"], "properties": { - "image": { - "description": "Details of the compressed extension image file in the tar file.", - "$ref": "#/$defs/ImageFile" - }, - "path": { - "description": "Absolute path of the extension image on the target OS. The file name MUST be `{name}.raw`, where `{name}` matches the suffix of the `extension-release.{name}` file inside the image. When omitted, the reader places the image in its default directory for the extension kind.", - "type": "string", - "pattern": "^/.+\\.raw$" - } + "image": { "$ref": "#/$defs/ImageFile" }, + "path": { "type": "string", "pattern": "^/.+\\.raw$" } } } } } ``` -Example: +Example (`extensions` fragment of `metadata.json`): ```json -"extensions": { - "sysexts": [ - { - "image": { - "path": "images/extensions/gpu-driver.rawzst", - "compressedSize": 41943040, - "uncompressedSize": 134217728, - "sha384": "3a1f9c0d4e5b6a7c8d9e0f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e" - }, - "path": "/var/lib/extensions/gpu-driver.raw" - } - ], - "confexts": [ - { - "image": { - "path": "images/extensions/fleet-config.rawzst", - "compressedSize": 262144, - "uncompressedSize": 1048576, - "sha384": "c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2" - } +{ + "sysexts": [{ + "image": { + "path": "images/extensions/gpu-driver.rawzst", + "compressedSize": 41943040, + "uncompressedSize": 134217728, + "sha384": "3a1f9c0d4e5b6a7c8d9e0f1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e" } - ] + }], + "confexts": [{ + "image": { + "path": "images/extensions/fleet-config.rawzst", + "compressedSize": 262144, + "uncompressedSize": 1048576, + "sha384": "c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2" + }, + "path": "/var/lib/confexts/fleet-config.raw" + }] } ``` -#### Two Arrays Rather Than One With a `kind` Field - -Sysexts and confexts have disjoint permitted directories, different defaults, -different `extension-release` locations, different identity fields and different -activation units. Every downstream rule branches on the kind. Two arrays make it -structural, and match the Host Configuration. - -#### Entry Contents - -`image` and an optional `path`. Everything else is derivable from the payload, -and each derivable field is a second source of truth. - -- **`path`** is kept: the destination is not derivable. Optional, defaulting as - `Extension.path` does. Existing rules apply — absolute, ending in `.raw`, in a - permitted directory for its kind, file name matching the `extension-release` - suffix. -- **Name** is rejected: derived from `extension-release.{name}`, which the - destination file name must already match. -- **`kind`** is rejected: structural. -- **Extension ID** is rejected: read from `extension-release`, and already the - identity Trident keys A/B state on. -- **Enable-on-first-boot** is rejected: systemd merges everything in the - extension directories, and the Host Configuration has no equivalent switch. -- **Version and compatibility metadata** is rejected: already in the DDI and - enforced by systemd. Copying it would let a COSI claim compatibility the - payload lacks. - -#### `sha384` Semantics - -`ImageFile.sha384` covers the compressed image; `Extension.sha384` covers the -raw file. For integrity this matches partition images, since Trident hashes the -compressed stream as it decompresses. - -`ExtensionData.sha384` serves both as change detector and as the key matching a -processed extension to its Host Configuration entry, so entries must record the -hash with its source and compare only within a source. Across sources the -comparison is always unequal and would report every extension as changed. - -Within a source it is sound but imprecise: compressed hashes are unstable across -recompressions, so identical DDIs at different ZSTD levels appear to differ. The -result is a redundant copy, never a missed update. +`ImageFile` supplies the tar path, compressed size for ranged reads, +uncompressed size for output bounds, and SHA-384 of the compressed member. +Trident checks both sizes and the hash against the bytes read and written. +It does not reuse Host Configuration's `Extension.sha384`, which hashes the +uncompressed DDI; a source switch must replace the image, not compare those +two hashes. Recompression may cause an unnecessary replacement, never a missed +one. + +`path` is deliberately optional, **not** a missing piece of metadata. Omission +leaves placement to Trident and may be the usual case for fully +Trident-managed extensions. Today Trident derives `{name}.raw` from +`extension-release.{name}` and uses `/var/lib/extensions/` or +`/var/lib/confexts/`. An explicit `path` must use a permitted directory and +match that name. [RFC 0838](https://github.com/microsoft/trident/pull/838) +proposes fully Trident-managed, slotted sysext storage without `path`; it +handles `/etc` layers separately. How bundled extensions select such a policy +is unresolved; omitting `path` alone must not opt in. + +There is no `url` (the tar is the source), `kind` (the array supplies it), or +separate name, ID, enable flag or compatibility fields (the DDI supplies them, +or systemd merges all discovered images). ### Versioning -COSI revision 1.3. `extensions` is optional; absent equals empty. A newer -Trident reading a 1.0–1.2 COSI behaves as today. - -An older Trident reading a 1.3 COSI accepts it, since the version check rejects -only `major != 1` and unknown fields must be ignored. It deploys the OS and -omits the extensions silently. This cannot be corrected in-band: any tripwire -field is one such a reader must ignore. Nor does it self-heal — the old reader -records the new COSI as deployed, so after upgrading Trident the same Host -Configuration is no longer an image change. Recovery needs a forced -redeployment. - -Mitigations: document the requirement; warn in -`validate_cosi_metadata_version` when `minor` exceeds the highest known -revision; and, where the target's Trident version cannot be controlled, gate the -update on a -[health check](../../Reference/Host-Configuration/API-Reference/Health.md) -asserting the extension is merged. Given that the omission does not self-heal, -the health check is the only real safeguard. +COSI revision 1.3 adds the optional section; older COSIs behave unchanged. +Existing Trident accepts any `1.x` metadata and ignores unknown fields, so it +silently skips bundled extensions. That omission does not self-heal after a +Trident upgrade because the image is already recorded as deployed. Require a +1.3-aware reader, warn on unknown minor versions in new readers, and use a +[health check](../../Reference/Host-Configuration/API-Reference/Health.md) for +required extensions when reader versions cannot be controlled. ### Trident-Side Consumption -Trident computes an **effective extension set**, the union of the Host -Configuration and COSI entries, and the code reading `ctx.spec.os.sysexts` and -`confexts` reads that instead. Each entry carries its source. Three consumers: - -- **`ExtensionsSubsystem`.** For a bundled entry only the source changes: the - image streaming pipeline decompresses the tar member into the staging - directory and verifies `ImageFile.sha384` over the compressed stream, in place - of the `FileReader` download. `update_host_configuration` fails when a - processed extension has no Host Configuration entry matching by hash, so it - must write back resolved paths for Host-Configuration-sourced entries only. -- **`osconfig`.** The merge units are enabled only when `ctx.spec.os.sysexts` or - `confexts` are non-empty, so a COSI-only host would never have them enabled. -- **`selinux`.** The validation raising `ExtensionImagesAndSelinuxUnsupported` - must also read the effective set, but that alone is insufficient: it runs - inside `configure`, which returns early for Runtime Updates, and - `SelinuxSubsystem` inherits `runs_on = REQUIRES_REBOOT`. It must move to a - hook running on every servicing type, or - [Extension-Only Updates](#extension-only-updates) bypass it. - -[Disk streaming](../../Explanation/Disk-Streaming.md) is excluded: `osconfig` -returns early when `is_stream_image` is set, so the units are never enabled and -the extensions would be placed and never merged. Trident refuses to stream a -COSI carrying extensions rather than produce that state. +Compute an effective set from COSI and Host Configuration entries, retaining +each entry's source. The extensions subsystem reads bundled members instead of +downloading them; `update_host_configuration` writes resolved paths back only +for Host Configuration entries. `osconfig` enables the sysext/confext merge +units for the effective set. Move the SELinux check into validation that also +runs for Runtime Updates; its current `configure` hook does not. + +[Disk streaming](../../Explanation/Disk-Streaming.md) must reject bundled +extensions: `osconfig` skips stream-image configuration, leaving the merge +units disabled. #### Deployed Extension Inventory -The Host Configuration records what an operator asked for, so `ctx.spec_old` -does not describe what is deployed. Without a second record Trident cannot tell -which bundled extensions to remove, detect collisions against the deployed set, -compare sets for servicing selection, or rebuild its state when finalize runs -separately from stage. - -Trident records a deployed extension inventory in the Host Status, alongside the -[deployed image summary](#deployed-image-summary): kind, extension ID, name, -resolved destination, source, and the hash under that source's semantics. -`extensions_old` is populated from it. - -Inventory and summary follow the discipline applied to `spec` and `spec_old`: a -staged operation records a pending record while the deployed one is retained. -Promotion follows the servicing type, not the finalize step. Runtime Update and -`ManualRollbackRuntime` promote on completion. Clean Install, A/B Update and -manual A/B rollback reach `*Finalized` before rebooting and commit after booting -the expected root, so their pending record must survive the reboot and be -discarded when the boot check fails. - -`ManualRollbackChainItem` carries only kind, `spec`, active volume and install -index, so it must carry the historical inventory and summary too. Even then a -manual runtime rollback has no payload: its context is built with `image: None` -and the superseded image is retained only for the operation that replaced it. -Either payloads are kept in a content-addressed store for as long as the -rollback chain references them, or that rollback is refused. Refusing is the -safe default. +Host Configuration does not record bundled extensions. Persist an inventory +(kind, ID, name, resolved placement, source and source hash) in Host Status so +Trident can compare old/new sets and resume separate stage/finalize calls. +Retain deployed and pending inventories and image summaries alongside +`spec_old` and `spec`. Promote after successful Runtime Update, or after +post-boot commit for A/B; discard pending data on rollback. Extend +`ManualRollbackChainItem` to carry historical inventory and summary. -#### Destination Validation +Manual runtime rollback also needs the old DDI bytes. Either retain them while +the rollback chain references them or refuse rollback across a bundled +extension change; metadata alone cannot restore them. -`validate_extension_images_locations` rejects destinations not on an A/B volume -when A/B is configured, but it is static validation over the Host Configuration -lists and runs before the COSI is read. +#### Destination Validation -The equivalent check for the effective set runs after the COSI loads, against -the storage graph. It must resolve the filesystem that actually backs the -destination for the servicing type: under root-verity the engine mounts a -writable `/etc` overlay during provision, backed by `/var/lib/trident-overlay` -on an A/B volume, so `/etc/extensions/` is writable and rolls back with the -slot. Rejecting verity-backed destinations outright would reject that case. The -same destination has no such backing on a Runtime Update. +Existing A/B location validation reads only Host Configuration entries before +loading COSI. Validate the *resolved* effective set after reading the image: +every destination must be writable and, when A/B is configured, slot-scoped. +A/B volumes qualify; the root-verity `/etc` overlay qualifies during +provisioning if its backing volume is A/B. Slotted storage from +[RFC 0838](https://github.com/microsoft/trident/pull/838) qualifies only with +its loader capability; `path` omission alone must not bypass the A/B check. #### Where Images Are Written -On Clean Install and A/B Update, `provision()` runs with the target root at -`mount_path` and extensions are placed inside it — for A/B, the inactive slot. -Staging currently uses a fixed `/var/lib/extensions/.staging` with a non-atomic -copy when the rename crosses a filesystem boundary; it should use a temporary -file on the destination's own filesystem. - -On Runtime Update `provision()` is not called, so partitions are never touched. +Clean Install and A/B Update place files in the target root (the inactive slot +for A/B). Runtime Update places them in the running root without provisioning +partitions. Stage temporary files on the destination filesystem for atomic +placement; the current fixed `/var/lib/extensions/.staging` falls back to a +non-atomic cross-filesystem copy. #### Rollback -Extension images are files in the slot's own filesystem, so where every -destination is on an A/B volume the previous slot retains the previous set and -an A/B rollback reverts it with the OS. - -That depends on the placement check covering bundled destinations; see -[Destination Validation](#destination-validation). A bundled extension on a -shared volume changes the running slot immediately and cannot be rolled back. +An A/B rollback restores the previous extension set only when placement is +slot-scoped. Writing a bundled image to a shared volume would change the running +slot and defeat rollback; see [Destination Validation](#destination-validation). #### Extension-Only Updates -`ab_update_required()` returns true whenever `os.image.sha384` differs, before -any subsystem is consulted, and the metadata hash covers the `extensions` -section. Changing only a bundled extension would force a full A/B update, -making bundling strictly more disruptive than the status quo. - -An extension-only COSI is the wrong shape for this. The metadata root requires -`images`, `disk` with at least one `gptRegions` entry, `bootloader` and -`osPackages`, and every consumer of `os.image` assumes a complete OS. The -distinction is drawn on content equality, not absent content. +Today any change to the COSI metadata hash forces A/B servicing, including an +extension-only change. A partition-less COSI is not an option: `os.image` +describes a complete OS. Compare the new COSI against a stored summary of the +deployed OS instead. ##### Deployed Image Summary -Trident retains only the URL and metadata hash of the applied image, and -re-fetching the previous COSI to diff it is not dependable, so it must record -what it deployed. - -The summary holds, per entry in `images[]` and `disk.gptRegions[]`, the -`image.sha384`, `uncompressedSize` and identity (partition number, mount point, -`fsType`, `fsUuid`, `partType`, verity root hash); plus `osArch`, `osRelease`, -`disk` geometry and `bootloader`. It excludes `extensions`, the subject of the -comparison; `osPackages`, which Trident validates but never acts on; -`compression.maxWindowLog`, which a new extension may legitimately raise; and -`id`, which identifies the file rather than its content. +Store each region's hash, uncompressed size and identity (partition number, +mount point, `fsType`, `fsUuid`, `partType`, verity root hash), plus disk geometry, +`osArch`, `osRelease` and `bootloader`. Exclude extensions, packages, compression +settings and `id`: none changes the deployed OS. A structured summary identifies +which region differs. If missing or of an unknown summary version, fall back to +the existing metadata-hash comparison. -A record rather than a digest, so Trident can report which partition differs. - -The summary is versioned. Where absent or unrecognised, Trident falls back to -comparing `os.image.sha384`, so existing hosts are unaffected until their next -deployment. - -| Summary | Extension set | Image requires | -| ------- | ------------- | ----------------- | -| Equal | Equal | Nothing. | -| Equal | Differs | A Runtime Update. | -| Differs | Any | An A/B update. | - -This is the image's requirement, not the outcome; `select_servicing_type` still -takes the maximum across subsystems. The no-op row is new, since -`os.image.sha384` changes whenever any part of the metadata does. +Equal summaries and equal extension sets require no servicing; equal summaries +with changed extensions permit Runtime Update; a changed summary requires A/B. +Other subsystems can still require A/B. A no-op needs a metadata-only Host Status +update, since `update` currently returns without recording the accepted image. ##### Constructing an Extension-Only Update -Summary equality requires byte-identical region images, and filesystem images -are not reproducible: UUIDs, inode timestamps, superblock times and allocation -order vary between builds, and ZSTD output varies with level and library -version. - -So the build operation is not "rebuild the image identically with a different -extension set", it is "copy an existing COSI, replacing its extension set". -Region images and their metadata entries are copied verbatim; only the extension -members, the `extensions` section, `compression.maxWindowLog` and `id` change. -Equality then holds by construction, and the operation is a tar rewrite. - -A COSI rebuilt from source will differ and Trident will select an A/B update. -The failure mode of an unreproducible build is a redundant A/B update, never a -skipped one. +Copy an existing COSI's region images and metadata entries verbatim; replace +only the extension members and metadata (and `compression.maxWindowLog` or `id` +if needed). Rebuilding filesystems changes bytes and hashes, even for the same +files, and safely selects A/B instead of skipping an OS change. ##### Requested and Verified, Never Inferred -An extension-only update proceeds when all of: - -1. A Runtime Update is explicitly requested. -2. The recorded summary equals the one computed from the new COSI. -3. The recorded image is the deployed one: servicing completed, no A/B update - pending, active volume matching the Host Status. -4. No other subsystem requires an A/B update. - -If 2, 3 or 4 fails, Trident fails with a structured error naming what differs. -It must not promote the operation to an A/B update, nor apply the extension -change while leaving other changes unapplied. - -A no-op must still record the accepted image and summary. -`select_servicing_type` returning `NoActiveServicing` makes `update` return -without persisting anything, so a metadata-only status transition is needed. +Runtime Update requires an explicit request, matching OS summaries, a +committed/active source slot, and no A/B requirement from any other subsystem. +If any condition fails, report an error; do not partially apply changes or +silently switch to A/B. ##### Rollback on This Path -A Runtime Update replaces files in the running slot, because -`set_up_extensions` skips removal of the superseded image only on Clean Install -and A/B Update. Rollback re-runs the subsystem with the specs reversed and must -restore the previous image, but cannot re-fetch it: finalize and rollback build -their context with `image: None`, and auto-rollback runs unconditionally after a -finalize failure. The superseded image must be retained for the operation. - -Rollback is therefore bounded by that retained payload rather than by booting an -untouched slot, which is why the path is requested rather than inferred. +Runtime Update changes the running slot. Retain the old DDI until finalize and +auto-rollback complete; both can run with `image: None`. Historical manual +rollback needs longer retention or must be refused. #### Interaction With Host Configuration `sysexts` and `confexts` -The sets are merged, and a collision is an error. - -Both are valid at once: bundled extensions are content the image author -considers part of the OS, Host Configuration entries are content the operator -adds. Silent precedence is rejected, since an override would run software the -image author never validated with no visible symptom. - -A collision is either the same destination path, detectable from metadata when -both sides specify `path`, or the same extension ID within a kind, detectable -only after mounting. ID uniqueness is documented but unenforced today, so it -should become an enforced check. - -Two entries identical in every respect are still a collision; the operator can -drop theirs. A per-extension override, if needed, belongs on the Host -Configuration side as an opt-in. See [Open Questions](#open-questions). +Merge both sources. Reject duplicate extension IDs per kind or destination +collisions after resolving placement, including omitted `path`. Silent +precedence could run an unvalidated combination; identical entries still +collide. The Host Configuration's documented ID-uniqueness rule must also be +enforced. ### Validation -#### COSI Metadata Validation - -New `CosiMetadataErrorKind` variants, following the existing `V1_` -prefix: - -| Variant | Condition | -| ------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| `V1_3ExtensionDestinationPathNotAbsolute` | `path` is present and not absolute. | -| `V1_3ExtensionDestinationPathInvalidFileExtension` | `path` is present and does not end in `.raw`. | -| `V1_3ExtensionDestinationPathInvalidDirectory` | `path`'s parent is not in `VALID_SYSEXT_DIRECTORIES` or `VALID_CONFEXT_DIRECTORIES` for the kind. | -| `V1_3DuplicateExtensionDestinationPath` | Two entries both specify `path` and specify the same one. | -| `V1_3DuplicateExtensionImagePath` | Two entries reference the same tar member. | -| `V1_3ExtensionImagePathCollidesWithRegionImage` | An entry's `image.path` is also referenced by `images[]` or `disk.gptRegions[]`. | - -These reuse the logic behind `Extension::validate_sysext` and -`validate_confext`. - -Destination collisions are only partly detectable here, since an entry omitting -`path` resolves using the `{name}` inside the DDI. Member existence is not -checked at load either: `Cosi::new` scans only as far as `metadata.json` and -discovers the rest on demand, and checking up front would conflict with sparse -reads. - -#### Deploy-Time Validation - -After the DDI is mounted. The first two are already enforced by -`read_extension_release` and apply unchanged. - -- File name matches the `extension-release` suffix. -- Exactly one `extension-release` file, with `SYSEXT_ID` or `CONFEXT_ID`. -- SHA-384 agreement, verified over the compressed stream during decompression. -- Resolved destination collisions across the effective set. -- Extension ID uniqueness across the effective set, per kind. -- Destination placement. See [Destination Validation](#destination-validation). -- `extension-release` OS compatibility: warn when a bundled extension declares - an `ID=` not matching the COSI's `osRelease`, since systemd will refuse - to merge it. Warn on `ID` alone — systemd accepts a `VERSION_ID` mismatch when - `SYSEXT_LEVEL` matches, so warning on version would flag valid extensions. - -Trident should not warn or refuse on `ID=_any`. It describes what the extension -is compatible with, not how it should be delivered, and bundling a portable -extension is how an air-gapped deployment is achieved. - -### SELinux +In `cosi/validation.rs`, add `V1_3` error kinds for invalid explicit paths +(absolute, `.raw`, allowed directory), duplicate explicit destinations, reused +tar members and collisions with disk-region images. Reuse +`Extension::validate_sysext`/`validate_confext` path rules. A missing member is +detected when read: `Cosi::new` only scans as far as `metadata.json`. -Unchanged. Extensions remain incompatible with SELinux in enforcing mode on -systemd 255, because merging the overlays mislabels `/usr`, `/opt` and `/etc`. +After decompressing and mounting the DDI, check actual compressed and +uncompressed sizes, the compressed SHA-384, exactly one `extension-release` +with the kind's ID, the `{name}.raw` destination suffix, resolved placement, +and duplicate destinations/IDs across both sources. Omitted paths cannot be +collision-checked from metadata alone. -What changes is where the check looks and when it runs: it inspects the Host -Configuration lists only, and runs inside `configure`, which returns early for -anything but Clean Install and A/B Update. +Warn on definite OS `ID` mismatch with COSI `osRelease`, not `VERSION_ID`: +systemd may accept a version mismatch via `SYSEXT_LEVEL` or `CONFEXT_LEVEL`. +Do not warn on `ID=_any`; it declares compatibility, not delivery method. -## Public API Design - -The COSI metadata format is a public contract with its own -[specification](../../Reference/Composable-OS-Image.md) and published schemas, so -the additions above are the public API change. They are additive and optional. +### SELinux -The Host Configuration API does not change. A host may now end up with more -extensions than it lists, and specifying one the image already carries is an -error. +Still unsupported in enforcing mode on systemd 255. Check the effective set in +validation for every servicing type, not only the Host Configuration lists in +`configure`, which skips Runtime Updates. ## Testing and Metrics -- **Schema.** Add `cosi-metadata-v1.3.schema.json` and samples under - `tests/cosi/metadata_samples/v1.3/`; the existing workflow picks up a new - revision automatically. Invalid samples cover what the schema can express: - `image.path` outside `images/`, a non-absolute destination, one not ending in - `.raw`. The permitted-directory allow-list is Trident policy, not a property - of the format, so it stays out of the schema and is tested in Rust. -- **Unit.** Each new error variant; effective set computation including - collisions and the source discriminator; default-path resolution; replacement - of an extension keeping its ID and destination but changing content. -- **Functional.** Clean Install of a COSI carrying a sysext and a confext places - both, enables the merge units, and both merge after boot. -- **Servicing.** A/B update from extension set A to B, with rollback restoring - A. An extension-only Runtime Update, asserting partitions are untouched. Its - finalize and rollback as distinct processes, asserting the previous image is - restored without access to either COSI. A pending inventory surviving an A/B - reboot and discarded when the boot check fails. -- **Negative.** Streaming a COSI carrying extensions; the same extension in both - sources; a bundled extension with SELinux `enforcing`, on both A/B and - extension-only Runtime Update; a bundled destination on a shared volume; a - Runtime Update between differing image summaries. - -## Servicing - -Additive at every layer. A 1.2 COSI, and a 1.3 COSI without `extensions`, behave -identically. A 1.3 COSI with extensions read by an older Trident deploys the OS -and omits them; see [Versioning](#versioning). Existing hosts have no recorded -image summary, so servicing type selection is unchanged until their next -deployment. +- v1.3 schema and valid/invalid samples; semantic path rules in Rust tests. +- Clean Install and A/B Update with both kinds, merge-unit activation and + rollback (including absent `path` and slotted placement once supported). +- Runtime Update with identical region images, distinct stage/finalize and + auto-rollback; same-ID/same-path replacement and historical rollback refusal. +- Reject mixed-source collisions, invalid destinations, enforcing SELinux, + changed OS summaries under requested Runtime Update, and disk streaming. ## Implementation Plan -0. Prerequisites from [Dependencies](#dependencies). The in-place replacement - fix is a standalone bug and should land independently. -1. COSI revision 1.3: the `extensions` section, schema and samples. -2. Trident reader: parse and validate `extensions`, plus the minor-version - warning. -3. Effective extension set and deployed extension inventory, plumbed into the - extensions, `osconfig` and `selinux` subsystems. -4. Stream a tar member into the staging directory; dynamic destination - validation. -5. Deployed image summary and the extension-only Runtime Update path. -6. Tests, then documentation updates to - [Sysexts](../../Explanation/Sysexts.md), - [Confexts](../../Explanation/Confexts.md) and - [How Trident Consumes COSI](../../Explanation/How-Trident-Consumes-COSI.md). - -Steps 1 and 2 are independently useful, and steps 3 and 4 are shippable without -step 5. Step 5 is still required for this RFC to be complete, since without it -every extension change is an A/B update. +1. Fix [prerequisites](#dependencies); publish COSI 1.3 schema and samples. +2. Read, validate and install bundled extensions; persist the effective set. +3. Add the image summary, copy-based writer and verified Runtime Update path. + +Step 2 can ship before step 3, but the RFC is not complete without both. ## Counter-Arguments ### Drawbacks -- **Cadence coupling.** A new version of the extension requires a new COSI. - [Extension-Only Updates](#extension-only-updates) remove the reboot and reduce - the build to a copy, but it must still be published. Free for content that - must match the OS; a real loss of agility for content versioned independently, - where the Host Configuration route stays correct. -- **Size.** Every host pays for every bundled extension, including unused ones. -- **No opt-out.** A host wanting the image but not one of its extensions cannot - say so. -- **Two sources.** Merging and its collision rules add complexity. -- **Forward compatibility.** An older reader omits the extensions silently. +Bundling still ties distribution to a new COSI, increases its size and gives +hosts no per-extension opt-out. Use Host Configuration for extensions that must +move independently. Older readers silently omit bundled extensions. ### Alternatives -**Host Configuration only (status quo).** Maximum decoupling, and a change is a -Runtime Update with no reboot. Remains supported, and is wrong only when the -extension must land in lockstep with a new OS. - -**Place the `.raw` files in the root filesystem image.** See -[above](#why-not-place-the-files-in-the-root-filesystem-image). - -**Relax `ImageFile.path` to allow a top-level `extensions/` prefix.** See -[Tar Layout](#tar-layout). - -**One array with a `kind` discriminator.** See -[above](#two-arrays-rather-than-one-with-a-kind-field). +Embedding `.raw` files in a partition image hides them from COSI metadata, +fails when the destination is a separately created filesystem, and changes +verity hashes. A separate Host Configuration URL remains preferable for +independent release cycles. -**Prior art.** Flatcar's sysext-bakery pairs pre-built DDIs with +Flatcar's sysext-bakery pairs pre-built DDIs with [`systemd-sysupdate`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysupdate.html), hosting artefacts over HTTP with a `SHA256SUMS` manifest. Like `os.sysexts`, it optimises for extensions that move independently of the OS, though without an @@ -634,35 +315,12 @@ OCI transport. Bundling optimises for extensions that must move with it. ## Open Questions -- **What does the deployed image summary cover?** Excluding too much permits a - Runtime Update between images differing in ways Trident does not model. - `version` is currently excluded. -- **A structured record or a digest?** A record costs more but names the - partition that differs. -- **Does the no-op outcome belong here?** It needs a metadata-only status - transition, which is a change independent of extensions. -- **Where does the retained superseded image live, and for how long?** The - staging directory covers only the operation; a content-addressed store bounded - by the rollback chain also covers manual runtime rollback, at the cost of - disk. -- **Should an override escape hatch exist?** It would permit pinning a hotfixed - extension, and equally permit a combination the image author did not validate. - Recommendation: ship strict. -- **Should a bundled extension be suppressible?** No analogue in the current API - and no concrete requester. -- **Should `path` be required?** Optional is proposed for symmetry with - `Extension.path`. -- **Naming.** `extensions` containing both arrays, against two top-level arrays. -- **Should Host Configuration extension ID uniqueness be enforced separately?** - It is a pre-existing gap this RFC depends on. -- **Must writers place payloads after all region images, or is SHOULD enough?** - -## Future Possibilities - -- **Portable service images**, DDIs with the same shape and placement problem. -- **Per-extension selection at deploy time**, keyed on extension ID. -- **Initrd-scoped extensions.** `SYSEXT_SCOPE=initrd` is parsed but not acted - on, and a bundled payload is available before the root filesystem. -- **Bundled extensions under disk streaming**, needing a way to enable the merge - units on a streamed install. -- **Reporting the extension inventory** through the CLI and gRPC API. +- **Slotted placement:** how does a COSI entry request the policy in + [RFC 0838](https://github.com/microsoft/trident/pull/838)? Omitting `path` + cannot silently change an existing image's placement or imply loader support. +- **Runtime request:** where is the explicit request recorded across + stage/finalize, and how does it interact with `forceAbUpdate`? +- **Manual runtime rollback:** keep historical DDIs as long as the rollback + chain needs them, or refuse rollback across bundled extension changes? +- **Operator override:** retain the hard error on bundled/Host Configuration + collisions, or permit an explicit override when a hotfix is needed? From bccf272bf1916d1467e7de9eef997e76cc247a63 Mon Sep 17 00:00:00 2001 From: Paco Date: Wed, 14 Oct 2026 00:04:54 -0700 Subject: [PATCH 15/15] docs: require extension identity in COSI metadata Add a required per-entry id matching the DDI's SYSEXT_ID or CONFEXT_ID, with duplicate and mismatch validation. Match extension updates by kind and ID; use the compressed hash to detect replacement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../RFCs/0789-cosi-extension-images.md | 55 +++++++++++++------ 1 file changed, 38 insertions(+), 17 deletions(-) diff --git a/docs/Development/RFCs/0789-cosi-extension-images.md b/docs/Development/RFCs/0789-cosi-extension-images.md index 31f7b5f422..d40c072455 100644 --- a/docs/Development/RFCs/0789-cosi-extension-images.md +++ b/docs/Development/RFCs/0789-cosi-extension-images.md @@ -22,7 +22,7 @@ available for content released independently. ### Requirements - COSI revision 1.3 adds optional `extensions.sysexts` and - `extensions.confexts` arrays, each entry `{ image, path? }`. + `extensions.confexts` arrays, each entry `{ id, image, path? }`. - Clean Install and A/B Update deploy the effective set from COSI and Host Configuration; collisions fail. - An extension-only change can use Runtime Update without rewriting partitions @@ -66,7 +66,7 @@ Older readers ignore the additional tar members. ### Metadata Schema The two arrays mirror Host Configuration and determine the kind; each entry -contains only an `image` and optional `path`. This is the proposed addition to +contains an `id`, an `image` and optional `path`. This is the proposed addition to the [1.2 schema](../../Reference/Composable-OS-Image/cosi-metadata-v1.2.schema.json): ```json @@ -84,8 +84,9 @@ the [1.2 schema](../../Reference/Composable-OS-Image/cosi-metadata-v1.2.schema.j }, "ExtensionImage": { "type": "object", - "required": ["image"], + "required": ["id", "image"], "properties": { + "id": { "type": "string", "minLength": 1 }, "image": { "$ref": "#/$defs/ImageFile" }, "path": { "type": "string", "pattern": "^/.+\\.raw$" } } @@ -99,6 +100,7 @@ Example (`extensions` fragment of `metadata.json`): ```json { "sysexts": [{ + "id": "gpu-driver", "image": { "path": "images/extensions/gpu-driver.rawzst", "compressedSize": 41943040, @@ -107,6 +109,7 @@ Example (`extensions` fragment of `metadata.json`): } }], "confexts": [{ + "id": "fleet-config", "image": { "path": "images/extensions/fleet-config.rawzst", "compressedSize": 262144, @@ -118,6 +121,16 @@ Example (`extensions` fragment of `metadata.json`): } ``` +`id` is the extension's stable logical identity, not the COSI root `id`, the +host OS's `ID=`, or the image filename. For a sysext it must equal `SYSEXT_ID=` inside +`/usr/lib/extension-release.d/extension-release.{name}`; for a confext it +must equal `CONFEXT_ID=` inside +`/etc/extension-release.d/extension-release.{name}`. Trident already requires +those fields in DDIs. `(kind, id)` identifies an extension across COSI versions; +its compressed hash triggers replacement. IDs must be unique within each +array. Metadata can now match old and new extensions before reading their +payloads, but Trident must still verify the declared ID against the DDI. + `ImageFile` supplies the tar path, compressed size for ranged reads, uncompressed size for output bounds, and SHA-384 of the compressed member. Trident checks both sizes and the hash against the bytes read and written. @@ -137,8 +150,10 @@ handles `/etc` layers separately. How bundled extensions select such a policy is unresolved; omitting `path` alone must not opt in. There is no `url` (the tar is the source), `kind` (the array supplies it), or -separate name, ID, enable flag or compatibility fields (the DDI supplies them, -or systemd merges all discovered images). +separate name, enable flag or compatibility fields (the DDI supplies them, or +systemd merges all discovered images). The name determines the required +`{name}.raw` filename and can affect systemd's merge precedence; it is not +Trident's update identity. ### Versioning @@ -153,9 +168,11 @@ required extensions when reader versions cannot be controlled. ### Trident-Side Consumption Compute an effective set from COSI and Host Configuration entries, retaining -each entry's source. The extensions subsystem reads bundled members instead of -downloading them; `update_host_configuration` writes resolved paths back only -for Host Configuration entries. `osconfig` enables the sysext/confext merge +each entry's source. Match old and new entries by `(kind, id)`; replace when +content changes, and reconcile changes to placement or filename. The extensions +subsystem reads bundled members instead of downloading them; +`update_host_configuration` writes resolved paths back only for Host +Configuration entries. `osconfig` enables the sysext/confext merge units for the effective set. Move the SELinux check into validation that also runs for Runtime Updates; its current `configure` hook does not. @@ -247,22 +264,24 @@ rollback needs longer retention or must be refused. Merge both sources. Reject duplicate extension IDs per kind or destination collisions after resolving placement, including omitted `path`. Silent precedence could run an unvalidated combination; identical entries still -collide. The Host Configuration's documented ID-uniqueness rule must also be -enforced. +collide. COSI ID duplicates can be rejected from metadata; Host Configuration +IDs still require reading their DDIs. Enforce the Host Configuration's +documented ID-uniqueness rule too. ### Validation In `cosi/validation.rs`, add `V1_3` error kinds for invalid explicit paths -(absolute, `.raw`, allowed directory), duplicate explicit destinations, reused -tar members and collisions with disk-region images. Reuse +(absolute, `.raw`, allowed directory), duplicate `(kind, id)` pairs, duplicate +explicit destinations, reused tar members and collisions with disk-region +images. Reuse `Extension::validate_sysext`/`validate_confext` path rules. A missing member is detected when read: `Cosi::new` only scans as far as `metadata.json`. After decompressing and mounting the DDI, check actual compressed and uncompressed sizes, the compressed SHA-384, exactly one `extension-release` -with the kind's ID, the `{name}.raw` destination suffix, resolved placement, -and duplicate destinations/IDs across both sources. Omitted paths cannot be -collision-checked from metadata alone. +whose `SYSEXT_ID` or `CONFEXT_ID` equals the metadata `id`, the `{name}.raw` +destination suffix, resolved placement, and duplicate destinations/IDs across +both sources. Omitted paths cannot be collision-checked from metadata alone. Warn on definite OS `ID` mismatch with COSI `osRelease`, not `VERSION_ID`: systemd may accept a version mismatch via `SYSEXT_LEVEL` or `CONFEXT_LEVEL`. @@ -276,13 +295,15 @@ validation for every servicing type, not only the Host Configuration lists in ## Testing and Metrics -- v1.3 schema and valid/invalid samples; semantic path rules in Rust tests. +- v1.3 schema and valid/invalid samples, including missing or empty IDs; + semantic path and duplicate-ID rules in Rust tests. - Clean Install and A/B Update with both kinds, merge-unit activation and rollback (including absent `path` and slotted placement once supported). - Runtime Update with identical region images, distinct stage/finalize and auto-rollback; same-ID/same-path replacement and historical rollback refusal. - Reject mixed-source collisions, invalid destinations, enforcing SELinux, - changed OS summaries under requested Runtime Update, and disk streaming. + mismatched metadata/DDI IDs, changed OS summaries under requested Runtime + Update, and disk streaming. ## Implementation Plan