diff --git a/tools/tailor/e2e/README.md b/tools/tailor/e2e/README.md index 2ed52da17..3300b7ada 100644 --- a/tools/tailor/e2e/README.md +++ b/tools/tailor/e2e/README.md @@ -1,6 +1,6 @@ # tailor end-to-end fixture -A small, real workspace the **E2E workflow** (`.github/workflows/e2e.yml`) builds with the actual +A small, real workspace the **E2E workflow** (`.github/workflows/tailor.yml`) builds with the actual `tailor` binary and Image Customizer — to prove the full product works, end to end. The images are built but **not booted**. @@ -44,3 +44,102 @@ tailor build # builds artifacts/appliance_{min,net}_amd64_cosi.cosi expected artifact slugs — add the new ones there. - **Exercise more IC features**: extend the `config:` (e.g. packages, scripts, users). Package installs reach the network inside the build, so prefer them only when worth the extra time. + +## SELinux janitor regression (Fedora VM only) + +`selinux_janitor_e2e.py` exercises real rootful Podman containers and the actual Tailor `clean` +command, **not** a mocked runtime. This is a manual harness, **not wired into CI**: there is no +dedicated SELinux-enforcing runner. Run it as a non-root user *inside* a disposable Fedora VM with +`getenforce` reporting `Enforcing`. The user needs access to a rootful Podman API socket; granting +that access is effectively granting root on the VM, so **never** use a shared or production runner. +The script aborts unless the daemon reports SELinux enabled and rootless false. It uses only a +unique child of `~/tailor-selinux-e2e-work/` in the guest, requires the bind source to be +`user_home_t`, and never binds `/`, changes SELinux policy, relabels any host directory, or runs +host-side `sudo`. Keep this fixture out of the normal Ubuntu-hosted Tailor job: Ubuntu's Docker +runner cannot prove an enforcing Fedora SELinux denial. + +Build this checkout's `tailor` binary and copy it beside the script in the VM. With the guest +Podman socket at `/run/podman/podman.sock`, run: + +```text +python3 selinux_janitor_e2e.py --tailor-bin ./tailor +``` + +This **diagnostic** mode positively demonstrates that an unmodified confined janitor (`container_t`) +is denied access to a root-owned `user_home_t` artifact, that the current non-root `tailor clean` +also fails, and that a container-scoped `label=disable` janitor removes the *same* artifact without +changing labels or policy. That option is a measured workaround, **not** a production fix: disabling +the janitor container's SELinux confinement has security implications that require review. A genuine +fix needs a separate design decision. Automatic `:z`/`:Z` is **not** a safe fallback: janitor +removal binds the *parent* so `rm` can delete the child, but a parent can be an image directory or +RPM source containing unrelated user files. Confined `rm` needs write access to that parent; +relabeling its bind source would also relabel those unrelated files. The tested per-container +`label=disable` approach does not change host labels or disable host SELinux, but it weakens +isolation for that janitor container. This test does not claim to run a full Image Customizer build. + +To compare a proposed fix **on the same running VM**, copy both binaries into the guest and run +the script twice: first `--tailor-bin ./tailor-baseline`, then +`--tailor-bin ./tailor-patched --expect-fixed`. Each invocation creates a uniquely +named disposable workspace, so the baseline cannot remove the patched test's fixture. The second +run is a **gating** check: the unmodified bind must still fail as a positive control, then +**Tailor itself** must remove the root-owned artifact as a non-root caller. Put that job on a +dedicated, ephemeral SELinux-enforcing Fedora VM runner (for example, a self-hosted runner labeled +`fedora-selinux-e2e`), not in a container on an Ubuntu runner. Provision Fedora 43 Cloud Base, +a 25 GiB guest disk, 4 GiB RAM, KVM, Podman, and a non-root runner with access to the rootful Podman +socket; assert +`getenforce=Enforcing` before running. Build Tailor from `tools/tailor/` with its pinned Rust +toolchain and public crates.io, then run this Python script inside the VM. Until Tailor's janitor +is fixed, `--expect-fixed` is expected to fail and must not be represented as a passing test. + +Before each fixture deletion, the harness reads the guest's `/sys/block/loop*/loop/backing_file` +(including entries ending in ` (deleted)`) and `/proc/self/mountinfo` (mount root, mountpoint, +and source). A reference under the exact target path or an unreadable/malformed input fails +closed; it never detaches a loop or unmounts anything. Run +`python3 test_selinux_janitor_liveness.py` for synthetic path, mountinfo-escape, and file-backed +scanner checks that create **no live devices or mounts**. These fixture-side checks only protect +the named disposable artifact: they do **not** prove Tailor will reject a live build-scratch +tree or close races/other mount namespaces. In particular, a client-side scan is not +authoritative for a remote container daemon: its loops and mounts may be invisible here. +Do not infer liveness from artifact timestamps, substitute this preflight for a daemon-side +safety contract, or use an unguarded host-root deletion fallback. The tested minimal +janitor-only SELinux prototype has no production liveness guard: do not place real live loops +or mounts under a deletable fixture to try to validate one. + +### Bounded pre-IC stale tools-dir probe + +The default harness runs `tailor clean base-os` on a disposable root-owned artifact. To check +the **distinct pre-IC build path**, run it twice on the same Fedora VM, adding +`--probe-stale-tools-dir` to both invocations and `--expect-fixed` to the patched one. It creates +a root-owned `user_home_t` stale `//tools-dir` only in the guest workspace +and invokes actual non-root `tailor build preic`. Both the named tools-dir source and the +intentionally *fake* IC toolchain use the small `base/core:3.0` image. Baseline fails at the +janitor's stale-copy removal with Permission denied; the SELinux janitor prototype removes the +stale marker and reaches the deliberately absent `customize` command. Both CLI builds exit +non-zero, for **different asserted reasons**. No image is built and post-build chown is not +exercised; do not report this as a successful IC build. Before running, `build --dry-run` must +show IC bind sources only inside that disposable guest workspace; the fixture disables the +otherwise unnecessary `/dev` bind. It never binds the guest or Ubuntu filesystem root. +For a future enforcing-Fedora CI gate, run both: + +```text +python3 selinux_janitor_e2e.py --tailor-bin ./tailor --expect-fixed +python3 selinux_janitor_e2e.py --tailor-bin ./tailor --probe-stale-tools-dir --expect-fixed +``` + +Each must exit zero, although the fake-IC `tailor build` subprocess deliberately exits +non-zero. This needs access to MCR's small `base/core:3.0` image but no Image Customizer +image or real base-image download. + +### What a full build regression would require + +An additional, separate guest-only test could use one real Azure Linux `minimal-os` input, a +single raw output, `toolsDir: {source: base}` with the Azure Linux `base/core:3.0` source, and +IC's `input-image-oci` and `tools-dir` preview features. Pre-create a root-owned stale +`//tools-dir` **only inside that unique disposable guest workspace**; assert +the guest has no live loop backing file or mount under that target before invoking non-root +`tailor build`. After a real successful IC run, verify the stale copy was replaced/reclaimed, +the resulting artifact is owned by the caller, and guest labels are unchanged. Do not detach +loops, unmount, or recursively clean up a failed build: stop and discard the disposable VM. +Use a larger dedicated runner than the measured *clean*-only VM (estimate at least 8 GiB RAM +and 40 GiB guest disk, plus outbound MCR access and a longer job timeout); these full-build +resources and the real IC path have **not** been measured by this harness. diff --git a/tools/tailor/e2e/selinux_janitor_e2e.py b/tools/tailor/e2e/selinux_janitor_e2e.py new file mode 100644 index 000000000..6d333d0c1 --- /dev/null +++ b/tools/tailor/e2e/selinux_janitor_e2e.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +"""Reproduce Tailor's janitor denial on a disposable SELinux-enforcing Fedora VM.""" + +import argparse +import os +from pathlib import Path +import re +import shlex +import subprocess +import uuid + + +JANITOR_IMAGE = "mcr.microsoft.com/azurelinux/base/core:3.0" +WORKSPACE_NAME = "tailor-selinux-e2e-work" +IMAGE_YAML = """\ +name: base-os +base: + azureLinux: + version: "3.0" + variant: minimal-os +outputs: + - format: raw +config: + os: + hostname: base-os +""" +TAILOR_YAML = """\ +schemaVersion: 1 +toolchains: + default: ic + entries: + - name: ic + container: mcr.microsoft.com/azurelinux/imagecustomizer + tag: latest +defaults: + outputs: + - format: raw +""" +PRE_IC_TAILOR_YAML = """\ +schemaVersion: 1 +toolchains: + default: fake + entries: + - name: fake + container: mcr.microsoft.com/azurelinux/base/core + tag: "3.0" +toolsDirSources: + - name: base + container: mcr.microsoft.com/azurelinux/base/core + tag: "3.0" +runtime: + mounts: + dev: false +defaults: + outputs: + - format: raw +""" +PRE_IC_IMAGE_YAML = """\ +name: preic +base: + path: ./base.raw +toolsDir: + source: base +outputs: + - format: raw +config: + previewFeatures: + - tools-dir + os: + hostname: preic +""" + + +def run(*argv: str, cwd: Path | None = None) -> subprocess.CompletedProcess[str]: + result = subprocess.run(argv, cwd=cwd, capture_output=True, text=True, check=False) + print(f"$ {' '.join(argv)}\nexit={result.returncode}") + if result.stdout.strip(): + print(result.stdout.strip()) + if result.stderr.strip(): + print(result.stderr.strip()) + return result + + +def require(condition: bool, message: str) -> None: + if not condition: + raise AssertionError(message) + + +def label(path: Path) -> bytes: + return os.getxattr(path, "security.selinux") + + +def under_prefix(value: str, prefix: Path) -> bool: + value = value.removesuffix(" (deleted)") + if not value.startswith("/"): + return False + return Path(os.path.normpath(value)).is_relative_to(prefix) + + +def unescape_mount_path(value: str) -> str: + encoded = value.encode("utf-8") + decoded = bytearray() + index = 0 + while index < len(encoded): + if encoded[index] != ord("\\"): + decoded.append(encoded[index]) + index += 1 + continue + digits = encoded[index + 1:index + 4] + if len(digits) != 3 or any(digit not in b"01234567" for digit in digits): + raise RuntimeError(f"refusing malformed mountinfo escape: {value}") + number = int(digits, 8) + if number > 255: + raise RuntimeError(f"refusing malformed mountinfo escape: {value}") + decoded.append(number) + index += 4 + try: + return decoded.decode("utf-8") + except UnicodeDecodeError as error: + raise RuntimeError(f"refusing malformed mountinfo encoding: {value}") from error + + +def assert_no_live_references( + target: Path, loop_backings: list[tuple[str, str]], mountinfo: str, +) -> None: + for device, backing in loop_backings: + if under_prefix(backing, target): + raise RuntimeError(f"refusing to delete live {device} backing file: {backing}") + + for line in mountinfo.splitlines(): + before, separator, after = line.partition(" - ") + fields, filesystem = before.split(), after.split() + if not separator or len(fields) < 5 or len(filesystem) < 2: + raise RuntimeError(f"refusing deletion with malformed mountinfo: {line}") + for field in (fields[3], fields[4], filesystem[1]): + value = unescape_mount_path(field) + if under_prefix(value, target): + raise RuntimeError(f"refusing to delete live mount reference: {value}") + + +def assert_no_local_live_references( + target: Path, sys_block: Path = Path("/sys/block"), + mountinfo: Path = Path("/proc/self/mountinfo"), +) -> None: + # The VM fixture uses a local daemon; this client-namespace scan is not a remote-engine barrier. + require(target.is_absolute() and target.exists() and target == target.resolve(strict=True), + f"refusing missing or symlinked deletion target: {target}") + loop_backings = [] + for block in sys_block.iterdir(): + if not re.fullmatch(r"loop\d+", block.name): + continue + try: + (block / "loop").stat() + except FileNotFoundError: + continue + loop_backings.append((block.name, (block / "loop/backing_file").read_text().strip())) + mount_data = mountinfo.read_text() + require(bool(mount_data.strip()), "refusing deletion with empty mountinfo") + assert_no_live_references(target, loop_backings, mount_data) + + +def probe_stale_tools_dir( + tailor: str, podman: list[str], endpoint: str, root: Path, expect_fixed: bool, +) -> None: + workspace = root / f"pre-ic-{uuid.uuid4().hex}" + image_dir = workspace / "preic" + image_dir.mkdir(parents=True) + (workspace / "tailor.yaml").write_text(PRE_IC_TAILOR_YAML) + (image_dir / "image.yaml").write_text(PRE_IC_IMAGE_YAML) + (image_dir / "base.raw").write_bytes(b"not an image: fake IC must never read this") + matrix = run(tailor, "matrix", "--format", "slugs", cwd=workspace) + slugs = matrix.stdout.strip().splitlines() + require(matrix.returncode == 0 and len(slugs) == 1 and re.fullmatch(r"[a-zA-Z0-9_.-]+", slugs[0]), + f"expected one safe cell slug, got {slugs!r}") + dry = run(tailor, "build", "preic", "--dry-run", cwd=workspace) + bind_sources = re.findall(r"(?m)^\s+-v (\S+?):", dry.stdout) + require(dry.returncode == 0 and bind_sources + and all(Path(source).is_relative_to(workspace) for source in bind_sources), + f"fake IC must bind only its disposable guest workspace, got {bind_sources!r}") + + parent = workspace / "artifacts/.tailor/build" / slugs[0] + parent.mkdir(parents=True) + stale = parent / "tools-dir" + marker = stale / "stale-marker" + bind = f"{parent}:{parent}" + seeded = run( + *podman, "run", "--rm", "--security-opt", "label=disable", + "-v", bind, JANITOR_IMAGE, "/bin/sh", "-c", + f"mkdir -p -- {shlex.quote(str(stale))} && printf stale > {shlex.quote(str(marker))}", + ) + require(seeded.returncode == 0 and marker.stat().st_uid == 0, + "could not seed disposable root-owned stale tools-dir marker") + parent_label = label(parent) + require(b":user_home_t:" in label(stale) and b":user_home_t:" in parent_label, + "pre-IC fixture must retain its user_home_t labels") + assert_no_local_live_references(stale) + + result = run(tailor, "--engine", "podman", "--host", + endpoint, "build", "preic", cwd=workspace) + output = result.stdout + result.stderr + require(result.returncode != 0, "fake IC must fail after the pre-IC janitor phase") + require(label(parent) == parent_label, "build attempt changed the guarded parent label") + if expect_fixed: + require(not marker.exists(), "stale tools-dir marker survived: pre-IC cleanup not proven") + require("Permission denied" not in output and "janitor exited" not in output, + "build failed in janitor rather than the intentionally fake IC toolchain") + require("customize" in output and "not found" in output, + "build did not reach the deliberately missing fake IC command") + print("PASS: non-root Tailor passed stale tools-dir pre-clean; fake IC failed later") + else: + require(marker.exists() and "Permission denied" in output and "janitor exited" in output, + "baseline did not fail at the root-owned stale tools-dir pre-clean") + print("PASS: baseline Tailor denied at stale tools-dir pre-clean") + print("NOTE: fake IC never produces an image; post-build chown is not exercised") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--tailor-bin", type=Path, required=True) + parser.add_argument("--podman-socket", default="/run/podman/podman.sock") + parser.add_argument("--expect-fixed", action="store_true", + help="require janitor cleanup to pass (the fake IC still fails in pre-IC mode)") + parser.add_argument("--probe-stale-tools-dir", action="store_true", + help="probe real pre-IC janitor removal using a deliberately failing toolchain") + args = parser.parse_args() + + require(os.geteuid() != 0, "run this test as an ordinary Fedora user, not root") + require(run("getenforce").stdout.strip() == "Enforcing", "SELinux must be enforcing") + info = run( + "podman", "--remote", "--url", f"unix://{args.podman_socket}", + "info", "--format", "{{.Host.Security.SELinuxEnabled}} {{.Host.Security.Rootless}}", + ) + require(info.returncode == 0 and info.stdout.strip() == "true false", + "the non-root test user must access a rootful, SELinux-enabled Podman API") + + root = Path.home() / WORKSPACE_NAME + require(not root.is_symlink(), f"refusing symlinked fixture root: {root}") + root.mkdir(exist_ok=True) + endpoint = f"unix://{args.podman_socket}" + podman = ["podman", "--remote", "--url", endpoint] + tailor = str(args.tailor_bin.resolve(strict=True)) + if args.probe_stale_tools_dir: + probe_stale_tools_dir(tailor, podman, endpoint, root, args.expect_fixed) + return + + workspace = root / f"run-{uuid.uuid4().hex}" + artifacts = workspace / "artifacts" + image_dir = workspace / "base-os" + artifacts.mkdir(parents=True) + image_dir.mkdir() + (workspace / "tailor.yaml").write_text(TAILOR_YAML) + (image_dir / "image.yaml").write_text(IMAGE_YAML) + artifact = artifacts / "base-os_amd64_raw.raw" + parent_label = label(artifacts) + require(b":user_home_t:" in parent_label, + f"expected user_home_t for positive control; got {parent_label!r}") + + bind = f"{artifacts}:{artifacts}" + # The rootful container stands in for IC's root-owned output; no sudo or host writes. + created = run( + *podman, "run", "--rm", "--security-opt", "label=disable", + "-v", bind, JANITOR_IMAGE, "/bin/sh", "-c", + f"printf fixture > '{artifact}'", + ) + require(created.returncode == 0, "could not create disposable root-owned IC-like output") + require(artifact.stat().st_uid == 0, "fixture artifact is not root-owned") + original_label = label(artifact) + require(b":user_home_t:" in original_label, f"unexpected artifact label: {original_label!r}") + print(f"positive control: artifact={artifact}, label={original_label!r}") + + container = run(*podman, "run", "--rm", JANITOR_IMAGE, "/bin/cat", + "/proc/self/attr/current") + require(container.returncode == 0 and "container_t:" in container.stdout, + "janitor container did not run confined as container_t") + + assert_no_local_live_references(artifact) + baseline = run(*podman, "run", "--rm", "-v", bind, JANITOR_IMAGE, + "/bin/rm", "-rf", "--", str(artifact)) + require(baseline.returncode != 0 and "Permission denied" in baseline.stderr + and artifact.exists(), "unmodified janitor bind must be denied by SELinux") + print("PASS: actual container_t janitor rm cannot remove a user_home_t artifact") + + assert_no_local_live_references(artifact) + actual = run(tailor, "--engine", "podman", "--host", + f"unix://{args.podman_socket}", "clean", "base-os", cwd=workspace) + if args.expect_fixed: + require(actual.returncode == 0 and not artifact.exists(), + "Tailor clean still fails against user_home_t with SELinux enforcing") + print("PASS: actual non-root Tailor janitor removed the root-owned artifact") + recreated = run( + *podman, "run", "--rm", "--security-opt", "label=disable", + "-v", bind, JANITOR_IMAGE, "/bin/sh", "-c", f"printf fixture > '{artifact}'", + ) + require(recreated.returncode == 0, "could not recreate artifact for workaround control") + else: + require(actual.returncode != 0 and "Permission denied" in actual.stderr + actual.stdout, + "Tailor clean did not fail with the expected SELinux permission denial") + require(artifact.exists() and artifact.stat().st_uid == 0, + "failed janitor unexpectedly removed or changed the positive-control artifact") + print("PASS: actual non-root Tailor clean fails against user_home_t with SELinux enforcing") + + # Demonstrate a container-scoped workaround without changing policy or relabeling the host. + assert_no_local_live_references(artifact) + fixed = run(*podman, "run", "--rm", "--security-opt", "label=disable", + "-v", bind, JANITOR_IMAGE, "/bin/rm", "-rf", "--", str(artifact)) + require(fixed.returncode == 0 and not artifact.exists(), + "container-scoped label=disable did not allow the same janitor removal") + require(label(artifacts) == parent_label, + "workaround must not relabel the disposable bind source") + print("PASS: same janitor rm succeeds with per-container label=disable; host policy/labels unchanged") + if args.expect_fixed: + print("PASS: patched Tailor janitor succeeded; no real Image Customizer build was run") + else: + print("NOTE: baseline Tailor is not fixed; no real Image Customizer build was run") + + +if __name__ == "__main__": + main() diff --git a/tools/tailor/e2e/test_selinux_janitor_liveness.py b/tools/tailor/e2e/test_selinux_janitor_liveness.py new file mode 100644 index 000000000..8eb34703d --- /dev/null +++ b/tools/tailor/e2e/test_selinux_janitor_liveness.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +"""Synthetic liveness inputs; no loop device or mount is created or cleaned up.""" + +from pathlib import Path +import unittest +import uuid + +from selinux_janitor_e2e import assert_no_live_references, assert_no_local_live_references + + +TARGET = Path("/disposable/work/artifacts/image.raw") + + +def mountinfo(root: str = "/", mountpoint: str = "/other", source: str = "/dev/vda") -> str: + return f"42 1 8:1 {root} {mountpoint} rw - ext4 {source} rw\n" + + +class LivenessChecks(unittest.TestCase): + def test_loop_backing_file_blocks_deletion_including_deleted_suffix(self) -> None: + for backing in (str(TARGET), f"{TARGET} (deleted)"): + with self.subTest(backing=backing), self.assertRaisesRegex( + RuntimeError, "live loop" + ): + assert_no_live_references(TARGET, [("loop7", backing)], "") + + def test_mountpoint_root_and_source_block_deletion(self) -> None: + for key, value in ( + ("mountpoint", str(TARGET)), + ("root", str(TARGET)), + ("source", str(TARGET)), + ): + with self.subTest(field=key), self.assertRaisesRegex( + RuntimeError, "live mount" + ): + assert_no_live_references(TARGET, [], mountinfo(**{key: value})) + + def test_mountinfo_escaped_spaces_are_decoded(self) -> None: + target = Path("/disposable/work/artifacts/my image.raw") + with self.assertRaisesRegex(RuntimeError, "live mount"): + assert_no_live_references( + target, [], mountinfo(mountpoint="/disposable/work/artifacts/my\\040image.raw") + ) + + def test_unrelated_sibling_paths_do_not_block(self) -> None: + assert_no_live_references( + TARGET, [("loop7", f"{TARGET}-backup")], + mountinfo(mountpoint=f"{TARGET}-backup"), + ) + + def test_malformed_mountinfo_fails_closed(self) -> None: + with self.assertRaisesRegex(RuntimeError, "malformed mountinfo"): + assert_no_live_references(TARGET, [], "not a mountinfo entry") + for escaped in (r"/some\777path", r"/some\04path", r"/some\q40path", r"/some\377path"): + with self.subTest(escaped=escaped), self.assertRaisesRegex( + RuntimeError, "malformed mountinfo" + ): + assert_no_live_references(TARGET, [], mountinfo(mountpoint=escaped)) + + def test_file_reader_rejects_synthetic_sysfs_and_mountinfo(self) -> None: + fixture = Path(__file__).resolve().parent / "artifacts" / f".liveness-{uuid.uuid4().hex}" + loop_dir = fixture / "sys-block/loop7/loop" + loop_dir.mkdir(parents=True) + target = fixture / "image.raw" + target.write_text("fixture") + backing = loop_dir / "backing_file" + mount_file = fixture / "mountinfo" + try: + backing.write_text(f"{target} (deleted)\n") + mount_file.write_text(mountinfo()) + with self.assertRaisesRegex(RuntimeError, "live loop"): + assert_no_local_live_references(target, fixture / "sys-block", mount_file) + + backing.write_text("") + mount_file.write_text(mountinfo(source=str(target))) + with self.assertRaisesRegex(RuntimeError, "live mount"): + assert_no_local_live_references(target, fixture / "sys-block", mount_file) + + mount_file.write_text("") + with self.assertRaisesRegex(AssertionError, "empty mountinfo"): + assert_no_local_live_references(target, fixture / "sys-block", mount_file) + finally: + for path in (backing, mount_file, target): + if path.exists(): + path.unlink() + for path in (loop_dir, loop_dir.parent, loop_dir.parent.parent, fixture): + path.rmdir() + + +if __name__ == "__main__": + unittest.main()