devex/mcdiff: Add MachineConfig diff, attribution, and node drift scanner CLI

devex/cmd/mcdiff/PR_DESCRIPTION.md

@@ -0,0 +1,61 @@
+# PR Title
+
+```
+devex/mcdiff: Add MachineConfig diff, attribution, and node drift scanner CLI
+```
+
+# Short Summary
+
+Replaces opaque MachineConfigDaemon byte-count errors with file-level expected content, last-writer attribution, and whole-node drift scanning.
+
+The existing `mcdiff diff MC1 MC2` dyff helper is unchanged. This PR adds `mcdiff file` and `mcdiff node` as a `devex` diagnostic: it consumes the **rendered** MachineConfig as source of truth (no client-side re-merge), attributes last writer using MCO merge order, and diffs against a local file, a live node (MCD `/rootfs`), or an unpacked must-gather archive.
+
+This is not remediation, MachineConfig editing, or a CI gate. Drift findings exit 0.
+
+# Scope of Changes
+
+All new and updated code lives under `devex/cmd/mcdiff`:
+
+| Package | Role |
+| --- | --- |
+| `devex/cmd/mcdiff` | CLI: `file`, `node`, existing `diff`, shell completion |
+| `internal/cluster` | Load pool rendered MC (`status.configuration`, else `spec.configuration`) |
+| `internal/ignition` | Decode Ignition files (`data:` base64 and percent-encoded) |
+| `internal/attribution` | Last-writer using MergeMachineConfigs order |
+| `internal/diff` | Byte compare, unified diff, mode compare |
+| `internal/node` | Live node read via machine-config-daemon exec (`/rootfs`) |
+| `internal/mustgather` | Offline Getter + NodeReader from unpacked archives |
+| `internal/scanner` | Whole-node scan + MCP detection from node labels |
+| `internal/report` | Text and JSON reports |
+
+Docs: `README.md`, `TESTING.md`, this file.
+
+# Verification Checklist for Reviewers
+
+- [ ] `go test ./devex/cmd/mcdiff/...` passes
+- [ ] `mcdiff file` works with `--pool`, `--from-file`, `--node`, and `--must-gather`
+- [ ] `mcdiff node` performs whole-node scan
+- [ ] Base64 and percent-encoded Ignition payloads decode seamlessly
+- [ ] Offline must-gather mode functions without kubeconfig
+
+See [TESTING.md](./TESTING.md) for unit, live-cluster, and must-gather steps.
+
+# Local verification (author)
+
+From the MCO repo root:
+
+```console
+go build ./devex/cmd/mcdiff/...
+go vet ./devex/cmd/mcdiff/...
+gofmt -s -l devex/cmd/mcdiff/
+go test ./devex/cmd/mcdiff/... -count=1
+```
+
+# Notes for reviewers
+
+- Expected bytes always come from the rendered MachineConfig, never `MergeMachineConfigs`.
+- Last-writer uses `configuration.source` and the same fragment sort as `pkg/controller/common.MergeMachineConfigs`.
+- Live `--node` execs into the existing `machine-config-daemon` pod (`k8s-app=machine-config-daemon` in `openshift-machine-config-operator`), host tree at `/rootfs`.
+- Standard must-gather does **not** dump `/etc`. Paths without a snapshot are **MISSING ON NODE**.
+- Unified diffs and `--show-content` can include secrets. Treat output as sensitive.
+- Do not run the live drift-injection scenario from TESTING.md on a production cluster.

devex/cmd/mcdiff/README.md

@@ -0,0 +1,145 @@
+# mcdiff
+
+MCDiff explains what the Machine Config Operator (MCO) thinks a file should contain, which MachineConfig last wrote it, and how that differs from a local copy, a live node, or a must-gather archive.
+
+The MCD reports on-disk mismatches as byte counts on purpose (those files can hold secrets). That is enough to know something drifted, and not enough to debug it. MCDiff is the explanation layer: it reads the **rendered** MachineConfig as source of truth, attributes last-writer using MCO merge order, and prints a unified diff when you ask it to compare.
+
+This is a `devex` helper. Do not use it as an unsupervised production remediation tool.
+
+## Build
+
+From the MCO repo root:
+
+```console
+go build -o mcdiff ./devex/cmd/mcdiff
+```
+
+Or: `make install-helpers`
+
+## Commands
+
+```console
+mcdiff file PATH --pool POOL [flags]
+mcdiff node NODE [flags]
+mcdiff diff MC1 MC2
+mcdiff completion bash|zsh|fish|powershell
+```
+
+`file` inspects one path. `node` scans every Ignition file in the node's rendered MachineConfig against the host filesystem. `diff` is the older helper that runs `dyff` between two MachineConfig objects.
+
+## Examples
+
+Inspect expected content and last writer (does not print file bytes by default):
+
+```console
+mcdiff file /etc/ssh/sshd_config --pool worker
+```
+
+Compare against a live node (execs into the machine-config-daemon pod, host root at `/rootfs`):
+
+```console
+mcdiff file /etc/ssh/sshd_config --pool worker --node worker-0
+```
+
+Compare against a local file:
+
+```console
+mcdiff file /etc/ssh/sshd_config --pool worker --from-file ./sshd_config
+```
+
+Offline analysis from an unpacked must-gather:
+
+```console
+mcdiff file /etc/ssh/sshd_config --pool worker --must-gather ./must-gather.local
+mcdiff file /etc/ssh/sshd_config --pool worker --node worker-0 --must-gather ./must-gather.local
+```
+
+Print expected bytes or JSON:
+
+```console
+mcdiff file /etc/ssh/sshd_config --pool worker --show-content
+mcdiff file /etc/ssh/sshd_config --pool worker -o json
+```
+
+Scan every managed file on a node (pool is detected from node labels):
+
+```console
+mcdiff node worker-0
+mcdiff node worker-0 --pool worker
+mcdiff node worker-0 --show-diffs
+mcdiff node worker-0 --must-gather ./must-gather.local --pool worker
+mcdiff node worker-0 -o json
+```
+
+Replace the KCS `oc debug` + `jq` + `base64`/`urldecode` walkthrough for a degraded MachineConfigDaemon:
+
+```console
+mcdiff file /etc/chrony.conf --pool worker --node worker-0
+mcdiff file /etc/resolv.conf --pool worker --node worker-0
+mcdiff file /etc/kubernetes/kubelet-ca.crt --pool master --node master-0
+```
+
+Ignition `data:,…` percent-encoding and `data:text/plain;charset=utf-8;base64,…` are decoded automatically. Missing host files (`could not stat file`) are reported as **MISSING ON NODE** without failing the command. Mode drift (for example 0644 vs 0755) is reported next to size and content deltas.
+
+## Flag matrix
+
+`--pool` is required for `file`. For `node` it is optional: omitted means detect the pool from the node's labels the same way the Machine Config Operator does. `--show-content` (file) / `--show-diffs` (node) and `-o json` are optional in every valid mode.
+
+| Mode | `--from-file` | `--node` | `--must-gather` | Result |
+| --- | --- | --- | --- | --- |
+| Live inspect | | | | Expected bytes + last writer from the cluster |
+| Local compare | yes | | | Diff expected vs a local file |
+| Live node compare | | yes | | Diff expected vs the file on the node |
+| Offline inspect | | | yes | Same as live inspect, from must-gather CRs (no kubeconfig) |
+| Offline node compare | | yes | yes | Diff expected vs a must-gather node snapshot |
+| **Invalid** | yes | yes | | Error: `cannot use --from-file and --node together` |
+| **Invalid** | yes | | yes | Error: `cannot use --must-gather and --from-file together` |
+| **Invalid** | yes | yes | yes | Same `--from-file` / `--node` error |
+
+`mcdiff node NODE` always compares against the node (live or must-gather). There is no `--from-file` on this command.
+
+| Mode | `--pool` | `--must-gather` | `--show-diffs` | Result |
+| --- | --- | --- | --- | --- |
+| Live whole-node scan | optional | | | Summary of every managed file vs the node |
+| Live whole-node scan with diffs | optional | | yes | Same, plus unified diffs for mismatches |
+| Offline whole-node scan | recommended | yes | | Same, from must-gather CRs and snapshots |
+| Pool override | yes | | | Skip label-based pool detection |
+
+Pass `--pool` when the node is unassigned, Windows, or matches more than one custom pool.
+
+Live modes use the standard kubeconfig flags (`--kubeconfig`, `--context`, `KUBECONFIG`). `--must-gather` skips kubeconfig.
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | Inspection succeeded. Includes MATCH, CONTENT MISMATCH, MODE MISMATCH, unmanaged paths, MISSING ON NODE, CLEAN, and DRIFT DETECTED. |
+| non-zero | The tool could not perform the inspection: invalid flags, missing pool / rendered MachineConfig / node, unreadable `--from-file` or must-gather directory, RBAC, or network errors. |
+
+MCDiff is a diagnostic tool, not a CI gate. A drift finding is still a successful inspection.
+
+## Must-gather caveat
+
+Standard `oc adm must-gather` archives include:
+
+- Cluster-scoped MachineConfig and MachineConfigPool YAML under `cluster-scoped-resources/machineconfiguration.openshift.io/`
+- Node objects under `cluster-scoped-resources/core/nodes/`
+- On degraded nodes, MCO’s `machine_config_ondisk/<node>/currentconfig`
+
+They do **not** snapshot the entire host `/etc` tree. `--node` with `--must-gather` (and `mcdiff node --must-gather`) only diffs a host file when the archive contains a snapshot (`nodes/<node>/host/...`, `host_files/<node>/...`, `machine_config_ondisk/<node>/files/...`) or when that path can be decoded from `currentconfig`. Files without a snapshot are reported as missing. Extract the archive to a directory first; do not pass a tarball.
+
+## Shell completion
+
+```console
+source <(mcdiff completion bash)
+source <(mcdiff completion zsh)
+mcdiff completion fish | source
+```
+
+## Testing
+
+Unit tests, live-cluster scenarios, and must-gather steps for reviewers and QE: [TESTING.md](./TESTING.md).
+
+Suggested PR title and description: [PR_DESCRIPTION.md](./PR_DESCRIPTION.md).
+
+Expected file contents are omitted unless `--show-content` is set. Unified diffs from `--from-file`, `--node`, or `mcdiff node --show-diffs` do print changed lines, because that is the comparison result. Treat those outputs as sensitive.

devex/cmd/mcdiff/TESTING.md

@@ -0,0 +1,158 @@
+# Testing MCDiff
+
+Instructions for peer reviewers and QE. Run all commands from the machine-config-operator repository root unless noted.
+
+MCDiff is a diagnostic. A content mismatch, mode mismatch, or **MISSING ON NODE** is a successful inspection (exit 0). Non-zero means the tool could not inspect (bad flags, missing pool/rendered MC/node, RBAC, or I/O).
+
+Do **not** run the live drift-injection scenario on a production cluster. Use a disposable cluster or skip to unit tests and must-gather.
+
+## 1. Local unit and package tests
+
+```console
+go test ./devex/cmd/mcdiff/... -v -count=1
+```
+
+Expect every package under `devex/cmd/mcdiff` to print `PASS`, including:
+
+- `internal/ignition` — base64 and percent-encoded `data:` URLs
+- `internal/diff` — content, CRLF, trailing newline, mode
+- `internal/scanner` — all-match, mismatch, missing, mode, pool detection
+- `devex/cmd/mcdiff` — `file` and `node` CLI, JSON, must-gather fixtures
+
+Optional gates used before opening a PR:
+
+```console
+go build ./devex/cmd/mcdiff/...
+go vet ./devex/cmd/mcdiff/...
+gofmt -s -l devex/cmd/mcdiff/
+```
+
+`gofmt -s -l` should print nothing.
+
+## 2. Binary compilation
+
+Build the whole `main` package (not `main.go` alone; the command is split across several files):
+
+```console
+mkdir -p bin
+go build -o bin/mcdiff ./devex/cmd/mcdiff
+./bin/mcdiff --help
+./bin/mcdiff file --help
+./bin/mcdiff node --help
+```
+
+Or: `make install-helpers` (installs all `devex/cmd` helpers).
+
+Confirm the help lists `file`, `node`, `diff`, and `completion`.
+
+## 3. Live OpenShift cluster testing
+
+Prerequisites:
+
+- `oc` logged in with rights to get MachineConfigPools, MachineConfigs, Nodes, and to exec into `machine-config-daemon` pods in `openshift-machine-config-operator`
+- Standard kubeconfig (`KUBECONFIG`, `--kubeconfig`, or `--context`)
+
+### a. Identify a worker node
+
+```console
+oc get nodes -l node-role.kubernetes.io/worker
+```
+
+Pick one `Ready` node. In the steps below, replace `<node-name>` with that name (for example `worker-0`).
+
+### b. Inspect expected content (no host read)
+
+```console
+./bin/mcdiff file /etc/ssh/sshd_config --pool worker
+```
+
+Expect: pool `worker`, a `rendered-worker-*` MachineConfig, `Exists: yes`, last writer (often `99-worker-ssh` or similar), expected content omitted unless `--show-content`.
+
+### c. Introduce intentional drift (disposable cluster only)
+
+```console
+oc debug node/<node-name> -- chroot /host sh -c "echo '# drift' >> /etc/ssh/sshd_config"
+```
+
+The node may go degraded. That is the point of this scenario.
+
+### d. Live single-file diff
+
+```console
+./bin/mcdiff file /etc/ssh/sshd_config --pool worker --node <node-name>
+```
+
+Expect: `CONTENT MISMATCH`, a size delta, and a unified diff that includes `+# drift`. Exit 0.
+
+Also useful KCS-style paths (no extra drift required):
+
+```console
+./bin/mcdiff file /etc/chrony.conf --pool worker --node <node-name>
+./bin/mcdiff file /etc/resolv.conf --pool worker --node <node-name>
+```
+
+### e. Whole-node scan
+
+```console
+./bin/mcdiff node <node-name> --show-diffs
+```
+
+Expect: `DRIFT DETECTED`, `/etc/ssh/sshd_config` in mismatched files, last writer, size delta, and a unified diff because `--show-diffs` is set. Other managed files should `MATCH` unless the node was already drifted.
+
+If pool detection fails (`not assigned` / multiple custom pools), add `--pool worker`.
+
+### f. Clean up drift
+
+```console
+oc debug node/<node-name> -- chroot /host sh -c "sed -i '/# drift/d' /etc/ssh/sshd_config"
+```
+
+Re-run the file diff and node scan. Expect `MATCH` / `CLEAN` unless other drift remains. The MCD may take a short time to clear degraded.
+
+## 4. Offline must-gather testing
+
+Standard `oc adm must-gather` does **not** snapshot all of `/etc`. Offline `--node` / `mcdiff node` only diffs a path when the archive has a host snapshot (`nodes/<node>/host/...`, `host_files/<node>/...`, `machine_config_ondisk/<node>/files/...`) or that path can be decoded from `currentconfig`. Other managed paths are **MISSING ON NODE**. Extract the tarball first; do not pass a `.tar.gz`.
+
+### a. Unpack
+
+```console
+mkdir -p /tmp/must-gather.local
+tar -C /tmp/must-gather.local -xf must-gather.tar.gz
+```
+
+Use the directory that contains `cluster-scoped-resources` (sometimes one level down under an image directory).
+
+### b. Offline inspect (no kubeconfig)
+
+```console
+unset KUBECONFIG
+./bin/mcdiff file /etc/ssh/sshd_config --pool worker --must-gather /tmp/must-gather.local
+```
+
+Expect: rendered MC and last writer from YAML in the archive. Does not need a live cluster.
+
+### c. Offline whole-node scan
+
+```console
+./bin/mcdiff node worker-0 --must-gather /tmp/must-gather.local --pool worker
+```
+
+Replace `worker-0` with a node name present in the archive (`cluster-scoped-resources/core/nodes/` or `nodes/`). Pass `--pool` if the Node object has no role labels.
+
+Expect: scan completes (exit 0). Files without snapshots are listed as missing. That is expected for a stock must-gather.
+
+## QE pass / fail
+
+| Check | Pass |
+| --- | --- |
+| Unit tests | All `devex/cmd/mcdiff` packages PASS |
+| `file --pool` | Prints pool, rendered MC, last writer; omits file bytes |
+| `file --from-file` | MATCH or CONTENT MISMATCH with unified diff |
+| `file --node` after step c | CONTENT MISMATCH, `+# drift`, exit 0 |
+| `file --node` missing path | **MISSING ON NODE**, exit 0 |
+| `node` scan after step c | DRIFT DETECTED includes sshd_config |
+| Encoding | Unit tests for base64 and `data:,` percent-encoding PASS; live inspect does not require manual decode |
+| Must-gather | Inspect works with `KUBECONFIG` unset; no API server required |
+| Invalid flags | `cannot use --from-file and --node together` |
+
+Treat `--show-content` and unified diffs as sensitive. Do not paste them into public bugs.