feat(migration): export a guest so another node can take it over - #1
Open
ericwang401 wants to merge 18 commits into
Open
ericwang401 wants to merge 18 commits into
ericwang401 wants to merge 18 commits into
Conversation
The serial console has never worked. `qm terminal` hands the chardev socket to `socat ... STDIO,raw,echo=0`; those termios options make socat call `tcgetattr(0)`, which fails with "Inappropriate ioctl for device" whenever stdin is a pipe rather than a TTY. `qm` runs it through `system()` without checking the status, so it prints its banner and still exits 0 — the failure never reached the caller. Verified on PVE 9.2.2: exit 255 for a missing device, but exit 0 with empty stdout for this. Anchor now connects to `/var/run/qemu-server/<vmid>.<iface>` itself, mirroring `qm terminal`'s selection rule (first of serial0..3 whose value is exactly `socket`). That removes socat, the pseudo-terminal it would otherwise need — allocated only for socat to disable it again — and the orphaned-child hazard of killing a Perl parent that owns a socat grandchild still holding the port open. VNC deliberately keeps going through `qm vncproxy`: that installs a one-time password over QMP which expires after 30 seconds, real console state Proxmox owns. A serial chardev has no equivalent to arrange. Every session now ends with a close frame, success or failure, carrying the reason. A dropped socket reaches the browser as a bare 1006 that cannot be told apart from a network fault, which is what made these failures unreadable from the panel. The cost is a coupling to two things PVE does not call public API: the `serial[n]` config keys and the chardev path. Both are overridable and recorded in HANDOFF.md.
Adds a `templates.install` capability: given an artifact URL, its SHA-256, a VMID and a storage, the agent downloads the archive, verifies it, restores it with `qmrestore` and converts the result into a template. This is coport's job, driven by the panel instead of by a human on the node. The work order travels inside the bearer token the way console targets do, so a captured token cannot be re-pointed at a different artifact, VMID or storage. Two details worth knowing: - Cofoundry vzdumps a stopped VM, not a template. Coport's users clone by hand so it never mattered there, but Convoy only offers guests reporting `template: 1` — hence the `qm template` step, without which an install would appear to succeed and then be invisible in the panel. - VMIDs are cluster-global while `/etc/pve/qemu-server` is a symlink to the local node's directory only, so conflict detection reads pmxcfs's `.vmlist`. It runs before the download, because the alternative is learning about a collision from `qmrestore` after several gigabytes of transfer. Job state is deliberately in memory: a restart loses in-flight installs, the panel's poller sees the id disappear and says so, and the admin re-runs it.
Enrollment used to send nothing but a token, which only worked because the token identified a row an admin had already filled in by hand. An enrollment key identifies nobody, so the machine has to say who it is. The report is gathered from /proc and /etc/pve directly rather than through pvesh: enrollment runs before there is any configuration, so there is no qm_path to trust yet, and a probe that spawns nothing cannot hang on a subprocess that never exits. Every field is optional and every probe is best-effort. A host that cannot answer one of these questions is still worth enrolling, and failing an install over a fact nobody schedules against would trade a working node for a tidier payload. Deliberately not reported: the PVE version and the cluster CA fingerprint, both of which the panel can ask the Proxmox API for over an authenticated channel using code it already has; and host addresses, because the panel records the source address it actually observed, which is the one reachability claim a machine cannot overstate. `--mode` is new and only consulted for key-based enrollment; on the targeted path the panel already knows. Enrollment failures now carry the panel's own words where it has any -- "this key does not admit an agent" is the whole answer, and a bare status code sends an operator to the panel's logs to rediscover it. Verified end to end against a live panel: the report round-trips into anchors.reported_facts with CPU topology, memory and capabilities intact.
PVE defines its entire API in JSON Schema -- that is how it generates its documentation and validates every call -- and ships the result on each node at /usr/share/pve-docs/api-viewer/apidoc.js. Reading it here means the panel can validate a hardware profile, and build a form for one, against the rules *this* node will actually enforce rather than against whatever PVE release the panel was written for. It is also the only honest way to get the enums: nobody should be hand-maintaining the list of valid `scsihw` values. The file is JavaScript only in the sense that it assigns a JSON array to a variable, so the array is sliced out rather than parsed as a program, and tolerantly -- none of the assignment's shape is contractual. Its own claim set, for the same reason template tokens have one: the panel mints these for itself, and they authorise reading a definition off this node rather than doing anything to it. An enum with one variant, because the next thing the panel wants to read off a node should be a new variant rather than a new endpoint and a new token type.
The file landed unformatted, so `cargo fmt --check` failed on a clean checkout. No behaviour change; this is `cargo fmt` output verbatim.
Adds `ExportSpec`, `ExportMode` and `Compression` beside `InstallSpec`, and the `Export`, `Fetch` and `Discard` actions to the template token. `ExportMode` has one variant on purpose. A snapshot-mode archive is crash-consistent as of the moment it began, so every write the guest made during the transfer would be silently lost at cutover; the panel stops the guest first and leaves it stopped until the destination is verified, and this states that requirement on the wire rather than leaving it to convention. `Fetch` and `Discard` are separate actions so a token minted to let a destination node download an artifact cannot also delete it. Each names one artifact, so neither is a bearer token for the node's dump directory. The contract calls the action enum `Command`; it stays `TemplateAction` here because that is the name the install side already shipped, and renaming it would churn the destination side for nothing. The variants and the `action` tag are the contract's. Config gains `vzdump_path`, `exports_enabled`, `export_dir` and `export_ttl_secs`. Exports are a separate switch from installs because reading a guest's whole disk out to a file another machine may download is a different grant from writing an image in.
`run_command` moves out of `install.rs` verbatim. The export pipeline needs the same three things out of a child process that the install pipeline does: progress read off stdout as it arrives, a cancellation that actually kills the process, and a non-zero exit that carries the tool's own stderr back to the admin instead of a bare status code. Reaching into `install` for that from `export` would read badly, and a second copy would drift. No behaviour change.
`run_command` drained the child's stderr and then awaited that task before returning. Any grandchild inherits the stderr pipe, so `read_to_string` saw no EOF until the last of them exited: cancelling a tool that had spawned one left the job sitting in its running phase long after the cancel was acknowledged, which is the exact hang a cancel exists to end. A `vzdump` killed mid-dump readily leaves one behind, and the same was already latent for `qmrestore`. On an interrupt the stderr task is now dropped and the child reaped directly. A cancelled job's stderr was never used for anything.
`qm migrate` is intra-cluster only and these nodes cannot be clustered; `qm remote-migrate` is marked EXPERIMENTAL by Proxmox itself and documented nowhere. `vzdump` and `qmrestore` are stable and are already how Anchor installs an image, so migration is offline by construction. An export runs in the existing job registry rather than a parallel one: the same record, the same cancellation, and the same concurrency permit as an install, because a `vzdump` costs the node the same disk and the same I/O bandwidth as a `qmrestore`. It adds one phase, `Dumping`, and shares `Verifying` with installs, where on both sides it means a whole archive being read through sha256 in fixed-size chunks. A finished export flattens `artifact`, `sha256`, `size` and `path` into its status body, which is the contract's shape; an install's body is unchanged. A guest is refused unless this node's own config directory holds it. `vzdump` can only dump a guest on the node it runs on, so a VMID that exists elsewhere in the cluster is as un-exportable here as one that does not exist at all, and saying so plainly beats letting Proxmox report a missing file. Two dispositions, and only one is Anchor's to clean up. With no storage named, the archive goes into a per-artifact directory under `export_dir` that discard and the sweeper may remove wholesale. With one, `vzdump` writes into the admin's own dump storage and only the files this export produced are touched. The registry maps an opaque id to an absolute path recorded at export time. A request never contributes to a path, so traversal is not something to filter but something the shape of the lookup rules out; the id is validated as a hyphenated UUID regardless. Artifacts carry a TTL and are swept whether or not any migration claimed them, and the export root is cleared at startup, because the failure being designed against is a multi-gigabyte file nobody deletes.
`GET /artifacts/{artifact}` streams a finished export to the destination node,
which pulls directly over the tailnet rather than through the panel.
Range requests are supported because the install side already retries a
dropped download three times, and without resumption that is three attempts at
the same doomed transfer rather than three chances to finish one. Open-ended,
closed and suffix ranges are honoured; a range starting past the end is a 416
with `Content-Range: bytes */size`; anything unparseable or multi-range falls
back to the whole representation, which RFC 9110 permits.
The body is a stream of fixed-size buffers, so an archive measured in
gigabytes is never held in memory, and the stream stops at the length already
promised in `Content-Length` rather than running to the end of the file.
Authorisation follows the existing template-token pattern rather than
inventing a third claim shape. The token is checked before the id is resolved,
so a caller without one cannot learn which artifacts exist from the difference
between a 401 and a 404.
`DELETE /artifacts/{artifact}` is `Discard`. `Status` and `Cancel` are one pair
of commands over both kinds of job, so they get a shared
`/api/v1/templates/jobs/{id}` route; the install path stays exactly where it
was because the panel is already polling it.
The install pipeline finalises every restore with `qm template`, because the only thing it ever served was an image import and Convoy offers a guest only when it reports `template: 1`. A migration restores the tenant's own machine, so that step leaves the customer with a guest that cannot be started, after a migration that reported success. `finalize_as_template` defaults to true, so every existing import caller stays correct without changing what it sends, and a migration opts out explicitly. That default is also why the destination half now needs a capability of its own. An agent old enough to lack the field ignores it and templates the guest anyway, which is precisely the failure the panel has to be able to refuse in advance -- so `migration.install` is advertised alongside the install pipeline and the panel gates on it before it moves anyone's machine.
The unit sets `ProtectSystem=strict` and named no `ReadWritePaths`, so the whole filesystem was read-only to the agent -- and, because a child inherits the mount namespace, to `vzdump` and `qmrestore` as well. An install therefore downloaded its entire multi-gigabyte artifact and then failed with "Read-only file system (os error 30)"; an export failed before writing a byte. This predates the export work and broke template installs on any systemd-managed install; it surfaced now because this is the first time the pipeline ran against one rather than a hand-started binary.
`vzdump` refuses to start without /var/log/pve/tasks and reports the missing log rather than the archive, which reads as a backup failure. /run/lock is where it and `qmrestore` take their guest locks.
With `CapabilityBoundingSet=` empty, `vzdump` dies part-way through chowning
its own task log to www-data ("Operation not permitted") and then cannot
unlink the half-written log, so a backup fails naming a log file rather than
the archive. Five capabilities, each for a named reason, nothing broader.
Less of a reduction than it appears: the unit already runs the agent as
root:root with write access to /etc/pve, so an empty set was largely
decorative, and it silently broke template installs as well as migration.
The unit was written as though Anchor were an ordinary network daemon. It is not: it starts QEMU. `PrivateDevices=yes` gives a private /dev of pseudo devices, and `vzdump` needs real ones to read a guest's disk -- /dev/kvm first, which fails as "KVM virtualisation configured, but not available" and reads like a BIOS problem, then /dev/net/tun for the guest's NIC, with /dev/vhost-net and loop devices behind those depending on the storage. Binding them one at a time is a queue of identical bugs discovered one restore at a time, in a service that already runs as root:root with write access to /etc/pve. The filesystem, kernel and capability protections above are kept.
vzdump's default notification path reaches for PVE's notification system, which shells out to postdrop. On a hardened unit postdrop cannot write to the mail queue and retries indefinitely: the backup never finishes, the job sits in `dumping` forever, and the task log fills with mail-queue warnings instead of anything about the archive. `legacy-sendmail` with no `--mailto` is how vzdump is told to notify nobody. An export the panel drove is not an event an operator subscribed to.
Written as though Anchor were an ordinary network daemon, the unit restricted the very things Anchor exists to do. A child inherits every setting, so each applied to Proxmox's own tooling and not merely to the agent. Ten settings each broke it outright, all only at runtime and all reporting something other than the cause: a backup blaming a missing log file, a QEMU blaming the BIOS, a bridge saying only "interface activation failed". This predates migration -- template installs failed the same way, downloading a multi-gigabyte artifact before dying on a read-only filesystem. Rewritten in one pass rather than patched further: keep what costs nothing, drop what cannot coexist with running virtual machines, and record the reason beside each relaxation so the next reader does not restore it and rediscover the same ten bugs.
The destination downloads through the install pipeline, which fetches `spec.url` with a bare GET and sets no headers -- it was built to pull public CDN images and has nowhere to put a bearer token. Requiring the header meant a real migration failed with 401 while every header-based test passed; the URL is the only channel available, so a query token is the normal case here and the header is the convenience. The query string is a second door, not a weaker one: the token still names one artifact and one action, and a token for another artifact is refused either way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the source half of server migration, and fixes the packaging that was
quietly breaking template installs already.
Export
The destination side of a migration is the install pipeline that already exists:
fetch a URL, verify a sha256,
qmrestore. This adds the source side thatproduces such a URL.
Export,FetchandDiscardon the wire, besideInstall.vzdumpin the existing job registry (same record,cancellation and concurrency permit) and hashes the archive in 1 MiB chunks.
GET /artifacts/{id}streams in 64 KiB chunks, supports open, closed andsuffix ranges, answers
416withbytes */size, and setsETagto thesha256. The install side retries a dropped download three times, which is
worthless without resumption.
DELETE /artifacts/{id}, a TTL sweeper, and a clear of the export root atstartup, because the failure being designed against is a multi-gigabyte file
nobody deletes.
as the header: the install pipeline fetches
spec.urlwith a bare GET andsets no headers, so the URL is the only channel it has.
No new dependencies.
Two bugs this found in existing code
Cancelling a job could hang.
run_commandawaited the stderr drain beforereturning, and grandchildren inherit that pipe, so
read_to_stringsaw no EOFuntil they exited. This affected
qmrestoreon the install path too.The systemd unit restricted the things Anchor exists to do. It was written
as though this were an ordinary network daemon; a child inherits every setting,
so each applied to
vzdumpandqmrestorerather than just the agent. Tensettings each broke it outright, all only at runtime and all reporting something
other than the cause: a backup blaming a missing log file, a QEMU blaming the
BIOS, a bridge saying only "interface activation failed".
This predates migration. Template installs failed the same way on any
systemd-managed install, downloading a whole artifact before dying on a
read-only filesystem. The unit is rewritten in one pass: keep what costs
nothing, drop what cannot coexist with running virtual machines, and record the
reason beside each relaxation.
Also:
vzdump's default notification path shells out topostdrop, which on ahardened unit cannot write the mail queue and retries forever, so an export
never finished. It now notifies nobody.
Verification
136 tests,
cargo fmt --check,cargo clippy --all-targets -- -D warnings.Driven end to end against two live Proxmox 9.2.2 nodes with the Convoy panel:
three real migrations, both directions, each leaving the guest present on the
destination, gone from the source, and no artifact behind.