Script the CVM deploy, and write down what a cluster move costs

[HONGJICAI]

Two thirds of a release were already scripted and one third was not. release.yml produces the attested artifact, switch.sh moves traffic between two CVMs — and in between sat a dashboard form. That form is not a convenience gap: allowed_envs, the CVM name, and the public_logs / public_sysinfo / public_tcbinfo / secure_time flags all land in the measured app-compose, so every field in it is an unrecorded input to app_id, and two blue/green sides can only be compared if they were filled in identically.

deploy/phala/deploy.sh is that record, plus a README section on what moving GATEWAY_DOMAIN actually costs.

deploy.sh

cp deploy.env.example deploy.env               # once
./deploy.sh preflight --side b                 # checks only, changes nothing
./deploy.sh deploy --side b --release latest

• Takes a release's digest-pinned docker-compose.release.yml, checked against the sha256 the release publishes (the API's asset digest, falling back to the notes), and refuses any gateway-image line still on a tag — including cvm-identity, which runs as root with the guest-agent socket mounted, and which must carry the same digest as gateway.
• Hands the CLI a mode-600 env file, not -e KEY=VALUE: an argv is world-readable in ps for the life of the call and four of these values are secrets.
• Reads back the two DNS records whose absence fails silently — the serving alias must name the cluster being deployed to, and the issuance switch must already point at this side (otherwise the side's first ACME run burns the 5-failed-validations-per-hour budget for the hostname, blocking the live side's renewal too).
• Waits on /readyz by app-id (<app_id>-443s.<base>), so the new CVM is verified before anything points at it. A cold side has to finish its first warmer sweep, so that window is minutes.
• Defaults: public_logs and public_sysinfo off (this CVM handles sealed prompts, and anything a container logs is disclosed), public_tcbinfo on (pcverify fetches app-compose.json through it), never --dev-os (it wants an SSH key, i.e. interactive root inside the enclave).
• Never moves traffic and never edits DNS — it only reads it. Cutting over stays switch.sh's separate decision.

It also corrects the CLI: phala cvm create is gone in favour of phala deploy, and --vcpu/--memory in favour of --instance-type. INSTANCE_TYPE is deliberately left with no default, because GOMEMLIMIT=24GiB in the compose was derived by hand from a 32 GiB CVM and nothing in the manifest can check the shape.

Two facts about the CVM name, which are independent

It is measured. dstack's AppCompose carries a name field and app_id is that manifest's hash, so the name is inside app_id — the live deployment's own pcverify output prints app name 0g-pc-gateway-a-1. It is not a label to be tidied later: a new name is a new app_id, i.e. a new CVM and a re-audit. So the default is 0g-pc-gateway-<ZG_PROM_ENV>-<side>; without the env, a staging CVM and a mainnet one on the same build would hash to one app_id, since DOMAIN and the router URL are ${…} references whose values are never measured.

This corrects blue-green.md, which said the image digest was the only side-distinguishing field in the measurement. That was true of the compose text and false of the manifest app_id hashes.

It is not an identifier. --cvm-id's help says it takes "UUID, app_id, instance_id, or name", but the CLI passes the value straight into the cvms/{id} path and its SDK schemas accept only id / uuid / app_id / instance_id — any name resolution is the API's, undocumented, and unverified. So names are resolved against phala cvms list and every later call uses the platform id, which the deploy output prints beside app_id.

The env separation

deploy.env feeds two different environments and they no longer bleed into each other:

• toward the CVM it was already an allowlist — ten names, and nothing else can reach a container, because the compose interpolates a fixed set of ${…} references and allowed_envs is measured;
• toward child processes it used to be export everything (inherited from switch.sh). That is fine for curl and openssl; it is not fine for the phala CLI, which has an env contract of its own. The Cloudflare token and the Prometheus password sat in the environment of every child, and a DEBUG=1 in deploy.env would have reached the CLI, where DEBUG=phala::api-client prints every HTTP request body.

Now four names a child genuinely needs are exported and the rest are plain shell variables. The script also warns on the two things a config file looks like it can do and cannot: a ZG_GATEWAY_* the compose does not reference (every other gateway setting is literal, measured text) and DEBUG.

Moving to another dstack cluster (README)

Changing GATEWAY_DOMAIN reads like a one-line edit. Three of the things it moves are verification inputs, so they get settled on a throwaway CVM before a cutover — the failure otherwise lands on users running pcverify, not on us:

1. the new cluster's OS image needs an osimages.json entry, or the os-image check fails for every user (that entry is Pin dstack-0.5.9 in the OS-image allowlist #100 for dstack-0.5.9);
2. its mr_config_id must still be the V1 layout, or compose_hash cannot be read out of a quote at all — a code change, not a redeploy;
3. app_id must still be compose_hash's leading 20 bytes, or the record model and the standby probe stop meaning what they say.

Plus: the SNI allowlist is per-cluster and fails as a bare TLS error with DNS and certificates looking perfect; the serving alias names the cluster and both sides share it, so blue and green cannot straddle two clusters and a cluster move is a cutover with a DNS-shaped gap; certificates start over from an empty volume against a per-hostname rate limit; and every enforced check is fail-closed, so a blocked egress host looks exactly like a CVM that never becomes ready.

Self-review

A review of the branch found eight things, all fixed in the last commit — the ones worth naming here:

• a failed phala cvms list was indistinguishable from a free name, so an API blip would have waved a duplicate of a measured name through the collision guard;
• under --allow-duplicate-name the resolver returned the first match — usually the CVM already there — so the fallback could have probed the old healthy CVM and pronounced the new one ready;
• the measured booleans were compared against literal true with nothing rejecting a near miss: PUBLIC_TCBINFO=True sent --no-public-tcbinfo while the plan printed True;
• verify and status were documented in forms that always die (the first needs to know which manifest to compare byte-for-byte, the second needs --side).

Every path was exercised against a stub CLI: preflight, dry-run, deploy, the not-ready failure (exit 1, and it says the CVM is still running), name collision, ambiguous name, list failure, bad boolean, floating tag, wrong-side issuance, cluster mismatch, missing config. pcverify's flag combination used by verify was confirmed to be accepted (exit 1 on an unreachable host, not exit 2 for a caller mistake).

Not exercised against the real platform. Nobody has run this against Phala Cloud yet, so two things want checking on the first real run: where phala deploy --json puts app_id (the script scans for any 40-hex value and falls back to cvms get), and whether INSTANCE_TYPE on the new cluster can actually give the 32 GiB the compose's GOMEMLIMIT assumes.

---

Generated by Claude Code