[andypalmi]

Builds on #362, which is now merged; main is merged into this branch.

Problem

A live T4 Spot test (n1-standard-4, europe-west4-b) on the default image family common-cu129-ubuntu-2204-nvidia-580 failed at boot with amicu: docker is not installed and Script "startup-script" failed with error: exit status 1. The Deep Learning VM image has the NVIDIA 580 driver but not Docker or the NVIDIA container toolkit. amicu apply did not notice. It kept polling /health for 20+ minutes after the script had exited.

Approach: install at boot on the Deep Learning VM image

We keep the Deep Learning VM image and install the missing tools from the startup script. Reasons:

• The driver is the slow and fragile part, and this image already has it. It also has python3, curl and gpg, which the Secret Manager fetch and the key check use.
• Container-Optimized OS would mean installing the driver at every boot with cos-extensions, mounting the GPU devices and libraries by hand instead of --gpus all, a read-only /opt, no python3, and a rewrite of the script.
• A pip install of vLLM is slower than the image pull, and its versions drift.
• The Ubuntu accelerator images also ship only the driver.

What the script does now:

• Install step (install Docker and the NVIDIA container toolkit). It adds Docker's apt repository (docker-ce docker-ce-cli containerd.io) and NVIDIA's libnvidia-container repository (nvidia-container-toolkit) for whichever tool is missing.
• Skip and repair. The step is skipped only when three things hold: the marker /var/lib/amicu/tools-ready (written after the restart) exists, systemctl is-active docker succeeds, and docker info lists the nvidia runtime. Otherwise it first runs dpkg --configure -a, retried for 10 minutes while the lock is held, so an install cut short by a reboot or preemption is finished.
• Keys. trust() downloads each key with curl --retry 5 --retry-connrefused and imports it into a scratch GNUPGHOME. It then exports only the pinned fingerprint to /etc/apt/keyrings/<name>.gpg, so apt trusts exactly the checked key:
  • Docker: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
  • NVIDIA: C95B321B61E88C1809C4F759DDCAE044F796ECB0
• apt. apt-get update errors from unrelated repositories on the image are tolerated. What must hold is that apt-cache policy shows a candidate for every needed package (3 tries). apt waits up to 10 minutes for the dpkg lock, because unattended-upgrades holds it on first boot.
• Runtime setup. Then nvidia-ctk runtime configure --runtime=docker and systemctl restart docker. The step checks that Docker lists the nvidia runtime before writing the marker.
• Steps and readiness. The image pull is its own step, with retries. After the container starts, the script waits on the VM for /health. It fails only if the container stops or restarts 3 times, and prints docker logs --tail 50. A slow model load is left to apply's own deadline, and the script has no deadline of its own.
• Unchanged: secrets still come from Secret Manager at boot into tmpfs (or the metadata fallback), TLS with the pinned certificate, the auth middleware, the port binding and the idle watchdog.
• Arm / A4X: the install is arch-neutral (dpkg --print-architecture, NVIDIA's $(ARCH) repository). A4X is still refused, but A4X_UNSUPPORTED, the placement error and the docs no longer say its images lack Docker. They now say amicu is untested on the Grace image and the arm64 vLLM container.

Fail fast

• Reporting from the VM. The script sets enable-guest-attributes=TRUE in instance metadata. At the start of every boot it writes amicu/status=booting and amicu/boot_id, so a ready from an earlier boot does not survive a restart. An EXIT trap writes amicu/detail, the last 4 KB of the script's own log, and then amicu/status=failed:<step>. On success it writes amicu/status=ready.
• Polling. While /health fails, wait_ready reads getGuestAttributes?queryPath=amicu/, at most every 15 s. Compute Engine allows 10 queries per minute per VM.
• On failed:<step> it returns GcpError::StartupFailed, with the step and the last 25 log lines.
• No dead serial hint. The error no longer suggests the serial console, because apply then deletes the VM. The gcloud compute instances get-serial-port-output <vm> --zone <zone> --project <project> command is printed only when that delete failed. That note also says the VM may still be running and billing in its zone, and that amicu destroy --target <dep> retries.
• If the report cannot be read, the wait falls back to the /health deadline. If the VM reports ready but the probe never gets through, the timeout names the firewall and allowed_source_cidrs.
• Preemption. If a Spot VM disappears during the wait, the zone's compute.instances.preempted operations are checked, matched by the instance id seen while it ran. The error then says it was preempted and suggests re-running amicu apply or provisioning = "standard", instead of "deleted".
• No orphans. The stop record is written to state before anything is created. A failed start deletes what this run created and keeps the record until a later apply or destroy confirms it is gone.

Client snippets for a VM

Plain curl fails on the pinned self-signed certificate, and --cacert also fails (exit 60) because the certificate names no host. apply now prints:

• the certificate's public key pin (base64 SHA-256 of its SubjectPublicKeyInfo), next to the existing fingerprint. It is read from the certificate file only when that file matches the recorded fingerprint.
• curl -k --pinnedpubkey 'sha256//<pin>' .... -k skips only the CA and host name checks, and the pin still refuses any other key.
• Python: ssl.create_default_context(cafile=<cert>) with check_hostname = False, passed as httpx.Client(verify=tls) to OpenAI(http_client=...). This still verifies the chain against that one certificate.

Checked locally against openssl s_server with an amicu-generated certificate:

• Python 3.12, and 3.13 with its strict X.509 flags, accept the right certificate and refuse another.
• httpx gives 200 with the right certificate and a ConnectError with another.
• curl returns 200 with the right pin and exits 90 with a wrong one.

Boot time

The live T4 Spot run on the first commit took 8m53s from apply to ready.

Tests

• startup.rs (content checks on the generated script):
  • the skip condition (marker, Docker active, nvidia runtime) and dpkg --configure -a before the install
  • import, then export by fingerprint, then the check, then install into keyrings/*.gpg, and --retry-connrefused
  • tolerant apt-get update with the candidate check
  • the order install, configure, restart, marker, pull
  • booting and boot_id written before the first step
  • the /health wait with no fixed limit
  • bash -n on every variant
• health.rs:
  • a startup failure ends the wait with the step and log, and no gcloud hint
  • guest attributes are read every 15 s, and a 403 is ignored
  • ready but unreachable gives the firewall hint
  • a Spot VM gone after a preemption with its id gets the preempted message; a preemption of another id, or unreadable operations, gives "deleted"
• tls.rs: the public key pin equals the hash of the key's SubjectPublicKeyInfo.
• amicu-cli:
  • the pinned snippets (curl pin, Python ssl context, a note when the certificate file is missing), and the printed pin in the end-to-end apply test
  • a failed install step stops apply at once without the serial hint, and destroy releases the record
  • with a VM delete denied, the error says it may be billing and gives the serial command with zone and project
• Runs:
  • ~/.claude/amicu-cargo.sh +stable clippy --workspace --all-targets -- -D warnings: clean
  • ~/.claude/amicu-cargo.sh test -p amicu-gcp -p amicu-cli: amicu-gcp 174 passed; amicu-cli 517 passed, 1 ignored
  • cargo fmt --check: clean