gce-hunter

A command-line tool that relentlessly tries to create a Google Compute Engine VM — typically with a GPU attached — until capacity is found.

On-demand GPUs are frequently out of stock. Creating a VM fails with a capacity error, and a stopped GPU VM often cannot be restarted because its zone ran dry in the meantime. gce-hunter automates both problems:

• create hunts across every candidate zone, over and over, until one of them has capacity for your VM.
• start detects the "stopped VM whose zone ran dry" situation and offers to hunt a replacement VM that boots from a snapshot of your current disk — in any zone — so none of your environment setup is lost.

Every cloud action shells out to the gcloud CLI, and every command line is echoed to your terminal prefixed with $ before it runs, so you always see exactly what is being done on your behalf.

Prerequisite: the gcloud CLI

gce-hunter drives the official Google Cloud CLI and never manages credentials itself. Before the first run:

1. Install the Google Cloud SDK
2. gcloud auth login
3. Make sure the Compute Engine API is enabled on your project (the first gcloud compute … call will print an activation link if it is not).

Every gce-hunter command starts with a preflight: it locates gcloud (on your PATH, or the executable named in GCE_HUNTER_GCLOUD; on Windows the SDK's gcloud.cmd shim is found too) and checks that an account is active (gcloud config get-value account). If either check fails, the command stops and tells you the fix. Credentials that expire in the middle of a hunt abort it with a gcloud auth login hint.

Installation

One line — macOS, Linux, Windows (WSL), Chromebook

curl -fsSL https://raw.githubusercontent.com/j-about/gce-hunter/master/install.sh | sh

install.sh detects your OS and CPU, downloads the newest vX.Y.Z release, verifies its SHA-256 against the release's SHA256SUMS file, and installs gce-hunter into /usr/local/bin — asking for your password only if that directory is not writable. It ends by running gce-hunter --version and warns you if /usr/local/bin is not on your PATH or if gcloud cannot be found.

• Windows: run the line inside your Ubuntu (WSL) terminal — gce-hunter must live where your gcloud lives. For a gcloud installed on Windows itself, see the PowerShell steps below.
• Chromebook (Linux/Crostini terminal): run sudo apt-get install -y curl first if curl is missing.
• Linux gets the fully static x86_64-unknown-linux-musl build (works on any distro) or aarch64-unknown-linux-gnu on ARM machines.

Variable	Effect
GCE_HUNTER_VERSION=v0.1.0	install this exact release instead of the newest
GCE_HUNTER_INSTALL_DIR=$HOME/.local/bin	install somewhere else (no password needed)
GCE_HUNTER_BASE_URL=https://…	download the archives from a mirror instead of GitHub

Uninstall with sudo rm /usr/local/bin/gce-hunter; the state file can be deleted too.

Windows (PowerShell, gcloud installed on Windows)

Paste into PowerShell:

$tag = (Invoke-RestMethod "https://api.github.com/repos/j-about/gce-hunter/releases?per_page=100" |
        Where-Object { $_.tag_name -match '^v\d+\.\d+\.\d+$' } | Select-Object -First 1).tag_name
$name = "gce-hunter-$tag-x86_64-pc-windows-msvc"
Invoke-WebRequest "https://github.com/j-about/gce-hunter/releases/download/$tag/$name.zip" -OutFile "$env:TEMP\$name.zip"
Expand-Archive "$env:TEMP\$name.zip" -DestinationPath $env:TEMP -Force
$dir = "$env:LOCALAPPDATA\Programs\gce-hunter"
New-Item -ItemType Directory -Force $dir | Out-Null
Copy-Item "$env:TEMP\$name\gce-hunter.exe" $dir -Force
$path = [Environment]::GetEnvironmentVariable("Path", "User")
if ($path -notlike "*$dir*") { [Environment]::SetEnvironmentVariable("Path", "$path;$dir", "User") }

Then open a new terminal and run gce-hunter --version.

Manual download

Each release on the releases page ships one archive per target, named gce-hunter-<tag>-<target>.tar.gz (.zip on Windows). It unpacks to a folder of the same name holding the gce-hunter binary, this README and the LICENSE; put the binary somewhere on your PATH. A SHA256SUMS file lists the hash of every archive: compare it with sha256sum <archive> (Linux) or shasum -a 256 <archive> (macOS).

Platform	Target
Linux x86_64, fully static — any distro, Chromebook (Crostini)	x86_64-unknown-linux-musl
Linux x86_64, glibc ≥ 2.35	x86_64-unknown-linux-gnu
Linux ARM64, glibc ≥ 2.35	aarch64-unknown-linux-gnu
macOS Intel	x86_64-apple-darwin
macOS Apple Silicon	aarch64-apple-darwin
Windows x86_64	x86_64-pc-windows-msvc

From source

cargo install --path . --bin gce-hunter

--bin is needed because the crate also builds mock-gcloud, the test double used by the integration tests.

Usage

Interactive or scripted

Every create parameter can be passed as a flag. When both stdin and stderr are terminals, anything you omit is asked interactively. Otherwise — in scripts, or with the output redirected — nothing ever prompts: --gpu-count, --max-attempts and --delay fall back to their defaults, an absent --region or --gpu-type means "all zones" / "no GPU", and every other missing parameter is reported in one error listing all the missing flags. Prompt suggestions never apply silently in scripts: they decide what the VM costs.

The interactive prompts, in order:

1. Google Cloud project — a Select over gcloud projects list, starting on your active gcloud project (free-text entry if listing fails).
2. Region filter — no default; empty hunts worldwide.
3. Attach a GPU? — defaults to Yes, then the GPU type (no default; list them with gcloud compute accelerator-types list) and count (1).
4. Machine type — a common default is prefilled. Each GPU type is only available on certain machine families: see Google's GPU machine types page, or gcloud compute machine-types list.
5. Image family — no default — and image project (prefilled according to the GPU choice). Browse the public images with gcloud compute images list (--project <image-project> --no-standard-images for a specific project); Google documents its public images and, for GPU-ready images, the Deep Learning VM images.
6. Boot disk size (suggested 100 GB, minimum 10) and type (pd-balanced).
7. Instance name — suggested <user>-vm from your gcloud account.

create — hunt a new VM

# Fully interactive:
gce-hunter create

# Fully scripted — a GPU VM hunted across every European zone that offers
# the GPU (fill in the values from Google's lists, see above):
gce-hunter create \
  --project my-project \
  --region europe \
  --gpu-type <gpu-type> \
  --machine-type <machine-type> \
  --image-family <image-family> \
  --image-project <image-project> \
  --boot-disk-size 100 \
  --boot-disk-type pd-balanced \
  --name my-vm

Flag	Interactive	In scripts
-p, --project	Select over your projects	required
-r, --region	asked, no default	optional; --region "" also means no filter
-g, --gpu-type	"Attach a GPU?" then type	optional; absent = CPU-only VM
--gpu-count	1	1; needs --gpu-type; at least 1
-m, --machine-type	asked, a default is prefilled	required
--image-family	asked, no default	required
--image-project	asked, prefilled according to the GPU choice	required
--boot-disk-size	suggested 100 (GB)	required; at least 10
--boot-disk-type	suggested pd-balanced	required
--name	suggested <user>-vm	required; lowercase letter first, then letters, digits or -, max 63
-n, --max-attempts	0 = keep hunting forever	same
-d, --delay	120 seconds between passes	same; at least 1

--region is a zone-name prefix at any granularity — europe, europe-west, europe-west1, europe-west1-b — with a numeric boundary: europe-west1 does not match europe-west10-a.

gce-hunter manages one VM at a time: create refuses to run while a VM is recorded in the state file (delete clears it). GPU VMs are always created with --maintenance-policy=TERMINATE (mandatory for GPU instances) and --metadata=install-nvidia-driver=True (Deep Learning VM images install the NVIDIA driver on first boot when it is not preinstalled).

How the hunt works. gce-hunter first asks gcloud which zones can host your request (zones offering the GPU type, or all UP zones for CPU-only VMs) and applies the --region prefix filter. Then it sweeps: one pass tries every candidate zone once, in random order, with a 2-second courtesy delay between tries. Each failed create is classified from gcloud's error text:

• Retried next pass — out of stock, rate-limited (HTTP 429), transient backend errors (5xx).
• Zone skipped for the rest of the hunt — the machine type or accelerator is not offered there; the location is not accessible to your project (allowlist-only regions such as me-central2, or an organization policy).
• Fatal, hunt aborted with the verbatim gcloud error plus a targeted hint — quota exceeded (GPUS_ALL_REGIONS defaults to 0 on new projects: request an increase in the Google Cloud console, IAM & Admin → Quotas), permission or billing problems, expired credentials, image family not found, invalid parameters, an instance of that name already existing.
• Unrecognized errors are retried, but the same one three times in a row aborts the hunt rather than looping forever.

Between passes it waits --delay seconds — a live countdown on a terminal, a single log line in scripts. The hunt ends with one of: success (the VM is recorded locally and a summary with the gcloud compute ssh command is printed); no zone in the selected scope can host this machine/GPU combination (every candidate zone was skipped — widen or drop --region, or change the machine or GPU type); no capacity found after N passes when --max-attempts is set (re-run the same command to keep hunting); or Ctrl-C (exit 130, with a warning if a create call was in flight and may have produced an instance server-side).

start, stop, status, delete — manage the recorded VM

The lifecycle commands take no targeting flags: they operate on the single VM recorded by the last successful create (managing one VM at a time is an explicit v1 scope decision).

gce-hunter stop     # warns that a stopped GPU VM may be unable to restart
gce-hunter start    # starts it again — or offers disk recovery (see below)
gce-hunter status   # recorded state (and its file path) + live status and external IP
gce-hunter delete   # deletes the VM and its boot disk; type its name to confirm

-y, --yes (global) skips confirmation prompts — it never fills in missing values. In scripts delete refuses to run without it, and start refuses to begin disk recovery without it. If the VM was deleted outside gce-hunter, status reports it as NOT FOUND and delete just clears the stale state.

Disk recovery: when start finds the zone dry

If start fails with a capacity error, your VM is intact but its zone has no room for it right now. gce-hunter then offers to:

1. Snapshot the stopped VM's boot disk as gce-hunter-rec-<timestamp> (the VM is not modified — snapshots are global, so any zone can use it). The exact command to remove the snapshot by hand is printed right away.
2. Hunt a replacement VM booted from that snapshot, with the same machine type and GPU as the stopped VM (a different one could be incompatible with what is installed on the disk). You are asked for the region filter, prefilled with the recorded one: press Enter to keep it, shorten it to widen the hunt — the typical move, the original zone may stay dry for a while — or delete it to hunt worldwide (start accepts -n/--max-attempts and -d/--delay for this hunt; scripts reuse the recorded filter silently).
3. Only after the replacement VM is RUNNING and recorded: delete the old VM (its boot disk auto-deletes with it) and the snapshot.

The old VM and its disk are never touched before the replacement is up. On any failure or Ctrl-C the old VM stays intact and the snapshot is deleted best-effort — if that cleanup fails, the exact gcloud command to remove it is printed. If the state file cannot be written once the replacement is running, nothing is deleted: both VMs are listed with the commands to clean up by hand. The replacement gets an incremented name (my-vm → my-vm-2 → my-vm-3).

On a gcloud too old to accept --source-snapshot when creating an instance, the hunt switches to pre-creating a boot disk from the snapshot in each zone it tries (removing it again when the zone fails) and suggests gcloud components update.

The state file

The recorded VM lives in state.json under the platform config directory: ~/.config/gce-hunter/ on Linux and ChromeOS, ~/Library/Application Support/gce-hunter/ on macOS, %APPDATA%\gce-hunter\config\ on Windows. create writes it, delete removes it, status prints its path. Deleting the file by hand simply makes gce-hunter forget the VM.

Exit codes

0 success · 1 runtime error · 2 command-line usage error · 130 interrupted with Ctrl-C.

The wagon branch

The wagon branch is a specialization of this tool for learners of Le Wagon's Data Science & AI bootcamp: same engine, but with curriculum-aligned choices baked in — a fixed GPU priority order, one pinned image, a fixed disk and naming scheme — so a learner cannot misconfigure their VM. Learners choose only their project and an optional region filter, and install with the same one-line installer (pointed at the wagon branch).

Development

cargo test          # unit + integration tests (gcloud is always mocked)
cargo clippy --all-targets -- -D warnings
cargo fmt --check

The integration tests run the real binary against a scripted mock-gcloud double; no test ever contacts Google Cloud. CI runs the lints and the tests on Linux, macOS and Windows. Internals used by the tests: GCE_HUNTER_GCLOUD (path of the gcloud executable to use — also handy for exotic installations), GCE_HUNTER_CONFIG_DIR (state-file directory), GCE_HUNTER_DELAY_OVERRIDE, GCE_HUNTER_ZONE_DELAY_MS, GCE_HUNTER_SEED.

Releasing. Pushing a tag builds the six targets and publishes the archives plus SHA256SUMS as a GitHub release. Tags are vX.Y.Z on master and vX.Y.Z-wagon on wagon — both branches share the releases page, and each branch's install.sh only ever picks tags of its own channel. To test the installer without a release, serve a fake one locally (<dir>/<tag>/gce-hunter-<tag>-<target>.tar.gz + SHA256SUMS, e.g. with python3 -m http.server) and run it with GCE_HUNTER_BASE_URL, GCE_HUNTER_VERSION and a temporary GCE_HUNTER_INSTALL_DIR.

License

MIT — see LICENSE.