From 750099e71af18d3f6f1c97d1c5afbf6b47ea03de Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 15:48:50 -0400 Subject: [PATCH 01/18] docs(flash): move the flash lane to the Runpod SDK 2.0 (rp flash) Flash ships in runpod 2.0 as SDK apps (runpod.App, @app.queue/@app.api/@app.task) with the rp CLI, and runpod-flash is deprecated on October 29, 2026. - flash skill: SDK-first SKILL.md, setup/CLI, API and patterns references checked against runpod 2.0.0rc1 (--help, source, sdk-runtime) - new references: migrate-from-runpod-flash.md, runpod-flash-legacy.md - examples/: five single-file apps, each passing rp flash deploy --build-only - evals rewritten for the SDK, plus a migration eval; dev-loop fixture now fails on a missing dependency (the body-only NameError no longer exists) - flash concept rules rewritten with public docs/source evidence; citing rules in flashboot, job, model-store, runpodctl, serverless-handler and volume-cache updated - router, READMEs, getting-started and endpoint/pod references use rp - golden paths 03-B, 07, 08 ported to the SDK and marked ported (build-only checked; the live runs used runpod-flash) --- AGENTS.md | 2 +- README.md | 4 +- plugins/runpod/README.md | 6 +- plugins/runpod/skills/flash/SKILL.md | 339 ++++++++---------- .../flash/evals/client-external-image.eval.md | 28 +- .../evals/connect-existing-endpoint.eval.md | 24 +- .../flash/evals/cpu-gpu-pipeline.eval.md | 26 +- .../flash/evals/dev-loop-iteration.eval.md | 70 ++-- .../flash/evals/fixtures/dev-loop/main.py | 30 +- .../flash/evals/lb-multi-route-api.eval.md | 26 +- .../migrate-runpod-flash-project.eval.md | 35 ++ .../flash/evals/qb-gpu-function.eval.md | 24 +- .../evals/volume-cache-model-load.eval.md | 35 +- .../runpod/skills/flash/examples/README.md | 25 ++ .../flash/examples/api_model_service.py | 40 +++ .../skills/flash/examples/call_deployed.py | 28 ++ .../skills/flash/examples/cpu_gpu_pipeline.py | 37 ++ .../runpod/skills/flash/examples/queue_gpu.py | 28 ++ .../skills/flash/examples/train_then_serve.py | 50 +++ plugins/runpod/skills/flash/reference/api.md | 254 +++++++------ .../reference/migrate-from-runpod-flash.md | 153 ++++++++ .../runpod/skills/flash/reference/patterns.md | 262 +++++++++----- .../flash/reference/runpod-flash-legacy.md | 93 +++++ .../skills/flash/reference/setup-and-cli.md | 169 +++++---- .../skills/runpod-usage/concepts/flash.yaml | 183 +++++++--- .../runpod-usage/concepts/flashboot.yaml | 10 +- .../skills/runpod-usage/concepts/job.yaml | 2 +- .../runpod-usage/concepts/model-store.yaml | 22 +- .../runpod-usage/concepts/runpodctl.yaml | 6 +- .../concepts/serverless-handler.yaml | 4 +- .../runpod-usage/concepts/volume-cache.yaml | 11 +- .../reference/development-loop.md | 2 +- .../reference/endpoint-workflows.md | 8 +- .../runpod-usage/reference/getting-started.md | 6 +- .../runpod-usage/reference/pod-workflows.md | 2 +- plugins/runpod/skills/runpod/SKILL.md | 14 +- .../03-whisper-endpoint/README.md | 4 +- .../03-whisper-endpoint/variant-b-flash.md | 157 ++++---- .../golden-paths/07-network-volume-handoff.md | 72 ++-- .../golden-paths/08-finetune-to-serverless.md | 82 +++-- .../skills/runpod/golden-paths/README.md | 9 +- .../runpod/skills/serverless-deploy/SKILL.md | 2 +- 42 files changed, 1546 insertions(+), 838 deletions(-) create mode 100644 plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md create mode 100644 plugins/runpod/skills/flash/examples/README.md create mode 100644 plugins/runpod/skills/flash/examples/api_model_service.py create mode 100644 plugins/runpod/skills/flash/examples/call_deployed.py create mode 100644 plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py create mode 100644 plugins/runpod/skills/flash/examples/queue_gpu.py create mode 100644 plugins/runpod/skills/flash/examples/train_then_serve.py create mode 100644 plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md create mode 100644 plugins/runpod/skills/flash/reference/runpod-flash-legacy.md diff --git a/AGENTS.md b/AGENTS.md index 2d0f7ed9..4bd87e34 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ lane is unclear, then follows its decision table into a lane's `SKILL.md`. skills/runpod/ router / entrypoint — decides the lane skills/runpod-mcp/ manage infra via the Runpod MCP server (structured tool calls) skills/runpodctl/ manage infra via the CLI (+ Hub, file transfer, SSH, doctor) -skills/flash/ write & deploy your own code on Runpod serverless (@remote) +skills/flash/ write & deploy your own code on Runpod serverless (Runpod SDK apps, rp flash) skills/companion-clis/ prerequisite CLIs (hf, gh, docker, aws) skills/runpod-usage/ conceptual knowledge ("how Runpod works") — not a tool reference/*.md detailed topics, loaded on demand diff --git a/README.md b/README.md index cdcde6ae..e9de6a38 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Gemini, opencode, +more). The first line is all you need; drop any others you wo ```bash npx skills add runpod/runpod-plugins-official # the skills — any agent (essential) curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl) -uv tool install runpod-flash # flash — deploy your own code (optional; or: pip install runpod-flash) +pip install "runpod>=2.0.0rc1" # Runpod SDK + rp CLI — deploy your own code (optional; flash lane) npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth (optional) ``` @@ -180,7 +180,7 @@ Start with the **`runpod`** router; it points at the right lane. | [`runpod`](plugins/runpod/skills/runpod/SKILL.md) | Router / entrypoint. Start here when the right skill is unclear. | | [`runpod-mcp`](plugins/runpod/skills/runpod-mcp/SKILL.md) | Manage infra via the Runpod **MCP server**'s structured tool calls. | | [`runpodctl`](plugins/runpod/skills/runpodctl/SKILL.md) | Manage infra from the **CLI**, plus Hub, file transfer, SSH, `doctor`. | -| [`flash`](plugins/runpod/skills/flash/SKILL.md) | **Write and deploy your own code** on Runpod serverless (`@remote`). | +| [`flash`](plugins/runpod/skills/flash/SKILL.md) | **Write and deploy your own code** on Runpod serverless with the Runpod SDK (`runpod.App`, `rp flash`). | | [`companion-clis`](plugins/runpod/skills/companion-clis/SKILL.md) | Prerequisite CLIs: `hf`, `docker`, `gh`, `aws`. | | [`runpod-usage`](plugins/runpod/skills/runpod-usage/SKILL.md) | **Concepts** — pods/serverless, containers, storage, GPU selection, gotchas. | | [`runpod-templates`](plugins/runpod/skills/runpod-templates/SKILL.md) | **Official pod templates** — what each image ships, ports/paths, readiness, gotchas. | diff --git a/plugins/runpod/README.md b/plugins/runpod/README.md index e4d5c8e5..881e4b4b 100644 --- a/plugins/runpod/README.md +++ b/plugins/runpod/README.md @@ -17,7 +17,7 @@ lane below. If you already know the lane, go straight to it. | [`runpod`](skills/runpod/SKILL.md) | Router / entrypoint. Start here when the right skill is unclear. | | [`runpod-mcp`](skills/runpod-mcp/SKILL.md) | Manage infra (pods, endpoints, jobs, templates, volumes, catalog, billing) via the Runpod **MCP server**'s structured tool calls. | | [`runpodctl`](skills/runpodctl/SKILL.md) | Manage infra from the **CLI**, plus Hub deploys, file transfer (`send`/`receive`), SSH, and `doctor` setup. | -| [`flash`](skills/flash/SKILL.md) | **Write and deploy your own code** on Runpod serverless — `@remote`/`@Endpoint`, `flash dev`, `flash deploy`. | +| [`flash`](skills/flash/SKILL.md) | **Write and deploy your own code** on Runpod serverless with the Runpod SDK — `@app.queue`/`@app.api`/`@app.task`, `rp flash dev`, `rp flash deploy`; migrates legacy `runpod-flash` projects. | | [`companion-clis`](skills/companion-clis/SKILL.md) | Prerequisite CLIs: `hf` (models), `docker` (images), `gh` (repos/releases), `aws` (S3 to volumes). | | [`runpod-usage`](skills/runpod-usage/SKILL.md) | **Concepts** — how pods/serverless work, building containers, storage, GPU selection, gotchas. | | [`runpod-templates`](skills/runpod-templates/SKILL.md) | **Official pod templates** — what each image ships, ports/paths, readiness/runtime gotchas, version pinning; ComfyUI, PyTorch, Ubuntu. | @@ -123,7 +123,7 @@ For one-off actions, just ask — the router picks the lane: - "Create a pod with an RTX 4090" - "Deploy a serverless endpoint from this image" - "Which GPU should I use for a 13B model?" -- "Write an `@remote` function and run it on a GPU" +- "Write a Python function and run it on a GPU with the Runpod SDK" - "Download a model, containerize it, and deploy it" - "Find the missing model links in this ComfyUI workflow and download the verified files to my Pod" @@ -156,7 +156,7 @@ skills/ golden-paths/ worked end-to-end reference tasks (indexed by the router) runpod-mcp/ Runpod MCP server (structured tool calls) runpodctl/ Runpod CLI (+ Hub, transfer, SSH, doctor) - flash/ write & deploy your own code (@remote) + flash/ write & deploy your own code (Runpod SDK apps) companion-clis/ hf / gh / docker / aws prerequisites runpod-usage/ concepts + reference/*.md runpod-templates/ official pod templates + repair guide + scripts diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index bdc22345..a36d3fc3 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -1,234 +1,195 @@ --- name: flash description: >- - Write and deploy code-first Runpod Serverless with runpod-flash: @Endpoint/@remote, - flash dev, resource configuration, and debugging. Use runpod-build-template for - explicit Dockerfile/template project authoring, and runpodctl/runpod-mcp for - infrastructure management. + Code-first serverless with the Runpod Python SDK (`runpod` 2.0): define a + `runpod.App` with `@app.queue` / `@app.api` / `@app.task`, iterate on remote + GPUs/CPUs with `rp flash dev`, ship with `rp flash deploy`. Also covers + maintaining and migrating legacy `runpod-flash` projects (`from runpod_flash + import Endpoint`). Use runpod-build-template for explicit Dockerfile/template + project authoring, and runpodctl/runpod-mcp for infrastructure management. user-invocable: true metadata: author: runpod version: "1.8.0" # x-release-please-version - concepts: [flash, serverless-endpoint, serverless-handler, worker, volume-cache] + concepts: [flash, serverless-endpoint, serverless-handler, worker, volume-cache, model-store] license: Apache-2.0 --- -# Runpod Flash +# Runpod SDK apps (Flash) For an explicit Dockerfile/template project request, use [runpod-build-template](../runpod-build-template/SKILL.md). Do not replace that deliverable with Flash unless the user chooses its code-first workflow. -Write code locally, iterate with `flash dev` — it runs your functions on remote Runpod GPUs/CPUs with hot-reload and live worker logs — then `flash deploy` to ship. `Endpoint` handles provisioning. +Write Python locally, run it on remote Runpod GPUs/CPUs with `rp flash dev`, then +`rp flash deploy` to ship. Flash is now part of the `runpod` package (2.0): one SDK, one +`rp` CLI, one login. The standalone `runpod-flash` package is deprecated on +**October 29, 2026** and gets no fixes after that date. -`runpod-flash` releases on its own cadence, so **`flash --help` and `flash --help` -are authoritative for the command surface** — this skill is the mental model, the decision -rules, and the gotchas that help output does not carry. Confirm the installed version with -`pip show runpod-flash` before concluding a subcommand or flag is unavailable. +**`rp --help` and `rp flash --help` are authoritative for the command surface.** +This skill is the mental model, the decision rules, and the gotchas that help output does not +carry. Confirm the installed version with `pip show runpod` before concluding a command or +parameter is unavailable. -**Worked examples first for anything multi-step.** Flash appears in verified end-to-end -paths — [03 variant B (whisper endpoint via flash)](../runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md) -and [08 (fine-tune → serve)](../runpod/golden-paths/08-finetune-to-serverless.md); the full -index is [runpod/golden-paths/README.md](../runpod/golden-paths/README.md). Open the matching -path before planning a deploy — it carries the ordering and the cost cleanup this skill only -summarizes. +## Which SDK is this project on? -**Load on demand — this skill keeps the mental model + gotchas inline; details live in [`reference/`](reference/):** +Check before writing code. The two APIs do not mix. -| Need | Read | -|------|------| -| Install, auth, `flash init`, and the full `flash` command list | [reference/setup-and-cli.md](reference/setup-and-cli.md) | -| `Endpoint(...)` constructor params, `NetworkVolume`/`PodTemplate`/`EndpointJob`, GPU & CPU enum tables | [reference/api.md](reference/api.md) | -| Worked patterns — choosing a model, warm-worker model loading, CPU→GPU pipeline, parallel calls | [reference/patterns.md](reference/patterns.md) | - -Quick start: `uv tool install runpod-flash` → `flash login` (or `export RUNPOD_API_KEY=...`) → `flash init my-project` → `flash dev`. Details in [reference/setup-and-cli.md](reference/setup-and-cli.md). - -## Dev vs Deploy - -- `flash dev` — **iterate.** Local server at `:8888`, but your decorated functions - execute on **remote GPU/CPU workers**. Hot-reloads on save and **streams the worker's - logs live** to the terminal. No build/upload/deploy wait — use this the whole time you - develop. -- `flash deploy` — **ship.** Builds an artifact and deploys a stable endpoint. Slow - (build + upload + provision); only do this once the code works under `flash dev`. - -`flash dev` ships **only the function body** to the worker, so a `NameError` for a -module-level name surfaces immediately here. `flash deploy` imports the whole module and -can mask that bug (see Gotcha #1). Develop against `flash dev` and you catch it first. - -## Autonomous Dev Loop - -`flash dev` is a long-running server. Three rules: -- **Run it in the background** — don't block on it. -- **Capture its output** to a log file. -- **Drive it over HTTP.** - -The captured log is the remote worker's live stream (cold start, model load, `print`s, -tracebacks) — read it to debug. - -```bash -flash dev > /tmp/flash-dev.log 2>&1 & # background; never run it blocking -for i in $(seq 1 60); do grep -q "flash dev localhost:" /tmp/flash-dev.log && break; sleep 2; done # bounded ~2min; if it never appears, check the log for errors -URL=$(grep -o "localhost:[0-9]*" /tmp/flash-dev.log | head -1) # actual port (8888 bumps if taken) -curl -s "$URL/main/predict" -d '{"data": {...}}' # dispatches to the remote worker -``` - -- **Read the real URL from the log** — flash auto-bumps the port if 8888 is in use, and - prints `✓ flash dev localhost:` plus the route table. -- **Routes are namespaced by file**: `main.py`'s `/predict` is served at `/main/predict`. -- **Two route shapes, two body shapes** (mismatch → `422` naming the missing field in `loc`): - - **Load-balanced** (`@api.post("/predict")`) → `POST /main/predict`, body is the arg - at top level: a handler `def predict(data: dict)` wants `{"data": {...}}` (not the bare object). - - **Queue-based** (bare `@Endpoint` decorator) → `POST /main/runsync` (the local dev - server only generates `/runsync`; production also exposes `/run`), - body is **double-wrapped** in `input`: a handler `def synthesize(data: dict)` wants - `{"input": {"data": {...}}}`. The outer `input` is the queue envelope; the inner key is - the handler's param name. -- Edit a handler and save — hot-reload re-syncs the body; just re-send the request, no - redeploy. Add `--auto-provision` to skip the first-call cold start. `kill %1` when done. - -## Endpoint: Three Modes - -Full constructor params and the GPU/CPU enum tables are in [reference/api.md](reference/api.md). +| You see | Project is on | Do | +|---|---|---| +| `import runpod`, `runpod.App(...)`, `@app.queue`, `rp flash ...` | **Runpod SDK 2.0** | This file + `reference/` | +| `from runpod_flash import Endpoint`, `@Endpoint(...)`, `flash dev`, `runpod-flash` in deps | **Legacy runpod-flash** | [reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md) to maintain it, [reference/migrate-from-runpod-flash.md](reference/migrate-from-runpod-flash.md) to move it | +| A new project | Runpod SDK 2.0 | Start below | -### Mode 1: Your Code (Queue-Based Decorator) +Write new code on the SDK. For an existing `runpod-flash` project, propose migrating and +check its *Known differences* first; don't migrate silently. -One function = one endpoint with its own workers. +**Worked examples first for anything multi-step.** Flash appears in end-to-end paths: +[03 variant B (whisper endpoint)](../runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md), +[07 (network-volume handoff)](../runpod/golden-paths/07-network-volume-handoff.md) and +[08 (fine-tune → serve)](../runpod/golden-paths/08-finetune-to-serverless.md); the full index is +[runpod/golden-paths/README.md](../runpod/golden-paths/README.md). Runnable single-file apps +live in [`examples/`](examples/). -```python -from runpod_flash import Endpoint, GpuGroup - -@Endpoint(name="my-worker", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"]) -async def compute(data): - import torch # MUST import inside function (cloudpickle) - return {"sum": torch.tensor(data, device="cuda").sum().item()} +**Load on demand:** -result = await compute([1, 2, 3]) -``` +| Need | Read | +|------|------| +| Install, `rp login`, `rp flash init`, every `rp` command | [reference/setup-and-cli.md](reference/setup-and-cli.md) | +| Decorator parameters, call styles, `Queue`/`Api`/`Endpoint` clients, volumes, `Model`, `Secret`, GPU & CPU names | [reference/api.md](reference/api.md) | +| Worked patterns: choosing a model, warm model loading, CPU→GPU pipeline, fan-out, tasks, raw HTTP | [reference/patterns.md](reference/patterns.md) | +| Moving a `runpod-flash` project to the SDK | [reference/migrate-from-runpod-flash.md](reference/migrate-from-runpod-flash.md) | +| Maintaining a `runpod-flash` project as it is | [reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md) | -### Mode 2: Your Code (Load-Balanced Routes) +Quick start: `uv add "runpod>=2.0.0rc1"` (or `pip install "runpod>=2.0.0rc1"`; this resolves +to the release candidate until 2.0.0 is stable) → `rp login` (or `export RUNPOD_API_KEY=...`) +→ `rp flash init my-app` → `cd my-app && rp flash dev main.py`. -Multiple HTTP routes share one pool of workers. +## The app model ```python -from runpod_flash import Endpoint, GpuGroup +import runpod -api = Endpoint(name="my-api", gpu=GpuGroup.ADA_24, workers=(1, 5), dependencies=["torch"]) +app = runpod.App("my-app") # names the group of resources -@api.post("/predict") -async def predict(data: list[float]): +@app.queue(gpu="4090", dependencies=["torch"]) # queue-based serverless endpoint +def infer(x: list[float]): import torch - return {"result": torch.tensor(data, device="cuda").sum().item()} + return {"sum": torch.tensor(x, device="cuda").sum().item()} -@api.get("/health") -async def health(): - return {"status": "ok"} +@runpod.local_entrypoint # runs on YOUR machine +def main(): + print(infer.remote([1.0, 2.0, 3.0])) # runs on a Runpod worker ``` -### Mode 3: External Image (Client) - -Deploy a pre-built Docker image and call it via HTTP. +| Decorator | Creates | Use for | +|---|---|---| +| `@app.queue(...)` on a function | Autoscaling queue-based endpoint | Request/response inference, batch jobs | +| `@app.api(...)` on a class | Load-balanced endpoint serving routes (`@runpod.get`/`@runpod.post`, setup in `@runpod.init`) | HTTP services, multi-route APIs, load-once model state | +| `@app.task(...)` on a function | One ephemeral **Pod per call**, terminated when it returns | Training, fine-tuning, long jobs | + +Call styles on a queue or task handle: `fn.remote(...)` (blocks), `await fn.remote.aio(...)`, +`fn.spawn(...)` (returns a `Job`), `fn.stream(...)` (generator functions), `fn.local(...)` +(runs on your machine). API classes are called by route: `MyApi.post("/path", body)`. + +## Dev vs deploy + +- `rp flash dev main.py` — **iterate.** Creates temporary endpoints (`dev---`), + runs the `@runpod.local_entrypoint` on your machine, streams the workers' `print` output to + your terminal, and re-runs after every file save. Ctrl-C deletes the endpoints. There is **no + local HTTP server**; the entrypoint is how you exercise the code. +- `rp flash dev main.py --once` — run the entrypoint once, tear down, exit non-zero on failure. + The smoke test for CI and for "does the migrated code work". +- `rp flash deploy` — **ship.** Builds an artifact (≤1,500 MB, torch excluded on built-in GPU + images) and creates persistent endpoints in the `default` environment (`--env NAME` for + others). Re-run to update. `--build-only` packages without deploying, a cheap offline check + that discovery and dependency resolution work. + +Dev and deploy run the **same module source**: the whole file ships, so module-level imports, +constants and helpers behave identically in both. The app must live in **one file** today +(multi-file projects are a known gap). + +## Autonomous dev loop + +A dev session is long-running. Three rules: +- **Run it in the background with stdin closed** (`< /dev/null`). With no stdin it re-runs + the entrypoint on every file save, so you never need to press Enter. +- **Capture its output** to a log file and read it. Worker `print`s, tracebacks, `✗ remote execution failed: ...` and the per-run + `✓ done in ...` / `✗ failed in ...` lines land there. +- **Stop it with SIGINT**, never `kill -9`, so it deletes its endpoints. Confirm the log ends + with `cleaning up N deleted`. -```python -from runpod_flash import Endpoint, GpuGroup, PodTemplate - -server = Endpoint( - name="my-server", - image="my-org/my-image:latest", - gpu=GpuGroup.AMPERE_80, - workers=1, - env={"HF_TOKEN": "xxx"}, - template=PodTemplate(containerDiskInGb=100), -) - -# LB-style -result = await server.post("/v1/completions", {"prompt": "hello"}) -models = await server.get("/v1/models") - -# QB-style -job = await server.run({"prompt": "hello"}) # optional: webhook="https://..." for completion callback -await job.wait() -print(job.output) -``` - -Connect to an existing endpoint by ID (no provisioning): - -```python -ep = Endpoint(id="abc123") -job = await ep.runsync({"prompt": "hello"}) # runsync wraps this as {"input": {"prompt": "hello"}} -print(job.output) +```bash +rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 & +DEV=$! +# poll the log (bounded) for the run result: "✓ done in ..." or "✗ failed in ..." +for i in $(seq 1 120); do grep -qE "✓ done in|✗ failed in" /tmp/rp-dev.log && break; sleep 3; done +tail -40 /tmp/rp-dev.log +# edit main.py and save -> the session re-scans, refreshes endpoints, re-runs main() +# (each run appends a new result line; read the last one) +kill -INT $DEV; wait $DEV; tail -3 /tmp/rp-dev.log # expect "cleaning up N deleted" ``` -## How Mode Is Determined - -| Parameters | Mode | -|-----------|------| -| `name=` only | Decorator (your code) | -| `image=` set | Client (deploys image, then HTTP calls) | -| `id=` set | Client (connects to existing, no provisioning) | - -The table above is *how* the mode is picked from params. *When* to reach for `image=`: - -### When to use `image=` (custom container) vs your own code +For a single check, `rp flash dev main.py --once` is simpler and cleans up on its own. If a +session died without cleanup, list leftovers with `runpodctl serverless list` and delete the +`dev--...` endpoints **this session created**, by ID. -Default to writing Python (decorator / routes) — it runs arbitrary code with -`dependencies=[...]`/`system_dependencies=[...]` and needs no Dockerfile. Even large -HuggingFace models stay in decorator mode (weights stream at runtime — see -[reference/patterns.md → Loading ML models](reference/patterns.md#loading-ml-models-warm-workers)). -Reach for `image=` **only** when you need: +## Choosing the resource shape -- **a pre-built inference server** — vLLM, TensorRT-LLM (`image="vllm/vllm-openai:latest"`, or `runpod/worker-vllm`, `runpod/worker-comfy`) -- **system-level deps not pip-installable** — a specific CUDA/cuDNN, OS libraries -- **models baked into the image** — to skip the runtime download entirely -- **an existing Runpod Serverless worker** — you already have a working image +Default to plain Python on the Runpod runtime image: `dependencies=[...]` and +`system_dependencies=[...]` install on the worker, so no Dockerfile is needed, and even large +HuggingFace models stay in code (see [reference/patterns.md](reference/patterns.md#loading-ml-models)). -Trade-off: `image=` mode **can't run arbitrary Python** (the image owns all logic) and the -image must implement a Runpod Serverless handler. Full list + examples: -https://docs.runpod.io/flash/custom-docker-images +- **`image="org/image:tag"`** on any decorator runs your function on that image; it needs + Python 3, not a Serverless handler. Use it for system libraries or a specific CUDA stack. + `registry_auth=` names saved credentials for a private image (`rp registry add`). +- **A prebuilt inference server image** (vLLM, worker-comfyui) that already implements a + Serverless handler is not an SDK app. Deploy it with runpodctl or runpod-mcp and call it + with `runpod.Endpoint("ENDPOINT_ID")`. +- **An existing endpoint** by ID: `runpod.Endpoint("ENDPOINT_ID").run_sync({...})`. +- **A resource in another deployed app**: `runpod.Queue(app="media", name="transcribe")` / + `runpod.Api(app="media", name="inference")`. ## Gotchas -1. **Only the function body ships to the worker** -- most common error. Put imports *and* any module-level constants/helpers the function uses *inside* the decorated body. `flash deploy` imports the whole module so module globals happen to work; `flash dev` ships just the body, so a module-level name raises `NameError`. A handler that works deployed can break under dev — fix it by moving everything inside. -2. **Forgetting await** -- all decorated functions and client methods need `await`. -3. **Missing dependencies** -- must list in `dependencies=[]`. -4. **gpu/cpu are exclusive** -- pick one per Endpoint. -5. **idle_timeout is seconds** -- default 60s, not minutes. -6. **10MB payload limit** -- pass URLs, not large objects. Return binary (audio/images/files) as base64 in the JSON (`{"audio_b64": ...}`) and decode client-side; for larger outputs write to a NetworkVolume or upload to storage and return a URL. -7. **Client vs decorator** -- `image=`/`id=` = client. Otherwise = decorator. -8. **Auto GPU switching requires workers >= 5** -- pass a list of GPU types (e.g. `gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_80]`) and set `workers=5` or higher. The platform only auto-switches GPU types based on supply when max workers is at least 5. -9. **`runsync` timeout is 60s** -- cold starts can exceed 60s. Use `ep.runsync(data, timeout=120)` for first requests or use `ep.run()` + `job.wait()` instead. -10. **Request body shape (raw/external HTTP callers only)** -- match the request shape to the endpoint type: - - **LB routes** (`@api.post(...)`): send the handler arg at the top level — `{"data": {...}}`. - - **QB endpoints** (bare `@Endpoint`, hit via `.../run` or `.../runsync`): the worker calls - **`handler(**job_input)`**, so the request's `input` keys must match the handler's parameter - names — `def transcribe(input_data: dict)` wants `{"input": {"input_data": {...}}}`, and - `def read(input: dict)` wants `{"input": {"input": {...}}}`. A mismatch fails with - `got an unexpected keyword argument …`. Use `**kwargs` if the handler ignores the payload. - - **Never send an empty `input`.** A QB request with `{"input": {}}` is rejected by the - worker SDK as `Job has missing field(s): id or input` — always include at least one key. - - *Context:* the flash client (`ep.runsync(x)`, `api.post(...)`) hides the spreading, so this - only bites raw HTTP/external callers (mismatch behavior verified 2026-07-10 via worker logs). - See *Autonomous Dev Loop*. -11. **Load a model once per worker (not per call)** -- for real inference use a class `@Endpoint` whose `__init__` loads the model once per worker (see [reference/patterns.md → Loading ML models](reference/patterns.md#loading-ml-models-warm-workers)). In function-form, reconcile with #1 by caching in a module global *inside* the body so it works under both `flash dev` and `deploy`: - ```python - global _MODEL - try: _MODEL - except NameError: _MODEL = load_model() # runs once per worker, reused across calls - ``` -12. **Native CUDA libs go in `dependencies=[]` too** -- e.g. CTranslate2/faster-whisper needs `nvidia-cublas-cu12` + `nvidia-cudnn-cu12` or it silently falls back to CPU. Add them alongside the Python package. -13. **Silent 401 auth failure** -- a set `RUNPOD_API_KEY` env var overrides the `flash login` token, so a bad/expired key wins. The failure is quiet: provisioning logs `GraphQL request failed: 401`, but `flash dev` still prints its normal ready line ("failed endpoints deploy on-demand"), so it *looks* healthy. When endpoints fail to provision: - 1. Check the provisioning log for `GraphQL request failed: 401`. - 2. Verify the current key independently: `curl -s -o /dev/null -w '%{http_code}' https://api.runpod.io/v2/serverless -H "Authorization: Bearer $RUNPOD_API_KEY"` (200 = good, 401 = bad). - 3. Fix it: `unset RUNPOD_API_KEY` to fall back to the `flash login` token, or `export` a valid key. -14. **`system_dependencies=` adds to cold start** -- apt packages (e.g. `["ffmpeg", "espeak-ng"]`) install on the worker before first use, so the initial call is slower (on top of any model download); warm calls are unaffected. -15. **Teardown a deployed app with `flash app delete `** -- `flash undeploy list` may show "no endpoints" for an app that is deployed and serving; `flash app delete` (or `runpodctl serverless delete `) reliably removes it. +1. **Queue arguments and results must be JSON-serializable.** Queue jobs are plain JSON in dev + and deploy alike; a non-JSON argument fails with `arguments must be json-serializable`. Pass + URLs or base64, not objects. +2. **One file per app.** Discovery imports the module you point at. Helpers in a sibling local + module are not shipped yet; keep them in the same file or in an installable package. +3. **Keep blocking work out of module scope.** The SDK imports your module to discover + resources, locally and on every worker. Put heavy imports (`torch`) and work inside + functions; guard local-only code with `runpod.is_local()`. +4. **`gpu=` and `cpu=` are exclusive** (`gpu and cpu are mutually exclusive`). GPU strings can + be short (`"4090"`, `"H100"`) or a list of acceptable GPUs. +5. **Default `workers=(0, 3)`** (runpod-flash was `(0, 1)`). Set it explicitly when cost or + concurrency matters. `idle_timeout` is seconds (default 60). +6. **Auto GPU switching needs `workers` max ≥ 5** when you pass a list of GPUs. +7. **10 MB payload limit** on requests. Return large outputs via a volume or object storage URL. +8. **Volumes outlive everything.** Dev sessions and `rp flash undeploy` delete endpoints, not + the `NetworkVolume`s they created, and those keep billing. Queue and API resources mount one + volume at `/runpod-volume`; read `vol.path` inside the remote function, never at import time. +9. **Tasks bill until they return.** A `.spawn()`ed task that you never `wait()` or `cancel()` + keeps its Pod running after your script exits. +10. **Native CUDA libraries go in `dependencies` too**, e.g. faster-whisper needs + `nvidia-cublas-cu12` + `nvidia-cudnn-cu12` or it silently falls back to CPU. +11. **Local Python must be 3.10–3.14.** Dev sessions and tasks ship code pickled by your + interpreter, so the worker matches your local version. `rp flash deploy --python-version` + sets it for deploys. +12. **A set `RUNPOD_API_KEY` wins over `rp login`.** A stale exported key causes 401s while the + saved login is fine. Check with `runpodctl user`, then `unset RUNPOD_API_KEY` or export a + good key. +13. **Check the artifact when a dependency needs torch.** Auto-exclusion drops `torch` only when + it is listed directly. In `runpod` 2.0.0rc1, a package that depends on it (`peft`, + `accelerate`, `sentence-transformers`) pulls `torch` into the artifact even on GPU images + and with `--exclude torch`, so the artifact grows by hundreds of MB. Run + `rp flash deploy --build-only` and check the size before deploying. +14. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` + removes one environment's endpoints; `rp flash app delete APP` removes every environment. + Then delete volumes you no longer need. ## Resources -- Setup & CLI: [reference/setup-and-cli.md](reference/setup-and-cli.md) · API & compute enums: [reference/api.md](reference/api.md) · Patterns: [reference/patterns.md](reference/patterns.md) -- Flash source: https://github.com/runpod/flash -- Runnable examples: https://github.com/runpod/flash-examples — clone and adapt the closest one -- Package (PyPI): https://pypi.org/project/runpod-flash/ -- Docs: https://docs.runpod.io/flash/overview - - Custom Docker images (when + how): https://docs.runpod.io/flash/custom-docker-images - - Storage / network volumes: https://docs.runpod.io/flash/configuration/storage +- Setup & CLI: [reference/setup-and-cli.md](reference/setup-and-cli.md) · API: [reference/api.md](reference/api.md) · Patterns: [reference/patterns.md](reference/patterns.md) · Examples: [examples/](examples/) +- SDK source and upstream examples: https://github.com/runpod/runpod-python (`examples/apps/`) +- Docs: https://docs.runpod.io/sdk/overview · CLI: https://docs.runpod.io/sdk/cli · Migration: https://docs.runpod.io/sdk/migrate-from-flash diff --git a/plugins/runpod/skills/flash/evals/client-external-image.eval.md b/plugins/runpod/skills/flash/evals/client-external-image.eval.md index 8d2f8501..6d061465 100644 --- a/plugins/runpod/skills/flash/evals/client-external-image.eval.md +++ b/plugins/runpod/skills/flash/evals/client-external-image.eval.md @@ -1,23 +1,27 @@ -# Deploy a prebuilt vLLM image and call it over HTTP +# Serve a prebuilt vLLM worker image and call it ## Prompt -I have a prebuilt Docker image `myorg/vllm-server:latest` that serves an -OpenAI-compatible API on an A100. Using runpod-flash, deploy it to a Runpod -serverless GPU endpoint and send a completion request to `/v1/completions`. +I have a prebuilt Docker image `myorg/vllm-worker:latest`. It is a complete Runpod Serverless +worker (it implements the handler) for OpenAI-style completions on an A100. Deploy it to a +Runpod serverless endpoint and send it a completion request from Python. ## Expected behavior The agent should: -1. Create an `Endpoint` with `image="myorg/vllm-server:latest"` (client mode) plus `name=`, `gpu=GpuGroup.AMPERE_80` -2. Recognize that `image=` means client mode (deploys the image, then calls it via HTTP) — no decorated function -3. Call the deployed endpoint with `await server.post("/v1/completions", {...})` +1. Recognize that a complete worker image is not an SDK app: there is no Python function to + decorate +2. Create the endpoint with an infra lane (runpodctl or runpod-mcp) from the image, on an A100 + pool +3. Call it from Python with `runpod.Endpoint("ENDPOINT_ID").run_sync({...}, timeout=...)` or + `.run(...)` ## Assertions -- Creates an `Endpoint(name=..., image="myorg/vllm-server:latest", gpu=GpuGroup.AMPERE_80, ...)` -- Does NOT wrap a Python function in a decorator (client mode, not decorator mode) -- Calls the endpoint with `await .post("/v1/completions", )` -- Uses `await` on the HTTP call -- Does NOT set `id=` together with `image=` (mutually exclusive) +- Does NOT wrap the image in `@app.queue(image=...)` with an empty or placeholder function +- Creates the endpoint through runpodctl or the Runpod MCP tools, referencing the image +- Calls it with `runpod.Endpoint(...)` and puts the request under the client's `request_input` + (the client sends `{"input": {...}}`) +- Raises the `run_sync` timeout or uses `run()` + `output(timeout=...)` for the first, cold request +- Does NOT import from `runpod_flash` diff --git a/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md b/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md index 6f3b9412..f27d6ef7 100644 --- a/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md +++ b/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md @@ -2,22 +2,24 @@ ## Prompt -I already have a Runpod serverless endpoint with ID `abc123xyz`. Using -runpod-flash, send it a synchronous job `{"prompt": "hello"}` and print the -output. The first request may cold-start and take longer than a minute. +I already have a Runpod serverless endpoint with ID `abc123xyz`. Using the Runpod Python SDK, +send it a synchronous job `{"prompt": "hello"}` and print the output. The first request may +cold-start and take longer than a minute. ## Expected behavior The agent should: -1. Create `Endpoint(id="abc123xyz")` — connects to the existing endpoint, no provisioning -2. Submit the job with `runsync`, raising the timeout above the 60s default to survive cold start -3. Print `job.output` +1. Create `runpod.Endpoint("abc123xyz")` — no provisioning +2. Submit with `run_sync({"prompt": "hello"}, timeout=...)` and a timeout that covers a cold + start, or `run(...)` then `job.output(timeout=...)` +3. Print the result ## Assertions -- Creates `Endpoint(id="abc123xyz")` (no `name=`, `gpu=`, or `image=` needed) -- Uses `await ep.runsync({"prompt": "hello"}, timeout=...)` with a timeout > 60 (e.g. 120) OR uses `await ep.run(...)` + `await job.wait()` to avoid the 60s cap -- Accesses the result via `job.output` -- Uses `await` on the call -- Does NOT pass `id=` together with `image=` (`id=` + `name=` is legal and harmless) +- Uses `runpod.Endpoint("abc123xyz")` +- Passes `{"prompt": "hello"}` as the input (the client wraps it in `{"input": ...}`; the agent + does NOT double-wrap it as `{"input": {"prompt": ...}}`) +- Sets a timeout above 60 seconds or uses `run()` + `output(timeout=...)` +- Does NOT create an `App` or decorate a function for this +- Does NOT import from `runpod_flash` diff --git a/plugins/runpod/skills/flash/evals/cpu-gpu-pipeline.eval.md b/plugins/runpod/skills/flash/evals/cpu-gpu-pipeline.eval.md index 9a36bcd2..ffb4c3e1 100644 --- a/plugins/runpod/skills/flash/evals/cpu-gpu-pipeline.eval.md +++ b/plugins/runpod/skills/flash/evals/cpu-gpu-pipeline.eval.md @@ -2,24 +2,26 @@ ## Prompt -Using runpod-flash, build a two-stage pipeline: a CPU stage that cleans raw data -with pandas, then a GPU stage that runs inference with torch. The CPU stage -should use a compute CPU instance and the GPU stage an A100. Wire them together. +With the Runpod Python SDK, build a two-stage pipeline: a CPU stage that cleans raw data +with pandas, then a GPU stage that runs inference with torch on an A100. Wire them together +and run several batches in parallel. ## Expected behavior The agent should: -1. Define a CPU `@Endpoint(cpu=CpuInstanceType., dependencies=["pandas"])` function -2. Define a GPU `@Endpoint(gpu=GpuGroup.AMPERE_80, dependencies=["torch"])` function +1. Define a CPU `@app.queue(cpu="cpu5c-..." or similar, dependencies=["pandas"])` function +2. Define a GPU `@app.queue(gpu="AMPERE_80" or "A100", ...)` function 3. Import `pandas`/`torch` inside the respective functions -4. Chain them: `await infer(await preprocess(raw))` +4. Chain them: `score.remote(preprocess.remote(rows))` +5. Fan out with `.spawn()` and collect with `job.result()` (or `asyncio.gather` over `.remote.aio`) ## Assertions -- CPU stage uses `cpu=CpuInstanceType.` and does NOT set `gpu=` -- GPU stage uses `gpu=GpuGroup.AMPERE_80` and does NOT set `cpu=` -- `pandas` listed in the CPU stage `dependencies`, `torch` in the GPU stage `dependencies` -- Imports are inside each decorated function -- Stages are chained with `await` (the GPU call awaits the result of the CPU call) -- Does NOT put `gpu=` and `cpu=` on the same Endpoint +- CPU stage sets `cpu=` to a CPU flavor id and does NOT set `gpu=` +- GPU stage sets `gpu=` to an A100 pool or name and does NOT set `cpu=` +- `pandas` is in the CPU stage `dependencies`; the GPU stage either lists `torch` or relies on + the built-in GPU image's torch, and says which +- Stages exchange JSON-serializable values (lists/dicts, not DataFrames or tensors) +- Parallel calls use `.spawn()` / `.remote.aio`, not sequential `.remote()` in a loop +- Does NOT import from `runpod_flash` diff --git a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md index 958f2c1d..18c755b1 100644 --- a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md +++ b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md @@ -1,4 +1,4 @@ -# Iterate on a flash GPU handler against a live remote worker +# Iterate on an SDK app against a live remote worker > **LIVE eval.** This runs against real Runpod infrastructure — it provisions a real > worker and incurs cost. It is graded on what actually happened at runtime, not on what @@ -6,58 +6,48 @@ ## Setup -- Requires `RUNPOD_API_KEY` in the environment (flash CLI authenticated, v1.17.0+). +- Requires `runpod>=2.0.0rc1` installed in the active environment and a valid key + (`rp login` or `RUNPOD_API_KEY`). - Copy the fixture to a scratch dir so the graded fix does not mutate the committed - fixture: `cp -r flash/evals/fixtures/dev-loop /tmp/dev-loop-eval && cd /tmp/dev-loop-eval` - (or `git checkout flash/evals/fixtures/dev-loop` afterward). -- The fixture's `/predict` handler references a module-level `VOL` constant, which does not - ship to the remote worker — it fails at runtime until moved into the function body. + fixture: `cp -r flash/evals/fixtures/dev-loop /tmp/dev-loop-eval && cd /tmp/dev-loop-eval`. +- The fixture's `describe` function imports `humanize`, which is not in its `dependencies`, so + it fails on the worker with `No module named 'humanize'` until the dependency is declared. ## Prompt -Use the runpod-flash project in this directory (an image-to-3D style endpoint). The -`/predict` route fails when it actually runs on a worker. Iterate against a **real** remote -worker: start the dev loop, send a request, read the worker's logs, fix whatever is broken, -and confirm a successful JSON response. I don't want to re-deploy on every change. +Use the Runpod SDK app in this directory. `main()` fails when `describe` runs on a worker. +Iterate against a **real** remote worker: start the dev loop, read the worker's output, fix +what is broken, and confirm `main()` succeeds. I don't want to deploy on every change. ## Expected behavior The agent should actually execute (not merely describe) the following: -1. Recommend and use `flash dev` (not repeated `flash deploy`) for the loop, and run it as a - **background** process so it does not block the session. -2. Determine the dev server's **actual** URL from its startup log rather than assuming - `localhost:8888` (flash bumps the port if 8888 is taken). -3. Send a real request to the correct **file-namespaced** route (`main.py` → `/main/predict`), - which provisions and dispatches to the remote worker. -4. Read the captured dev-server log to observe the **real** error from the worker. -5. Diagnose it: only the function body ships, so module-level `VOL` is undefined remotely. - Fix by moving `VOL` inside the handler and rely on hot-reload (no redeploy). -6. Re-send the request and confirm a real successful response. -7. Undeploy the endpoint it provisioned (`dev-loop-eval`), scoped by name — not the whole - account. +1. Use `rp flash dev main.py` (not `rp flash deploy`) and run it in the **background** with + stdin closed and output captured to a log, or use `rp flash dev main.py --once` per attempt. +2. Read the captured log to observe the **real** error from the worker. +3. Diagnose it: the worker lacks `humanize`. Fix by adding `"humanize"` to the decorator's + `dependencies`, not by vendoring or `pip install` inside the function. +4. Re-run without deploying: save the file and let the background session re-run, or run + `--once` again. +5. Confirm the entrypoint succeeds (`✓` result line, the `assert` passes). +6. End the session with SIGINT (or let `--once` exit) and confirm the log reports its + temporary endpoints deleted. ## Assertions -- Runs `flash dev` as a background / non-blocking process (does NOT run it as a plain - blocking command and hang) -- Determines the actual host:port from the dev-server output (does NOT hardcode `8888` when - it was bumped) -- Sends the request to the file-namespaced route (`/main/predict`), not the bare `/predict` -- Observes the **verbatim** runtime error `NameError: name 'VOL' is not defined` in the - worker's streamed logs (it is reported from the live run, not guessed) -- Fixes the bug by moving `VOL` into the function body and re-tests via hot-reload, without - running `flash deploy` -- Obtains a real **HTTP 200** whose body contains `"ok": true` (e.g. - `{"ok":true,"vol":"/runpod-volume/models","echo":...}`) -- Runs `flash undeploy dev-loop-eval --force` — scoped to the endpoint it created, NOT - `flash undeploy --all` (which would delete unrelated endpoints in the account) — and - confirms the `dev-loop-eval` endpoint is gone +- Runs `rp flash dev` non-blocking (background with captured output) or with `--once`; does + NOT hang on an interactive session +- Does NOT run `rp flash deploy` during the loop +- Reports the **verbatim** worker error containing `No module named 'humanize'`, from the live run +- Fixes it by adding `humanize` to `dependencies=[...]` +- Obtains a successful entrypoint run: `main()` prints `3.0 MB` and the log has no `✗` failure line +- Confirms cleanup from the log (`cleaning up ... deleted`); if cleanup failed, deletes only the + `dev-dev-loop-eval-...` endpoint by id with `runpodctl serverless delete` ## Cleanup -- `flash undeploy dev-loop-eval --force` must report the `dev-loop-eval` endpoint deleted, - and `flash undeploy list` must no longer list it. Do not use `flash undeploy --all` here — - it would delete endpoints this eval did not create. -- The agent must only stop processes/ports it started. +- The dev session must delete its temporary endpoint. Verify with `runpodctl serverless list` + that no `dev-dev-loop-eval-` endpoint remains. +- The agent must only stop processes it started. - Restore the fixture if it was edited in place: `git checkout flash/evals/fixtures/dev-loop`. diff --git a/plugins/runpod/skills/flash/evals/fixtures/dev-loop/main.py b/plugins/runpod/skills/flash/evals/fixtures/dev-loop/main.py index 4b0986a6..d0d91fcf 100644 --- a/plugins/runpod/skills/flash/evals/fixtures/dev-loop/main.py +++ b/plugins/runpod/skills/flash/evals/fixtures/dev-loop/main.py @@ -1,21 +1,21 @@ -from runpod_flash import Endpoint, GpuGroup +import runpod -# BUG (intentional — do NOT "pre-fix" this): a module-level constant referenced -# inside the handler. Under `flash dev` only the function body ships to the -# remote worker, so this raises `NameError: name 'VOL' is not defined` remotely -# until it is moved inside predict(). `flash deploy` imports the whole module and -# masks the bug; `flash dev` surfaces it. The eval's job is to reproduce, observe -# in the live worker logs, and fix it. -VOL = "/runpod-volume/models" +app = runpod.App("dev-loop-eval") -api = Endpoint(name="dev-loop-eval", gpu=GpuGroup.AMPERE_16, workers=(0, 1), dependencies=[]) +# BUG (intentional — do NOT "pre-fix" this): describe() imports humanize, but +# the decorator does not declare it, so the worker raises +# `No module named 'humanize'`. The eval's job is to reproduce it in a live dev +# session, read the worker error, and fix the decorator. +@app.queue(cpu="cpu3c-1-2", workers=(0, 1), dependencies=[]) +def describe(size_bytes: int) -> str: + import humanize -@api.post("/predict") -async def predict(data: dict): - return {"ok": True, "vol": VOL, "echo": data} + return humanize.naturalsize(size_bytes) -@api.get("/health") -async def health(): - return {"status": "ok"} +@runpod.local_entrypoint +def main(): + out = describe.remote(3_000_000) + print(out) + assert out == "3.0 MB", out diff --git a/plugins/runpod/skills/flash/evals/lb-multi-route-api.eval.md b/plugins/runpod/skills/flash/evals/lb-multi-route-api.eval.md index 4c4c8764..cf20e6f3 100644 --- a/plugins/runpod/skills/flash/evals/lb-multi-route-api.eval.md +++ b/plugins/runpod/skills/flash/evals/lb-multi-route-api.eval.md @@ -2,24 +2,26 @@ ## Prompt -Using runpod-flash, I want a single GPU endpoint that exposes two HTTP routes: +With the Runpod Python SDK, I want a single GPU endpoint that exposes two HTTP routes: `POST /predict` for inference and `GET /health` for a health check, sharing the -same pool of workers (1 to 5). Write the code. +same pool of workers (1 to 5). The model should load once per worker. Write the code. ## Expected behavior The agent should: -1. Create an `Endpoint` INSTANCE (not a decorator on a function): `api = Endpoint(name=..., gpu=..., workers=(1, 5), ...)` -2. Register routes with `@api.post("/predict")` and `@api.get("/health")` -3. Put heavy imports inside the route handlers -4. Make handlers `async def` +1. Decorate a **class** with `@app.api(gpu=..., workers=(1, 5), ...)` +2. Load the model in a `@runpod.init` method +3. Register routes with `@runpod.post("/predict")` and `@runpod.get("/health")` methods +4. Show how to call it: `MyApi.post("/predict", {...})` in a dev session, or a raw POST of the + JSON body itself to the deployed endpoint ## Assertions -- Creates an `Endpoint(...)` instance assigned to a variable (load-balanced mode) -- Uses `@.post("/predict")` and `@.get("/health")` to register routes -- Uses `workers=(1, 5)` (explicit min/max tuple), NOT `workers=5` -- Does NOT define each route as its own separate `@Endpoint(name=...)` decorator (that would be separate endpoints, not shared workers) -- Route handlers are `async def` -- Heavy/GPU imports are inside the handler functions +- Uses `@app.api(...)` on a class, not one `@app.queue` per route +- Uses `workers=(1, 5)`, NOT `workers=5` (which means `(0, 5)`) +- Loads the model in `@runpod.init`, not inside the route methods +- Routes are `@runpod.post("/predict")` and `@runpod.get("/health")` +- The POST method takes one `body` argument; any raw HTTP example sends the object itself, + not `{"data": {...}}` +- Does NOT import from `runpod_flash` diff --git a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md new file mode 100644 index 00000000..2835d48b --- /dev/null +++ b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md @@ -0,0 +1,35 @@ +# Migrate a runpod-flash project to the Runpod SDK + +## Prompt + +This repo uses runpod-flash: `main.py` has `from runpod_flash import Endpoint, GpuGroup`, a +queue `@Endpoint(name="embed", gpu=GpuGroup.ADA_24, dependencies=["sentence-transformers"])` +function, and an `api = Endpoint(name="search", cpu="cpu5c-2-4")` with `@api.post("/query")`. +CI runs `flash deploy`. I heard runpod-flash is deprecated. Move it over. + +## Expected behavior + +The agent should: + +1. Inventory the `runpod_flash` imports, every `Endpoint`, and every `flash` CLI call (CI too) +2. Check the known differences (multi-file project, `id=`/`template=`, path params, volume lists) + and report any that apply before converting +3. Install `runpod>=2.0.0rc1`, convert to `runpod.App` with `@app.queue` for `embed` and an + `@app.api` class with `@runpod.post("/query")` for `search`, and add a + `@runpod.local_entrypoint` +4. Update CI from `flash deploy` to `rp flash deploy` +5. Smoke-test with `rp flash dev main.py --once` before deploying +6. Tell the user the load-balanced route's raw HTTP body changes (no `{"data": ...}` wrapper), + that default `workers` is now `(0, 3)`, and to remove the old runpod-flash endpoints after + the new deploy works + +## Assertions + +- Removes every `runpod_flash` import; uses `import runpod` +- `embed` becomes `@app.queue(gpu="ADA_24" or GpuGroup.ADA_24, dependencies=[...])` +- `search` becomes an `@app.api` class whose POST method takes one `body` argument +- CI calls `rp flash deploy`, not `flash deploy` +- Runs or recommends `rp flash dev main.py --once` before `rp flash deploy` +- Mentions the `workers` default change or sets `workers` explicitly +- Does NOT delete the old endpoints before the new deployment is confirmed +- Does NOT claim the October 29, 2026 deprecation stops existing endpoints diff --git a/plugins/runpod/skills/flash/evals/qb-gpu-function.eval.md b/plugins/runpod/skills/flash/evals/qb-gpu-function.eval.md index e18e89f1..b1781b2c 100644 --- a/plugins/runpod/skills/flash/evals/qb-gpu-function.eval.md +++ b/plugins/runpod/skills/flash/evals/qb-gpu-function.eval.md @@ -3,24 +3,24 @@ ## Prompt I have a Python function that runs a PyTorch model on a GPU. I want to run it on -Runpod serverless using runpod-flash, with up to 5 workers. Write the code. +Runpod serverless with the Runpod Python SDK, with up to 5 workers. Write the code. ## Expected behavior The agent should: -1. Import `Endpoint` and `GpuGroup` from `runpod_flash` -2. Decorate the function with `@Endpoint(name=..., gpu=GpuGroup., workers=5, dependencies=["torch"])` -3. Put the `import torch` (and any other deps) INSIDE the decorated function -4. Make the function `async def` -5. Call it with `await` +1. `import runpod` and create `app = runpod.App("...")` +2. Decorate the function with `@app.queue(gpu=..., workers=5 or (0, 5), dependencies=[...])` +3. Import heavy packages inside the function +4. Call it from a `@runpod.local_entrypoint` with `.remote(...)` and run it with + `rp flash dev FILE` (or `--once`) ## Assertions -- Uses `@Endpoint(...)` as a decorator with a `name=` (queue-based mode) -- Sets `gpu=` to a `GpuGroup` member and `workers` to 5 -- Lists `torch` in `dependencies=[...]` -- The `import torch` statement is INSIDE the function body, not at module top level -- The function is `async def` and is invoked with `await` -- Does NOT use the deprecated `@remote` decorator +- Uses `runpod.App` and `@app.queue(...)` (queue-based) +- Sets `gpu=` (a string such as `"4090"`, a `GpuGroup`, or a list) and a worker max of 5 +- Lists the function's pip packages in `dependencies=[...]` +- Calls the function with `.remote(...)` (or `await fn.remote.aio(...)`), not a bare `await fn(...)` +- Does NOT import from `runpod_flash` or use `@Endpoint` / `@remote` - Does NOT set both `gpu=` and `cpu=` +- Arguments passed to `.remote()` are JSON-serializable diff --git a/plugins/runpod/skills/flash/evals/volume-cache-model-load.eval.md b/plugins/runpod/skills/flash/evals/volume-cache-model-load.eval.md index 44e70b09..8efcaad6 100644 --- a/plugins/runpod/skills/flash/evals/volume-cache-model-load.eval.md +++ b/plugins/runpod/skills/flash/evals/volume-cache-model-load.eval.md @@ -2,7 +2,7 @@ ## Prompt -I have a Flash GPU endpoint that loads Qwen2.5-7B-Instruct with `from_pretrained`, and it +I have a Runpod SDK GPU endpoint that loads Qwen2.5-7B-Instruct with `from_pretrained`, and it scales to zero, so every cold start re-downloads the model from HuggingFace. I attached a network volume. How do I make cold starts faster? @@ -12,29 +12,28 @@ The agent should: 1. Keep the HF cache on the worker's **container disk** (for example `HF_HUB_CACHE=/opt/hf-cache`), not under `/runpod-volume` -2. Wrap the model load in `VolumeCache(dirs=[])` from `runpod.serverless`, inside - the class `@Endpoint`'s `__init__` so it runs once per worker, and add `runpod>=1.12.0` to - `dependencies` +2. Wrap the model load in `VolumeCache(dirs=[])` from `runpod.serverless`, inside an + `@app.api` class's `@runpod.init` method (or a once-per-worker module-global cache in a + queue function), with the volume mounted via `mounts={"/runpod-volume": vol}` 3. Explain the mechanism: the first worker downloads from the Hub and syncs a mirror to the volume, and later cold starts copy the mirror onto local disk and load from there -4. Name the gotchas: errors are swallowed by default (`best_effort=False` while debugging), - Hub access stays on because symlinks aren't copied, size the volume for the mirror, - size the container disk for the whole model (`PodTemplate(containerDiskInGb=...)`; the - Flash default is 64 GB), and let one worker populate the mirror first -5. Offer Model Store as the no-volume alternative for a HuggingFace model on GPU: attach it after - `flash deploy` with `runpodctl serverless update --model-reference`, with the - caveat that worker start waits for a host that holds the model +4. Name the gotchas: errors are swallowed by default (`best_effort=False` while debugging), Hub + access stays on because symlinks aren't copied, size the volume for the mirror, size + `container_disk_gb` for the whole model (the SDK default is 30 GB on GPU), and let one worker + populate the mirror first +5. Offer `model=runpod.Model("Qwen/Qwen2.5-7B-Instruct")` as the no-volume alternative, with + the caveat that worker start waits until the host has the weights 6. Suggest timing the whole cold request before and after, not quoting a speedup ## Assertions -- Wraps the load in `VolumeCache` and imports it from `runpod.serverless` -- Lists `runpod>=1.12.0` in the `@Endpoint` `dependencies` -- In the `VolumeCache` answer, points the HF cache at a local path, not at `/runpod-volume` - (the Model Store alternative's `/runpod-volume/huggingface-cache/hub` is correct) +- Wraps the load in `VolumeCache` imported from `runpod.serverless` +- Points the HF cache at a local path in the `VolumeCache` answer, not at `/runpod-volume` +- Mounts the volume with `mounts={"/runpod-volume": ...}` - Mentions that a failing cache falls back silently to a normal cold start -- Checks that the container disk can hold the whole local cache -- Mentions Model Store (`--model-reference`) as an alternative, with the worker-wait caveat +- Raises `container_disk_gb` so the local cache fits +- Mentions `runpod.Model` as an alternative, with the worker-wait caveat - Does NOT set `HF_HUB_OFFLINE=1` alongside `VolumeCache` -- Does NOT tell the user to load the model directly from `/runpod-volume` as the fast path +- Does NOT load the model directly from `/runpod-volume` as the fast path - Does NOT quote a fixed speedup figure as a guarantee +- Does NOT use `runpod_flash` imports or `PodTemplate` diff --git a/plugins/runpod/skills/flash/examples/README.md b/plugins/runpod/skills/flash/examples/README.md new file mode 100644 index 00000000..64d7e3b0 --- /dev/null +++ b/plugins/runpod/skills/flash/examples/README.md @@ -0,0 +1,25 @@ +# Runpod SDK app examples + +Single-file apps for the Runpod Python SDK (`runpod` 2.0). Each file carries a +`@runpod.local_entrypoint` that asserts its own result, so a passing run proves it works. + +```bash +rp flash dev queue_gpu.py --once # temporary endpoints, run main() once, clean up +rp flash deploy queue_gpu.py # persistent endpoints +rp flash deploy queue_gpu.py --build-only # offline check: discovery + dependency resolution +``` + +| File | Shows | +|---|---| +| [`queue_gpu.py`](queue_gpu.py) | `@app.queue` on a GPU list with `workers=(0, 5)` | +| [`api_model_service.py`](api_model_service.py) | `@app.api` class, model loaded once in `@runpod.init`, `POST /generate` + `GET /health` | +| [`cpu_gpu_pipeline.py`](cpu_gpu_pipeline.py) | CPU stage feeding a GPU stage, `.spawn()` fan-out | +| [`train_then_serve.py`](train_then_serve.py) | `@app.task` writes a checkpoint to a `NetworkVolume`, a queue endpoint reads it | +| [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | + +Cost and cleanup: dev sessions delete their endpoints on exit. `train_then_serve.py` creates +the `train-serve-ckpts` network volume, which stays and bills until you delete it. Remove +deployed apps with `rp flash app delete APP --yes`. + +The upstream SDK ships more examples in +[runpod-python `examples/apps`](https://github.com/runpod/runpod-python/tree/main/examples/apps). diff --git a/plugins/runpod/skills/flash/examples/api_model_service.py b/plugins/runpod/skills/flash/examples/api_model_service.py new file mode 100644 index 00000000..71d961fa --- /dev/null +++ b/plugins/runpod/skills/flash/examples/api_model_service.py @@ -0,0 +1,40 @@ +"""A load-balanced HTTP service that loads a model once per worker. + + rp flash dev api_model_service.py --once + rp flash deploy api_model_service.py + +@runpod.init runs once when a worker starts, before it takes traffic, so +the pipeline stays in memory across requests. In a dev session you call +routes through the class; once deployed, the endpoint serves them over HTTP. +""" + +import runpod + +app = runpod.App("text-gen") + + +@app.api(gpu="ADA_24", workers=(0, 3), idle_timeout=300, dependencies=["transformers"]) +class TextGen: + @runpod.init + def load(self): + from transformers import pipeline + + # swap the model id for the real one once the wiring works + self.pipe = pipeline("text-generation", model="gpt2", device=0) + + @runpod.post("/generate") + async def generate(self, body: dict): + out = self.pipe(body["prompt"], max_new_tokens=body.get("max_new_tokens", 20)) + return {"text": out[0]["generated_text"]} + + @runpod.get("/health") + async def health(self): + return {"status": "ok"} + + +@runpod.local_entrypoint +def main(): + print(TextGen.get("/health")) + resp = TextGen.post("/generate", {"prompt": "Runpod is"}) + print(resp) + assert resp["text"].startswith("Runpod is"), resp diff --git a/plugins/runpod/skills/flash/examples/call_deployed.py b/plugins/runpod/skills/flash/examples/call_deployed.py new file mode 100644 index 00000000..2ac05c5b --- /dev/null +++ b/plugins/runpod/skills/flash/examples/call_deployed.py @@ -0,0 +1,28 @@ +"""Call deployed resources from any Python code (no rp flash dev needed). + + python call_deployed.py + +Assumes `rp flash deploy queue_gpu.py` and `rp flash deploy +api_model_service.py` have run. Set ENDPOINT_ID to call any queue +endpoint by id, including ones the SDK did not create. +""" + +import os + +import runpod + +# a resource in a deployed app, by app and resource name (default environment) +gpu_sum = runpod.Queue(app="gpu-sum", name="gpu_sum") +print(gpu_sum.remote(values=[1.0, 2.0, 3.0])) # keyword arguments only + +job = gpu_sum.spawn(values=[4.0, 5.0]) +print(job.id, job.result()) + +text_gen = runpod.Api(app="text-gen", name="TextGen") +print(text_gen.post("/generate", {"prompt": "Hello"})) + +# any queue endpoint by id +endpoint_id = os.environ.get("ENDPOINT_ID") +if endpoint_id: + ep = runpod.Endpoint(endpoint_id) + print(ep.run_sync({"prompt": "hello"}, timeout=120)) # sent as {"input": {...}} diff --git a/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py new file mode 100644 index 00000000..53352504 --- /dev/null +++ b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py @@ -0,0 +1,37 @@ +"""Two stages on different hardware, plus parallel fan-out. + + rp flash dev cpu_gpu_pipeline.py --once + +The CPU stage cleans rows with pandas; the GPU stage scores them with +torch. .spawn() submits jobs without waiting, so several batches run on +separate workers at once. +""" + +import runpod + +app = runpod.App("pipeline") + + +@app.queue(cpu="cpu5c-4-8", dependencies=["pandas"]) +def preprocess(rows: list[dict]) -> list[list[float]]: + import pandas as pd + + df = pd.DataFrame(rows).dropna() + return df.astype(float).values.tolist() + + +@app.queue(gpu="AMPERE_80", workers=(0, 5)) +def score(matrix: list[list[float]]) -> list[float]: + import torch + + return torch.tensor(matrix, device="cuda").mean(dim=1).tolist() + + +@runpod.local_entrypoint +def main(): + rows = [{"a": 1, "b": 3}, {"a": 2, "b": None}, {"a": 5, "b": 7}] + print("one batch:", score.remote(preprocess.remote(rows))) + + batches = [[{"a": i, "b": i + 1}] for i in range(4)] + jobs = [score.spawn(preprocess.remote(b)) for b in batches] + print("fan-out:", [job.result() for job in jobs]) diff --git a/plugins/runpod/skills/flash/examples/queue_gpu.py b/plugins/runpod/skills/flash/examples/queue_gpu.py new file mode 100644 index 00000000..df602e98 --- /dev/null +++ b/plugins/runpod/skills/flash/examples/queue_gpu.py @@ -0,0 +1,28 @@ +"""A GPU function behind an autoscaling queue endpoint. + + rp flash dev queue_gpu.py --once # run main() once against a temporary endpoint + rp flash deploy queue_gpu.py # persistent endpoint in app "gpu-sum" + +Any of the listed GPUs is acceptable, and a max of 5 workers lets the +platform switch between them by supply. +""" + +import runpod + +app = runpod.App("gpu-sum") + + +@app.queue(gpu=["4090", "L4", "A5000"], workers=(0, 5), dependencies=["numpy"]) +def gpu_sum(values: list[float]) -> dict: + # torch ships on the built-in GPU image; import heavy packages inside the function + import torch + + t = torch.tensor(values, device="cuda") + return {"gpu": torch.cuda.get_device_name(0), "sum": t.sum().item()} + + +@runpod.local_entrypoint +def main(): + out = gpu_sum.remote([1.0, 2.0, 3.0]) + print(out) + assert out["sum"] == 6.0, out diff --git a/plugins/runpod/skills/flash/examples/train_then_serve.py b/plugins/runpod/skills/flash/examples/train_then_serve.py new file mode 100644 index 00000000..64471d5f --- /dev/null +++ b/plugins/runpod/skills/flash/examples/train_then_serve.py @@ -0,0 +1,50 @@ +"""Train on a Pod, then serve the checkpoint from a queue endpoint. + + rp flash dev train_then_serve.py --once + +train() is a task: one Pod per call, terminated when it returns. predict() +is a queue endpoint. Both mount the same network volume, so the endpoint +reads the checkpoint the task wrote. The volume outlives the session and +keeps billing; delete it when you are done. +""" + +import runpod + +app = runpod.App("train-serve") + +ckpts = runpod.NetworkVolume("train-serve-ckpts", size=10) + + +@app.task(gpu="4090", mounts={"/ckpts": ckpts}) +def train(steps: int = 200) -> str: + import torch + + x = torch.randn(1024, 1, device="cuda") + y = 3 * x + 1 + model = torch.nn.Linear(1, 1).cuda() + opt = torch.optim.SGD(model.parameters(), lr=0.05) + for _ in range(steps): + loss = ((model(x) - y) ** 2).mean() + opt.zero_grad() + loss.backward() + opt.step() + path = ckpts.path / "linear.pt" # resolve the mount inside the function + torch.save(model.state_dict(), path) + return path.name + + +@app.queue(cpu="cpu3c-2-4", mounts={"/runpod-volume": ckpts}, dependencies=["torch"]) +def predict(name: str, x: float) -> float: + import torch + + model = torch.nn.Linear(1, 1) + model.load_state_dict(torch.load(ckpts.path / name, map_location="cpu")) + return model(torch.tensor([[x]])).item() + + +@runpod.local_entrypoint +def main(): + name = train.remote() + y = predict.remote(name, 2.0) + print(f"f(2.0) = {y:.2f}") + assert abs(y - 7.0) < 0.5, y diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 0de251cd..cf016808 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -1,138 +1,184 @@ --- -concepts: [cpu-flavor] +concepts: [flash, cpu-flavor, gpu-pool, network-volume, model-store, runpod-secret] --- -# Flash — Endpoint API & compute-type reference +# Runpod SDK — app API & compute reference -## Endpoint Constructor +Everything is importable from `runpod` (`runpod.App`, `runpod.NetworkVolume`, ...). + +## App and decorators ```python -Endpoint( - name="endpoint-name", # required (unless id= set) - id=None, # connect to existing endpoint - gpu=GpuGroup.AMPERE_80, # GpuGroup tier, GpuType model, or list of either (default: GpuGroup.ANY) - cpu=CpuInstanceType.CPU5C_4_8, # CPU type (mutually exclusive with gpu) - workers=5, # shorthand for (0, 5) - workers=(1, 5), # explicit (min, max) - max_concurrency=1, # concurrent requests per worker (default 1) - idle_timeout=60, # seconds before scale-down (default: 60) - dependencies=["torch"], # pip packages for remote exec - system_dependencies=["ffmpeg"], # apt-get packages - image="org/image:tag", # pre-built Docker image (client mode) - env={"KEY": "val"}, # environment variables - volume=NetworkVolume(...), # persistent storage - datacenter=DataCenter.US_CA_2, # DataCenter | list | str (default: None) - gpu_count=1, # GPUs per worker - template=PodTemplate(containerDiskInGb=100), - flashboot=True, # fast cold starts - accelerate_downloads=True, # speed up model/file downloads (default True) - min_cuda_version=CudaVersion.V12_8, # minimum CUDA version (default 12.8) - scaler_type=ServerlessScalerType.QUEUE_DELAY, # default unset; or REQUEST_COUNT - scaler_value=4, # scaler threshold (default 4) - execution_timeout_ms=0, # max execution time (0 = unlimited) -) +import runpod + +app = runpod.App("my-app") # App(name, *, env="default") ``` -- `gpu=` and `cpu=` are mutually exclusive -- `gpu=` accepts a `GpuGroup`, a `GpuType`, or a list of either (see GPU Types below) -- `workers=5` means `(0, 5)`. Default is `(0, 1)` -- `max_concurrency` -- requests handled concurrently per worker (default 1). Raise it for I/O-bound LB routes so one worker serves multiple requests -- `idle_timeout` default is **60 seconds** -- `flashboot=True` (default) -- enables fast cold starts via snapshot restore -- `gpu_count` -- GPUs per worker (default 1), use >1 for multi-GPU models -- `datacenter` -- a `DataCenter` enum, list, or string; defaults to `None` (unset) -- `scaler_type` -- defaults to `QUEUE_DELAY` for queue-based endpoints and `REQUEST_COUNT` for load-balanced endpoints; pass `ServerlessScalerType.QUEUE_DELAY` or `REQUEST_COUNT` to override -- `DataCenter`, `CudaVersion`, and `ServerlessScalerType` are importable from `runpod_flash` +| Decorator | Wraps | Creates | +|---|---|---| +| `@app.queue(...)` | a function (sync, async, or generator) | queue-based Serverless endpoint | +| `@app.api(...)` | a class with `@runpod.get` / `@runpod.post` methods and an optional `@runpod.init` setup method | load-balanced Serverless endpoint | +| `@app.task(...)` | a function | one Pod per call, terminated when the function returns | +| `@runpod.local_entrypoint` | your `main()` | the code `rp flash dev` runs on your machine | + +### Parameters + +| Parameter | Default | Queue | API | Task | Notes | +|---|---|---|---|---|---| +| `name` | function or class name | ✓ | ✓ | ✓ | endpoint name inside the app | +| `gpu` | any GPU | ✓ | ✓ | ✓ | `"4090"`, `"H100"`, a `GpuGroup`, a `GpuType`, or a list of acceptable ones | +| `cpu` | `None` | ✓ | ✓ | ✓ | a CPU flavor id such as `"cpu5c-4-8"`; exclusive with `gpu` | +| `gpu_count` | `1` | ✓ | ✓ | ✓ | GPUs per worker or Pod | +| `workers` | `(0, 3)` | ✓ | ✓ | | `int` n means `(0, n)` | +| `idle_timeout` | `60` | ✓ | ✓ | | seconds before an idle worker scales down | +| `max_concurrency` | `1` | ✓ | | | concurrent jobs per worker | +| `dependencies` | `None` | ✓ | ✓ | ✓ | pip requirements, installed before your code runs | +| `system_dependencies` | `None` | ✓ | ✓ | ✓ | apt packages; they add to cold start | +| `mounts` | `None` | ✓ | ✓ | ✓ | `{"/runpod-volume": vol}`; see *Storage* | +| `env` | `None` | ✓ | ✓ | ✓ | plain values or `runpod.Secret("name")` | +| `datacenter` | any | ✓ | ✓ | ✓ | a data center id or a list, such as `"US-CA-2"` | +| `image` | Runpod runtime image | ✓ | ✓ | ✓ | any image with Python 3 | +| `registry_auth` | `None` | ✓ | ✓ | ✓ | name of credentials saved with `rp registry add` | +| `model` | `None` | ✓ | ✓ | | a `runpod.Model`, staged on the host before the worker starts | +| `execution_timeout_ms` | `0` (no limit) | ✓ | ✓ | | per-job limit | +| `flashboot` | `True` | ✓ | ✓ | | | +| `scaler_type`, `scaler_value` | by resource type, `4` | ✓ | ✓ | | | +| `min_cuda_version` | `None` | ✓ | ✓ | ✓ | such as `"12.8"` | +| `accelerate_downloads` | `True` | ✓ | ✓ | ✓ | | +| `container_disk_gb` | 30 (GPU), 10 (CPU) | ✓ | ✓ | ✓ | size it for local model caches | + +> **These are SDK defaults, not platform defaults.** REST v2, the console and runpodctl each +> use their own values for `workers`, idle timeout and execution timeout. Set the ones that +> matter explicitly. + +## Calling resources + +On a queue or task handle: + +| Call | Does | +|---|---| +| `fn.remote(*args, **kwargs)` | runs remotely, blocks for the result; no timeout by default | +| `await fn.remote.aio(...)` | async form | +| `fn.spawn(...)` | submits, returns a job immediately | +| `fn.stream(...)` | yields chunks from a generator function | +| `fn.local(...)` | runs the body in the local process | +| `fn.with_options(webhook=..., execution_timeout=..., ttl=..., low_priority=...)` | a copy of the handle with per-job options | + +Queue jobs from `.spawn()`: `job.id`, `job.status()`, `job.result(timeout=...)`, `job.stream()`, +`job.cancel()`, `job.retry()`; each has an `.aio` async form. + +Task jobs from `.spawn()` own a Pod: `await job.wait(timeout=...)` returns the result and then +terminates the Pod (also on timeout or failure); `await job.cancel()` abandons it and +terminates the Pod. If the SDK cannot delete the Pod it warns and keeps the id in `job.pod_id`. + +On an `@app.api` class: `MyApi.post("/path", body)` / `MyApi.get("/path")`, with +`await MyApi.post.aio(...)` as the async form. + +### Deployed resources from other code -> **These defaults are flash-SDK defaults, and differ from the Runpod platform defaults** -> (Console / `runpodctl` / API): `idle_timeout` 60s here vs **5s** on the platform, -> `workers` (0, 1) here vs max **3** on the platform, `execution_timeout` unlimited here vs -> **600s** on the platform. When you read a default, note which layer it belongs to. +```python +import runpod -### NetworkVolume +transcribe = runpod.Queue(app="media", name="transcribe") # env="default" unless given +text = transcribe.remote(audio_url="https://example.com/a.mp3") # keyword arguments only +job = transcribe.spawn(audio_url="...") -```python -NetworkVolume(name="my-vol", size=100) # size in GB, default 100 +inference = runpod.Api(app="media", name="inference") +resp = inference.post("/generate", {"prompt": "hi"}) ``` -### PodTemplate +`Queue(..., id="ENDPOINT_ID")` and `Api(..., id=...)` target an endpoint by id instead of name. + +### Any queue endpoint by id ```python -PodTemplate( - containerDiskInGb=64, # container disk size (default 64) - dockerArgs="", # extra docker arguments - ports="", # exposed ports - startScript="", # script to run on start -) +ep = runpod.Endpoint("ENDPOINT_ID") # also works for endpoints the SDK did not create +out = ep.run_sync({"prompt": "hello"}, timeout=120) # sends {"input": {...}} +job = ep.run({"prompt": "hello"}) +job.status(); job.output(timeout=300); job.cancel() +ep.health(); ep.purge_queue() ``` -## EndpointJob +### Raw HTTP to a deployed queue resource -Returned by `ep.run()` and `ep.runsync()` in client mode. +A deployed queue function reads the job's `input` as its keyword arguments, so a function +`def transcribe(audio_url: str)` takes: -```python -job = await ep.run({"data": [1, 2, 3]}) -await job.wait(timeout=120) # poll until done -print(job.id, job.output, job.error, job.done) -await job.cancel() +```bash +curl -s -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"input": {"audio_url": "https://example.com/a.mp3"}}' ``` -## GPU Types +A function with no parameters still needs a non-empty `input`; the SDK sends +`{"input": {"__empty": true}}`. -`gpu=` accepts a `GpuGroup` (a supply pool by VRAM tier), a `GpuType` (a pinned GPU model), or a list of either. `GpuGroup` picks the cheapest available GPU within a tier; `GpuType` pins a specific model. +### Raw HTTP to a deployed API resource -### GpuGroup (supply pool) +A POST route receives the request's JSON body as its one argument, and a GET route takes none. +For `@runpod.post("/generate") async def generate(self, body: dict)`: -| Enum | GPU | VRAM | -|------|-----|------| -| `ANY` | any | varies | -| `AMPERE_16` | RTX A4000 / A4500 / RTX 4000 Ada / RTX 2000 Ada | 16GB | -| `AMPERE_24` | RTX A5000 / L4 / RTX 3090 | 24GB | -| `AMPERE_48` | A40 / RTX A6000 | 48GB | -| `AMPERE_80` | A100 (PCIe / SXM4) | 80GB | -| `ADA_24` | RTX 4090 | 24GB | -| `ADA_32_PRO` | RTX 5090 | 32GB | -| `ADA_48_PRO` | RTX 6000 Ada / L40 / L40S | 48GB | -| `ADA_80_PRO` | H100 PCIe (80GB) / H100 HBM3 (80GB) / H100 NVL (94GB) | 80GB+ | -| `HOPPER_141` | H200 | 141GB | -| `BLACKWELL_96` | RTX PRO 6000 Blackwell | 96GB | -| `BLACKWELL_180` | B200 | 180GB | +```bash +curl -s -X POST "https://$ENDPOINT_ID.api.runpod.ai/generate" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"prompt": "hi"}' +``` -### GpuType (pinned model) +The body is the object itself, with no wrapper key. -Pin an exact GPU model. Members include `NVIDIA_GEFORCE_RTX_4090`, `NVIDIA_GEFORCE_RTX_5090`, `NVIDIA_RTX_6000_ADA_GENERATION`, `NVIDIA_H100_80GB_HBM3`, `NVIDIA_A100_80GB_PCIe`, `NVIDIA_A100_SXM4_80GB`, `NVIDIA_H200`, `NVIDIA_B200`, the `NVIDIA_RTX_PRO_6000_BLACKWELL_*` editions (Server / Workstation / Max-Q), and the Ampere/Ada RTX A-series models (`NVIDIA_RTX_A4000`, `A4500`, `A5000`, `A6000`, `NVIDIA_L4`, `NVIDIA_A40`, `NVIDIA_GEFORCE_RTX_3090`, `NVIDIA_RTX_4000_ADA_GENERATION`, `NVIDIA_RTX_2000_ADA_GENERATION`). +## Storage ```python -from runpod_flash import Endpoint, GpuType - -@Endpoint(name="pinned", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, dependencies=["torch"]) -async def report_gpu(data): - import torch - return {"gpu": torch.cuda.get_device_name(0)} +models = runpod.NetworkVolume("models", size=100, datacenter="US-CA-2") # created on first use +shared = runpod.GlobalVolume("shared") # not tied to a data center +existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must already exist ``` -## CPU Types (CpuInstanceType) - -| Enum | vCPU | RAM | Max Disk | Type | -|------|------|-----|----------|------| -| `CPU3G_1_4` | 1 | 4GB | 10GB | General | -| `CPU3G_2_8` | 2 | 8GB | 20GB | General | -| `CPU3G_4_16` | 4 | 16GB | 40GB | General | -| `CPU3G_8_32` | 8 | 32GB | 80GB | General | -| `CPU3C_1_2` | 1 | 2GB | 10GB | Compute | -| `CPU3C_2_4` | 2 | 4GB | 20GB | Compute | -| `CPU3C_4_8` | 4 | 8GB | 40GB | Compute | -| `CPU3C_8_16` | 8 | 16GB | 80GB | Compute | -| `CPU5C_1_2` | 1 | 2GB | 15GB | Compute (5th gen) | -| `CPU5C_2_4` | 2 | 4GB | 30GB | Compute (5th gen) | -| `CPU5C_4_8` | 4 | 8GB | 60GB | Compute (5th gen) | -| `CPU5C_8_16` | 8 | 16GB | 120GB | Compute (5th gen) | +- Queue and API resources mount **one** volume, at `/runpod-volume`: + `mounts={"/runpod-volume": models}`. +- Tasks mount up to one network volume and one global volume, at distinct paths. +- Global volumes need GPU compute. +- A new network volume is placed in a data center that has network storage and stock for every + resource sharing it; resources using it are scheduled in that data center. +- Read `models.path` **inside** the remote function. Volumes are not mounted locally. +- Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. + +## Models and secrets ```python -from runpod_flash import Endpoint, CpuInstanceType +llama = runpod.Model("meta-llama/Llama-3.1-8B-Instruct") # "owner/name" or "owner/name:revision" -@Endpoint(name="cpu-work", cpu=CpuInstanceType.CPU5C_4_8, workers=5, dependencies=["pandas"]) -async def process(data): - import pandas as pd - return pd.DataFrame(data).describe().to_dict() +@app.queue(gpu="H100", model=llama, env={"HF_TOKEN": runpod.Secret("hf-token")}) +def chat(prompt: str): + import vllm + llm = vllm.LLM(model=str(llama.path)) # staged weights; transformers/vllm also find the HF cache + ... ``` + +The platform stages the weights on the host before the worker starts, so a cold start does +not download them. Gated models need `HF_TOKEN` on the same resource. Create the secret with +`rp secret add hf-token`. + +## GPU and CPU names + +`gpu=` accepts: + +- **A fragment of the device name**: `"4090"`, `"H100"`, `"B200"`, `"L4"`. Every device whose + name contains it matches, so `"H100"` covers the H100 variants. +- **A `runpod.GpuGroup` pool**: `ANY`, `AMPERE_16`, `AMPERE_24`, `AMPERE_48`, `AMPERE_80`, + `ADA_24`, `ADA_32_PRO`, `ADA_48_PRO`, `ADA_80_PRO`, `HOPPER_141`, `BLACKWELL_96`, + `BLACKWELL_180`. The VRAM tiers match the GPU pool table in + [gpu-pool](../../runpod-usage/concepts/gpu-pool.yaml). +- **A `runpod.GpuType` member** for an exact model, such as `GpuType.NVIDIA_GEFORCE_RTX_4090`. +- **A list** of any of these. Automatic switching between listed GPUs needs a max of at least 5 + workers. + +`cpu=` takes a flavor id: `cpu3g-{1-4,2-8,4-16,8-32}` (general), `cpu3c-{1-2,2-4,4-8,8-16}` +(compute), `cpu5c-{1-2,2-4,4-8,8-16}` (compute, 5th gen), where the numbers are vCPUs and GB of +RAM. `runpod.CpuInstanceType` holds the same values. + +## Exact rules + +- [flash](../../runpod-usage/concepts/flash.yaml) +- [cpu-flavor](../../runpod-usage/concepts/cpu-flavor.yaml) +- [model-store](../../runpod-usage/concepts/model-store.yaml) diff --git a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md new file mode 100644 index 00000000..5a595eff --- /dev/null +++ b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md @@ -0,0 +1,153 @@ +--- +concepts: [flash] +--- + +# Migrate a `runpod-flash` project to the Runpod SDK + +`runpod-flash` is deprecated on October 29, 2026 and gets no fixes after that date. Its +features live in the `runpod` package (2.0) as SDK apps with the `rp flash` CLI. Endpoints +already deployed with `runpod-flash` keep running; migrating changes how you build and deploy, +not what is running. + +Run this as a plan with the user, not a silent rewrite: inventory, check *Known differences*, +convert, smoke-test with `rp flash dev --once`, deploy, then remove the old endpoints. + +## 1. Inventory + +```bash +grep -rnE "runpod_flash|runpod-flash|@Endpoint|flash (dev|deploy|build|login|undeploy|app|env)" \ + --include='*.py' --include='*.toml' --include='*.txt' --include='*.cfg' \ + --include='Makefile' --include='*.yml' --include='*.yaml' --include='*.sh' . +``` + +List every `Endpoint`, its mode (decorator, load-balanced routes, `image=`, `id=`), and every +CI step that calls `flash`. + +## 2. Check known differences first + +These patterns behave differently in the SDK. If the project relies on one, tell the user +before converting, and test it carefully or keep the project on `runpod-flash`: + +- Projects split across multiple local files. Discovery ships one module per app. +- Classes as remote resources, other than `@app.api` classes. +- Path and query parameters on HTTP routes. +- Resources that reference an existing template or endpoint with `template=` or `id=`. +- `volume=[...]` lists across data centers. The SDK mounts one volume per queue or API + resource. +- Local testing that curls `localhost:8888`. Dev sessions run no local server. + +## 3. Install and log in + +```bash +pip uninstall runpod-flash && pip install "runpod>=2.0.0rc1" +# uv: uv remove runpod-flash && uv add "runpod>=2.0.0rc1" +rp login # a key saved by `flash login` keeps working +``` + +## 4. Convert the code + +| runpod-flash | Runpod SDK | +|---|---| +| `from runpod_flash import Endpoint, GpuGroup, GpuType` | `import runpod` | +| project folder = app | `app = runpod.App("APP_NAME")` | +| `@Endpoint(name="x", gpu=..., ...)` on a function | `@app.queue(name="x", gpu=...)` | +| `api = Endpoint(name=..., ...)` + `@api.post("/p")` functions | `@app.api(...)` on a class with `@runpod.post("/p")` methods | +| class `@Endpoint` with `__init__` loading a model | `@app.api` class with a `@runpod.init` method | +| `await fn(x)` | `fn.remote(x)` or `await fn.remote.aio(x)` | +| `asyncio.gather(fn(a), fn(b))` | `[fn.spawn(a), fn.spawn(b)]` then `job.result()` | +| `if __name__ == "__main__": asyncio.run(main())` | `@runpod.local_entrypoint` on `main`, run with `rp flash dev main.py` | +| `gpu=GpuType.NVIDIA_GEFORCE_RTX_4090` | `gpu="4090"` (or `runpod.GpuType...`) | +| `gpu=GpuGroup.ADA_24` | `gpu="ADA_24"` (or `runpod.GpuGroup.ADA_24`) | +| `cpu=CpuInstanceType.CPU5C_4_8` | `cpu="cpu5c-4-8"` | +| `NetworkVolume(name="m", size=100, datacenter=DataCenter.US_CA_2)` | `runpod.NetworkVolume("m", size=100, datacenter="US-CA-2")` | +| `volume=vol`, files at `/runpod-volume/...` | `mounts={"/runpod-volume": vol}`, read `vol.path` inside the function | +| `template=PodTemplate(containerDiskInGb=100)` | `container_disk_gb=100` | +| `min_cuda_version=CudaVersion.V12_8` | `min_cuda_version="12.8"` | +| `env={"HF_TOKEN": "hf_..."}` | `env={"HF_TOKEN": runpod.Secret("hf-token")}` after `rp secret add hf-token` | +| `Endpoint(id="ID")` + `await ep.runsync(...)` | `runpod.Endpoint("ID").run_sync({...}, timeout=120)` | +| `Endpoint(name=..., image="IMG")` (image is a Serverless worker) | `image="IMG"` on a decorator (any Python 3 image); a prebuilt worker image goes through runpodctl/runpod-mcp + `runpod.Endpoint` | +| `idle_timeout`, `execution_timeout_ms`, `flashboot`, `scaler_type`, `scaler_value`, `accelerate_downloads`, `datacenter`, `dependencies`, `system_dependencies`, `gpu_count`, `max_concurrency` | same names on the decorator | + +Before and after for a queue function: + +```python +# runpod-flash +from runpod_flash import Endpoint, GpuType + +@Endpoint(name="hello-gpu", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, dependencies=["numpy"]) +async def hello(): + import torch + return {"gpu": torch.cuda.get_device_name(0)} +``` + +```python +# Runpod SDK +import runpod + +app = runpod.App("hello-gpu") + +@app.queue(gpu="4090", dependencies=["numpy"]) +def hello(): + import torch + return {"gpu": torch.cuda.get_device_name(0)} + +@runpod.local_entrypoint +def main(): + print(hello.remote()) +``` + +Load-balanced routes become a class: + +```python +# runpod-flash +api = Endpoint(name="my-api", gpu=GpuGroup.ADA_24, workers=(1, 5)) + +@api.post("/predict") +async def predict(data: list[float]): ... +``` + +```python +# Runpod SDK +@app.api(gpu="ADA_24", workers=(1, 5)) +class MyApi: + @runpod.post("/predict") + async def predict(self, body: dict): ... # body is the request JSON +``` + +## 5. Behavior that changes without an error + +- **Default `workers` is `(0, 3)`**, not `(0, 1)`. Set it explicitly to keep the old ceiling. +- **The whole module ships** in dev and deploy alike, so the runpod-flash "only the function + body ships under `flash dev`" workaround (imports and constants inside the body, the + `try: _MODEL except NameError` cache) is no longer needed. It still works. +- **Queue arguments must be JSON-serializable**, in dev as well as deployed. +- **Raw HTTP body shapes.** Queue endpoints keep the runpod-flash shape: the function's + keyword arguments go in `input` (`{"input": {"data": {...}}}` for `def f(data)`). + Load-balanced routes change: a POST route receives the whole JSON body, so callers that sent + `{"data": {...}}` to a runpod-flash route `def predict(data)` now send the object itself. +- **`.flashignore` becomes `.runpodignore`**, and `.gitignore` files apply too. +- **Local Python must be 3.10–3.14** and matches the worker's in dev sessions and tasks. +- **Volumes outlive undeploy** and keep billing. + +## 6. Smoke-test, deploy, clean up + +```bash +rp flash dev main.py --once # exits non-zero if the entrypoint fails +rp flash deploy # creates the new endpoints (default environment) +rp flash app list # confirm +``` + +Then remove the endpoints deployed with `runpod-flash`, scoped to that app: +`flash app delete OLD_APP` (with the old CLI still installed), or +`runpodctl serverless delete ENDPOINT_ID` for each old endpoint id. Update CI to call `rp flash` +instead of `flash`, and update any client that called the old endpoint ids. + +## Silencing the notice in CI + +`runpod-flash` 1.21.0 and later print a deprecation notice on every `flash` command and a +`FutureWarning` on import. Set `RUNPOD_FLASH_SILENCE_DEPRECATION=1` to silence both while a +pipeline still runs the old package. + +## Exact rules + +- [flash](../../runpod-usage/concepts/flash.yaml) diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index 2d5c7d80..7fe2de07 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -1,128 +1,198 @@ --- concepts: [flash, serverless-handler, network-volume, container-disk, volume-cache, model-store] --- -# Flash — Common Patterns +# Runpod SDK — common patterns + +Complete runnable versions live in [`../examples/`](../examples/). ## Choosing a model -Flash has **no model catalog** — name a HuggingFace repo id in code and it downloads to the -worker at runtime (see *Loading ML models* below). Other sources: a custom image's `MODEL_NAME` env -(vLLM etc.), a URL, or your own weights on a NetworkVolume. +The SDK has **no model catalog**: name a HuggingFace repo id in code and the worker downloads +it at runtime, or attach it with `runpod.Model` so the platform stages it on the host first. - **Start with the smallest model that proves the pipeline** (`gpt2`, `stabilityai/sd-turbo`, - a 0.5–1B variant) — it provisions in seconds, so you validate the `@Endpoint` wiring, deps, - GPU, and I/O fast under `flash dev`, then change *only the id string* to the real model. + a 0.5–1B variant). It loads in seconds, so you validate the decorator, dependencies, GPU and + I/O fast under `rp flash dev`, then change only the id string. - **Match the model to GPU VRAM** (fp16 ≈ params × 2 bytes + overhead): | Model (fp16) | ~VRAM | `gpu=` | |---|---|---| - | ≤3B / SD1.5 / sd-turbo | ≤8 GB | `GpuGroup.AMPERE_16` or `GpuGroup.ADA_24` | - | 7–8B | ~16 GB | `GpuGroup.ADA_24` or `GpuGroup.AMPERE_24` | - | 13B | ~28 GB | `GpuGroup.ADA_32_PRO` or `GpuGroup.AMPERE_48` | - | 70B | ~140 GB | `GpuGroup.HOPPER_141` / `GpuGroup.BLACKWELL_180` (or quantize) | - -- A ready-made hosted model with **no code** is [Runpod Public Endpoints / Hub](https://docs.runpod.io/hub) — a different product, not Flash. - -> **Big model? Naming it still works — but mind the re-download.** Model size is not a -> flash limit: naming a large HF repo streams the weights to the worker at runtime. The -> catch is that a scaled-to-zero worker **re-downloads on every cold start**. For a large -> model you call often, cache it so it isn't re-pulled each time — persist to a -> **NetworkVolume** with `VolumeCache` (see *Loading ML models* below), or on the runpodctl/serverless side use -> the **HF model cache** (`--model-reference`, attached after `flash deploy`) or bake it into the image. This is the same -> tradeoff as the delivery-methods table in -> [`runpodctl/reference/model-caching.md`](../../runpodctl/reference/model-caching.md): easy -> streaming vs. faster/cheaper cold starts for reused weights — not a contradiction. - -## Loading ML models (warm workers) - -Model **weights are not part of the 1.5GB build artifact** — that cap is your code + pip -deps (torch is auto-excluded). Weights download on the worker at runtime (HuggingFace, -etc.), so **model size is not a Flash limit**. Two things make this fast and cheap: - -- **Load once per worker, not per request** — use a *class* `@Endpoint`: `__init__` loads - the model into VRAM once when the worker starts; methods handle requests and reuse it. -- **Keep the weights close so a cold worker doesn't re-pull them from the Hub.** Two ways: - - **Model Store**, when the model is on HuggingFace and the endpoint is GPU: no volume and - no cache code. Set `env={"HF_HUB_CACHE": "/runpod-volume/huggingface-cache/hub"}` on the - `@Endpoint`, then after `flash deploy` attach the model with - `runpodctl serverless update --model-reference https://huggingface.co//:main`. - Later `flash deploy` runs keep the reference. Worker start waits until a host holds the - model, so time the whole cold request; if that wait is too long, use the volume below. - - **A NetworkVolume mirrored with `VolumeCache`**: a cold worker copies the weights an - earlier worker downloaded onto its local disk and loads from there, not over the network - mount. Keep the HF cache on container disk (not under `/runpod-volume`) and wrap the load - in `VolumeCache` from the Runpod Python SDK (`runpod>=1.12.0`). It hydrates the local cache - from the volume's mirror, then syncs new files back: + | ≤3B / SD1.5 / sd-turbo | ≤8 GB | `"AMPERE_16"` or `"4090"` | + | 7–8B | ~16 GB | `"4090"`, `"L4"`, `"AMPERE_24"` | + | 13B | ~28 GB | `"5090"` or `"AMPERE_48"` | + | 70B | ~140 GB | `"H200"` / `"B200"` (or quantize) | + +- A hosted model with **no code** is a [Public Endpoint](https://docs.runpod.io/hub/public-endpoints), + a different product. + +## Loading ML models + +Model weights are not part of the 1,500 MB build artifact; the artifact is your code plus pip +dependencies. Two things make loading fast and cheap. + +**Load once per worker, not per request.** Put the load in an `@app.api` class's +`@runpod.init` method. It runs once when a worker starts, and every route reuses `self`: ```python -from runpod_flash import Endpoint, GpuType, DataCenter, NetworkVolume - -vol = NetworkVolume(name="model-cache", size=100, datacenter=DataCenter.US_GA_2) - -@Endpoint( - name="sd", - gpu=GpuType.NVIDIA_GEFORCE_RTX_5090, - workers=(0, 3), - idle_timeout=300, # keep workers warm between calls - datacenter=DataCenter.US_GA_2, - volume=vol, - env={"HF_HUB_CACHE": "/opt/hf-cache"}, # local container disk, mirrored by VolumeCache - dependencies=["torch", "diffusers", "transformers", "accelerate", "runpod>=1.12.0"], -) +import runpod + +app = runpod.App("sd") + +@app.api(gpu="5090", workers=(0, 3), idle_timeout=300, + dependencies=["diffusers", "transformers", "accelerate"]) class SD: - def __init__(self): # runs ONCE per worker + @runpod.init + def load(self): import torch from diffusers import StableDiffusionPipeline - from runpod.serverless import VolumeCache + self.pipe = StableDiffusionPipeline.from_pretrained( + "stabilityai/sd-turbo", torch_dtype=torch.float16).to("cuda") + + @runpod.post("/generate") + async def generate(self, body: dict): + import base64, io + image = self.pipe(prompt=body["prompt"], num_inference_steps=1).images[0] + buf = io.BytesIO(); image.save(buf, format="PNG") + return {"png_b64": base64.b64encode(buf.getvalue()).decode()} +``` - with VolumeCache(dirs=["/opt/hf-cache"]): # hydrate from the volume, then sync back - self.pipe = StableDiffusionPipeline.from_pretrained( - "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16 - ).to("cuda") +In a queue function, cache in a module global. Deployed workers import the module once, so the +load runs once per worker: - async def generate(self, prompt: str) -> dict: # per request, reuses self.pipe - image = self.pipe(prompt=prompt).images[0] - image.save("/runpod-volume/out.png") # /runpod-volume/ persists; elsewhere is wiped - return {"saved": "/runpod-volume/out.png"} +```python +_PIPE = None + +@app.queue(gpu="4090", dependencies=["transformers"]) +def generate(prompt: str) -> str: + global _PIPE + if _PIPE is None: + from transformers import pipeline + _PIPE = pipeline("text-generation", model="gpt2", device=0) + return _PIPE(prompt, max_new_tokens=20)[0]["generated_text"] ``` -- **Gated** models: pass `env={"HF_TOKEN": "..."}`. -- **`VolumeCache` is best-effort.** Any cache error is logged and swallowed, and the worker - does a normal cold start, so a broken cache looks like a slow cold start. Pass - `VolumeCache(..., best_effort=False)` while debugging. It never copies symlinks, which the HF - cache uses for `snapshots/`. Leave Hub access on (don't set `HF_HUB_OFFLINE=1`) so - `from_pretrained` can re-link the snapshot with a metadata call. Size the volume for the - mirror (`/runpod-volume/.cache/`), and size the **container disk** for the whole - local cache plus headroom. It defaults to 64 GB, so a 70B fp16 model (~140 GB) needs - `template=PodTemplate(containerDiskInGb=200)` on the `@Endpoint`. Let one worker populate it first: workers - that cold-start together on an empty mirror each download and sync a full copy. Full - behavior: [`concepts/volume-cache.yaml`](../../runpod-usage/concepts/volume-cache.yaml). -- `workers=(1, n)` keeps one worker warm (no cold start on the first request); `(0, n)` scales to zero and cold-starts after `idle_timeout`. -- The class form is the cleanest way to load once. In function-form `@Endpoint` the same effect needs the module-global cache trick (see Gotcha #11 in the skill); the class form is preferred for real inference. - -## CPU + GPU Pipeline +**Keep the weights close to the worker.** A scaled-to-zero worker re-downloads on every cold +start unless you cache. Pick one: + +- **`runpod.Model`** (HuggingFace model, GPU resource). The platform stages the weights on the + host and holds the worker until they are ready, so the cold request includes that wait. Time + the whole cold request before deciding it is fast enough. + + ```python + qwen = runpod.Model("Qwen/Qwen2.5-7B-Instruct") + + @app.queue(gpu="4090", model=qwen, dependencies=["transformers", "accelerate"]) + def chat(prompt: str) -> str: + from transformers import pipeline + pipe = pipeline("text-generation", model=str(qwen.path), device=0) + return pipe(prompt, max_new_tokens=64)[0]["generated_text"] + ``` + +- **A `NetworkVolume` mirrored with `VolumeCache`.** A cold worker copies weights an earlier + worker downloaded onto its local disk and loads from there instead of over the network mount. + Keep the HF cache on container disk (not under `/runpod-volume`) and wrap the load: + + ```python + cache = runpod.NetworkVolume("model-cache", size=100) + + @app.api(gpu="5090", mounts={"/runpod-volume": cache}, container_disk_gb=60, + env={"HF_HUB_CACHE": "/opt/hf-cache"}, dependencies=["transformers", "accelerate"]) + class Chat: + @runpod.init + def load(self): + from runpod.serverless import VolumeCache + from transformers import pipeline + with VolumeCache(dirs=["/opt/hf-cache"]): # hydrate from the volume, then sync back + self.pipe = pipeline("text-generation", model="Qwen/Qwen2.5-7B-Instruct", device=0) + ``` + + `rp flash deploy` vendors your local `runpod` package into the artifact, so + `runpod.serverless.VolumeCache` is importable on the worker without listing `runpod` in + `dependencies`. + +`VolumeCache` details that bite: +- **Best-effort.** Cache errors are logged and swallowed, and the worker cold-starts normally, + so a broken cache looks like a slow cold start. Use `VolumeCache(..., best_effort=False)` + while debugging. +- It never copies symlinks, which the HF cache uses for `snapshots/`. Leave Hub access on + (don't set `HF_HUB_OFFLINE=1`) so `from_pretrained` can re-link with a metadata call. +- Size the volume for the mirror and `container_disk_gb` for the whole local cache plus + headroom; the default is 30 GB on GPU. +- Let one worker populate the mirror first. Workers that cold-start together on an empty mirror + each download and sync a full copy. + +Full behavior: [`concepts/volume-cache.yaml`](../../runpod-usage/concepts/volume-cache.yaml). + +## CPU + GPU pipeline ```python -from runpod_flash import Endpoint, GpuGroup, CpuInstanceType - -@Endpoint(name="preprocess", cpu=CpuInstanceType.CPU5C_4_8, workers=5, dependencies=["pandas"]) -async def preprocess(raw): +@app.queue(cpu="cpu5c-4-8", dependencies=["pandas"]) +def preprocess(rows: list[dict]) -> list[list[float]]: import pandas as pd - return pd.DataFrame(raw).to_dict("records") + return pd.DataFrame(rows).dropna().astype(float).values.tolist() -@Endpoint(name="infer", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"]) -async def infer(clean): +@app.queue(gpu="AMPERE_80") +def score(matrix: list[list[float]]) -> list[float]: import torch - t = torch.tensor([[v for v in r.values()] for r in clean], device="cuda") - return {"predictions": t.mean(dim=1).tolist()} + return torch.tensor(matrix, device="cuda").mean(dim=1).tolist() + +@runpod.local_entrypoint +def main(): + print(score.remote(preprocess.remote([{"a": 1, "b": 3}, {"a": 5, "b": 7}]))) +``` + +Each stage is its own endpoint with its own workers and hardware. A remote function can also +call another resource's `.remote()` directly. + +## Parallel execution -async def pipeline(data): - return await infer(await preprocess(data)) +```python +jobs = [score.spawn(batch) for batch in batches] # submit all, don't wait +results = [job.result() for job in jobs] # collect + +# async code +results = await asyncio.gather(*(score.remote.aio(b) for b in batches)) +``` + +## Training jobs as tasks + +`@app.task` gets a Pod per call and terminates it when the function returns, which fits +training. Share artifacts through a volume: + +```python +ckpts = runpod.NetworkVolume("ckpts", size=50) + +@app.task(gpu="H100", gpu_count=2, mounts={"/ckpts": ckpts}) +def finetune(steps: int = 1000) -> str: + ... + return str(ckpts.path / "adapter") ``` -## Parallel Execution +A queue endpoint that mounts the same volume at `/runpod-volume` reads what the task wrote; see +[`train_then_serve.py`](../examples/train_then_serve.py). For a long run, `job = finetune.spawn()` +then `await job.wait(timeout=...)`, which terminates the Pod when it finishes or times out. + +## Streaming results ```python -import asyncio -results = await asyncio.gather(compute(a), compute(b), compute(c)) +@app.queue(cpu="cpu3c-1-2") +def tokens(prompt: str): + for word in prompt.split(): + yield word + +for chunk in tokens.stream("hello streaming world"): + print(chunk) ``` + +## Returning binary output + +Requests and results go through the job API as JSON, capped at 10 MB. Return images and audio +as base64 (`{"png_b64": ...}`) and decode on the client. For larger outputs, write to a volume +or object storage and return a path or URL. + +## Calling from outside Python + +Deployed queue resources take their keyword arguments as the job `input`: +`def generate(prompt: str)` takes `{"input": {"prompt": "hi"}}` on `/run` or `/runsync`. +Deployed API routes take the JSON body itself on `https://ENDPOINT_ID.api.runpod.ai/PATH`. +Full examples in [api.md](api.md#raw-http-to-a-deployed-queue-resource). diff --git a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md new file mode 100644 index 00000000..1647f198 --- /dev/null +++ b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md @@ -0,0 +1,93 @@ +--- +concepts: [flash, serverless-handler, job, container-disk] +--- + +# Legacy `runpod-flash` reference + +Use this only to maintain a project that still imports `runpod_flash`. `runpod-flash` is +deprecated on October 29, 2026 and gets no fixes after that date; propose +[migrating](migrate-from-runpod-flash.md). New code goes on the Runpod SDK +([../SKILL.md](../SKILL.md)). `flash --help` is authoritative for the installed version. + +## Setup and CLI + +```bash +uv tool install runpod-flash # Python 3.10-3.13; or: pip install runpod-flash +flash login # saves a key to ~/.runpod/config.toml (--no-open for headless) +flash init my-project # scaffold (writes AGENTS.md + CLAUDE.md) +flash dev # local server on :8888; functions run on remote workers +flash dev --auto-provision # pre-provision endpoints to skip the first cold start +flash deploy [--env NAME] # build (<=1500 MB, torch excluded) and deploy +flash app list | get APP | delete APP +flash env list | create NAME | get NAME | delete NAME +flash undeploy NAME [--force] # one endpoint +flash update # upgrade the CLI +``` + +1.21.0 and later print a deprecation notice; `RUNPOD_FLASH_SILENCE_DEPRECATION=1` silences it. + +## Modes + +```python +from runpod_flash import Endpoint, GpuGroup, PodTemplate + +@Endpoint(name="worker", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"]) +async def compute(data): # queue-based: one function, one endpoint + import torch + return {"sum": torch.tensor(data, device="cuda").sum().item()} + +api = Endpoint(name="my-api", gpu=GpuGroup.ADA_24, workers=(1, 5)) +@api.post("/predict") # load-balanced: routes share one worker pool +async def predict(data: list[float]): ... + +server = Endpoint(name="srv", image="org/img:tag", gpu=GpuGroup.AMPERE_80, + template=PodTemplate(containerDiskInGb=100)) # client mode: prebuilt worker image +ep = Endpoint(id="abc123") # client mode: existing endpoint +job = await ep.runsync({"prompt": "hello"}, timeout=120) +``` + +`name=` only is decorator mode; `image=` or `id=` is client mode. `gpu=` and `cpu=` are +exclusive. A class `@Endpoint` whose `__init__` loads a model runs that load once per worker. + +## Defaults + +`Endpoint` defaults: `workers` `(0, 1)`, `idle_timeout` 60 seconds, `execution_timeout_ms` 0 +(unlimited), `flashboot=True`, `min_cuda_version` 12.8 (the `runpod/flash` image needs it; lower +it only with an image that supports older CUDA), `PodTemplate(containerDiskInGb=64)`. These +differ from the platform defaults. + +## Dev server loop + +```bash +flash dev > /tmp/flash-dev.log 2>&1 & +for i in $(seq 1 60); do grep -q "flash dev localhost:" /tmp/flash-dev.log && break; sleep 2; done +URL=$(grep -o "localhost:[0-9]*" /tmp/flash-dev.log | head -1) # the port bumps if 8888 is taken +``` + +- Routes are namespaced by file: `main.py`'s `/predict` is `/main/predict`. +- Load-balanced routes take the handler argument at the top level: `def predict(data)` wants + `{"data": {...}}`. +- Queue endpoints are served at `/main/runsync` and want the body double-wrapped: + `def synthesize(data)` wants `{"input": {"data": {...}}}`. The worker calls + `handler(**job_input)`, so `input` keys must match parameter names, and an empty `input` is + rejected with `Job has missing field(s): id or input`. + +## Gotchas + +1. **`flash dev` ships only the function body.** Module-level constants and imports raise + `NameError` on the worker under `flash dev`, while `flash deploy` imports the whole module + and hides the bug. Put imports and helpers inside the function. To cache a model in + function form, use a module global set inside the body + (`global _M` / `try: _M` / `except NameError: _M = load()`). +2. **Every decorated function and client call needs `await`.** +3. **`runsync` times out after 60 seconds**; raise `timeout=` or use `run()` + `job.wait()` for + cold starts. +4. **A set `RUNPOD_API_KEY` overrides `flash login`.** A bad key logs + `GraphQL request failed: 401` while `flash dev` still prints its ready line. +5. **Teardown with `flash app delete APP`.** `flash undeploy list` can miss a deployed app. +6. **10 MB payload limit**; native CUDA libraries go in `dependencies`; `system_dependencies` + add to cold start; automatic GPU switching needs `workers` ≥ 5. These carry over to the SDK. + +## Exact rules + +- [flash](../../runpod-usage/concepts/flash.yaml) diff --git a/plugins/runpod/skills/flash/reference/setup-and-cli.md b/plugins/runpod/skills/flash/reference/setup-and-cli.md index 0ceeb5d5..eaec89e3 100644 --- a/plugins/runpod/skills/flash/reference/setup-and-cli.md +++ b/plugins/runpod/skills/flash/reference/setup-and-cli.md @@ -1,90 +1,103 @@ -# Flash — Setup & CLI reference +--- +concepts: [flash, api-key] +--- + +# Runpod SDK — setup & `rp` CLI reference ## Setup +The SDK and the `rp` CLI ship together in the `runpod` package (2.0 and later). Install it +into the project's virtual environment, because dev sessions run your local interpreter. + ```bash -# install the CLI — requires Python 3.10-3.13 (NOT 3.14+ yet) -uv tool install runpod-flash -pip install runpod-flash -# on Python 3.14+ the install fails — pin an older interpreter for the tool: -uv tool install --python 3.13 runpod-flash - -# auth option 1: browser-based login (saves token locally) -flash login -# headless: print URL instead of opening a browser -flash login --no-open -# max seconds to wait for browser auth (default 600) -flash login --timeout 300 - -# auth option 2: API key via environment variable -export RUNPOD_API_KEY=your_key - -# scaffold a new project in ./my-project (writes AGENTS.md + CLAUDE.md) -flash init my-project -# scaffold in the current directory -flash init . -# overwrite existing files (-f) -flash init my-project --force -# update the CLI to the latest version -flash update -# pin a specific version (-V also works) -flash update --version 1.16.0 +# requires Python 3.10-3.14 +uv add "runpod>=2.0.0rc1" # uv project +pip install "runpod>=2.0.0rc1" # pip / venv +pip show runpod # confirm the version (2.x) +rp update # upgrade the package in place; rp update --version X pins one ``` -`flash init` writes `AGENTS.md` (+ a `CLAUDE.md` symlink). To add them to an existing project: `python -c "from runpod_flash.rules import install_agent_files; from pathlib import Path; install_agent_files(Path.cwd())"`. +The `>=2.0.0rc1` specifier also accepts the 2.0 release candidate, so it resolves to 2.x +whether or not a stable 2.0.0 is published. A plain `pip install runpod` on an index with only +the release candidate installs 1.x, which has no `rp flash`. -**Auth precedence:** a set `RUNPOD_API_KEY` env var **overrides** the saved `flash login` token, so an exported bad/expired key silently beats a good login — a common trap (see Gotcha #13 in the skill). +The CLI is also installed as `runpod`, so `runpod flash deploy` and `rp flash deploy` are the +same command. -## CLI +### Auth + +```bash +rp login # browser approval; saves the key to ~/.runpod/config.toml +rp login --no-open # headless: print the approval URL instead of opening a browser +rp login --api-key "$KEY" # store a key directly, no browser +export RUNPOD_API_KEY=your_key # headless agents and CI +``` -`flash dev` is the canonical dev-server command (`flash run` still works as a hidden alias). +- The SDK reads `RUNPOD_API_KEY` first, then `~/.runpod/config.toml`. A set env var wins, so + a stale exported key causes 401s even after a good `rp login`. +- `runpodctl` reads the same `config.toml`, so one `rp login` serves both. A key saved by the + legacy `flash login` keeps working. +- With no key at all, calls fail with "no api key configured. run `rp login` or set RUNPOD_API_KEY." + +### Scaffold ```bash -# local server at :8888, but functions run on REMOTE GPU/CPU workers; -# hot-reloads on save and streams the worker's logs live to your terminal -flash dev -# same, but pre-provision endpoints (no cold start on first call) -flash dev --auto-provision -# custom port/host; --reload/--no-reload toggles autoreload -flash dev --port 9000 --host 0.0.0.0 -# build + deploy (auto-selects env if only one) -flash deploy -# build + deploy to "staging" environment -flash deploy --env staging -# deploy a specific app to an environment -flash deploy --app my-app --env prod -# build + launch local preview in Docker -flash deploy --preview -# build flags below also apply to deploy -flash deploy --no-deps --python-version 3.11 -# list deployment environments -flash env list -# create "staging" environment -flash env create staging -# show environment details + resources -flash env get staging -# delete environment + tear down resources -flash env delete staging -# list flash apps in your account -flash app list -# create a flash app -flash app create my-app -# show an app's environments + builds -flash app get my-app -# delete an app and all its resources -flash app delete my-app -# list all active endpoints -flash undeploy list -# remove a specific endpoint -flash undeploy my-endpoint -# remove all endpoints (--interactive/-i to pick, --force/-f to skip prompts) -flash undeploy --all -# remove endpoints whose code no longer exists locally -flash undeploy --cleanup-stale - -# build-only (no deploy) — mainly for debugging the artifact; `flash deploy` builds for you -# package the artifact without deploying (1500MB limit; torch auto-excluded) -flash build -# build flags: --no-deps, --exclude pkg1,pkg2, --output name.tar.gz, --python-version 3.11 -flash build --no-deps +rp flash init my-app # writes my-app/main.py, requirements.txt, .runpodignore +rp flash init . # scaffold into the current directory +rp flash init my-app -f # overwrite existing files ``` + +The scaffold is a CPU queue function plus a `@runpod.local_entrypoint`, so +`cd my-app && rp flash dev main.py --once` is a working first run. + +## CLI + +```bash +rp --help # top level: flash, login, logs, pod, registry, secret, ssh, update + +# develop +rp flash dev main.py # temporary endpoints + run the entrypoint, re-run on save, ^C cleans up +rp flash dev main.py --once # run once, tear down, exit non-zero on failure (CI smoke test) + +# deploy +rp flash deploy # every app found in the cwd -> "default" environment +rp flash deploy main.py --env staging # one target file, another environment +rp flash deploy --python-version 3.12 # worker Python for dependency wheels (defaults to the local one) +rp flash deploy --exclude scipy,numba # leave packages out of the artifact (they must come from the image) +rp flash deploy --build-only # write -artifact.tar.gz, deploy nothing + +# apps and environments +rp flash app list # every app and its environments +rp flash app get my-app +rp flash app delete my-app --yes # undeploys every environment in the app +rp flash env list --app my-app +rp flash env add staging --app my-app +rp flash env get staging --app my-app # an environment with its endpoints +rp flash env delete staging --app my-app --yes + +# tear down one environment (the app and its build history remain) +rp flash undeploy --app my-app --env default --yes + +# platform helpers in the same CLI +rp secret add hf-token --value "$HF_TOKEN" # read in code with runpod.Secret("hf-token") +rp registry add my-registry --username USER --password TOKEN # private images (registry_auth="my-registry") +rp logs POD_ID --follow # a Pod's container and system logs +rp ssh POD_ID +``` + +Take exact flags from `rp --help`; the list above is the shape of the surface. + +### Build artifact + +- `rp flash deploy` packages your code plus resolved `dependencies` into one artifact of up to + **1,500 MB**. On the built-in GPU images `torch`, `torchvision`, `torchaudio` and `triton` are + left out automatically, because the image ships them. +- `.runpodignore` and the project's `.gitignore` files decide what source goes in. `.env` + files, keys and credential files are always left out. +- `--build-only` resolves dependencies and packages without touching your account, which makes + it a quick check that discovery and dependency resolution succeed. + +## Exact rules + +- [flash](../../runpod-usage/concepts/flash.yaml) +- [api-key](../../runpod-usage/concepts/api-key.yaml) diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 0df7b379..e43a0d60 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -2,117 +2,186 @@ id: flash name: Flash kind: tool summary: > - Runpod's Python SDK and CLI (`runpod-flash`) for code-first serverless. You - write decorated functions or routes, iterate with `flash dev` against remote - workers, and ship with `flash deploy`. + Code-first serverless in the Runpod Python SDK (`runpod` 2.0). You define a + `runpod.App` with queue, API and task resources, iterate with `rp flash dev` + against remote workers, and ship with `rp flash deploy`. The older standalone + `runpod-flash` package is deprecated. product: flash is_a: null part_of: runpod-platform -aliases: [Runpod Flash, runpod-flash, flash CLI] +aliases: [Runpod Flash, runpod-flash, flash CLI, rp flash, Runpod SDK apps, Apps SDK] relations: - { type: runs_on, target: serverless-endpoint, cardinality: "1..n" } - { type: uses, target: network-volume, cardinality: "0..n" } + - { type: uses, target: model-store, cardinality: "0..n" } - { type: requires, target: api-key, cardinality: "1" } rules: + - id: flash.part-of-runpod-sdk + statement: > + Flash ships in the `runpod` package from version 2.0, with the `rp` CLI + (`rp flash init`, `dev`, `deploy`, `app`, `env`, `undeploy`). The CLI is + also installed as `runpod`. Install `runpod>=2.0.0rc1`; that specifier + also accepts the 2.0 release candidate. + status: verified + evidence: + - { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" } + - { source: live-probe, ref: "pip install runpod==2.0.0rc1; rp --help and rp flash --help", seen: "2026-10-16" } + + - id: flash.runpod-flash-deprecated + statement: > + The standalone `runpod-flash` package is deprecated on October 29, 2026 + and gets no fixes after that date. Endpoints it deployed keep running. + Write new code on the Runpod SDK and migrate existing projects. + status: documented + evidence: + - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } + - { source: skill, path: plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md } + - id: flash.not-flashboot statement: > - Flash is the `runpod-flash` SDK and CLI. FlashBoot is a separate endpoint + Flash is the code-first SDK and CLI. FlashBoot is a separate endpoint setting for faster restarts; Flash exposes it through the `flashboot` - argument but does not depend on it. + parameter but does not depend on it. status: documented see: [flashboot.flash-sdk-default-on] evidence: - { source: skill, path: plugins/runpod/skills/flash/SKILL.md } - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: The flashboot argument of Endpoint. } + - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: The flashboot parameter of the decorators. } - - id: flash.modes + - id: flash.resource-types statement: > - Flash has three modes. A bare `@Endpoint` decorator is queue-based with - its own worker pool. Routes such as `@api.post` are load-balanced and - share one pool. Passing `image=` or `id=` is client mode, against a - prebuilt image or an existing endpoint. `gpu=` and `cpu=` cannot be used - together. + An app has three resource types. `@app.queue` on a function creates a + queue-based endpoint. `@app.api` on a class creates a load-balanced + endpoint whose `@runpod.get` and `@runpod.post` methods are routes, with + setup in a `@runpod.init` method. `@app.task` on a function runs one Pod + per call and terminates it when the function returns. `gpu=` and `cpu=` + cannot be used together. status: documented evidence: - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Modes section and Gotchas 4 and 7. } - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md } + - { source: public-docs, url: "https://docs.runpod.io/sdk/overview", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/spec.py", seen: "2026-10-16", note: gpu and cpu are mutually exclusive. } + + - id: flash.dev-session + statement: > + `rp flash dev FILE` creates temporary endpoints, runs the + `@runpod.local_entrypoint` on your machine, streams the workers' output + and runs again after each file change. Ending the session deletes the + endpoints. It runs no local HTTP server. `--once` runs the entrypoint + once, cleans up and exits non-zero on failure. + status: documented + evidence: + - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/rp_cli/main.py", seen: "2026-10-16", note: With stdin closed the session re-runs on file changes only. } + + - id: flash.same-module-in-dev-and-deploy + statement: > + Dev sessions and deploys run the whole module source on the worker, so + module-level imports, constants and helpers behave the same in both. An + app lives in one file; helpers in sibling local modules are not shipped. + status: documented + evidence: + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/serialization.py", seen: "2026-10-16", note: get_function_source returns the full module source. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16", note: Known differences lists projects split across multiple local files. } + + - id: flash.json-queue-arguments + statement: > + Queue job arguments are plain JSON, in a dev session as well as deployed, + so a non-JSON argument fails before the job is sent. A deployed queue + function receives the job's `input` object as keyword arguments. + status: documented + see: [job.input-envelope] + evidence: + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-16", note: "require_json_args runs on live queue calls; deployed targets send {\"input\": kwargs}." } + + - id: flash.api-route-body + statement: > + A POST route on an `@app.api` class receives the request's JSON body as + its one argument, so callers send the object itself with no wrapper key. A + GET route takes no argument. + status: documented + evidence: + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/examples/apps/web_service.py", seen: "2026-10-16", note: "Counter.post(\"/bump\", {\"by\": 3}) reaches bump(self, body) as body[\"by\"]." } - id: flash.artifact-limits statement: > - `flash build` packages an artifact of up to 1500 MB and leaves out `torch` - automatically. The CLI needs Python 3.10 to 3.13, and `--python-version` - picks the Python version the worker runs. + `rp flash deploy` packages an artifact of up to 1500 MB. On the built-in + GPU images it leaves out `torch`, `torchvision`, `torchaudio` and + `triton`. Local Python must be 3.10 to 3.14, and `--python-version` picks + the worker's Python for deploys. status: documented evidence: + - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/deploy", seen: "2026-10-16" } - { source: skill, path: plugins/runpod/skills/flash/reference/setup-and-cli.md } - id: flash.sdk-defaults statement: > - The SDK uses its own defaults: `workers` (0, 1), `idle_timeout` 60 - seconds, unlimited execution time, `flashboot` `True` and - `min_cuda_version` 12.8. They differ from the platform's defaults, so set - them explicitly when they matter. + The SDK uses its own defaults: `workers` (0, 3), `idle_timeout` 60 + seconds, `execution_timeout_ms` 0 (no limit), `flashboot` `True`, and + `container_disk_gb` 30 on GPU or 10 on CPU. They differ from the + platform's defaults, so set them explicitly when they matter. status: documented see: [serverless-endpoint.defaults-by-layer] evidence: - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Endpoint constructor. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16" } + - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Parameters table. } - - id: flash.cuda-floor + - id: flash.volume-mounts statement: > - Flash endpoints default to `min_cuda_version` 12.8 because the - `runpod/flash` image needs it; on older hosts that image crash-loops. - Lower `min_cuda_version` only with an image that supports the older CUDA - version. + Queue and API resources mount one volume, at `/runpod-volume`. Tasks mount + up to one network volume and one global volume at distinct paths. Read + `volume.path` inside the remote function. Undeploying or ending a dev + session leaves the volumes, and they keep billing. status: documented + see: [serverless-endpoint.network-volume-pins-dc] evidence: - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md } + - { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/storage", seen: "2026-10-16" } - - id: flash.volume-sets-dc + - id: flash.cli-mediated statement: > - A `NetworkVolume` lives in one data center, so give the `Endpoint` the - same `datacenter` as its volume and pick a data center with stock for the - GPU you need. + Deploy and tear down through the CLI rather than raw REST or GraphQL + calls: `rp flash deploy`, `rp flash undeploy --app APP --env ENV` for one + environment, and `rp flash app delete APP` for every environment. status: documented - see: [serverless-endpoint.network-volume-pins-dc] evidence: - - { source: skill, path: plugins/runpod/skills/flash/reference/patterns.md, note: The volume and the endpoint share one data center. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" } + - { source: skill, path: plugins/runpod/skills/flash/reference/setup-and-cli.md } - - id: flash.cli-mediated + - id: flash.api-key-precedence statement: > - Deploy and tear down through the `flash` CLI (`flash deploy` and `flash - app delete`) rather than raw REST or GraphQL calls. `flash init` writes an - `AGENTS.md` that tells coding agents the same. `flash undeploy list` can - miss a deployed app. + A set `RUNPOD_API_KEY` overrides the key saved by `rp login` in + `~/.runpod/config.toml`, so a stale exported key causes 401 errors while + the saved login is valid. status: documented evidence: - - { source: skill, path: plugins/runpod/skills/flash/reference/setup-and-cli.md } - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 15. } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/utils/network.py", seen: "2026-10-16", note: RUNPOD_API_KEY is read before the stored credentials. } + - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 12. } - - id: flash.dev-ships-body-only + - id: flash.client-limits statement: > - `flash dev` ships only the decorated function's body to the worker, so put - imports and helpers inside it. `flash deploy` imports the whole module, - which can hide this mistake. + Payloads are limited to 10 MB, so pass URLs or write to a volume. + `.remote()` and `job.wait()` impose no timeout by default; the endpoint + client's `run_sync` takes a `timeout` in seconds. status: documented evidence: - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 1. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. } + - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 7. } - - id: flash.api-key-precedence + - id: flash.legacy-dev-ships-body-only statement: > - A set `RUNPOD_API_KEY` overrides the token from `flash login`. A bad key - fails quietly with "GraphQL request failed: 401" in the provisioning log - while `flash dev` still reports that it is ready. + In the legacy `runpod-flash` package, `flash dev` ships only the decorated + function's body to the worker, so imports and helpers go inside it, while + `flash deploy` imports the whole module and can hide the mistake. status: documented evidence: - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 13. } + - { source: skill, path: plugins/runpod/skills/flash/reference/runpod-flash-legacy.md, note: Gotcha 1. } - - id: flash.client-limits + - id: flash.legacy-defaults statement: > - Payloads are limited to 10 MB, so pass URLs or write to a volume. The - SDK's `runsync` times out after 60 seconds, so use `run()` and - `job.wait()` when a cold start is likely. + Legacy `runpod-flash` endpoints default to `workers` (0, 1), a 64 GB + container disk and `min_cuda_version` 12.8, because the `runpod/flash` + image needs CUDA 12.8; on older hosts that image crash-loops. status: documented evidence: - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotchas 6 and 9. } + - { source: skill, path: plugins/runpod/skills/flash/reference/runpod-flash-legacy.md, note: Defaults section. } diff --git a/plugins/runpod/skills/runpod-usage/concepts/flashboot.yaml b/plugins/runpod/skills/runpod-usage/concepts/flashboot.yaml index bf252174..a0fc7258 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flashboot.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flashboot.yaml @@ -43,10 +43,12 @@ rules: - id: flashboot.flash-sdk-default-on statement: > - The Flash SDK exposes FlashBoot as the `flashboot` argument of `Endpoint`, - and it defaults to `True`. FlashBoot itself is an endpoint setting, not - part of Flash. + The Flash SDK exposes FlashBoot as the `flashboot` parameter of + `@app.queue` and `@app.api` (and of the legacy `Endpoint`), and it + defaults to `True`. FlashBoot itself is an endpoint setting, not part of + Flash. status: documented see: [flash.not-flashboot] evidence: - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Endpoint constructor. } + - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Parameters table. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16" } diff --git a/plugins/runpod/skills/runpod-usage/concepts/job.yaml b/plugins/runpod/skills/runpod-usage/concepts/job.yaml index d0585974..6a20a054 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/job.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/job.yaml @@ -80,7 +80,7 @@ rules: status: documented evidence: - { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md } - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 10. } + - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Raw HTTP to a deployed queue resource. } - { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests" } - { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/operation-reference", seen: "2026-10-14", note: Maximum payload sizes for /run and /runsync. } diff --git a/plugins/runpod/skills/runpod-usage/concepts/model-store.yaml b/plugins/runpod/skills/runpod-usage/concepts/model-store.yaml index f0c7ee03..c800d0ed 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/model-store.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/model-store.yaml @@ -114,12 +114,26 @@ rules: - { source: public-docs, url: "https://docs.runpod.io/serverless/development/huggingface-models", seen: "2026-10-13" } - { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/model-delivery.md, note: Connect configuration to the loader. } + - id: model-store.flash-model-parameter + statement: > + In a Flash SDK app, attach a Hugging Face model with + `model=runpod.Model("owner/name")` on `@app.queue` or `@app.api`. The + platform stages the weights on the host before the worker starts, and + `Model.path` gives the staged directory. Gated models need `HF_TOKEN` on + the same resource. + status: documented + see: [model-store.attach-by-reference, model-store.best-effort-locality] + evidence: + - { source: public-docs, url: "https://docs.runpod.io/sdk/overview", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/model.py", seen: "2026-10-16" } + - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: Models and secrets. } + - id: model-store.flash-attach-after-deploy statement: > - For a Flash endpoint, attach the model after `flash deploy` with - `runpodctl serverless update --model-reference :`, - which prints the saved `modelReferences`. Later `flash deploy` runs keep - the reference. Point `HF_HUB_CACHE` at + For a legacy `runpod-flash` endpoint, attach the model after `flash + deploy` with `runpodctl serverless update --model-reference + :`, which prints the saved `modelReferences`. Later `flash + deploy` runs keep the reference. Point `HF_HUB_CACHE` at `/runpod-volume/huggingface-cache/hub` so the load finds it. status: verified see: [model-store.attach-by-reference, model-store.best-effort-locality] diff --git a/plugins/runpod/skills/runpod-usage/concepts/runpodctl.yaml b/plugins/runpod/skills/runpod-usage/concepts/runpodctl.yaml index a9fe5b6a..18f4fd06 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/runpodctl.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/runpodctl.yaml @@ -87,9 +87,9 @@ rules: - id: runpodctl.credential-resolution statement: > `runpodctl` reads the API key from `RUNPOD_API_KEY`, then a `.env` file, - then `~/.runpod/config.toml`. `flash login` saves a key to that file, so - one key serves `runpodctl` and Flash, and the same key can be sent to the - hosted MCP server as a Bearer token. + then `~/.runpod/config.toml`. `rp login` (and the legacy `flash login`) + saves a key to that file, so one key serves `runpodctl` and Flash, and the + same key can be sent to the hosted MCP server as a Bearer token. status: documented see: [api-key.bearer-only] evidence: diff --git a/plugins/runpod/skills/runpod-usage/concepts/serverless-handler.yaml b/plugins/runpod/skills/runpod-usage/concepts/serverless-handler.yaml index a8080598..b8abfd93 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/serverless-handler.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/serverless-handler.yaml @@ -53,7 +53,7 @@ rules: status: documented evidence: - { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md, note: Cold starts and timeouts. } - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 11. } + - { source: skill, path: plugins/runpod/skills/flash/reference/patterns.md, note: Loading ML models. } - id: serverless-handler.input-shape statement: > @@ -62,7 +62,7 @@ rules: status: documented see: [job.input-envelope] evidence: - - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 10. } + - { source: skill, path: plugins/runpod/skills/flash/reference/runpod-flash-legacy.md, note: Dev server loop. } - id: serverless-handler.local-serve-flag statement: > diff --git a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml index f566702c..98feb6fa 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml @@ -77,14 +77,17 @@ rules: statement: > The worker's container disk holds the whole local cache, because both the hydrate and a fallback Hub download write there. Size the container disk - for the full model plus runtime headroom. Flash endpoints default to 64 - GB, so raise it with `template=PodTemplate(containerDiskInGb=...)`, and on - a template use `runpodctl template create --container-disk-in-gb`. + for the full model plus runtime headroom. Flash SDK resources default to + 30 GB on GPU, so raise it with `container_disk_gb=` (legacy `runpod-flash` + endpoints default to 64 GB and use + `template=PodTemplate(containerDiskInGb=...)`), and on a template use + `runpodctl template create --container-disk-in-gb`. status: documented see: [container-disk.what-survives] evidence: - { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14", note: VolumeCache restores cached files into the local directories. } - - { source: skill, path: plugins/runpod/skills/flash/reference/api.md, note: PodTemplate containerDiskInGb defaults to 64. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16", note: container_disk_gb defaults to 30 on GPU and 10 on CPU. } + - { source: skill, path: plugins/runpod/skills/flash/reference/runpod-flash-legacy.md, note: PodTemplate containerDiskInGb defaults to 64. } - { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md } - id: volume-cache.silent-without-namespace diff --git a/plugins/runpod/skills/runpod-usage/reference/development-loop.md b/plugins/runpod/skills/runpod-usage/reference/development-loop.md index b2bd746b..7fd6d3c0 100644 --- a/plugins/runpod/skills/runpod-usage/reference/development-loop.md +++ b/plugins/runpod/skills/runpod-usage/reference/development-loop.md @@ -82,7 +82,7 @@ blocked**. Don't spin or fake progress. `--stop-after`: they were never enforced and v2.12.0 removed them. - Serverless → `--workers-min 0` (scale-to-zero, ~$0 idle). - Delete test resources when done (`runpodctl pod remove` / `serverless delete` / - `flash app delete`; then any network volume). + `rp flash app delete`; then any network volume). ## Which sub-loop? diff --git a/plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md b/plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md index 8b811b57..f1459bc7 100644 --- a/plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md +++ b/plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md @@ -13,8 +13,8 @@ Proven on the Whisper golden path (both variants). Select the execution lane using the [router capability matrix](../../runpod/SKILL.md#runpod-mcp-vs-runpodctl-the-overlap). Best when a good worker ships what you need. -2. **flash (from scratch, custom/light)** — write an `@Endpoint` handler and - `flash deploy`. Best when you need your own model size / I/O schema / a lighter +2. **flash (from scratch, custom/light)** — write an `@app.queue` function with the + Runpod SDK and `rp flash deploy`. Best when you need your own model size / I/O schema / a lighter image, or no good Hub worker exists. 3. **Custom image + endpoint** — write a handler, `docker build --platform=linux/amd64`, push (private image → registry auth), create the @@ -27,7 +27,7 @@ Proven on the Whisper golden path (both variants). # Hub runpodctl serverless create --hub-id --name --workers-min 0 --workers-max 3 # flash -flash deploy # @Endpoint(workers=(0,3)) in the code +rp flash deploy # @app.queue(workers=(0, 3)) in the code # Custom image — TWO steps: serverless create takes --template-id/--hub-id, NOT an image. runpodctl template create --name --serverless \ --image /: --container-disk-in-gb 15 \ @@ -96,7 +96,7 @@ output. Bound any poll loop. Give the user the endpoint id + a copy-paste `curl` and the input schema (how to pass a URL and/or base64). Scale-to-zero means it's safe to leave; delete with -`runpodctl serverless delete ` (or `flash app delete `). +`runpodctl serverless delete ` (or `rp flash app delete --yes`). See [`golden-paths/03-whisper-endpoint/`](../../runpod/golden-paths/03-whisper-endpoint/README.md) for a fully worked example of both the Hub and flash variants (a README plus one diff --git a/plugins/runpod/skills/runpod-usage/reference/getting-started.md b/plugins/runpod/skills/runpod-usage/reference/getting-started.md index c03551e3..3dc41840 100644 --- a/plugins/runpod/skills/runpod-usage/reference/getting-started.md +++ b/plugins/runpod/skills/runpod-usage/reference/getting-started.md @@ -16,7 +16,7 @@ lane's `SKILL.md` (linked). You don't need all of them. | Lane | Quick install | Source / docs | | --- | --- | --- | | **runpodctl** (CLI) | `curl -sSL https://cli.runpod.net \| bash` (or `brew install runpod/runpodctl/runpodctl`) | [source](https://github.com/runpod/runpodctl) · [`runpodctl/SKILL.md`](../../runpodctl/SKILL.md) | -| **flash** (deploy your own code) | `uv tool install runpod-flash` (or `pip install runpod-flash`; Python 3.10–3.13) | [source](https://github.com/runpod/flash) · [`flash/SKILL.md`](../../flash/SKILL.md) | +| **flash** (deploy your own code) | `pip install "runpod>=2.0.0rc1"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`flash/SKILL.md`](../../flash/SKILL.md) | | **runpod-mcp** (hosted) | `npx @runpod/mcp-server@latest add` (guided; OAuth) | [source](https://github.com/runpod/runpod-mcp) · [`runpod-mcp/SKILL.md`](../../runpod-mcp/SKILL.md) | | **runpod-mcp** (local stdio) | `claude mcp add runpod -e RUNPOD_API_KEY=... -- npx -y @runpod/mcp-server` | same as above | | **companion CLIs** (`hf`/`gh`/`docker`/`aws`) | per-tool; see the skill | [`companion-clis/SKILL.md`](../../companion-clis/SKILL.md) | @@ -29,7 +29,7 @@ Nothing to install for the **hosted MCP** beyond configuring your client, and th ```bash npx skills add runpod/runpod-plugins-official # the skills — works with any agent curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl) -uv tool install runpod-flash # flash — deploy your own code (or: pip install runpod-flash) +pip install "runpod>=2.0.0rc1" # Runpod SDK + rp CLI — deploy your own code (flash lane) npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth ``` @@ -50,7 +50,7 @@ yourself (e.g. an `Authorization: Bearer` header for a direct API call), prefer | --- | --- | --- | | **runpodctl** | `export RUNPOD_API_KEY=...` | Non-interactive — runpodctl reads it. Best for agents/CI/scripts. | | runpodctl (human) | `runpodctl doctor` | Interactive; stores the key **and** sets up SSH keys. Prompts, so not for agents. | -| **flash** | `export RUNPOD_API_KEY=...` | Or `flash login` — browser OAuth that **saves a real API key to `~/.runpod/config.toml`**, which runpodctl reads too, so one login serves both. Human-only (needs a browser). | +| **flash** | `export RUNPOD_API_KEY=...` | Or `rp login` — browser approval that **saves a real API key to `~/.runpod/config.toml`**, which runpodctl reads too, so one login serves both. Human-only (needs a browser); `rp login --api-key` stores a key without one. | | **runpod-mcp (hosted)** | "Sign in with Runpod" OAuth on first connect | No key on disk. Or pass `Authorization: Bearer $RUNPOD_API_KEY`. | | **runpod-mcp (local)** | `RUNPOD_API_KEY` env in the MCP client config | Forwarded to the API. | diff --git a/plugins/runpod/skills/runpod-usage/reference/pod-workflows.md b/plugins/runpod/skills/runpod-usage/reference/pod-workflows.md index a7e3c662..91978061 100644 --- a/plugins/runpod/skills/runpod-usage/reference/pod-workflows.md +++ b/plugins/runpod/skills/runpod-usage/reference/pod-workflows.md @@ -135,7 +135,7 @@ does, read the service log on the volume to diagnose. Re-run setup/service commands over the same SSH channel. Logs and artifacts on the volume survive restarts. Use a framework's hot-reload/dev mode when it has one -(e.g. `flash dev` in the flash lane). +(e.g. `rp flash dev` in the flash lane). ## 8. Escalate on manual steps diff --git a/plugins/runpod/skills/runpod/SKILL.md b/plugins/runpod/skills/runpod/SKILL.md index 72d44277..4b166419 100644 --- a/plugins/runpod/skills/runpod/SKILL.md +++ b/plugins/runpod/skills/runpod/SKILL.md @@ -28,7 +28,7 @@ the lane; otherwise read the matching skill's `SKILL.md` next. | --- | --- | | **runpod-mcp** | Manage infra (pods, endpoints, jobs, templates, volumes, registries, catalog, billing) via **structured tool calls** — when the Runpod MCP tools are connected in this session. Routes on to the journey skill for the task (discovery, lifecycle-crud, serverless-deploy, pod-deploy, pod-doctor, endpoint-ops, cost-audit). | | **runpodctl** | Manage the same infra from a **terminal/CI/script**, including Hub browse/deploy, file transfer (`send`/`receive`), SSH keys, `doctor` setup, and model cache. | -| **flash** | **Write Python** that runs on Runpod serverless — `@remote`/`@Endpoint` functions, `flash dev` hot-reload, `flash deploy`. Code-first, not infra management. | +| **flash** | **Write Python** that runs on Runpod serverless with the Runpod SDK — `runpod.App` with `@app.queue`/`@app.api`/`@app.task`, `rp flash dev`, `rp flash deploy`; also maintains or migrates legacy `runpod-flash` projects. Code-first, not infra management. | | **companion-clis** | **Prerequisite artifacts**: download a model (`hf`), build/push an image (`docker`), repos/releases (`gh`), move data to a network volume over S3 (`aws`). | | **runpod-usage** | **Understand** how Runpod works before acting — pods vs serverless, building a container, storage, GPU selection, gotchas. Knowledge only. | | **runpod-templates** | **Official prebuilt pod templates** (ComfyUI, PyTorch, …): is there one for this workload, and what does its image ship — ports, paths, autostart, readiness, what's missing on first boot. Reference + routing hub; deploy via runpod-mcp/runpodctl. | @@ -45,7 +45,7 @@ of you**: | --- | --- | | runpodctl | `runpodctl --help`, `runpodctl --help`, `runpodctl version`. For failure shapes, run a command wrong on purpose and read the JSON | | runpod-mcp | your client's tool list (`/mcp` in Claude Code) — each tool carries its own parameter descriptions | -| flash | `flash --help`, and the deploy/dev output | +| flash | `rp --help`, `rp flash --help`, `pip show runpod`, and the dev/deploy output | | companion-clis | that CLI's own `--help` (`hf`, `gh`, `docker`, `aws`) | | runpod-templates | `runpodctl template list --type official`, `template get ` (its readme is authoritative) | | REST v2 | the live spec at `https://api.runpod.io/v2/openapi.json` | @@ -64,7 +64,7 @@ fixed or removed. runpodctl before v2.12.0 accepts `pod create --stop-after` / deadline. Update each tool you are about to use: ```bash runpodctl update && runpodctl version # runpodctl -flash update # flash +rp update # Runpod SDK + rp CLI (flash lane) ``` The MCP needs no step: the hosted server is always current, and a local install started with `npx @runpod/mcp-server@latest` fetches the latest release. @@ -89,15 +89,15 @@ unlocks every tool — it authenticates **runpodctl + flash + the hosted MCP** ( stay blocked, so you hit a wall on any CLI-only task (`send`/`receive`, SSH, `doctor`, model cache/Model Repository, CPU endpoints). ⚠️ **OAuth-only is a half-setup.** If nothing's set up, stop and get a key, in order: -1. **`flash login`** — browser OAuth that saves a real key to `~/.runpod/config.toml` (runpodctl - + flash read it; reuse it for the MCP Bearer). One step, unlocks all. Human-only. +1. **`rp login`** — browser approval that saves a real key to `~/.runpod/config.toml` (runpodctl + + the Runpod SDK read it; reuse it for the MCP Bearer). One step, unlocks all. Human-only. 2. **`export RUNPOD_API_KEY=…`** (https://console.runpod.io/user/settings) — same full unlock; best for headless agents. 3. **MCP OAuth only** (`/mcp` → *Sign in*) — last resort, MCP-only work; CLIs stay unauthed. **Then:** if a lane already works, use it — but if *only* the MCP is OAuth'd, still get a key before any CLI-only step. Missing a CLI? `curl -sSL https://cli.runpod.net | bash` (runpodctl) · -`uv tool install runpod-flash` (flash) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: +`pip install "runpod>=2.0.0rc1"` (Runpod SDK + `rp`, flash lane) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: [`runpod-usage/reference/getting-started.md`](../runpod-usage/reference/getting-started.md). ## How to route @@ -266,7 +266,7 @@ Example — "deploy `openai/gpt-oss-20b` to a serverless endpoint": ## Auth Everything is one key: **`RUNPOD_API_KEY`** (https://console.runpod.io/user/settings). -Each lane just makes that key resolvable — `runpodctl doctor`, `flash login`, MCP +Each lane just makes that key resolvable — `runpodctl doctor`, `rp login`, MCP stdio env var, or MCP hosted "Sign in with Runpod" (OAuth, no key on disk). Companion CLIs use their **own** credentials (HuggingFace token, GitHub auth, Docker Hub PAT, Runpod **S3** keys for `aws`) — do not reuse `RUNPOD_API_KEY` for diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/README.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/README.md index ce7a168f..0138e29c 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/README.md @@ -44,7 +44,7 @@ Rejected: | Effort | ~2 min, zero code | ~15 min, you own the handler | | Customization | Take the worker's model + I/O schema | Own the model size, I/O schema, pre/post-processing | | Image | Heavy prebuilt WhisperX image | Lighter/cheaper image you control | -| Iteration | None needed (prebuilt) | `flash dev` on a real remote GPU (hot-reload) | +| Iteration | None needed (prebuilt) | `rp flash dev` on a real remote GPU (re-runs on save) | | Pick when | A **heavy, prebuilt, known** model has a good Hub worker | **Custom/small** workload, or no good Hub worker fits | Rule of thumb: for a big prebuilt model with a solid Hub worker, flash is just @@ -110,7 +110,7 @@ resources when done: ```bash runpodctl serverless delete -# flash apps can also be removed with: flash app delete +# SDK apps can also be removed with: rp flash app delete --yes ``` See each variant file for the exact teardown command it used. diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index 6d6f7650..e77d9db7 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -4,121 +4,136 @@ mcp: none concepts: [flash, serverless-endpoint, worker, serverless-handler, gpu-type] --- -# Golden path 03 — Whisper — Variant B: from scratch with flash - -**Status:** COVERED — live-verified 2026-07-07. **Lane:** flash (code-first -serverless). **When to use this variant:** you want a **custom/lighter** worker than -any Hub image — your own model size, your own I/O schema, your own pre/post- -processing — or no good Hub worker fits. You write and own a faster-whisper -handler, iterate on a real GPU with `flash dev`, then `flash deploy`. For a heavy -prebuilt model with a solid Hub worker, prefer -[Variant A — Hub](variant-a-hub.md) (flash would just be re-implementing it). -Shared schema, gotchas, and cost notes live in the [folder README](README.md); the -flash skill is at [../../../flash/SKILL.md](../../../flash/SKILL.md). +# Golden path 03 — Whisper — Variant B: from scratch with the Runpod SDK + +**Status:** PORTED — the `runpod-flash` version of this path was live-verified 2026-07-07; this +Runpod SDK (`runpod` 2.0) version is checked with `rp flash deploy --build-only` and awaits a +live re-run. **Lane:** flash (code-first serverless). **When to use this variant:** you want a +**custom/lighter** worker than any Hub image — your own model size, your own I/O schema, your +own pre/post-processing — or no good Hub worker fits. You write a faster-whisper function, +iterate on a real GPU with `rp flash dev`, then `rp flash deploy`. For a heavy prebuilt model +with a solid Hub worker, prefer [Variant A — Hub](variant-a-hub.md). Shared schema, gotchas, +and cost notes live in the [folder README](README.md); the flash skill is at +[../../../flash/SKILL.md](../../../flash/SKILL.md). ## Prerequisites -- Python 3.10–3.13 and the flash CLI: +- Python 3.10–3.14 and the Runpod SDK in the project environment: ```bash - uv tool install runpod-flash # or: pip install runpod-flash + pip install "runpod>=2.0.0rc1" # installs the rp CLI ``` -- A Runpod API key exported for non-interactive use: +- A Runpod API key for non-interactive use: ```bash - export RUNPOD_API_KEY=... # https://console.runpod.io/user/settings + export RUNPOD_API_KEY=... # https://console.runpod.io/user/settings (or: rp login) ``` ## Walkthrough -1. **Write the handler — declare deps + GPU IN the decorator (NOT pyproject.toml).** - flash ships only the function body to the worker, so imports and the model cache - live *inside* the body; the pip deps and GPU tier are declared on `@Endpoint`. +1. **Write the app — declare dependencies and GPU on the decorator.** ```python - # whisper_worker.py — deps + GPU declared in the decorator (NOT pyproject.toml) - from runpod_flash import Endpoint, GpuGroup + # main.py + import runpod - @Endpoint( - name="whisper-flash", - gpu=GpuGroup.AMPERE_16, # whisper base needs <2GB; broad supply + app = runpod.App("whisper") + + _MODEL = None + + + @app.queue( + gpu="AMPERE_16", # whisper base needs <2 GB; broad supply workers=(0, 3), idle_timeout=60, # scale-to-zero dependencies=["faster-whisper", "nvidia-cublas-cu12", "nvidia-cudnn-cu12"], # CTranslate2 GPU libs ) - async def transcribe(input_data: dict) -> dict: + def transcribe(audio_url: str = "", audio_base64: str = "") -> dict: import base64, tempfile, urllib.request from faster_whisper import WhisperModel - global _MODEL # load once per worker (see flash gotcha 11) - try: _MODEL - except NameError: _MODEL = WhisperModel("base", device="cuda", compute_type="float16") - # download input_data["audio_url"] (or decode audio_base64) -> temp file -> transcribe + + global _MODEL + if _MODEL is None: # once per worker + _MODEL = WhisperModel("base", device="cuda", compute_type="float16") + + with tempfile.NamedTemporaryFile(suffix=".audio") as f: + if audio_url: + f.write(urllib.request.urlopen(audio_url).read()) + else: + f.write(base64.b64decode(audio_base64)) + f.flush() + segments, info = _MODEL.transcribe(f.name) + return {"text": "".join(s.text for s in segments).strip(), "language": info.language} + + + @runpod.local_entrypoint + def main(): + out = transcribe.remote(audio_url="https://github.com/runpod-workers/sample-inputs/raw/main/audio/gettysburg.wav") + print(out) + assert "four score" in out["text"].lower(), out ``` - Two flash rules are load-bearing here (both from - [../../../flash/SKILL.md](../../../flash/SKILL.md)): **native CUDA libs go in - `dependencies=[]` too** — `nvidia-cublas-cu12` + `nvidia-cudnn-cu12`, or - CTranslate2/faster-whisper silently falls back to CPU (gotcha 12); and **load the - model once per worker** via the module-level `global _MODEL` cache *inside* the - body so it works under both `flash dev` and `flash deploy` (gotchas 1 + 11). - -2. **Scaffold a project** (outside your git repos): + Two rules matter here (both in [../../../flash/SKILL.md](../../../flash/SKILL.md)): + **native CUDA libraries go in `dependencies` too** — `nvidia-cublas-cu12` + + `nvidia-cudnn-cu12`, or faster-whisper silently falls back to CPU; and **load the model + once per worker** with the module-level cache. The whole module ships to the worker, so the + global works in dev and deploy alike. + +2. **Check the build offline** (no account changes): ```bash - flash init ~/whisper-flash # scaffold (writes AGENTS.md + CLAUDE.md) + rp flash deploy main.py --build-only ``` -3. **Iterate on a real remote GPU with hot-reload** — cheap, and this is where you - catch the payload shape before shipping: +3. **Iterate on a real remote GPU.** This is where you catch the payload shape and the CUDA + libraries before shipping: ```bash - flash dev # runs the function on a remote GPU, streams live worker logs + rp flash dev main.py --once # temporary endpoint, runs main(), cleans up ``` + For repeated edits, run `rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 &`; each + save re-runs `main()` against the refreshed endpoint. Stop it with `kill -INT`. -4. **Ship it** — builds an artifact and deploys a stable endpoint: +4. **Ship it:** ```bash - flash deploy # returns an endpoint id + rp flash deploy main.py # app "whisper", environment "default" + rp flash env get default --app whisper # shows the endpoint id ``` ## Verify it works -Call it over the Runpod job API. **The payload nests under the handler's parameter -name** — because the handler param is `input_data`, the wire body is -`{"input":{"input_data":{...}}}`, not a plain `{"input":{...}}` (see the gotcha -below): +From Python: + +```python +import runpod +transcribe = runpod.Queue(app="whisper", name="transcribe") +print(transcribe.remote(audio_url="https://github.com/runpod-workers/sample-inputs/raw/main/audio/gettysburg.wav")) +``` + +Over the Runpod job API, the function's keyword arguments are the `input` object: ```bash curl -s https://api.runpod.ai/v2//runsync \ -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ - -d '{"input":{"input_data":{"audio_url":"https://github.com/runpod-workers/sample-inputs/raw/main/audio/gettysburg.wav"}}}' + -d '{"input":{"audio_url":"https://github.com/runpod-workers/sample-inputs/raw/main/audio/gettysburg.wav"}}' ``` -Verified: cold ~55–75 s (image pull + model download), **warm <1 s**; returns -`{text, language, ...}` with the correct Gettysburg transcript ("Four score and -seven years ago…"). For the first (cold) call, cold start can exceed `runsync`'s -60 s — use `/run` + poll `/status/`, or `ep.runsync(data, timeout=120)` from the -flash client, then `runsync` once warm. +Expect `{text, language}` with the Gettysburg transcript ("Four score and seven years ago…"). +The `runpod-flash` run measured cold ~55–75 s (image pull + model download) and warm <1 s. +For the first, cold call use `/run` + poll `/status/`, or `.remote()` from the SDK, which +has no timeout by default. ## Variant-specific gotchas -- **Raw-HTTP callers nest under the parameter name** (flash gotcha 10). When - something other than the flash client hits the deployed endpoint - (`curl .../runsync`, another service), the wire body is - `{"input": {"": }}`. A handler - `async def transcribe(input_data: dict)` therefore expects - `{"input":{"input_data":{...}}}`. **Name the parameter `input`** if you want the - plain Runpod contract (matching Variant A's schema). The flash client's - `ep.runsync(x)` hides this — it's only a gotcha for external callers. -- **Only the function body ships to the worker** (flash gotcha 1). Put imports and - any module-level constants/helpers *inside* the decorated body, or `flash dev` - raises `NameError` (and `flash deploy` can mask it). This is why the model cache - is written as an in-body `global _MODEL` (gotcha 11). -- **Native CUDA libs must be listed in `dependencies=[]`** (flash gotcha 12) — - `nvidia-cublas-cu12` + `nvidia-cudnn-cu12` alongside `faster-whisper`, or it - silently runs on CPU. -- **`runsync` is 60 s / payload limit 10 MB** (flash gotchas 9 + 6) — pass a URL for - large audio, not bytes; use `/run` + poll for cold starts. +- **The `input` keys are the function's parameter names.** `transcribe(audio_url=...)` takes + `{"input": {"audio_url": ...}}`, the same plain contract as Variant A. (The `runpod-flash` + version used one `input_data: dict` parameter and needed `{"input":{"input_data":{...}}}`.) +- **Arguments must be JSON-serializable** — pass a URL or base64, not bytes. +- **Native CUDA libraries must be listed in `dependencies`** — or faster-whisper runs on CPU. +- **10 MB payload limit** — pass a URL for large audio. +- **The artifact is ~1.05 GB of the 1,500 MB cap**, mostly the cuDNN and cuBLAS wheels. Adding + another large package can push it over; check with `--build-only`. ## Cost & cleanup (link back to README for shared) -Scale-to-zero (`workers=(0, 3)` in the decorator) means ~$0 while idle. Teardown: +Scale-to-zero (`workers=(0, 3)`) means ~$0 while idle. Teardown: ```bash -flash app delete whisper-flash # reliable even if `flash undeploy list` shows "no endpoints" +rp flash app delete whisper --yes # every environment in the app # or: runpodctl serverless delete ``` diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index 1e5e5cac..f53715c0 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -10,8 +10,9 @@ concepts: [pod, network-volume, data-center, serverless-endpoint, worker, server **serverless endpoint** read that same data — the pattern behind "fine-tune on a pod, then serve the adapter from an endpoint." **Status:** COVERED — live-verified 2026-07-10 end to end (pod wrote `/workspace/hello.txt`; -a flash serverless worker read it back at `/runpod-volume/hello.txt` and returned the exact -contents). The earlier read-back failures were a handler-signature / empty-input bug, not the +a `runpod-flash` serverless worker read it back at `/runpod-volume/hello.txt` and returned the +exact contents). Step 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with +`rp flash deploy --build-only`; the SDK reader awaits a live re-run. The earlier read-back failures were a handler-signature / empty-input bug, not the handoff — see [Root cause](#root-cause-why-the-reader-first-timed-out-found-via-mcp-worker-logs). **Lane(s):** runpodctl (volume + pod) + flash (serverless reader) + Runpod MCP (`stream-worker-logs`, for diagnosis) @@ -36,8 +37,8 @@ The volume is **pinned to one data center**; the pod AND the endpoint workers mu that DC. ## Prerequisites -- `RUNPOD_API_KEY` resolvable (both runpodctl and flash read it). -- `runpodctl` + `flash` installed. +- `RUNPOD_API_KEY` resolvable (runpodctl and the Runpod SDK both read it). +- `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc1"`, which provides `rp`). - An SSH key registered **before** creating the pod — Runpod injects registered keys into the pod at startup, so a key added after boot won't work until a restart (`runpodctl ssh list-keys`; see @@ -73,51 +74,59 @@ Verified: the file was written and read back on the pod (48 bytes), and after `pod remove` the volume still listed — data persists independent of the pod. ### 3. Serverless endpoint attaches the SAME volume and reads at `/runpod-volume` -Use **flash** for a code-first reader. Attach the existing volume **by id** (deterministic -— no name matching), pin the same DC: +Use the **Runpod SDK** for a code-first reader. Reference the existing volume **by id** with +`create=False` (deterministic, and it fails instead of creating a new one), and pin the same DC: ```python # main.py import os -from runpod_flash import Endpoint, DataCenter, NetworkVolume -from runpod_flash.core.resources.cpu import CpuInstanceType +import runpod -vol = NetworkVolume(id="", datacenter=DataCenter.EU_RO_1) # attach EXISTING by id +app = runpod.App("handoff") -@Endpoint(name="handoff-reader", cpu=CpuInstanceType.CPU3C_1_2, - workers=(0, 1), datacenter=DataCenter.EU_RO_1, volume=vol) -async def read(**kwargs) -> dict: # see Root cause: flash calls read(**job_input) - p = "/runpod-volume/hello.txt" # NOTE: /runpod-volume, not /workspace +vol = runpod.NetworkVolume("", create=False) # attach EXISTING by id (or name) + +@app.queue(cpu="cpu3c-1-2", workers=(0, 1), datacenter="EU-RO-1", + mounts={"/runpod-volume": vol}) +def read(path: str = "hello.txt") -> dict: # input keys = parameter names + p = os.path.join(str(vol.path), path) # /runpod-volume, not /workspace return {"exists": os.path.exists(p), - "content": open(p).read() if os.path.exists(p) else None} + "content": open(p).read() if os.path.exists(p) else None, + "runpod_volume_listing": sorted(os.listdir(vol.path))} + +@runpod.local_entrypoint +def main(): + print(read.remote()) ``` -**Handler signature matters** — flash invokes the handler as `read(**job_input)`, spreading -the request's `input` dict as keyword arguments. Use `**kwargs` (or name parameters to match -the input keys); a plain `def read(input: dict)` fails with +**Handler signature matters** — the worker calls the function as `read(**job_input)`, so +the request's `input` keys must be parameter names. A plain `def read(input: dict)` fails with `read() got an unexpected keyword argument …`. See [Root cause](#root-cause-why-the-reader-first-timed-out-found-via-mcp-worker-logs). ```bash -RUNPOD_API_KEY=... flash deploy +rp flash dev main.py --once # optional: read it once from a temporary endpoint +rp flash deploy main.py # app "handoff", environment "default" runpodctl serverless list # confirm the endpoint's networkVolumeId == your volume id ``` -Verified: the deployed endpoint reported `networkVolumeId` equal to the volume the pod -wrote to — the **same volume is attached to both sides**. `NetworkVolume(id=...)` attaches -the existing volume deterministically (there is also a `name=`/`dataCenterId=` form). +The `runpod-flash` run verified that the deployed endpoint reported `networkVolumeId` equal to +the volume the pod wrote to — the **same volume is attached to both sides**. ### 4. Invoke and read the artifact back -Send a **non-empty** `input` (an empty `{}` is rejected by the worker SDK as "missing input"): +Send a **non-empty** `input` whose keys are the function's parameters (an empty `{}` is +rejected by the worker SDK as "missing input"): ```bash curl -s https://api.runpod.ai/v2//run -H "Authorization: Bearer $RUNPOD_API_KEY" \ - -H 'Content-Type: application/json' -d '{"input":{"noop":true}}' # then poll /status/ + -H 'Content-Type: application/json' -d '{"input":{"path":"hello.txt"}}' # then poll /status/ ``` +From Python: `runpod.Queue(app="handoff", name="read").remote(path="hello.txt")`. Verified result (2026-07-10) — the serverless worker read exactly what the pod wrote: ```json "output": { "exists": true, "content": "handoff via MCP+skills 2026-07-10\n", "runpod_volume_listing": ["hello.txt"] }, "status": "COMPLETED" ``` -> **Redeploy note:** flash keeps warm workers on the **old** artifact after a `flash deploy` -> until they recycle. If a code change doesn't take effect, `flash undeploy --force` -> then `flash deploy` for fresh workers. +> **Redeploy note:** with `runpod-flash`, warm workers kept the **old** artifact after a +> `flash deploy` until they recycled. If a code change doesn't take effect after +> `rp flash deploy`, `rp flash undeploy --app handoff --env default --yes` then deploy again +> for fresh workers. ## Root cause: why the reader first timed out (found via MCP worker logs) Early runs returned `"job timed out after 1 retries"` with the worker `IN_PROGRESS` but never @@ -140,6 +149,7 @@ from a broken worker — get the logs before assuming the worker is bad. This ru ## Cost & cleanup ```bash +rp flash app delete handoff --yes # SDK reader; or delete endpoints by id: runpodctl serverless delete # (flash may create a new endpoint per runpodctl serverless delete # compute-type change — delete each) runpodctl network-volume delete # pod must already be removed @@ -162,11 +172,11 @@ this specific fine-tune variant hasn't been run end to end (spec). ## Skill facts confirmed / folded back - `storage.md` already documents the `/workspace` (pod) vs `/runpod-volume` (serverless) mount-path split — confirmed correct in practice. -- flash attaches an existing volume via `NetworkVolume(id=...)` (deterministic); the pod - side uses `--network-volume-id`. Both must be in the volume's DC. -- **flash handler contract:** flash calls `read(**job_input)` — the handler must accept the - input keys as kwargs (`**kwargs` is safest), and the `input` must be non-empty (an empty - `{}` is rejected as "missing input"). A `def read(input: dict)` handler fails. +- The SDK attaches an existing volume via `runpod.NetworkVolume("", create=False)` + (deterministic); the pod side uses `--network-volume-id`. Both must be in the volume's DC. +- **Handler contract:** the worker calls `read(**job_input)` — the `input` keys must be the + function's parameter names, and the `input` must be non-empty (an empty `{}` is rejected as + "missing input"). A `def read(input: dict)` handler fails. - **Diagnosis:** the Runpod MCP `stream-worker-logs` distinguishes a job-reject (healthy worker, bad payload/handler) from a genuinely broken worker — reach for it before assuming the worker image is bad. Requires the prod v2 REST API to be reachable. diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 582696c8..9b6d814f 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -13,8 +13,9 @@ fast/cheap validation that **scales to larger models by changing three things** GPU, dataset) — nothing structural changes. **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the -volume; a flash serverless worker then loaded base + adapter from `/runpod-volume` and -generated. The live run used a **minimal peft + transformers `Trainer`** script (below) — +volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` +and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with +`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run. The live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. Composes the verified network-volume handoff (golden path [07](07-network-volume-handoff.md)). @@ -45,7 +46,7 @@ A 1.1B LoRA on a 4090 trains in **minutes** — fast enough to iterate the loop spending on a real run. ## Prerequisites -- `RUNPOD_API_KEY` resolvable; `runpodctl` + `flash` installed. +- `RUNPOD_API_KEY` resolvable; `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc1"`, which provides `rp`). - An SSH key registered **before** creating the pod (`runpodctl ssh list-keys`; see `../../runpod-usage/reference/getting-started.md`). - `HF_TOKEN` only if you scale to a **gated** base model (TinyLlama is ungated). @@ -151,44 +152,54 @@ runpodctl pod remove > long run so it survives SSH disconnect. ## 3. Serve on serverless (load base + adapter from `/runpod-volume`) -A flash endpoint attaches the **same volume** and loads the adapter once per worker. -Mind the two things from 07: mount path is `/runpod-volume`, and the handler is called as -`handler(**job_input)` (use `**kwargs`; empty input is rejected). - -This is the exact handler that ran. Note the flash rules from 07: **function** handler with a -`global` cache loaded once per worker (only the function *body* ships to the worker, so load -inside it, not at module level), `**kwargs`, and paths inlined (module-level constants don't -ship): +A Runpod SDK app attaches the **same volume** and loads the adapter once per worker. +Mind the two things from 07: mount path is `/runpod-volume`, and the worker calls the function +as `generate(**job_input)`, so the `input` keys are its parameter names (empty input is +rejected). + +The `runpod-flash` handler that ran used a function with a `global` cache and `**kwargs`. +The SDK version keeps the same model code; the whole module ships, so the paths can be +module-level constants: ```python # main.py -from runpod_flash import Endpoint, GpuGroup, DataCenter, NetworkVolume +import runpod -vol = NetworkVolume(id="", datacenter=DataCenter.EU_RO_1) # the SAME volume +app = runpod.App("ft-serve") -@Endpoint(name="ft-serve", gpu=GpuGroup.ADA_24, workers=(0, 1), idle_timeout=60, - datacenter=DataCenter.EU_RO_1, volume=vol, - env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache - dependencies=["torch", "transformers", "peft", "accelerate"]) -async def generate(**kwargs) -> dict: # flash spreads input as kwargs +vol = runpod.NetworkVolume("", create=False) # the SAME volume, by id +BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" +_M = _T = None + + +@app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="EU-RO-1", + mounts={"/runpod-volume": vol}, + env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache + dependencies=["transformers", "peft", "accelerate"]) # torch ships on the GPU image +def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict: import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel + global _M, _T - try: - _M - except NameError: # load once per worker - _T = AutoTokenizer.from_pretrained("/runpod-volume/outputs/lora-out") - base = AutoModelForCausalLM.from_pretrained( - "TinyLlama/TinyLlama-1.1B-Chat-v1.0", torch_dtype=torch.float16).to("cuda") - _M = PeftModel.from_pretrained(base, "/runpod-volume/outputs/lora-out").eval() - text = f"### Instruction:\n{kwargs.get('prompt', 'Hello')}\n\n### Response:\n" + adapter = str(vol.path / "outputs" / "lora-out") + if _M is None: # load once per worker + _T = AutoTokenizer.from_pretrained(adapter) + base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16).to("cuda") + _M = PeftModel.from_pretrained(base, adapter).eval() + text = f"### Instruction:\n{prompt}\n\n### Response:\n" ids = _T(text, return_tensors="pt").to("cuda") with torch.no_grad(): - out = _M.generate(**ids, max_new_tokens=int(kwargs.get("max_new_tokens", 64)), do_sample=False) + out = _M.generate(**ids, max_new_tokens=int(max_new_tokens), do_sample=False) return {"output": _T.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)} + + +@runpod.local_entrypoint +def main(): + print(generate.remote(prompt="Give me one tip about GPUs.")) ``` ```bash -RUNPOD_API_KEY=... flash deploy +rp flash dev main.py --once # optional: one request against a temporary endpoint +rp flash deploy main.py # app "ft-serve", environment "default" runpodctl serverless list # confirm the endpoint's networkVolumeId == ``` @@ -199,7 +210,9 @@ curl -s https://api.runpod.ai/v2//run -H "Authorization: Bearer $RU -H 'Content-Type: application/json' \ -d '{"input":{"prompt":"Give me one tip about GPUs."}}' # then poll /status/ ``` -Verified result (2026-07-10): cold call `COMPLETED` in ~48 s queue + 9.7 s exec (fast — the +From Python: `runpod.Queue(app="ft-serve", name="generate").remote(prompt="...")`. + +Verified result with `runpod-flash` (2026-07-10): cold call `COMPLETED` in ~48 s queue + 9.7 s exec (fast — the base model was already cached on the volume from training), output: ```json "output": {"output": "Sure, here's a tip for training LLMs:\n\n1. Start with small models: ..."} @@ -211,8 +224,8 @@ Green when the first (cold) call returns generated text. The pipeline is proven; `GpuGroup` to a bigger-VRAM tier, and dataset/steps. Add `HF_TOKEN` for gated models. Use **QLoRA** (axolotl `load_in_4bit: true`) when the base won't fit for full LoRA. **Doesn't change:** the loop — one volume, pod writes the adapter to `/workspace/outputs`, -serverless reads it from `/runpod-volume/outputs`, same DC, `**kwargs` handler, non-empty -input. For high-throughput serving, swap the transformers handler for **vLLM with LoRA** +serverless reads it from `/runpod-volume/outputs`, same DC, `input` keys matching the +parameters, non-empty input. For high-throughput serving, swap the transformers handler for **vLLM with LoRA** (load the base once, hot-swap adapters) — same volume handoff. ## Cost & cleanup @@ -226,9 +239,10 @@ scale-to-zero (`workers=(0,1)`), ~$0 idle. Keep the volume only while iterating. ## Gotchas this path inherits - **Mount-path swap:** `/workspace` on the pod == `/runpod-volume` on serverless — same volume, different path ([07](07-network-volume-handoff.md), `storage.md`). -- **flash handler contract:** `handler(**job_input)` — use `**kwargs`; invoke with non-empty - `input` (flash SKILL gotcha "Request body shape"). -- **Load the model once** in the class `__init__`, not per request (flash SKILL gotcha). +- **Handler contract:** the worker calls `generate(**job_input)` — `input` keys are the + function's parameter names; invoke with a non-empty `input`. +- **Load the model once per worker**, not per request: a module-level cache in a queue + function, or `@runpod.init` in an `@app.api` class (flash `reference/patterns.md`). - **PEP 668** on the training pod: `pip install --break-system-packages …` (`on-pod-setup.md`). - **Diagnosis:** if the endpoint job times out, pull worker logs before assuming a broken worker — it's usually a payload/handler issue. `runpodctl serverless logs --source diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 07942c38..1d905b73 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -70,12 +70,12 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | | ↳ [Variant B — prebuilt official image](02-comfyui-pod/variant-b-prebuilt.md) | | official template (auto-starts) | ✅ (faster) | | 03 | [Whisper endpoint (audio → text)](03-whisper-endpoint/README.md) | serverless | — | ✅ live-verified | | | ↳ [Variant A — Runpod Hub worker](03-whisper-endpoint/variant-a-hub.md) | | runpodctl + Hub (recommended) | ✅ | -| | ↳ [Variant B — from scratch with flash](03-whisper-endpoint/variant-b-flash.md) | | flash code-first handler | ✅ | +| | ↳ [Variant B — from scratch with the Runpod SDK](03-whisper-endpoint/variant-b-flash.md) | | Runpod SDK `@app.queue` (`rp flash`) | 🔄 ported to the SDK; `runpod-flash` version ✅ | | 04 | [LoRA fine-tune (training run) on a pod](04-finetune-pod.md) | pod / batch job | runpodctl pod + volume | ✅ live-verified (training phase, via 08) | | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | -| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + flash | ✅ live-verified | -| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + flash | ✅ live-verified | +| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK reader ported | +| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK endpoint ported | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified | @@ -136,6 +136,9 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac commands and the "observed output" blocks are from that run. - **📘 documented (launch only)** (used in the table above) — the provisioning/launch steps are written and grounded in the CLI/REST specs, but a full end-to-end run wasn't performed. +- **🔄 ported** (used in the table above) — a live-verified path rewritten for a newer tool + (the Runpod SDK in place of `runpod-flash`). The new code passes `rp flash deploy --build-only`; + the live run that produced the "observed output" blocks used the older tool. ## Cross-cutting requirements (every path) diff --git a/plugins/runpod/skills/serverless-deploy/SKILL.md b/plugins/runpod/skills/serverless-deploy/SKILL.md index 80ab160e..0d27f539 100644 --- a/plugins/runpod/skills/serverless-deploy/SKILL.md +++ b/plugins/runpod/skills/serverless-deploy/SKILL.md @@ -21,7 +21,7 @@ You take *any* deployable source and turn it into a working, queue-backed Server Two decisions are yours to make without being told: the **engine family** (which worker image is correct for this modality and model format) and the **provisioning path** (bake weights into the image vs download-on-start vs stage on a network volume, each with a different cold-start cost). Size hardware from the model's real metadata, never a guess. When several suitable workers exist for the modality, prefer the smallest, most-proven image for a smoke-able deploy. A giant image turns the first job into a cold start of many minutes, and the smoke test's job is proof of the journey, not maximal quality. Name the trade-off if the user's request implies a heavier engine. Creating an endpoint is billable: state the hourly cost of the GPU class before you create it, and remember a smoke job costs money. -For code-first Python (writing endpoint functions in a project and iterating with Runpod's Python dev loop, runpod-flash), this MCP-only bundle has no path. Say so instead of improvising one. This skill deploys an existing source (Hub repo, container image, ComfyUI workflow, Hub release) to an endpoint through the MCP tools. +For code-first Python (writing endpoint functions in a project and iterating with the Runpod SDK's `rp flash dev` loop), this MCP-only bundle has no path. Say so instead of improvising one. This skill deploys an existing source (Hub repo, container image, ComfyUI workflow, Hub release) to an endpoint through the MCP tools. ## Required capabilities From bf0dec0303e9ec92770f97ad2f4f15ac0748547c Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 16:04:31 -0400 Subject: [PATCH 02/18] docs(flash): restore SIGINT for background dev sessions so cleanup runs A non-interactive shell starts & jobs with SIGINT ignored, and SIGTERM exits rp flash dev without deleting its temporary endpoints. Verified live: the python exec wrapper restores SIGINT and the session reports its endpoints deleted. --- plugins/runpod/skills/flash/SKILL.md | 10 +++++++--- .../03-whisper-endpoint/variant-b-flash.md | 5 +++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index a36d3fc3..353e6f0c 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -116,11 +116,15 @@ A dev session is long-running. Three rules: the entrypoint on every file save, so you never need to press Enter. - **Capture its output** to a log file and read it. Worker `print`s, tracebacks, `✗ remote execution failed: ...` and the per-run `✓ done in ...` / `✗ failed in ...` lines land there. -- **Stop it with SIGINT**, never `kill -9`, so it deletes its endpoints. Confirm the log ends - with `cleaning up N deleted`. +- **Stop it with SIGINT**, so it deletes its endpoints, and confirm the log ends with + `cleaning up N deleted`. A non-interactive shell starts `&` jobs with SIGINT ignored, and + SIGTERM or `kill -9` exit without cleanup, so launch through the wrapper below, which + restores SIGINT. When one run is enough, prefer `--once`; it cleans up on its own. ```bash -rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 & +# restore SIGINT for the background job, then exec rp +python3 -c 'import os,signal,sys; signal.signal(signal.SIGINT, signal.SIG_DFL); os.execvp(sys.argv[1], sys.argv[1:])' \ + rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 & DEV=$! # poll the log (bounded) for the run result: "✓ done in ..." or "✗ failed in ..." for i in $(seq 1 120); do grep -qE "✓ done in|✗ failed in" /tmp/rp-dev.log && break; sleep 3; done diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index e77d9db7..0927d8c2 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -85,8 +85,9 @@ and cost notes live in the [folder README](README.md); the flash skill is at ```bash rp flash dev main.py --once # temporary endpoint, runs main(), cleans up ``` - For repeated edits, run `rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 &`; each - save re-runs `main()` against the refreshed endpoint. Stop it with `kill -INT`. + For repeated edits, run a background session as shown in the flash skill's *Autonomous dev + loop* (it restores SIGINT so `kill -INT` cleans up); each save re-runs `main()` against the + refreshed endpoint. 4. **Ship it:** ```bash From 6e4d5fe0ea4c83380368f7cd20571f9097ab22ae Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 16:32:51 -0400 Subject: [PATCH 03/18] docs(flash): record live results on runpod 2.0.0rc1 - examples: queue_gpu, api_model_service and cpu_gpu_pipeline pass rp flash dev --once live; cpu_gpu_pipeline now casts to float32 because whole-number floats cross the job boundary as ints - golden path 03-B live-verified on the SDK; pin av<19 (PyAV 19 dropped the metadata_errors argument faster-whisper passes) - gotcha 14: dev-session runtime images predate runpod.NetworkVolume, so volume apps (train_then_serve, golden paths 07 and 08) fail in rp flash dev --- plugins/runpod/skills/flash/SKILL.md | 11 +++++++++-- plugins/runpod/skills/flash/examples/README.md | 14 +++++++------- .../skills/flash/examples/cpu_gpu_pipeline.py | 2 +- .../runpod/skills/flash/reference/patterns.md | 2 +- .../03-whisper-endpoint/variant-b-flash.md | 16 +++++++++++----- .../golden-paths/07-network-volume-handoff.md | 3 ++- .../golden-paths/08-finetune-to-serverless.md | 3 ++- .../runpod/skills/runpod/golden-paths/README.md | 2 +- 8 files changed, 34 insertions(+), 19 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 353e6f0c..ac5acc45 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -158,7 +158,8 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. 1. **Queue arguments and results must be JSON-serializable.** Queue jobs are plain JSON in dev and deploy alike; a non-JSON argument fails with `arguments must be json-serializable`. Pass - URLs or base64, not objects. + URLs or base64, not objects. Whole-number floats can arrive as integers (`1.0` → `1`), so + cast numeric inputs explicitly (`torch.tensor(x, dtype=torch.float32)`). 2. **One file per app.** Discovery imports the module you point at. Helpers in a sibling local module are not shipped yet; keep them in the same file or in an installable package. 3. **Keep blocking work out of module scope.** The SDK imports your module to discover @@ -188,7 +189,13 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. `accelerate`, `sentence-transformers`) pulls `torch` into the artifact even on GPU images and with `--exclude torch`, so the artifact grows by hundreds of MB. Run `rp flash deploy --build-only` and check the size before deploying. -14. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` +14. **Dev sessions run the runtime image's own `runpod`.** Deploys vendor your local package, + but `rp flash dev` workers import the `runpod` baked into the built-in image. The + `:latest` runtime images used by `runpod` 2.0.0rc1 predate `runpod.NetworkVolume` and + `runpod.GlobalVolume`, so an app that declares one fails on the worker with + `module 'runpod' has no attribute 'NetworkVolume'`. Read the worker traceback; an + `AttributeError` on a `runpod.` name means the image is behind your SDK. +15. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` removes one environment's endpoints; `rp flash app delete APP` removes every environment. Then delete volumes you no longer need. diff --git a/plugins/runpod/skills/flash/examples/README.md b/plugins/runpod/skills/flash/examples/README.md index 64d7e3b0..c319fbe1 100644 --- a/plugins/runpod/skills/flash/examples/README.md +++ b/plugins/runpod/skills/flash/examples/README.md @@ -9,13 +9,13 @@ rp flash deploy queue_gpu.py # persistent endpoints rp flash deploy queue_gpu.py --build-only # offline check: discovery + dependency resolution ``` -| File | Shows | -|---|---| -| [`queue_gpu.py`](queue_gpu.py) | `@app.queue` on a GPU list with `workers=(0, 5)` | -| [`api_model_service.py`](api_model_service.py) | `@app.api` class, model loaded once in `@runpod.init`, `POST /generate` + `GET /health` | -| [`cpu_gpu_pipeline.py`](cpu_gpu_pipeline.py) | CPU stage feeding a GPU stage, `.spawn()` fan-out | -| [`train_then_serve.py`](train_then_serve.py) | `@app.task` writes a checkpoint to a `NetworkVolume`, a queue endpoint reads it | -| [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | +| File | Shows | Live check (`runpod` 2.0.0rc1, `--once`) | +|---|---|---| +| [`queue_gpu.py`](queue_gpu.py) | `@app.queue` on a GPU list with `workers=(0, 5)` | ✅ 2026-10-16 | +| [`api_model_service.py`](api_model_service.py) | `@app.api` class, model loaded once in `@runpod.init`, `POST /generate` + `GET /health` | ✅ 2026-10-16 | +| [`cpu_gpu_pipeline.py`](cpu_gpu_pipeline.py) | CPU stage feeding a GPU stage, `.spawn()` fan-out | ✅ 2026-10-16 | +| [`train_then_serve.py`](train_then_serve.py) | `@app.task` writes a checkpoint to a `NetworkVolume`, a queue endpoint reads it | ❌ runtime image lacks `runpod.NetworkVolume` (SKILL gotcha 14) | +| [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | needs the two apps deployed; not run | Cost and cleanup: dev sessions delete their endpoints on exit. `train_then_serve.py` creates the `train-serve-ckpts` network volume, which stays and bills until you delete it. Remove diff --git a/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py index 53352504..e177e00b 100644 --- a/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py +++ b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py @@ -24,7 +24,7 @@ def preprocess(rows: list[dict]) -> list[list[float]]: def score(matrix: list[list[float]]) -> list[float]: import torch - return torch.tensor(matrix, device="cuda").mean(dim=1).tolist() + return torch.tensor(matrix, dtype=torch.float32, device="cuda").mean(dim=1).tolist() @runpod.local_entrypoint diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index 7fe2de07..d72b827c 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -134,7 +134,7 @@ def preprocess(rows: list[dict]) -> list[list[float]]: @app.queue(gpu="AMPERE_80") def score(matrix: list[list[float]]) -> list[float]: import torch - return torch.tensor(matrix, device="cuda").mean(dim=1).tolist() + return torch.tensor(matrix, dtype=torch.float32, device="cuda").mean(dim=1).tolist() @runpod.local_entrypoint def main(): diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index 0927d8c2..ccfabb52 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -6,9 +6,9 @@ concepts: [flash, serverless-endpoint, worker, serverless-handler, gpu-type] # Golden path 03 — Whisper — Variant B: from scratch with the Runpod SDK -**Status:** PORTED — the `runpod-flash` version of this path was live-verified 2026-07-07; this -Runpod SDK (`runpod` 2.0) version is checked with `rp flash deploy --build-only` and awaits a -live re-run. **Lane:** flash (code-first serverless). **When to use this variant:** you want a +**Status:** COVERED — live-verified on the Runpod SDK (`runpod` 2.0.0rc1) 2026-10-16 with +`rp flash dev main.py --once`: the worker returned the Gettysburg transcript in 70 s from cold. +The `runpod-flash` version was live-verified 2026-07-07. **Lane:** flash (code-first serverless). **When to use this variant:** you want a **custom/lighter** worker than any Hub image — your own model size, your own I/O schema, your own pre/post-processing — or no good Hub worker fits. You write a faster-whisper function, iterate on a real GPU with `rp flash dev`, then `rp flash deploy`. For a heavy prebuilt model @@ -42,7 +42,7 @@ and cost notes live in the [folder README](README.md); the flash skill is at @app.queue( gpu="AMPERE_16", # whisper base needs <2 GB; broad supply workers=(0, 3), idle_timeout=60, # scale-to-zero - dependencies=["faster-whisper", + dependencies=["faster-whisper", "av<19", # av 19 drops an arg faster-whisper passes "nvidia-cublas-cu12", "nvidia-cudnn-cu12"], # CTranslate2 GPU libs ) def transcribe(audio_url: str = "", audio_base64: str = "") -> dict: @@ -113,7 +113,10 @@ curl -s https://api.runpod.ai/v2//runsync \ -d '{"input":{"audio_url":"https://github.com/runpod-workers/sample-inputs/raw/main/audio/gettysburg.wav"}}' ``` -Expect `{text, language}` with the Gettysburg transcript ("Four score and seven years ago…"). +Verified result (2026-10-16, dev session, cold): +``` +{'language': 'en', 'text': 'Four score and seven years ago, our fathers brought forth on this continent a new nation, ...'} +``` The `runpod-flash` run measured cold ~55–75 s (image pull + model download) and warm <1 s. For the first, cold call use `/run` + poll `/status/`, or `.remote()` from the SDK, which has no timeout by default. @@ -125,6 +128,9 @@ has no timeout by default. version used one `input_data: dict` parameter and needed `{"input":{"input_data":{...}}}`.) - **Arguments must be JSON-serializable** — pass a URL or base64, not bytes. - **Native CUDA libraries must be listed in `dependencies`** — or faster-whisper runs on CPU. +- **Pin `av<19`.** PyAV 19 removed the `metadata_errors` argument that faster-whisper passes + to `av.open`, so an unpinned install fails with `open() got an unexpected keyword argument + 'metadata_errors'`. - **10 MB payload limit** — pass a URL for large audio. - **The artifact is ~1.05 GB of the 1,500 MB cap**, mostly the cuDNN and cuBLAS wheels. Adding another large package can push it over; check with `--build-only`. diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index f53715c0..43d22c75 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -12,7 +12,8 @@ then serve the adapter from an endpoint." **Status:** COVERED — live-verified 2026-07-10 end to end (pod wrote `/workspace/hello.txt`; a `runpod-flash` serverless worker read it back at `/runpod-volume/hello.txt` and returned the exact contents). Step 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with -`rp flash deploy --build-only`; the SDK reader awaits a live re-run. The earlier read-back failures were a handler-signature / empty-input bug, not the +`rp flash deploy --build-only`; the SDK reader awaits a live re-run; on 2026-10-16 a dev session could not run it because the +built-in runtime images lack `runpod.NetworkVolume` (flash SKILL gotcha 14). The earlier read-back failures were a handler-signature / empty-input bug, not the handoff — see [Root cause](#root-cause-why-the-reader-first-timed-out-found-via-mcp-worker-logs). **Lane(s):** runpodctl (volume + pod) + flash (serverless reader) + Runpod MCP (`stream-worker-logs`, for diagnosis) diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 9b6d814f..8e35b2b2 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -15,7 +15,8 @@ GPU, dataset) — nothing structural changes. TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with -`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run. The live run used a **minimal peft + transformers `Trainer`** script (below) — +`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run; on 2026-10-16 a dev session could not run it because the +built-in runtime images lack `runpod.NetworkVolume` (flash SKILL gotcha 14). The live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. Composes the verified network-volume handoff (golden path [07](07-network-volume-handoff.md)). diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 1d905b73..2336f424 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -70,7 +70,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | | ↳ [Variant B — prebuilt official image](02-comfyui-pod/variant-b-prebuilt.md) | | official template (auto-starts) | ✅ (faster) | | 03 | [Whisper endpoint (audio → text)](03-whisper-endpoint/README.md) | serverless | — | ✅ live-verified | | | ↳ [Variant A — Runpod Hub worker](03-whisper-endpoint/variant-a-hub.md) | | runpodctl + Hub (recommended) | ✅ | -| | ↳ [Variant B — from scratch with the Runpod SDK](03-whisper-endpoint/variant-b-flash.md) | | Runpod SDK `@app.queue` (`rp flash`) | 🔄 ported to the SDK; `runpod-flash` version ✅ | +| | ↳ [Variant B — from scratch with the Runpod SDK](03-whisper-endpoint/variant-b-flash.md) | | Runpod SDK `@app.queue` (`rp flash`) | ✅ (SDK, 2026-10-16) | | 04 | [LoRA fine-tune (training run) on a pod](04-finetune-pod.md) | pod / batch job | runpodctl pod + volume | ✅ live-verified (training phase, via 08) | | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | From b4e3f638a826b23e55e7298dc6ce1ee3cd044107 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 17:17:31 -0400 Subject: [PATCH 04/18] docs(flash): verify the SDK volume handoff via rp flash deploy Deployed SDK workers attach the network volume, but on the current runtime images vol.path raises 'is not mounted in the current execution'. Reading the literal mount path works: on 2026-10-16 one deployed endpoint wrote to a volume and another read the exact contents back. Golden paths 07 and 08 and train_then_serve.py now read literal paths and run via rp flash deploy. --- plugins/runpod/skills/flash/SKILL.md | 7 +++++-- .../runpod/skills/flash/examples/train_then_serve.py | 6 +++--- plugins/runpod/skills/flash/reference/api.md | 4 +++- .../runpod/golden-paths/07-network-volume-handoff.md | 12 ++++++------ .../runpod/golden-paths/08-finetune-to-serverless.md | 8 ++++---- plugins/runpod/skills/runpod/golden-paths/README.md | 2 +- 6 files changed, 22 insertions(+), 17 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index ac5acc45..528b84a9 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -193,8 +193,11 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. but `rp flash dev` workers import the `runpod` baked into the built-in image. The `:latest` runtime images used by `runpod` 2.0.0rc1 predate `runpod.NetworkVolume` and `runpod.GlobalVolume`, so an app that declares one fails on the worker with - `module 'runpod' has no attribute 'NetworkVolume'`. Read the worker traceback; an - `AttributeError` on a `runpod.` name means the image is behind your SDK. + `module 'runpod' has no attribute 'NetworkVolume'`. Deployed workers on the same images + attach the volume, but `vol.path` raises `is not mounted in the current execution`; read + the literal mount path instead (`/runpod-volume` for queue and API resources, the + `mounts` key for tasks). An `AttributeError` on a `runpod.` name in a worker traceback + means the image is behind your SDK. 15. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` removes one environment's endpoints; `rp flash app delete APP` removes every environment. Then delete volumes you no longer need. diff --git a/plugins/runpod/skills/flash/examples/train_then_serve.py b/plugins/runpod/skills/flash/examples/train_then_serve.py index 64471d5f..83b60a7d 100644 --- a/plugins/runpod/skills/flash/examples/train_then_serve.py +++ b/plugins/runpod/skills/flash/examples/train_then_serve.py @@ -28,9 +28,9 @@ def train(steps: int = 200) -> str: opt.zero_grad() loss.backward() opt.step() - path = ckpts.path / "linear.pt" # resolve the mount inside the function + path = "/ckpts/linear.pt" # literal mount path (SKILL gotcha 14) torch.save(model.state_dict(), path) - return path.name + return "linear.pt" @app.queue(cpu="cpu3c-2-4", mounts={"/runpod-volume": ckpts}, dependencies=["torch"]) @@ -38,7 +38,7 @@ def predict(name: str, x: float) -> float: import torch model = torch.nn.Linear(1, 1) - model.load_state_dict(torch.load(ckpts.path / name, map_location="cpu")) + model.load_state_dict(torch.load(f"/runpod-volume/{name}", map_location="cpu")) return model(torch.tensor([[x]])).item() diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index cf016808..2bd5cdf3 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -140,7 +140,9 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must - Global volumes need GPU compute. - A new network volume is placed in a data center that has network storage and stock for every resource sharing it; resources using it are scheduled in that data center. -- Read `models.path` **inside** the remote function. Volumes are not mounted locally. +- Read `models.path` **inside** the remote function. Volumes are not mounted locally. If + `.path` raises `is not mounted in the current execution`, the runtime image is older than + your SDK; read the literal mount path (`/runpod-volume`) instead. - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. ## Models and secrets diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index 43d22c75..51257281 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -11,9 +11,10 @@ concepts: [pod, network-volume, data-center, serverless-endpoint, worker, server then serve the adapter from an endpoint." **Status:** COVERED — live-verified 2026-07-10 end to end (pod wrote `/workspace/hello.txt`; a `runpod-flash` serverless worker read it back at `/runpod-volume/hello.txt` and returned the -exact contents). Step 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with -`rp flash deploy --build-only`; the SDK reader awaits a live re-run; on 2026-10-16 a dev session could not run it because the -built-in runtime images lack `runpod.NetworkVolume` (flash SKILL gotcha 14). The earlier read-back failures were a handler-signature / empty-input bug, not the +exact contents). Step 3 is ported to the Runpod SDK (`runpod` 2.0.0rc1): on 2026-10-16 an SDK +app deployed with `rp flash deploy` wrote to a network volume from one endpoint and read the +exact contents back from another. Run it with `rp flash deploy`, not `rp flash dev`; dev +sessions on the built-in runtime images fail on volume apps (flash SKILL gotcha 14). The earlier read-back failures were a handler-signature / empty-input bug, not the handoff — see [Root cause](#root-cause-why-the-reader-first-timed-out-found-via-mcp-worker-logs). **Lane(s):** runpodctl (volume + pod) + flash (serverless reader) + Runpod MCP (`stream-worker-logs`, for diagnosis) @@ -90,10 +91,10 @@ vol = runpod.NetworkVolume("", create=False) # attach EXISTING by id @app.queue(cpu="cpu3c-1-2", workers=(0, 1), datacenter="EU-RO-1", mounts={"/runpod-volume": vol}) def read(path: str = "hello.txt") -> dict: # input keys = parameter names - p = os.path.join(str(vol.path), path) # /runpod-volume, not /workspace + p = os.path.join("/runpod-volume", path) # /runpod-volume, not /workspace return {"exists": os.path.exists(p), "content": open(p).read() if os.path.exists(p) else None, - "runpod_volume_listing": sorted(os.listdir(vol.path))} + "runpod_volume_listing": sorted(os.listdir("/runpod-volume"))} @runpod.local_entrypoint def main(): @@ -103,7 +104,6 @@ def main(): the request's `input` keys must be parameter names. A plain `def read(input: dict)` fails with `read() got an unexpected keyword argument …`. See [Root cause](#root-cause-why-the-reader-first-timed-out-found-via-mcp-worker-logs). ```bash -rp flash dev main.py --once # optional: read it once from a temporary endpoint rp flash deploy main.py # app "handoff", environment "default" runpodctl serverless list # confirm the endpoint's networkVolumeId == your volume id ``` diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 8e35b2b2..a02874a2 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -15,8 +15,9 @@ GPU, dataset) — nothing structural changes. TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with -`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run; on 2026-10-16 a dev session could not run it because the -built-in runtime images lack `runpod.NetworkVolume` (flash SKILL gotcha 14). The live run used a **minimal peft + transformers `Trainer`** script (below) — +`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run. Run it with +`rp flash deploy`: dev sessions on the built-in runtime images fail on volume apps, and the +code reads the literal `/runpod-volume` path (flash SKILL gotcha 14). The live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. Composes the verified network-volume handoff (golden path [07](07-network-volume-handoff.md)). @@ -182,7 +183,7 @@ def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict: from peft import PeftModel global _M, _T - adapter = str(vol.path / "outputs" / "lora-out") + adapter = "/runpod-volume/outputs/lora-out" # literal mount path (gotcha 14) if _M is None: # load once per worker _T = AutoTokenizer.from_pretrained(adapter) base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16).to("cuda") @@ -199,7 +200,6 @@ def main(): print(generate.remote(prompt="Give me one tip about GPUs.")) ``` ```bash -rp flash dev main.py --once # optional: one request against a temporary endpoint rp flash deploy main.py # app "ft-serve", environment "default" runpodctl serverless list # confirm the endpoint's networkVolumeId == ``` diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 2336f424..8616fe9a 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -74,7 +74,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | 04 | [LoRA fine-tune (training run) on a pod](04-finetune-pod.md) | pod / batch job | runpodctl pod + volume | ✅ live-verified (training phase, via 08) | | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | -| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK reader ported | +| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16) | | 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK endpoint ported | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | From fb91d175b26d36454665486ffa5715e417986c44 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 17:42:47 -0400 Subject: [PATCH 05/18] docs(flash): verify golden path 08 and call_deployed on the SDK; document deploy crash loop - golden path 08 serve step verified via rp flash deploy (TinyLlama + LoRA from a network volume); call_deployed.py verified against deployed apps - gotcha 13: torch-dependent packages (peft, accelerate, sentence-transformers) vendor a CUDA 13 torch that replaces the image's and crash-loops GPU workers (libcudart.so.13); install them with pip --no-deps at runtime (08 shows how) - patterns.md drops accelerate from GPU examples for the same reason - api.md: rp flash deploy maps GPU names to whole pools; pin after deploy - concept rules flash.transitive-torch-breaks-gpu-deploy and flash.deploy-gpu-names-map-to-pools with live-probe evidence --- plugins/runpod/skills/flash/SKILL.md | 14 ++++++---- .../migrate-runpod-flash-project.eval.md | 4 ++- .../runpod/skills/flash/examples/README.md | 2 +- plugins/runpod/skills/flash/reference/api.md | 5 ++++ .../runpod/skills/flash/reference/patterns.md | 6 ++--- .../skills/runpod-usage/concepts/flash.yaml | 22 +++++++++++++++ .../golden-paths/08-finetune-to-serverless.md | 27 +++++++++++++++---- .../skills/runpod/golden-paths/README.md | 2 +- 8 files changed, 66 insertions(+), 16 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 528b84a9..4eb1a0c7 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -184,11 +184,15 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. 12. **A set `RUNPOD_API_KEY` wins over `rp login`.** A stale exported key causes 401s while the saved login is fine. Check with `runpodctl user`, then `unset RUNPOD_API_KEY` or export a good key. -13. **Check the artifact when a dependency needs torch.** Auto-exclusion drops `torch` only when - it is listed directly. In `runpod` 2.0.0rc1, a package that depends on it (`peft`, - `accelerate`, `sentence-transformers`) pulls `torch` into the artifact even on GPU images - and with `--exclude torch`, so the artifact grows by hundreds of MB. Run - `rp flash deploy --build-only` and check the size before deploying. +13. **Don't list torch-dependent packages for a GPU deploy.** Auto-exclusion drops `torch` only + when it is listed directly. In `runpod` 2.0.0rc1, a package that depends on it (`peft`, + `accelerate`, `sentence-transformers`) pulls a CUDA 13 `torch` into the artifact, even with + `--exclude torch`. That copy replaces the GPU image's torch, the worker fails its CUDA check + (`libcudart.so.13: cannot open shared object file`), and the endpoint crash-loops. Install such + packages on first call with `pip install --no-deps` instead (the `_ensure` helper in + [golden path 08](../runpod/golden-paths/08-finetune-to-serverless.md)), and list their + non-torch dependencies (`psutil`, `pyyaml`) in `dependencies`. Dev sessions are unaffected: + they pip-install on the worker, where torch is already satisfied. 14. **Dev sessions run the runtime image's own `runpod`.** Deploys vendor your local package, but `rp flash dev` workers import the `runpod` baked into the built-in image. The `:latest` runtime images used by `runpod` 2.0.0rc1 predate `runpod.NetworkVolume` and diff --git a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md index 2835d48b..942955dd 100644 --- a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md +++ b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md @@ -26,7 +26,9 @@ The agent should: ## Assertions - Removes every `runpod_flash` import; uses `import runpod` -- `embed` becomes `@app.queue(gpu="ADA_24" or GpuGroup.ADA_24, dependencies=[...])` +- `embed` becomes `@app.queue(gpu="ADA_24" or GpuGroup.ADA_24, ...)`, and the agent flags that + `sentence-transformers` depends on `torch`: listed in `dependencies` it makes the GPU deploy + crash-loop, so it installs it with `--no-deps` at runtime (flash SKILL gotcha 13) - `search` becomes an `@app.api` class whose POST method takes one `body` argument - CI calls `rp flash deploy`, not `flash deploy` - Runs or recommends `rp flash dev main.py --once` before `rp flash deploy` diff --git a/plugins/runpod/skills/flash/examples/README.md b/plugins/runpod/skills/flash/examples/README.md index c319fbe1..3b4f9596 100644 --- a/plugins/runpod/skills/flash/examples/README.md +++ b/plugins/runpod/skills/flash/examples/README.md @@ -15,7 +15,7 @@ rp flash deploy queue_gpu.py --build-only # offline check: discovery + depende | [`api_model_service.py`](api_model_service.py) | `@app.api` class, model loaded once in `@runpod.init`, `POST /generate` + `GET /health` | ✅ 2026-10-16 | | [`cpu_gpu_pipeline.py`](cpu_gpu_pipeline.py) | CPU stage feeding a GPU stage, `.spawn()` fan-out | ✅ 2026-10-16 | | [`train_then_serve.py`](train_then_serve.py) | `@app.task` writes a checkpoint to a `NetworkVolume`, a queue endpoint reads it | ❌ runtime image lacks `runpod.NetworkVolume` (SKILL gotcha 14) | -| [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | needs the two apps deployed; not run | +| [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | ✅ 2026-10-16 (against both apps deployed) | Cost and cleanup: dev sessions delete their endpoints on exit. `train_then_serve.py` creates the `train-serve-ckpts` network volume, which stays and bills until you delete it. Remove diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 2bd5cdf3..39fbee4b 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -175,6 +175,11 @@ not download them. Gated models need `HF_TOKEN` on the same resource. Create the - **A list** of any of these. Automatic switching between listed GPUs needs a max of at least 5 workers. +`rp flash deploy` in `runpod` 2.0.0rc1 turns device names into their pools with no exclusions: +`gpu=["4090", "L4", "A5000"]` deployed as `ADA_24` + `AMPERE_24`, and a worker landed on a +24 GB RTX PRO 6000 Blackwell MIG slice. When exact hardware matters, read the endpoint after +deploy and pin it with the Runpod MCP `set-endpoint-gpus` tool. + `cpu=` takes a flavor id: `cpu3g-{1-4,2-8,4-16,8-32}` (general), `cpu3c-{1-2,2-4,4-8,8-16}` (compute), `cpu5c-{1-2,2-4,4-8,8-16}` (compute, 5th gen), where the numbers are vCPUs and GB of RAM. `runpod.CpuInstanceType` holds the same values. diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index d72b827c..0e983017 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -39,7 +39,7 @@ import runpod app = runpod.App("sd") @app.api(gpu="5090", workers=(0, 3), idle_timeout=300, - dependencies=["diffusers", "transformers", "accelerate"]) + dependencies=["diffusers", "transformers"]) class SD: @runpod.init def load(self): @@ -81,7 +81,7 @@ start unless you cache. Pick one: ```python qwen = runpod.Model("Qwen/Qwen2.5-7B-Instruct") - @app.queue(gpu="4090", model=qwen, dependencies=["transformers", "accelerate"]) + @app.queue(gpu="4090", model=qwen, dependencies=["transformers"]) def chat(prompt: str) -> str: from transformers import pipeline pipe = pipeline("text-generation", model=str(qwen.path), device=0) @@ -96,7 +96,7 @@ start unless you cache. Pick one: cache = runpod.NetworkVolume("model-cache", size=100) @app.api(gpu="5090", mounts={"/runpod-volume": cache}, container_disk_gb=60, - env={"HF_HUB_CACHE": "/opt/hf-cache"}, dependencies=["transformers", "accelerate"]) + env={"HF_HUB_CACHE": "/opt/hf-cache"}, dependencies=["transformers"]) class Chat: @runpod.init def load(self): diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index e43a0d60..167ea8d3 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -168,6 +168,28 @@ rules: - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. } - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 7. } + - id: flash.transitive-torch-breaks-gpu-deploy + statement: > + In `runpod` 2.0.0rc1, a GPU app whose `dependencies` include a package + that depends on `torch` (`peft`, `accelerate`, `sentence-transformers`) + deploys with a CUDA 13 `torch` in its artifact. That copy replaces the + image's torch, the worker fails its CUDA check with `libcudart.so.13: + cannot open shared object file`, and the endpoint crash-loops. Install + such packages on the worker with `pip install --no-deps` instead. + status: verified + evidence: + - { source: live-probe, ref: "rp flash deploy of a gpu queue function with dependencies=[\"peft\"]: 645.5 MB artifact, worker logs 'Fitness check failed: _cuda_init_check | libcudart.so.13'; the same function installing peft with --no-deps at runtime returned torch 2.9.1+cu128 with cuda True", seen: "2026-10-16" } + + - id: flash.deploy-gpu-names-map-to-pools + statement: > + In `runpod` 2.0.0rc1, `rp flash deploy` turns GPU device names into their + pools without exclusions, so a worker can run on any GPU in those pools. + Pin exact GPU types on the endpoint after deploy when the hardware + matters. + status: verified + evidence: + - { source: live-probe, ref: "gpu=[\"4090\", \"L4\", \"A5000\"] deployed as gpu.pools [ADA_24, AMPERE_24]; the worker reported NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb", seen: "2026-10-16" } + - id: flash.legacy-dev-ships-body-only statement: > In the legacy `runpod-flash` package, `flash dev` ships only the decorated diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index a02874a2..2ffde78a 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -14,10 +14,12 @@ GPU, dataset) — nothing structural changes. **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` -and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0) and checked with -`rp flash deploy --build-only`; the SDK endpoint awaits a live re-run. Run it with -`rp flash deploy`: dev sessions on the built-in runtime images fail on volume apps, and the -code reads the literal `/runpod-volume` path (flash SKILL gotcha 14). The live run used a **minimal peft + transformers `Trainer`** script (below) — +and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0.0rc1): on 2026-10-16 the +section 3 code, deployed with `rp flash deploy`, loaded TinyLlama plus a LoRA +adapter from a network volume and generated text (the adapter was an untrained one saved by a +helper function, standing in for the pod training step). Run it with `rp flash deploy`: dev +sessions on the built-in runtime images fail on volume apps (flash SKILL gotcha 14). +The `runpod-flash` live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. Composes the verified network-volume handoff (golden path [07](07-network-volume-handoff.md)). @@ -159,6 +161,12 @@ Mind the two things from 07: mount path is `/runpod-volume`, and the worker call as `generate(**job_input)`, so the `input` keys are its parameter names (empty input is rejected). +**Why `peft` is installed at runtime:** listing `peft` or `accelerate` in `dependencies` makes +`rp flash deploy` vendor their `torch` dependency into the artifact (`runpod` 2.0.0rc1), and that +copy replaces the GPU image's CUDA torch. The worker then fails its CUDA check +(`libcudart.so.13: cannot open shared object file`) and crash-loops. `_ensure` installs them with +`--no-deps` on first call instead (flash SKILL gotcha 13). + The `runpod-flash` handler that ran used a function with a `global` cache and `**kwargs`. The SDK version keeps the same model code; the whole module ships, so the paths can be module-level constants: @@ -173,11 +181,20 @@ BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" _M = _T = None +def _ensure(*pkgs): + # install torch-dependent packages without deps, so the GPU image's CUDA torch stays in place + import importlib.util, subprocess, sys + missing = [p for p in pkgs if importlib.util.find_spec(p) is None] + if missing: + subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", "--no-deps", *missing]) + + @app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="EU-RO-1", mounts={"/runpod-volume": vol}, env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache - dependencies=["transformers", "peft", "accelerate"]) # torch ships on the GPU image + dependencies=["transformers", "psutil", "pyyaml"]) # NOT peft/accelerate: see below def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict: + _ensure("peft", "accelerate") import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 8616fe9a..d58217d9 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -75,7 +75,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | | 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16) | -| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK endpoint ported | +| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step via `rp flash deploy` (2026-10-16) | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified | From 7db04f317e2c857af7543fef77ae40045e49da37 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 16 Oct 2026 17:44:51 -0400 Subject: [PATCH 06/18] docs(flash): make train_then_serve verifiable and verify it live The volume version could not run in a dev session on the current runtime images. The task now returns its weights as JSON to the queue endpoint (the volume handoff is covered by golden path 07). Verified 2026-10-16 with rp flash dev --once: the task Pod trained, predict returned f(2.0) = 7.00, and the Pod and endpoint were cleaned up. --- .../runpod/skills/flash/examples/README.md | 8 ++--- .../skills/flash/examples/train_then_serve.py | 34 ++++++++----------- .../runpod/skills/flash/reference/patterns.md | 5 +-- 3 files changed, 21 insertions(+), 26 deletions(-) diff --git a/plugins/runpod/skills/flash/examples/README.md b/plugins/runpod/skills/flash/examples/README.md index 3b4f9596..419b0c02 100644 --- a/plugins/runpod/skills/flash/examples/README.md +++ b/plugins/runpod/skills/flash/examples/README.md @@ -14,12 +14,12 @@ rp flash deploy queue_gpu.py --build-only # offline check: discovery + depende | [`queue_gpu.py`](queue_gpu.py) | `@app.queue` on a GPU list with `workers=(0, 5)` | ✅ 2026-10-16 | | [`api_model_service.py`](api_model_service.py) | `@app.api` class, model loaded once in `@runpod.init`, `POST /generate` + `GET /health` | ✅ 2026-10-16 | | [`cpu_gpu_pipeline.py`](cpu_gpu_pipeline.py) | CPU stage feeding a GPU stage, `.spawn()` fan-out | ✅ 2026-10-16 | -| [`train_then_serve.py`](train_then_serve.py) | `@app.task` writes a checkpoint to a `NetworkVolume`, a queue endpoint reads it | ❌ runtime image lacks `runpod.NetworkVolume` (SKILL gotcha 14) | +| [`train_then_serve.py`](train_then_serve.py) | `@app.task` trains on a GPU Pod, a queue endpoint serves the returned weights | ✅ 2026-10-16 | | [`call_deployed.py`](call_deployed.py) | `runpod.Queue`, `runpod.Api` and `runpod.Endpoint` from plain Python | ✅ 2026-10-16 (against both apps deployed) | -Cost and cleanup: dev sessions delete their endpoints on exit. `train_then_serve.py` creates -the `train-serve-ckpts` network volume, which stays and bills until you delete it. Remove -deployed apps with `rp flash app delete APP --yes`. +Cost and cleanup: dev sessions delete their endpoints on exit, and task Pods terminate when the +task returns. Remove deployed apps with `rp flash app delete APP --yes`. For a checkpoint shared +through a network volume, see [golden path 07](../../runpod/golden-paths/07-network-volume-handoff.md). The upstream SDK ships more examples in [runpod-python `examples/apps`](https://github.com/runpod/runpod-python/tree/main/examples/apps). diff --git a/plugins/runpod/skills/flash/examples/train_then_serve.py b/plugins/runpod/skills/flash/examples/train_then_serve.py index 83b60a7d..b27bc634 100644 --- a/plugins/runpod/skills/flash/examples/train_then_serve.py +++ b/plugins/runpod/skills/flash/examples/train_then_serve.py @@ -1,22 +1,20 @@ -"""Train on a Pod, then serve the checkpoint from a queue endpoint. +"""Train on a Pod, then serve the result from a queue endpoint. rp flash dev train_then_serve.py --once train() is a task: one Pod per call, terminated when it returns. predict() -is a queue endpoint. Both mount the same network volume, so the endpoint -reads the checkpoint the task wrote. The volume outlives the session and -keeps billing; delete it when you are done. +is a queue endpoint. The weights are small, so the task returns them as +JSON and main() passes them on. For checkpoints too big for a 10 MB +payload, share a NetworkVolume instead (golden path 07 shows the handoff). """ import runpod app = runpod.App("train-serve") -ckpts = runpod.NetworkVolume("train-serve-ckpts", size=10) - -@app.task(gpu="4090", mounts={"/ckpts": ckpts}) -def train(steps: int = 200) -> str: +@app.task(gpu="4090") +def train(steps: int = 200) -> dict: import torch x = torch.randn(1024, 1, device="cuda") @@ -28,23 +26,19 @@ def train(steps: int = 200) -> str: opt.zero_grad() loss.backward() opt.step() - path = "/ckpts/linear.pt" # literal mount path (SKILL gotcha 14) - torch.save(model.state_dict(), path) - return "linear.pt" - + return {"weight": model.weight.item(), "bias": model.bias.item()} -@app.queue(cpu="cpu3c-2-4", mounts={"/runpod-volume": ckpts}, dependencies=["torch"]) -def predict(name: str, x: float) -> float: - import torch - model = torch.nn.Linear(1, 1) - model.load_state_dict(torch.load(f"/runpod-volume/{name}", map_location="cpu")) - return model(torch.tensor([[x]])).item() +@app.queue(cpu="cpu3c-1-2") +def predict(params: dict, x: float) -> float: + # whole-number floats can arrive as ints, so cast before use + return float(params["weight"]) * float(x) + float(params["bias"]) @runpod.local_entrypoint def main(): - name = train.remote() - y = predict.remote(name, 2.0) + params = train.remote() + print("trained:", params) + y = predict.remote(params, 2.0) print(f"f(2.0) = {y:.2f}") assert abs(y - 7.0) < 0.5, y diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index 0e983017..9ae47014 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -168,8 +168,9 @@ def finetune(steps: int = 1000) -> str: return str(ckpts.path / "adapter") ``` -A queue endpoint that mounts the same volume at `/runpod-volume` reads what the task wrote; see -[`train_then_serve.py`](../examples/train_then_serve.py). For a long run, `job = finetune.spawn()` +A queue endpoint that mounts the same volume at `/runpod-volume` reads what the task wrote (golden +path 07 shows the volume handoff). Small results can skip the volume: return them from the task +as JSON, as [`train_then_serve.py`](../examples/train_then_serve.py) does. For a long run, `job = finetune.spawn()` then `await job.wait(timeout=...)`, which terminates the Pod when it finishes or times out. ## Streaming results From cdd1dc8eb6db8ccd0b9412d79e0204fc1c4fc265 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 20 Oct 2026 13:58:42 -0400 Subject: [PATCH 07/18] docs(flash): drop SDK workarounds now fixed in runpod 2.0.0rc2 Assumes runpod 2.0.0rc2, which carries the SDK fixes for these issues: - The build leaves torch out even when a dependency pulls it in, so golden path 08 lists peft and accelerate directly and the --no-deps workaround and its gotcha are gone. - GPU device names pin exact devices, so the set-endpoint-gpus advice is gone. - vol.path works on deployed workers. - Dev sessions clean up on kill or SIGINT, so the SIGINT wrapper is gone. - Only workers that call other resources get the API key. Install pins move to runpod>=2.0.0rc2. Concept rules are replaced to match. --- plugins/runpod/skills/flash/SKILL.md | 43 ++++++-------- .../flash/evals/dev-loop-iteration.eval.md | 4 +- .../migrate-runpod-flash-project.eval.md | 8 +-- plugins/runpod/skills/flash/reference/api.md | 21 ++++--- .../reference/migrate-from-runpod-flash.md | 13 +++- .../skills/flash/reference/setup-and-cli.md | 6 +- .../skills/runpod-usage/concepts/flash.yaml | 59 ++++++++++++++----- .../runpod-usage/reference/getting-started.md | 4 +- plugins/runpod/skills/runpod/SKILL.md | 2 +- .../03-whisper-endpoint/variant-b-flash.md | 4 +- .../golden-paths/07-network-volume-handoff.md | 2 +- .../golden-paths/08-finetune-to-serverless.md | 38 +++++------- .../skills/runpod/golden-paths/README.md | 2 +- .../runpodctl/reference/model-caching.md | 4 +- 14 files changed, 118 insertions(+), 92 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 4eb1a0c7..e3102ee1 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -61,7 +61,7 @@ live in [`examples/`](examples/). | Moving a `runpod-flash` project to the SDK | [reference/migrate-from-runpod-flash.md](reference/migrate-from-runpod-flash.md) | | Maintaining a `runpod-flash` project as it is | [reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md) | -Quick start: `uv add "runpod>=2.0.0rc1"` (or `pip install "runpod>=2.0.0rc1"`; this resolves +Quick start: `uv add "runpod>=2.0.0rc2"` (or `pip install "runpod>=2.0.0rc2"`; this resolves to the release candidate until 2.0.0 is stable) → `rp login` (or `export RUNPOD_API_KEY=...`) → `rp flash init my-app` → `cd my-app && rp flash dev main.py`. @@ -116,22 +116,20 @@ A dev session is long-running. Three rules: the entrypoint on every file save, so you never need to press Enter. - **Capture its output** to a log file and read it. Worker `print`s, tracebacks, `✗ remote execution failed: ...` and the per-run `✓ done in ...` / `✗ failed in ...` lines land there. -- **Stop it with SIGINT**, so it deletes its endpoints, and confirm the log ends with - `cleaning up N deleted`. A non-interactive shell starts `&` jobs with SIGINT ignored, and - SIGTERM or `kill -9` exit without cleanup, so launch through the wrapper below, which - restores SIGINT. When one run is enough, prefer `--once`; it cleans up on its own. +- **Stop it with `kill` (SIGTERM) or SIGINT**, so it deletes its endpoints, and confirm the + log ends with `cleaning up N deleted`. A second signal stops waiting for a running + entrypoint and goes straight to cleanup. `kill -9` skips cleanup. When one run is enough, + prefer `--once`; it cleans up on its own. ```bash -# restore SIGINT for the background job, then exec rp -python3 -c 'import os,signal,sys; signal.signal(signal.SIGINT, signal.SIG_DFL); os.execvp(sys.argv[1], sys.argv[1:])' \ - rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 & +rp flash dev main.py < /dev/null > /tmp/rp-dev.log 2>&1 & DEV=$! # poll the log (bounded) for the run result: "✓ done in ..." or "✗ failed in ..." for i in $(seq 1 120); do grep -qE "✓ done in|✗ failed in" /tmp/rp-dev.log && break; sleep 3; done tail -40 /tmp/rp-dev.log # edit main.py and save -> the session re-scans, refreshes endpoints, re-runs main() # (each run appends a new result line; read the last one) -kill -INT $DEV; wait $DEV; tail -3 /tmp/rp-dev.log # expect "cleaning up N deleted" +kill $DEV; wait $DEV; tail -3 /tmp/rp-dev.log # expect "cleaning up N deleted" ``` For a single check, `rp flash dev main.py --once` is simpler and cleans up on its own. If a @@ -184,24 +182,17 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. 12. **A set `RUNPOD_API_KEY` wins over `rp login`.** A stale exported key causes 401s while the saved login is fine. Check with `runpodctl user`, then `unset RUNPOD_API_KEY` or export a good key. -13. **Don't list torch-dependent packages for a GPU deploy.** Auto-exclusion drops `torch` only - when it is listed directly. In `runpod` 2.0.0rc1, a package that depends on it (`peft`, - `accelerate`, `sentence-transformers`) pulls a CUDA 13 `torch` into the artifact, even with - `--exclude torch`. That copy replaces the GPU image's torch, the worker fails its CUDA check - (`libcudart.so.13: cannot open shared object file`), and the endpoint crash-loops. Install such - packages on first call with `pip install --no-deps` instead (the `_ensure` helper in - [golden path 08](../runpod/golden-paths/08-finetune-to-serverless.md)), and list their - non-torch dependencies (`psutil`, `pyyaml`) in `dependencies`. Dev sessions are unaffected: - they pip-install on the worker, where torch is already satisfied. +13. **The torch family comes from the GPU image; never pin it.** Builds leave `torch`, + `torchvision`, `torchaudio` and `triton` out of the artifact whether you list them or a + dependency pulls them in (`peft`, `accelerate`, `sentence-transformers`), plus the wheels + only torch needs. GPU workers use the image's build (`torch 2.9.1+cu128`), so a version + pin on these packages is ignored. In an app that mixes CPU and GPU resources, a CPU + resource whose dependencies need torch installs it at cold start; give it + `container_disk_gb` room for that (the build warns when this applies). 14. **Dev sessions run the runtime image's own `runpod`.** Deploys vendor your local package, - but `rp flash dev` workers import the `runpod` baked into the built-in image. The - `:latest` runtime images used by `runpod` 2.0.0rc1 predate `runpod.NetworkVolume` and - `runpod.GlobalVolume`, so an app that declares one fails on the worker with - `module 'runpod' has no attribute 'NetworkVolume'`. Deployed workers on the same images - attach the volume, but `vol.path` raises `is not mounted in the current execution`; read - the literal mount path instead (`/runpod-volume` for queue and API resources, the - `mounts` key for tasks). An `AttributeError` on a `runpod.` name in a worker traceback - means the image is behind your SDK. + but `rp flash dev` workers import the `runpod` baked into the built-in image. An + `AttributeError` on a `runpod.` name in a dev worker traceback means the image is behind + your SDK; `rp flash deploy` runs your installed version. 15. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` removes one environment's endpoints; `rp flash app delete APP` removes every environment. Then delete volumes you no longer need. diff --git a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md index 18c755b1..e040744e 100644 --- a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md +++ b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md @@ -6,7 +6,7 @@ ## Setup -- Requires `runpod>=2.0.0rc1` installed in the active environment and a valid key +- Requires `runpod>=2.0.0rc2` installed in the active environment and a valid key (`rp login` or `RUNPOD_API_KEY`). - Copy the fixture to a scratch dir so the graded fix does not mutate the committed fixture: `cp -r flash/evals/fixtures/dev-loop /tmp/dev-loop-eval && cd /tmp/dev-loop-eval`. @@ -31,7 +31,7 @@ The agent should actually execute (not merely describe) the following: 4. Re-run without deploying: save the file and let the background session re-run, or run `--once` again. 5. Confirm the entrypoint succeeds (`✓` result line, the `assert` passes). -6. End the session with SIGINT (or let `--once` exit) and confirm the log reports its +6. End the session with `kill` (SIGTERM) or SIGINT (or let `--once` exit) and confirm the log reports its temporary endpoints deleted. ## Assertions diff --git a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md index 942955dd..612897aa 100644 --- a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md +++ b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md @@ -14,7 +14,7 @@ The agent should: 1. Inventory the `runpod_flash` imports, every `Endpoint`, and every `flash` CLI call (CI too) 2. Check the known differences (multi-file project, `id=`/`template=`, path params, volume lists) and report any that apply before converting -3. Install `runpod>=2.0.0rc1`, convert to `runpod.App` with `@app.queue` for `embed` and an +3. Install `runpod>=2.0.0rc2`, convert to `runpod.App` with `@app.queue` for `embed` and an `@app.api` class with `@runpod.post("/query")` for `search`, and add a `@runpod.local_entrypoint` 4. Update CI from `flash deploy` to `rp flash deploy` @@ -26,9 +26,9 @@ The agent should: ## Assertions - Removes every `runpod_flash` import; uses `import runpod` -- `embed` becomes `@app.queue(gpu="ADA_24" or GpuGroup.ADA_24, ...)`, and the agent flags that - `sentence-transformers` depends on `torch`: listed in `dependencies` it makes the GPU deploy - crash-loop, so it installs it with `--no-deps` at runtime (flash SKILL gotcha 13) +- `embed` becomes `@app.queue(gpu="ADA_24" or GpuGroup.ADA_24, ...)` with + `sentence-transformers` in `dependencies`. The agent does not pin `torch` or install it with + `--no-deps`: the build leaves torch to the GPU image (flash SKILL gotcha 13) - `search` becomes an `@app.api` class whose POST method takes one `body` argument - CI calls `rp flash deploy`, not `flash deploy` - Runs or recommends `rp flash dev main.py --once` before `rp flash deploy` diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 39fbee4b..77a09002 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -140,9 +140,7 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must - Global volumes need GPU compute. - A new network volume is placed in a data center that has network storage and stock for every resource sharing it; resources using it are scheduled in that data center. -- Read `models.path` **inside** the remote function. Volumes are not mounted locally. If - `.path` raises `is not mounted in the current execution`, the runtime image is older than - your SDK; read the literal mount path (`/runpod-volume`) instead. +- Read `models.path` **inside** the remote function. Volumes are not mounted locally. - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. ## Models and secrets @@ -175,15 +173,24 @@ not download them. Gated models need `HF_TOKEN` on the same resource. Create the - **A list** of any of these. Automatic switching between listed GPUs needs a max of at least 5 workers. -`rp flash deploy` in `runpod` 2.0.0rc1 turns device names into their pools with no exclusions: -`gpu=["4090", "L4", "A5000"]` deployed as `ADA_24` + `AMPERE_24`, and a worker landed on a -24 GB RTX PRO 6000 Blackwell MIG slice. When exact hardware matters, read the endpoint after -deploy and pin it with the Runpod MCP `set-endpoint-gpus` tool. +Device names pin those exact devices. Endpoints select GPUs by pool, so the SDK selects each +device's pool and excludes the pool's other devices, reading pool membership from the live GPU +catalog: `gpu=["4090", "L4", "A5000"]` deploys as `ADA_24,AMPERE_24,-NVIDIA GeForce RTX 3090, +-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb`. A pool id allows every device in +the pool, including MIG slices the platform adds to it. `cpu=` takes a flavor id: `cpu3g-{1-4,2-8,4-16,8-32}` (general), `cpu3c-{1-2,2-4,4-8,8-16}` (compute), `cpu5c-{1-2,2-4,4-8,8-16}` (compute, 5th gen), where the numbers are vCPUs and GB of RAM. `runpod.CpuInstanceType` holds the same values. +## Credentials on workers + +A worker that calls another resource (`.remote()`, `.spawn()`, a `runpod.Queue`/`Api` client) +needs the account API key, so deploy and dev add `RUNPOD_API_KEY` to the endpoints whose code +can reach one. The build reads the resource's own body and the module-level helpers it calls. +Code it cannot follow (a local helper module, a nested function) gets the key too, so a caller +never goes without it. Resources that make no such call run without the key. + ## Exact rules - [flash](../../runpod-usage/concepts/flash.yaml) diff --git a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md index 5a595eff..c50f15f7 100644 --- a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md +++ b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md @@ -39,8 +39,8 @@ before converting, and test it carefully or keep the project on `runpod-flash`: ## 3. Install and log in ```bash -pip uninstall runpod-flash && pip install "runpod>=2.0.0rc1" -# uv: uv remove runpod-flash && uv add "runpod>=2.0.0rc1" +pip uninstall runpod-flash && pip install "runpod>=2.0.0rc2" +# uv: uv remove runpod-flash && uv add "runpod>=2.0.0rc2" rp login # a key saved by `flash login` keeps working ``` @@ -128,6 +128,15 @@ class MyApi: - **`.flashignore` becomes `.runpodignore`**, and `.gitignore` files apply too. - **Local Python must be 3.10–3.14** and matches the worker's in dev sessions and tasks. - **Volumes outlive undeploy** and keep billing. +- **The torch family still comes from the GPU image**, as in runpod-flash, including torch + pulled in by another dependency. A CPU resource in the same app that needs torch installs it + at cold start, so give it `container_disk_gb` room. +- **GPU device names pin the exact device**, as runpod-flash's pool exclusions did, now using + the platform's live pool membership, so pool members runpod-flash didn't know about (MIG + slices) are excluded too. +- **Only resources that call other resources get `RUNPOD_API_KEY`** in their worker env. A + helper that calls `.remote()` from a separate local module counts, so nested calls keep + working. ## 6. Smoke-test, deploy, clean up diff --git a/plugins/runpod/skills/flash/reference/setup-and-cli.md b/plugins/runpod/skills/flash/reference/setup-and-cli.md index eaec89e3..a32d0b49 100644 --- a/plugins/runpod/skills/flash/reference/setup-and-cli.md +++ b/plugins/runpod/skills/flash/reference/setup-and-cli.md @@ -11,13 +11,13 @@ into the project's virtual environment, because dev sessions run your local inte ```bash # requires Python 3.10-3.14 -uv add "runpod>=2.0.0rc1" # uv project -pip install "runpod>=2.0.0rc1" # pip / venv +uv add "runpod>=2.0.0rc2" # uv project +pip install "runpod>=2.0.0rc2" # pip / venv pip show runpod # confirm the version (2.x) rp update # upgrade the package in place; rp update --version X pins one ``` -The `>=2.0.0rc1` specifier also accepts the 2.0 release candidate, so it resolves to 2.x +The `>=2.0.0rc2` specifier also accepts the 2.0 release candidate, so it resolves to 2.x whether or not a stable 2.0.0 is published. A plain `pip install runpod` on an index with only the release candidate installs 1.x, which has no `rp flash`. diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 167ea8d3..380e100c 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -22,7 +22,7 @@ rules: statement: > Flash ships in the `runpod` package from version 2.0, with the `rp` CLI (`rp flash init`, `dev`, `deploy`, `app`, `env`, `undeploy`). The CLI is - also installed as `runpod`. Install `runpod>=2.0.0rc1`; that specifier + also installed as `runpod`. Install `runpod>=2.0.0rc2`; that specifier also accepts the 2.0 release candidate. status: verified evidence: @@ -168,27 +168,56 @@ rules: - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. } - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 7. } - - id: flash.transitive-torch-breaks-gpu-deploy + - id: flash.torch-family-from-gpu-image statement: > - In `runpod` 2.0.0rc1, a GPU app whose `dependencies` include a package - that depends on `torch` (`peft`, `accelerate`, `sentence-transformers`) - deploys with a CUDA 13 `torch` in its artifact. That copy replaces the - image's torch, the worker fails its CUDA check with `libcudart.so.13: - cannot open shared object file`, and the endpoint crash-loops. Install - such packages on the worker with `pip install --no-deps` instead. + From `runpod` 2.0.0rc2, builds leave `torch`, `torchvision`, `torchaudio` + and `triton` out of the artifact whether a requirement lists them or a + dependency pulls them in (`peft`, `accelerate`, `sentence-transformers`), + together with the wheels only they need. GPU workers on the built-in + images use the image's `torch 2.9.1+cu128`, and version pins on these + packages are ignored. In an app that mixes CPU and GPU resources, a CPU + resource whose dependencies need torch installs it at cold start, so it + needs container disk for it. status: verified evidence: - - { source: live-probe, ref: "rp flash deploy of a gpu queue function with dependencies=[\"peft\"]: 645.5 MB artifact, worker logs 'Fitness check failed: _cuda_init_check | libcudart.so.13'; the same function installing peft with --no-deps at runtime returned torch 2.9.1+cu128 with cuda True", seen: "2026-10-16" } + - { source: live-probe, ref: "rp flash deploy of a gpu queue function with dependencies=[\"peft\"]: artifact 80.7 MB with no torch (645.6 MB with torch 2.14.1 before the fix); the worker returned torch 2.9.1+cu128 from site-packages with cuda True", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/pull/623", seen: "2026-10-16", note: Removes transitive torch-family wheels after vendoring and scopes the cold-start install list to CPU resources. } - - id: flash.deploy-gpu-names-map-to-pools + - id: flash.gpu-device-names-pin-devices statement: > - In `runpod` 2.0.0rc1, `rp flash deploy` turns GPU device names into their - pools without exclusions, so a worker can run on any GPU in those pools. - Pin exact GPU types on the endpoint after deploy when the hardware - matters. + From `runpod` 2.0.0rc2, GPU device names in `gpu=` pin those devices. + Endpoints select GPUs by pool, so the SDK selects each device's pool and + excludes the pool's other devices, reading membership from the live GPU + catalog. A pool id allows every device in the pool, including MIG slices. status: verified + see: [gpu-pool.pin-a-card] evidence: - - { source: live-probe, ref: "gpu=[\"4090\", \"L4\", \"A5000\"] deployed as gpu.pools [ADA_24, AMPERE_24]; the worker reported NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb", seen: "2026-10-16" } + - { source: live-probe, ref: "gpu=[\"NVIDIA GeForce RTX 4090\", \"NVIDIA L4\", \"NVIDIA RTX A5000\"] stored gpuIds ADA_24,AMPERE_24,-NVIDIA GeForce RTX 3090,-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb; four parallel jobs all reported NVIDIA GeForce RTX 4090", seen: "2026-10-16" } + - { source: live-probe, ref: "GET https://api.runpod.io/v2/catalog/gpus lists NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb with pool AMPERE_24", seen: "2026-10-16" } + + - id: flash.worker-api-key-for-callers + statement: > + From `runpod` 2.0.0rc2, deploy and dev add the account API key + (`RUNPOD_API_KEY`) only to resources whose code can call another + resource: a reference to another resource, `.remote()` or `.spawn()`, a + `runpod` client, or code the build cannot follow (a local helper module, + a nested function). Other resources run without the key. + status: verified + evidence: + - { source: live-probe, ref: "deployed leaf + caller queues: only caller's template had RUNPOD_API_KEY; caller returned {'has_key': True, 'nested': {'has_key': False, 'x': 42}}", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/pull/628", seen: "2026-10-16" } + + - id: flash.dev-session-stop-signals + statement: > + From `runpod` 2.0.0rc2, a dev session deletes its endpoints when it + receives SIGINT or SIGTERM, including a background session started with + SIGINT ignored. A second signal stops waiting for a running entrypoint. + `kill -9` skips the cleanup. + status: verified + see: [flash.dev-session] + evidence: + - { source: live-probe, ref: "rp flash dev main.py < /dev/null & from a script, then kill -TERM and separately kill -INT: both logged 'cleaning up 1 deleted' and left no endpoints", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/pull/625", seen: "2026-10-16" } - id: flash.legacy-dev-ships-body-only statement: > diff --git a/plugins/runpod/skills/runpod-usage/reference/getting-started.md b/plugins/runpod/skills/runpod-usage/reference/getting-started.md index 3dc41840..a1ecc3ef 100644 --- a/plugins/runpod/skills/runpod-usage/reference/getting-started.md +++ b/plugins/runpod/skills/runpod-usage/reference/getting-started.md @@ -16,7 +16,7 @@ lane's `SKILL.md` (linked). You don't need all of them. | Lane | Quick install | Source / docs | | --- | --- | --- | | **runpodctl** (CLI) | `curl -sSL https://cli.runpod.net \| bash` (or `brew install runpod/runpodctl/runpodctl`) | [source](https://github.com/runpod/runpodctl) · [`runpodctl/SKILL.md`](../../runpodctl/SKILL.md) | -| **flash** (deploy your own code) | `pip install "runpod>=2.0.0rc1"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`flash/SKILL.md`](../../flash/SKILL.md) | +| **flash** (deploy your own code) | `pip install "runpod>=2.0.0rc2"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`flash/SKILL.md`](../../flash/SKILL.md) | | **runpod-mcp** (hosted) | `npx @runpod/mcp-server@latest add` (guided; OAuth) | [source](https://github.com/runpod/runpod-mcp) · [`runpod-mcp/SKILL.md`](../../runpod-mcp/SKILL.md) | | **runpod-mcp** (local stdio) | `claude mcp add runpod -e RUNPOD_API_KEY=... -- npx -y @runpod/mcp-server` | same as above | | **companion CLIs** (`hf`/`gh`/`docker`/`aws`) | per-tool; see the skill | [`companion-clis/SKILL.md`](../../companion-clis/SKILL.md) | @@ -29,7 +29,7 @@ Nothing to install for the **hosted MCP** beyond configuring your client, and th ```bash npx skills add runpod/runpod-plugins-official # the skills — works with any agent curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl) -pip install "runpod>=2.0.0rc1" # Runpod SDK + rp CLI — deploy your own code (flash lane) +pip install "runpod>=2.0.0rc2" # Runpod SDK + rp CLI — deploy your own code (flash lane) npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth ``` diff --git a/plugins/runpod/skills/runpod/SKILL.md b/plugins/runpod/skills/runpod/SKILL.md index 4b166419..01fcd497 100644 --- a/plugins/runpod/skills/runpod/SKILL.md +++ b/plugins/runpod/skills/runpod/SKILL.md @@ -97,7 +97,7 @@ set up, stop and get a key, in order: **Then:** if a lane already works, use it — but if *only* the MCP is OAuth'd, still get a key before any CLI-only step. Missing a CLI? `curl -sSL https://cli.runpod.net | bash` (runpodctl) · -`pip install "runpod>=2.0.0rc1"` (Runpod SDK + `rp`, flash lane) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: +`pip install "runpod>=2.0.0rc2"` (Runpod SDK + `rp`, flash lane) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: [`runpod-usage/reference/getting-started.md`](../runpod-usage/reference/getting-started.md). ## How to route diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index ccfabb52..be5fbfc9 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -20,7 +20,7 @@ and cost notes live in the [folder README](README.md); the flash skill is at - Python 3.10–3.14 and the Runpod SDK in the project environment: ```bash - pip install "runpod>=2.0.0rc1" # installs the rp CLI + pip install "runpod>=2.0.0rc2" # installs the rp CLI ``` - A Runpod API key for non-interactive use: ```bash @@ -86,7 +86,7 @@ and cost notes live in the [folder README](README.md); the flash skill is at rp flash dev main.py --once # temporary endpoint, runs main(), cleans up ``` For repeated edits, run a background session as shown in the flash skill's *Autonomous dev - loop* (it restores SIGINT so `kill -INT` cleans up); each save re-runs `main()` against the + loop* (`kill` stops it and deletes its endpoint); each save re-runs `main()` against the refreshed endpoint. 4. **Ship it:** diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index 51257281..97258992 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -40,7 +40,7 @@ that DC. ## Prerequisites - `RUNPOD_API_KEY` resolvable (runpodctl and the Runpod SDK both read it). -- `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc1"`, which provides `rp`). +- `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc2"`, which provides `rp`). - An SSH key registered **before** creating the pod — Runpod injects registered keys into the pod at startup, so a key added after boot won't work until a restart (`runpodctl ssh list-keys`; see diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 2ffde78a..dc206db7 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -14,11 +14,13 @@ GPU, dataset) — nothing structural changes. **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` -and generated. Section 3 is ported to the Runpod SDK (`runpod` 2.0.0rc1): on 2026-10-16 the -section 3 code, deployed with `rp flash deploy`, loaded TinyLlama plus a LoRA -adapter from a network volume and generated text (the adapter was an untrained one saved by a -helper function, standing in for the pod training step). Run it with `rp flash deploy`: dev -sessions on the built-in runtime images fail on volume apps (flash SKILL gotcha 14). +and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code, +deployed with `rp flash deploy` on a build of `runpod` with the transitive-torch, GPU-pinning, +`vol.path` and API-key fixes, loaded TinyLlama plus a LoRA adapter from a network volume and +generated text (the adapter was an untrained one saved by a helper function, standing in for the +pod training step). `peft` and `accelerate` were plain `dependencies` and the worker ran the +image's `torch 2.9.1+cu128`. Run it with `rp flash deploy` (flash SKILL gotcha 14 covers dev +sessions on older runtime images). The `runpod-flash` live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. @@ -50,7 +52,7 @@ A 1.1B LoRA on a 4090 trains in **minutes** — fast enough to iterate the loop spending on a real run. ## Prerequisites -- `RUNPOD_API_KEY` resolvable; `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc1"`, which provides `rp`). +- `RUNPOD_API_KEY` resolvable; `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc2"`, which provides `rp`). - An SSH key registered **before** creating the pod (`runpodctl ssh list-keys`; see `../../runpod-usage/reference/getting-started.md`). - `HF_TOKEN` only if you scale to a **gated** base model (TinyLlama is ungated). @@ -161,11 +163,8 @@ Mind the two things from 07: mount path is `/runpod-volume`, and the worker call as `generate(**job_input)`, so the `input` keys are its parameter names (empty input is rejected). -**Why `peft` is installed at runtime:** listing `peft` or `accelerate` in `dependencies` makes -`rp flash deploy` vendor their `torch` dependency into the artifact (`runpod` 2.0.0rc1), and that -copy replaces the GPU image's CUDA torch. The worker then fails its CUDA check -(`libcudart.so.13: cannot open shared object file`) and crash-loops. `_ensure` installs them with -`--no-deps` on first call instead (flash SKILL gotcha 13). +**Dependencies:** `peft` and `accelerate` go in `dependencies` like any package; the build leaves +their `torch` dependency to the GPU image (flash SKILL gotcha 13). The `runpod-flash` handler that ran used a function with a `global` cache and `**kwargs`. The SDK version keeps the same model code; the whole module ships, so the paths can be @@ -181,29 +180,20 @@ BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" _M = _T = None -def _ensure(*pkgs): - # install torch-dependent packages without deps, so the GPU image's CUDA torch stays in place - import importlib.util, subprocess, sys - missing = [p for p in pkgs if importlib.util.find_spec(p) is None] - if missing: - subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", "--no-deps", *missing]) - - @app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="EU-RO-1", mounts={"/runpod-volume": vol}, env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache - dependencies=["transformers", "psutil", "pyyaml"]) # NOT peft/accelerate: see below + dependencies=["transformers", "peft", "accelerate"]) def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict: - _ensure("peft", "accelerate") import torch - from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel + from transformers import AutoModelForCausalLM, AutoTokenizer global _M, _T - adapter = "/runpod-volume/outputs/lora-out" # literal mount path (gotcha 14) + adapter = str(vol.path / "outputs" / "lora-out") # written by the pod's training run if _M is None: # load once per worker _T = AutoTokenizer.from_pretrained(adapter) - base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float16).to("cuda") + base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16).to("cuda") _M = PeftModel.from_pretrained(base, adapter).eval() text = f"### Instruction:\n{prompt}\n\n### Response:\n" ids = _T(text, return_tensors="pt").to("cuda") diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index d58217d9..17bcd697 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -75,7 +75,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | | 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16) | -| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step via `rp flash deploy` (2026-10-16) | +| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step via `rp flash deploy` (2026-10-17) | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified | diff --git a/plugins/runpod/skills/runpodctl/reference/model-caching.md b/plugins/runpod/skills/runpodctl/reference/model-caching.md index c87270f5..e8eb63a6 100644 --- a/plugins/runpod/skills/runpodctl/reference/model-caching.md +++ b/plugins/runpod/skills/runpodctl/reference/model-caching.md @@ -150,8 +150,8 @@ runpod.serverless.start({"handler": handler}) `HF_HUB_OFFLINE=1` on a cache that `VolumeCache` restores. - **Size the container disk for the whole local cache** plus runtime headroom: the hydrate and any fallback download both write there. Set it on the template - (`runpodctl template create --container-disk-in-gb`, or `PodTemplate(containerDiskInGb=...)` in - Flash, which defaults to 64 GB). + (`runpodctl template create --container-disk-in-gb`), or with `container_disk_gb=` on a Runpod + SDK decorator, which defaults to 30 GB on GPU. - **Size the volume for the mirror** (`/runpod-volume/.cache/`). Large files you delete or rename locally stay on the volume, so usage grows as you swap models. - **Populate with one worker.** Workers that cold-start together on an empty mirror each From 0dcc42a843d4fb92a26c79da2050218c491d7a64 Mon Sep 17 00:00:00 2001 From: Justin Date: Mon, 19 Oct 2026 23:37:35 -0400 Subject: [PATCH 08/18] docs(flash): queue calls stop waiting after 300 s; scope one-file rule to dev --- plugins/runpod/skills/flash/SKILL.md | 11 ++++++----- plugins/runpod/skills/flash/reference/api.md | 2 +- .../runpod/skills/runpod-usage/concepts/flash.yaml | 14 ++++++++++---- .../03-whisper-endpoint/variant-b-flash.md | 2 +- 4 files changed, 18 insertions(+), 11 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index e3102ee1..daaa9148 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -105,9 +105,9 @@ Call styles on a queue or task handle: `fn.remote(...)` (blocks), `await fn.remo others). Re-run to update. `--build-only` packages without deploying, a cheap offline check that discovery and dependency resolution work. -Dev and deploy run the **same module source**: the whole file ships, so module-level imports, -constants and helpers behave identically in both. The app must live in **one file** today -(multi-file projects are a known gap). +Dev and deploy run the **same module source**, so module-level imports, constants and helpers +behave identically in both. Deploy packages the project directory, but a dev session ships +only the module's source, so keep the app in **one file** when you use `rp flash dev`. ## Autonomous dev loop @@ -158,8 +158,9 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. and deploy alike; a non-JSON argument fails with `arguments must be json-serializable`. Pass URLs or base64, not objects. Whole-number floats can arrive as integers (`1.0` → `1`), so cast numeric inputs explicitly (`torch.tensor(x, dtype=torch.float32)`). -2. **One file per app.** Discovery imports the module you point at. Helpers in a sibling local - module are not shipped yet; keep them in the same file or in an installable package. +2. **One file per app in dev.** A dev session ships only the module's source, so a sibling + local module imports in deploy but fails in dev. For dev sessions, keep helpers in the same + file or in an installable package. 3. **Keep blocking work out of module scope.** The SDK imports your module to discover resources, locally and on every worker. Put heavy imports (`torch`) and work inside functions; guard local-only code with `runpod.is_local()`. diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 77a09002..7540b24b 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -57,7 +57,7 @@ On a queue or task handle: | Call | Does | |---|---| -| `fn.remote(*args, **kwargs)` | runs remotely, blocks for the result; no timeout by default | +| `fn.remote(*args, **kwargs)` | runs remotely, blocks for the result; on a queue, stops waiting after 300 s with `TimeoutError` | | `await fn.remote.aio(...)` | async form | | `fn.spawn(...)` | submits, returns a job immediately | | `fn.stream(...)` | yields chunks from a generator function | diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 380e100c..467996a1 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -78,11 +78,14 @@ rules: - id: flash.same-module-in-dev-and-deploy statement: > Dev sessions and deploys run the whole module source on the worker, so - module-level imports, constants and helpers behave the same in both. An - app lives in one file; helpers in sibling local modules are not shipped. + module-level imports, constants and helpers behave the same in both. + Deploy packages the project directory, but a dev session ships only the + module's source, so helpers in sibling local modules import only in + deploy. An app run with dev sessions lives in one file. status: documented evidence: - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/serialization.py", seen: "2026-10-16", note: get_function_source returns the full module source. } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/deploy.py", seen: "2026-10-19", note: package_project walks the project directory into the artifact. } - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16", note: Known differences lists projects split across multiple local files. } - id: flash.json-queue-arguments @@ -161,11 +164,14 @@ rules: - id: flash.client-limits statement: > Payloads are limited to 10 MB, so pass URLs or write to a volume. - `.remote()` and `job.wait()` impose no timeout by default; the endpoint - client's `run_sync` takes a `timeout` in seconds. + Jobs get no execution timeout by default, but `.remote()` stops + waiting after 300 s on a queue and raises `TimeoutError` while the job + keeps running. For longer calls, `.spawn()` and `job.result(timeout=...)`. + The endpoint client's `run_sync` takes a `timeout` in seconds. status: documented evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-19", note: DEFAULT_TIMEOUT_SECONDS = 300.0 bounds invoke() and job result waits; _wait_terminal raises TimeoutError at the deadline. } - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 7. } - id: flash.torch-family-from-gpu-image diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index be5fbfc9..ccea9f92 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -119,7 +119,7 @@ Verified result (2026-10-16, dev session, cold): ``` The `runpod-flash` run measured cold ~55–75 s (image pull + model download) and warm <1 s. For the first, cold call use `/run` + poll `/status/`, or `.remote()` from the SDK, which -has no timeout by default. +waits up to 300 s; beyond that, `.spawn()` and `job.result(timeout=...)`. ## Variant-specific gotchas From 5ac431622253e5ea26200e073d3ef8709e9ca0c1 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 21 Oct 2026 18:04:45 -0400 Subject: [PATCH 09/18] style(concepts): format flash.yaml --- .../skills/runpod-usage/concepts/flash.yaml | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 467996a1..7b6c3b40 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -163,15 +163,15 @@ rules: - id: flash.client-limits statement: > - Payloads are limited to 10 MB, so pass URLs or write to a volume. - Jobs get no execution timeout by default, but `.remote()` stops - waiting after 300 s on a queue and raises `TimeoutError` while the job - keeps running. For longer calls, `.spawn()` and `job.result(timeout=...)`. - The endpoint client's `run_sync` takes a `timeout` in seconds. + Payloads are limited to 10 MB, so pass URLs or write to a volume. Jobs get + no execution timeout by default, but `.remote()` stops waiting after 300 s + on a queue and raises `TimeoutError` while the job keeps running. For + longer calls, `.spawn()` and `job.result(timeout=...)`. The endpoint + client's `run_sync` takes a `timeout` in seconds. status: documented evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. } - - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-19", note: DEFAULT_TIMEOUT_SECONDS = 300.0 bounds invoke() and job result waits; _wait_terminal raises TimeoutError at the deadline. } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-19", note: "DEFAULT_TIMEOUT_SECONDS = 300.0 bounds invoke() and job result waits; _wait_terminal raises TimeoutError at the deadline." } - { source: skill, path: plugins/runpod/skills/flash/SKILL.md, note: Gotcha 7. } - id: flash.torch-family-from-gpu-image @@ -204,10 +204,10 @@ rules: - id: flash.worker-api-key-for-callers statement: > From `runpod` 2.0.0rc2, deploy and dev add the account API key - (`RUNPOD_API_KEY`) only to resources whose code can call another - resource: a reference to another resource, `.remote()` or `.spawn()`, a - `runpod` client, or code the build cannot follow (a local helper module, - a nested function). Other resources run without the key. + (`RUNPOD_API_KEY`) only to resources whose code can call another resource: + a reference to another resource, `.remote()` or `.spawn()`, a `runpod` + client, or code the build cannot follow (a local helper module, a nested + function). Other resources run without the key. status: verified evidence: - { source: live-probe, ref: "deployed leaf + caller queues: only caller's template had RUNPOD_API_KEY; caller returned {'has_key': True, 'nested': {'has_key': False, 'x': 42}}", seen: "2026-10-16" } From 4d00f66b09ff087f607a47ecb81981480a02270b Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 22 Oct 2026 12:58:53 -0400 Subject: [PATCH 10/18] docs(flash): pin runpod>=2.0.1 and say every endpoint gets the API key runpod 2.0.0rc2 was cut before the build, GPU-pinning, dev-signal and vol.path fixes merged, so the docs now pin the next release, 2.0.1. The change that limited RUNPOD_API_KEY to endpoints that call other resources (runpod-python #628, then #641) was closed unmerged. Deploy and dev add the key to every endpoint, so flash.worker-api-key-for-callers is replaced by flash.worker-api-key-every-endpoint, cited to the SDK source, and api.md, the migration guide and golden path 08 say the same. --- plugins/runpod/skills/flash/SKILL.md | 3 +- .../flash/evals/dev-loop-iteration.eval.md | 2 +- .../migrate-runpod-flash-project.eval.md | 2 +- plugins/runpod/skills/flash/reference/api.md | 9 +++-- .../reference/migrate-from-runpod-flash.md | 10 +++--- .../skills/flash/reference/setup-and-cli.md | 9 +++-- .../skills/runpod-usage/concepts/flash.yaml | 33 +++++++++---------- .../runpod-usage/reference/getting-started.md | 4 +-- plugins/runpod/skills/runpod/SKILL.md | 2 +- .../03-whisper-endpoint/variant-b-flash.md | 2 +- .../golden-paths/07-network-volume-handoff.md | 2 +- .../golden-paths/08-finetune-to-serverless.md | 4 +-- 12 files changed, 39 insertions(+), 43 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index daaa9148..149ee9cc 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -61,8 +61,7 @@ live in [`examples/`](examples/). | Moving a `runpod-flash` project to the SDK | [reference/migrate-from-runpod-flash.md](reference/migrate-from-runpod-flash.md) | | Maintaining a `runpod-flash` project as it is | [reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md) | -Quick start: `uv add "runpod>=2.0.0rc2"` (or `pip install "runpod>=2.0.0rc2"`; this resolves -to the release candidate until 2.0.0 is stable) → `rp login` (or `export RUNPOD_API_KEY=...`) +Quick start: `uv add "runpod>=2.0.1"` (or `pip install "runpod>=2.0.1"`) → `rp login` (or `export RUNPOD_API_KEY=...`) → `rp flash init my-app` → `cd my-app && rp flash dev main.py`. ## The app model diff --git a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md index e040744e..c86c9f70 100644 --- a/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md +++ b/plugins/runpod/skills/flash/evals/dev-loop-iteration.eval.md @@ -6,7 +6,7 @@ ## Setup -- Requires `runpod>=2.0.0rc2` installed in the active environment and a valid key +- Requires `runpod>=2.0.1` installed in the active environment and a valid key (`rp login` or `RUNPOD_API_KEY`). - Copy the fixture to a scratch dir so the graded fix does not mutate the committed fixture: `cp -r flash/evals/fixtures/dev-loop /tmp/dev-loop-eval && cd /tmp/dev-loop-eval`. diff --git a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md index 612897aa..64387da5 100644 --- a/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md +++ b/plugins/runpod/skills/flash/evals/migrate-runpod-flash-project.eval.md @@ -14,7 +14,7 @@ The agent should: 1. Inventory the `runpod_flash` imports, every `Endpoint`, and every `flash` CLI call (CI too) 2. Check the known differences (multi-file project, `id=`/`template=`, path params, volume lists) and report any that apply before converting -3. Install `runpod>=2.0.0rc2`, convert to `runpod.App` with `@app.queue` for `embed` and an +3. Install `runpod>=2.0.1`, convert to `runpod.App` with `@app.queue` for `embed` and an `@app.api` class with `@runpod.post("/query")` for `search`, and add a `@runpod.local_entrypoint` 4. Update CI from `flash deploy` to `rp flash deploy` diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 7540b24b..d827336a 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -185,11 +185,10 @@ RAM. `runpod.CpuInstanceType` holds the same values. ## Credentials on workers -A worker that calls another resource (`.remote()`, `.spawn()`, a `runpod.Queue`/`Api` client) -needs the account API key, so deploy and dev add `RUNPOD_API_KEY` to the endpoints whose code -can reach one. The build reads the resource's own body and the module-level helpers it calls. -Code it cannot follow (a local helper module, a nested function) gets the key too, so a caller -never goes without it. Resources that make no such call run without the key. +Deploy and dev add the account API key as `RUNPOD_API_KEY` to every endpoint's template env, so +a worker can call another resource (`.remote()`, `.spawn()`, a `runpod.Queue`/`Api` client). +Endpoints that never make such a call get it too, and anyone who can read the template can read +the key. ## Exact rules diff --git a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md index c50f15f7..aaf8bd8a 100644 --- a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md +++ b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md @@ -39,8 +39,8 @@ before converting, and test it carefully or keep the project on `runpod-flash`: ## 3. Install and log in ```bash -pip uninstall runpod-flash && pip install "runpod>=2.0.0rc2" -# uv: uv remove runpod-flash && uv add "runpod>=2.0.0rc2" +pip uninstall runpod-flash && pip install "runpod>=2.0.1" +# uv: uv remove runpod-flash && uv add "runpod>=2.0.1" rp login # a key saved by `flash login` keeps working ``` @@ -134,9 +134,9 @@ class MyApi: - **GPU device names pin the exact device**, as runpod-flash's pool exclusions did, now using the platform's live pool membership, so pool members runpod-flash didn't know about (MIG slices) are excluded too. -- **Only resources that call other resources get `RUNPOD_API_KEY`** in their worker env. A - helper that calls `.remote()` from a separate local module counts, so nested calls keep - working. +- **Every resource gets `RUNPOD_API_KEY`** in its worker env. runpod-flash added it only to + endpoints that made remote calls, so after migrating, more endpoint templates hold the account + key. ## 6. Smoke-test, deploy, clean up diff --git a/plugins/runpod/skills/flash/reference/setup-and-cli.md b/plugins/runpod/skills/flash/reference/setup-and-cli.md index a32d0b49..ebd6a792 100644 --- a/plugins/runpod/skills/flash/reference/setup-and-cli.md +++ b/plugins/runpod/skills/flash/reference/setup-and-cli.md @@ -11,15 +11,14 @@ into the project's virtual environment, because dev sessions run your local inte ```bash # requires Python 3.10-3.14 -uv add "runpod>=2.0.0rc2" # uv project -pip install "runpod>=2.0.0rc2" # pip / venv +uv add "runpod>=2.0.1" # uv project +pip install "runpod>=2.0.1" # pip / venv pip show runpod # confirm the version (2.x) rp update # upgrade the package in place; rp update --version X pins one ``` -The `>=2.0.0rc2` specifier also accepts the 2.0 release candidate, so it resolves to 2.x -whether or not a stable 2.0.0 is published. A plain `pip install runpod` on an index with only -the release candidate installs 1.x, which has no `rp flash`. +The `>=2.0.1` floor keeps an existing 1.x install from satisfying the requirement; 1.x has no +`rp flash`. The CLI is also installed as `runpod`, so `runpod flash deploy` and `rp flash deploy` are the same command. diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 7b6c3b40..c89645dc 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -22,8 +22,7 @@ rules: statement: > Flash ships in the `runpod` package from version 2.0, with the `rp` CLI (`rp flash init`, `dev`, `deploy`, `app`, `env`, `undeploy`). The CLI is - also installed as `runpod`. Install `runpod>=2.0.0rc2`; that specifier - also accepts the 2.0 release candidate. + also installed as `runpod`. Install `runpod>=2.0.1`. status: verified evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" } @@ -176,8 +175,8 @@ rules: - id: flash.torch-family-from-gpu-image statement: > - From `runpod` 2.0.0rc2, builds leave `torch`, `torchvision`, `torchaudio` - and `triton` out of the artifact whether a requirement lists them or a + From `runpod` 2.0.1, builds leave `torch`, `torchvision`, `torchaudio` and + `triton` out of the artifact whether a requirement lists them or a dependency pulls them in (`peft`, `accelerate`, `sentence-transformers`), together with the wheels only they need. GPU workers on the built-in images use the image's `torch 2.9.1+cu128`, and version pins on these @@ -191,7 +190,7 @@ rules: - id: flash.gpu-device-names-pin-devices statement: > - From `runpod` 2.0.0rc2, GPU device names in `gpu=` pin those devices. + From `runpod` 2.0.1, GPU device names in `gpu=` pin those devices. Endpoints select GPUs by pool, so the SDK selects each device's pool and excludes the pool's other devices, reading membership from the live GPU catalog. A pool id allows every device in the pool, including MIG slices. @@ -201,24 +200,24 @@ rules: - { source: live-probe, ref: "gpu=[\"NVIDIA GeForce RTX 4090\", \"NVIDIA L4\", \"NVIDIA RTX A5000\"] stored gpuIds ADA_24,AMPERE_24,-NVIDIA GeForce RTX 3090,-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb; four parallel jobs all reported NVIDIA GeForce RTX 4090", seen: "2026-10-16" } - { source: live-probe, ref: "GET https://api.runpod.io/v2/catalog/gpus lists NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb with pool AMPERE_24", seen: "2026-10-16" } - - id: flash.worker-api-key-for-callers + - id: flash.worker-api-key-every-endpoint statement: > - From `runpod` 2.0.0rc2, deploy and dev add the account API key - (`RUNPOD_API_KEY`) only to resources whose code can call another resource: - a reference to another resource, `.remote()` or `.spawn()`, a `runpod` - client, or code the build cannot follow (a local helper module, a nested - function). Other resources run without the key. + `rp flash deploy` and `rp flash dev` add the account API key + (`RUNPOD_API_KEY`) to every endpoint's template env, including endpoints + whose code never calls another resource. Anyone who can read the + endpoint's template can read the key. status: verified evidence: - - { source: live-probe, ref: "deployed leaf + caller queues: only caller's template had RUNPOD_API_KEY; caller returned {'has_key': True, 'nested': {'has_key': False, 'x': 42}}", seen: "2026-10-16" } - - { source: other, url: "https://github.com/runpod/runpod-python/pull/628", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L315", seen: "2026-10-22", note: "Deploy sets RUNPOD_API_KEY in every template env unless the resource's env already sets it." } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/dev.py#L146", seen: "2026-10-22", note: "Dev sessions add the client's key to every dev endpoint." } + - { source: other, url: "https://github.com/runpod/runpod-python/pull/628", seen: "2026-10-22", note: "The change that limited the key to callers was closed unmerged, as was its replacement #641." } - id: flash.dev-session-stop-signals statement: > - From `runpod` 2.0.0rc2, a dev session deletes its endpoints when it - receives SIGINT or SIGTERM, including a background session started with - SIGINT ignored. A second signal stops waiting for a running entrypoint. - `kill -9` skips the cleanup. + From `runpod` 2.0.1, a dev session deletes its endpoints when it receives + SIGINT or SIGTERM, including a background session started with SIGINT + ignored. A second signal stops waiting for a running entrypoint. `kill -9` + skips the cleanup. status: verified see: [flash.dev-session] evidence: diff --git a/plugins/runpod/skills/runpod-usage/reference/getting-started.md b/plugins/runpod/skills/runpod-usage/reference/getting-started.md index a1ecc3ef..926a5265 100644 --- a/plugins/runpod/skills/runpod-usage/reference/getting-started.md +++ b/plugins/runpod/skills/runpod-usage/reference/getting-started.md @@ -16,7 +16,7 @@ lane's `SKILL.md` (linked). You don't need all of them. | Lane | Quick install | Source / docs | | --- | --- | --- | | **runpodctl** (CLI) | `curl -sSL https://cli.runpod.net \| bash` (or `brew install runpod/runpodctl/runpodctl`) | [source](https://github.com/runpod/runpodctl) · [`runpodctl/SKILL.md`](../../runpodctl/SKILL.md) | -| **flash** (deploy your own code) | `pip install "runpod>=2.0.0rc2"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`flash/SKILL.md`](../../flash/SKILL.md) | +| **flash** (deploy your own code) | `pip install "runpod>=2.0.1"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`flash/SKILL.md`](../../flash/SKILL.md) | | **runpod-mcp** (hosted) | `npx @runpod/mcp-server@latest add` (guided; OAuth) | [source](https://github.com/runpod/runpod-mcp) · [`runpod-mcp/SKILL.md`](../../runpod-mcp/SKILL.md) | | **runpod-mcp** (local stdio) | `claude mcp add runpod -e RUNPOD_API_KEY=... -- npx -y @runpod/mcp-server` | same as above | | **companion CLIs** (`hf`/`gh`/`docker`/`aws`) | per-tool; see the skill | [`companion-clis/SKILL.md`](../../companion-clis/SKILL.md) | @@ -29,7 +29,7 @@ Nothing to install for the **hosted MCP** beyond configuring your client, and th ```bash npx skills add runpod/runpod-plugins-official # the skills — works with any agent curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl) -pip install "runpod>=2.0.0rc2" # Runpod SDK + rp CLI — deploy your own code (flash lane) +pip install "runpod>=2.0.1" # Runpod SDK + rp CLI — deploy your own code (flash lane) npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth ``` diff --git a/plugins/runpod/skills/runpod/SKILL.md b/plugins/runpod/skills/runpod/SKILL.md index 01fcd497..a16d2852 100644 --- a/plugins/runpod/skills/runpod/SKILL.md +++ b/plugins/runpod/skills/runpod/SKILL.md @@ -97,7 +97,7 @@ set up, stop and get a key, in order: **Then:** if a lane already works, use it — but if *only* the MCP is OAuth'd, still get a key before any CLI-only step. Missing a CLI? `curl -sSL https://cli.runpod.net | bash` (runpodctl) · -`pip install "runpod>=2.0.0rc2"` (Runpod SDK + `rp`, flash lane) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: +`pip install "runpod>=2.0.1"` (Runpod SDK + `rp`, flash lane) · `npx @runpod/mcp-server@latest add` (MCP). Full setup: [`runpod-usage/reference/getting-started.md`](../runpod-usage/reference/getting-started.md). ## How to route diff --git a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md index ccea9f92..21e4fe39 100644 --- a/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md +++ b/plugins/runpod/skills/runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md @@ -20,7 +20,7 @@ and cost notes live in the [folder README](README.md); the flash skill is at - Python 3.10–3.14 and the Runpod SDK in the project environment: ```bash - pip install "runpod>=2.0.0rc2" # installs the rp CLI + pip install "runpod>=2.0.1" # installs the rp CLI ``` - A Runpod API key for non-interactive use: ```bash diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index 97258992..c379c631 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -40,7 +40,7 @@ that DC. ## Prerequisites - `RUNPOD_API_KEY` resolvable (runpodctl and the Runpod SDK both read it). -- `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc2"`, which provides `rp`). +- `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.1"`, which provides `rp`). - An SSH key registered **before** creating the pod — Runpod injects registered keys into the pod at startup, so a key added after boot won't work until a restart (`runpodctl ssh list-keys`; see diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index dc206db7..2a144eb6 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -16,7 +16,7 @@ TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the ad volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code, deployed with `rp flash deploy` on a build of `runpod` with the transitive-torch, GPU-pinning, -`vol.path` and API-key fixes, loaded TinyLlama plus a LoRA adapter from a network volume and +and `vol.path` fixes, loaded TinyLlama plus a LoRA adapter from a network volume and generated text (the adapter was an untrained one saved by a helper function, standing in for the pod training step). `peft` and `accelerate` were plain `dependencies` and the worker ran the image's `torch 2.9.1+cu128`. Run it with `rp flash deploy` (flash SKILL gotcha 14 covers dev @@ -52,7 +52,7 @@ A 1.1B LoRA on a 4090 trains in **minutes** — fast enough to iterate the loop spending on a real run. ## Prerequisites -- `RUNPOD_API_KEY` resolvable; `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.0rc2"`, which provides `rp`). +- `RUNPOD_API_KEY` resolvable; `runpodctl` + the Runpod SDK installed (`pip install "runpod>=2.0.1"`, which provides `rp`). - An SSH key registered **before** creating the pod (`runpodctl ssh list-keys`; see `../../runpod-usage/reference/getting-started.md`). - `HF_TOKEN` only if you scale to a **gated** base model (TinyLlama is ungated). From 5a037b7a19402a5f6cb3af8882a27e917e750a39 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 22 Oct 2026 13:05:29 -0400 Subject: [PATCH 11/18] docs(flash): fall back to runpod-flash when the SDK path fails Flash in the Runpod SDK can still hit SDK bugs that runpod-flash handled. The flash skill now says how to tell an SDK failure from a code failure, where to look for a fix (runpod-python CHANGELOG and runpod/apps commit history, runpod-flash history), and how to fall back: install runpod-flash as an isolated uv tool with runpod<2 and deploy the pre-migration code. The migration guide keeps the old endpoints until the new ones pass. New rule flash.legacy-fallback (live-verified side-by-side install) and eval sdk-failure-fallback. --- plugins/runpod/skills/flash/SKILL.md | 15 ++++++++++ .../flash/evals/sdk-failure-fallback.eval.md | 30 +++++++++++++++++++ .../reference/migrate-from-runpod-flash.md | 11 ++++++- .../flash/reference/runpod-flash-legacy.md | 10 +++---- .../skills/runpod-usage/concepts/flash.yaml | 15 ++++++++++ 5 files changed, 75 insertions(+), 6 deletions(-) create mode 100644 plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 149ee9cc..8ae31193 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -44,6 +44,21 @@ Check before writing code. The two APIs do not mix. Write new code on the SDK. For an existing `runpod-flash` project, propose migrating and check its *Known differences* first; don't migrate silently. +**If the SDK path fails, `runpod-flash` is the fallback.** A build, deploy or worker failure +can be an SDK bug rather than your code: a traceback inside +`runpod/apps/`, a worker that crashes before your function runs, or the same code working on +`runpod-flash`. Then: +1. Run `pip show runpod` and upgrade. Check the + [CHANGELOG](https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md) and the + [commit history for `runpod/apps`](https://github.com/runpod/runpod-python/commits/main/runpod/apps) + for a fix or a behavior change. The [runpod-flash history](https://github.com/runpod/flash/commits/main) + shows how the old package handled the same case. +2. Still broken: tell the user, and run that project on `runpod-flash` with the legacy reference + ([reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md)). Install it as an + isolated tool, `uv tool install runpod-flash --with "runpod<2"`, so the project's `runpod` + 2.x stays put. It gets no fixes after October 29, 2026, so move back to the SDK once the fix + ships. + **Worked examples first for anything multi-step.** Flash appears in end-to-end paths: [03 variant B (whisper endpoint)](../runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md), [07 (network-volume handoff)](../runpod/golden-paths/07-network-volume-handoff.md) and diff --git a/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md b/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md new file mode 100644 index 00000000..517bef47 --- /dev/null +++ b/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md @@ -0,0 +1,30 @@ +# Fall back to runpod-flash when the SDK path fails + +## Prompt + +I moved my project to `runpod` 2.x with `rp flash`. `rp flash deploy` succeeds, but the GPU +worker crash-loops before my function runs, with a traceback inside `runpod/apps/`. The same +code deployed fine with `runpod-flash` last week. I need this endpoint up today. + +## Expected behavior + +The agent should: + +1. Treat it as a possible SDK bug, not only a code bug: check the installed version with + `pip show runpod`, upgrade, and look at the runpod-python CHANGELOG and the commit history + for `runpod/apps` for a fix +2. If no fix is out, offer `runpod-flash` as the fallback for this project, installed as an + isolated tool with `uv tool install runpod-flash --with "runpod<2"` +3. Deploy with `flash deploy` from the pre-migration code (the previous commit), and keep any + working `runpod-flash` endpoints running +4. Tell the user `runpod-flash` gets no fixes after October 29, 2026, and to move back to the + SDK once the fix ships + +## Assertions + +- Checks the installed `runpod` version or upgrades before falling back +- Points to the runpod-python CHANGELOG or commit history +- Installs `runpod-flash` isolated from the project's `runpod` 2.x (`uv tool install ... --with "runpod<2"` or a separate environment) +- Does NOT install `runpod-flash` into the project environment in a way that downgrades `runpod` +- Does NOT delete working `runpod-flash` endpoints while the SDK version is broken +- Mentions the October 29, 2026 no-fixes date diff --git a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md index aaf8bd8a..fe83ee11 100644 --- a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md +++ b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md @@ -12,6 +12,14 @@ not what is running. Run this as a plan with the user, not a silent rewrite: inventory, check *Known differences*, convert, smoke-test with `rp flash dev --once`, deploy, then remove the old endpoints. +Keep a way back until the SDK version works: convert on a branch, and leave the +`runpod-flash` endpoints running until the new ones pass. If the SDK fails on something the old +package handled, roll back to the previous commit and keep deploying with `flash`, installed with +`uv tool install runpod-flash --with "runpod<2"`. Check the runpod-python +[CHANGELOG](https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md) and +[commit history](https://github.com/runpod/runpod-python/commits/main/runpod/apps) for a fix +before retrying. + ## 1. Inventory ```bash @@ -146,7 +154,8 @@ rp flash deploy # creates the new endpoints (default enviro rp flash app list # confirm ``` -Then remove the endpoints deployed with `runpod-flash`, scoped to that app: +Then, once the new endpoints pass, remove the ones deployed with `runpod-flash`, scoped to +that app: `flash app delete OLD_APP` (with the old CLI still installed), or `runpodctl serverless delete ENDPOINT_ID` for each old endpoint id. Update CI to call `rp flash` instead of `flash`, and update any client that called the old endpoint ids. diff --git a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md index 1647f198..56936e67 100644 --- a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md +++ b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md @@ -4,15 +4,15 @@ concepts: [flash, serverless-handler, job, container-disk] # Legacy `runpod-flash` reference -Use this only to maintain a project that still imports `runpod_flash`. `runpod-flash` is -deprecated on October 29, 2026 and gets no fixes after that date; propose -[migrating](migrate-from-runpod-flash.md). New code goes on the Runpod SDK -([../SKILL.md](../SKILL.md)). `flash --help` is authoritative for the installed version. +Use this to maintain a project that still imports `runpod_flash`, or as the fallback when the +Runpod SDK path fails on an SDK bug ([../SKILL.md](../SKILL.md), *Which SDK is this project +on?*). `runpod-flash` is deprecated on October 29, 2026 and gets no fixes after that date; +propose [migrating](migrate-from-runpod-flash.md). New code goes on the Runpod SDK. `flash --help` is authoritative for the installed version. ## Setup and CLI ```bash -uv tool install runpod-flash # Python 3.10-3.13; or: pip install runpod-flash +uv tool install runpod-flash --with "runpod<2" # Python 3.10-3.13; isolated, keeps runpod 1.x flash login # saves a key to ~/.runpod/config.toml (--no-open for headless) flash init my-project # scaffold (writes AGENTS.md + CLAUDE.md) flash dev # local server on :8888; functions run on remote workers diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index c89645dc..dc4ffbc2 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -38,6 +38,21 @@ rules: - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } - { source: skill, path: plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md } + - id: flash.legacy-fallback + statement: > + `runpod-flash` still installs alongside the Runpod SDK: its CLI is + `flash`, separate from `rp`, and `runpod_flash` imports with `runpod` 2.x + installed. Install it as an isolated tool with `uv tool install + runpod-flash --with "runpod<2"` so it keeps its own `runpod` 1.x and the + project keeps 2.x. Use it as a fallback when an SDK build, deploy or + worker step fails in the SDK rather than in your code. It gets no fixes + after October 29, 2026. + status: verified + see: [flash.runpod-flash-deprecated] + evidence: + - { source: live-probe, ref: "one venv with runpod 2.0.0rc2 + runpod-flash 1.21.0: import runpod_flash ok, flash --help and rp --version both run; uv tool install runpod-flash --with 'runpod<2' resolved runpod 1.12.0 + runpod-flash 1.21.0 and flash --help exited 0", seen: "2026-10-22" } + - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } + - id: flash.not-flashboot statement: > Flash is the code-first SDK and CLI. FlashBoot is a separate endpoint From 6411449fedc63cf327920a007d12722d28f482f7 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 22 Oct 2026 13:59:59 -0400 Subject: [PATCH 12/18] docs(flash): cite SDK source instead of PRs; drop build history Concept evidence now points at the code on runpod-python main rather than pull requests, the README install pin moves to runpod>=2.0.1, and golden path 08's status no longer lists the fixes the test build carried. --- README.md | 2 +- .../runpod/skills/runpod-usage/concepts/flash.yaml | 5 ++--- .../runpod/golden-paths/08-finetune-to-serverless.md | 11 +++++------ 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index e9de6a38..fe9fa4f7 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Gemini, opencode, +more). The first line is all you need; drop any others you wo ```bash npx skills add runpod/runpod-plugins-official # the skills — any agent (essential) curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl) -pip install "runpod>=2.0.0rc1" # Runpod SDK + rp CLI — deploy your own code (optional; flash lane) +pip install "runpod>=2.0.1" # Runpod SDK + rp CLI — deploy your own code (optional; flash lane) npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth (optional) ``` diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index dc4ffbc2..6dbe9d7b 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -201,7 +201,7 @@ rules: status: verified evidence: - { source: live-probe, ref: "rp flash deploy of a gpu queue function with dependencies=[\"peft\"]: artifact 80.7 MB with no torch (645.6 MB with torch 2.14.1 before the fix); the worker returned torch 2.9.1+cu128 from site-packages with cuda True", seen: "2026-10-16" } - - { source: other, url: "https://github.com/runpod/runpod-python/pull/623", seen: "2026-10-16", note: Removes transitive torch-family wheels after vendoring and scopes the cold-start install list to CPU resources. } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/build.py#L322", seen: "2026-10-22", note: Removes transitive torch-family wheels after vendoring and scopes the cold-start install list to CPU resources. } - id: flash.gpu-device-names-pin-devices statement: > @@ -225,7 +225,6 @@ rules: evidence: - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L315", seen: "2026-10-22", note: "Deploy sets RUNPOD_API_KEY in every template env unless the resource's env already sets it." } - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/dev.py#L146", seen: "2026-10-22", note: "Dev sessions add the client's key to every dev endpoint." } - - { source: other, url: "https://github.com/runpod/runpod-python/pull/628", seen: "2026-10-22", note: "The change that limited the key to callers was closed unmerged, as was its replacement #641." } - id: flash.dev-session-stop-signals statement: > @@ -237,7 +236,7 @@ rules: see: [flash.dev-session] evidence: - { source: live-probe, ref: "rp flash dev main.py < /dev/null & from a script, then kill -TERM and separately kill -INT: both logged 'cleaning up 1 deleted' and left no endpoints", seen: "2026-10-16" } - - { source: other, url: "https://github.com/runpod/runpod-python/pull/625", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L498", seen: "2026-10-22", note: rp flash dev handles SIGINT and SIGTERM and ignores further signals during teardown. } - id: flash.legacy-dev-ships-body-only statement: > diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 2a144eb6..8ccfb861 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -15,12 +15,11 @@ GPU, dataset) — nothing structural changes. TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code, -deployed with `rp flash deploy` on a build of `runpod` with the transitive-torch, GPU-pinning, -and `vol.path` fixes, loaded TinyLlama plus a LoRA adapter from a network volume and -generated text (the adapter was an untrained one saved by a helper function, standing in for the -pod training step). `peft` and `accelerate` were plain `dependencies` and the worker ran the -image's `torch 2.9.1+cu128`. Run it with `rp flash deploy` (flash SKILL gotcha 14 covers dev -sessions on older runtime images). +deployed with `rp flash deploy` on a pre-release `runpod` 2.0 build, loaded TinyLlama plus a +LoRA adapter from a network volume and generated text (the adapter was an untrained one saved by +a helper function, standing in for the pod training step). `peft` and `accelerate` were plain +`dependencies` and the worker ran the image's `torch 2.9.1+cu128`. Run it with +`rp flash deploy` (flash SKILL gotcha 14 covers dev sessions on older runtime images). The `runpod-flash` live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. From 5925d7d16a235d7fab629187a8d1a9dfdb299871 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 22 Oct 2026 15:24:35 -0400 Subject: [PATCH 13/18] docs(flash): address review round 1 - Deploy rejects queue/API names under 3 characters: rename the pattern's class, use a real name in the migration table, add gotcha 16 and rule flash.resource-name-min-length. - Migration guide: dev ships one module but deploy packages the project; keep `flash` installed for cleanup and rollback; -y on pip uninstall; split decorator options by kind (max_concurrency is queue-only, tasks take no scaling options). - Fallback from the SDK lives in the legacy reference, in its own venv with the project's requirements and runpod<2 (verified); SKILL.md keeps a pointer. - Verification status says what ran: examples on 2.0.0rc1, golden path 08 marked ported (its SDK run used VolumeCache), 07 notes the literal path. - Examples: float32 tensor, parallel CPU stage. - Python 3.10-3.14 applies to dev and tasks; deploy falls back to 3.12. - Default ignores drop tests and virtualenvs; stream/with_options are queue-only; the JSON error text is dev-only; app delete needs --yes. - Deprecated now, no fixes after October 29, 2026; two-sentence skill description; gotcha 17 for the API key on every endpoint. --- plugins/runpod/skills/flash/SKILL.md | 54 +++++++++--------- .../evals/connect-existing-endpoint.eval.md | 6 +- .../flash/evals/sdk-failure-fallback.eval.md | 4 +- .../runpod/skills/flash/examples/README.md | 4 ++ .../api_model_service.cpython-314.pyc | Bin 0 -> 2796 bytes .../__pycache__/call_deployed.cpython-314.pyc | Bin 0 -> 1416 bytes .../cpu_gpu_pipeline.cpython-314.pyc | Bin 0 -> 3078 bytes .../__pycache__/queue_gpu.cpython-314.pyc | Bin 0 -> 1832 bytes .../train_then_serve.cpython-314.pyc | Bin 0 -> 3145 bytes .../skills/flash/examples/cpu_gpu_pipeline.py | 3 +- .../runpod/skills/flash/examples/queue_gpu.py | 2 +- plugins/runpod/skills/flash/reference/api.md | 6 +- .../reference/migrate-from-runpod-flash.md | 26 ++++----- .../runpod/skills/flash/reference/patterns.md | 2 +- .../flash/reference/runpod-flash-legacy.md | 34 +++++++++-- .../skills/flash/reference/setup-and-cli.md | 5 +- .../skills/runpod-usage/concepts/flash.yaml | 40 +++++++++---- .../golden-paths/07-network-volume-handoff.md | 5 +- .../golden-paths/08-finetune-to-serverless.md | 6 +- .../skills/runpod/golden-paths/README.md | 4 +- 20 files changed, 126 insertions(+), 75 deletions(-) create mode 100644 plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc create mode 100644 plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc create mode 100644 plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc create mode 100644 plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc create mode 100644 plugins/runpod/skills/flash/examples/__pycache__/train_then_serve.cpython-314.pyc diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 8ae31193..7b21385d 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -3,10 +3,10 @@ name: flash description: >- Code-first serverless with the Runpod Python SDK (`runpod` 2.0): define a `runpod.App` with `@app.queue` / `@app.api` / `@app.task`, iterate on remote - GPUs/CPUs with `rp flash dev`, ship with `rp flash deploy`. Also covers - maintaining and migrating legacy `runpod-flash` projects (`from runpod_flash - import Endpoint`). Use runpod-build-template for explicit Dockerfile/template - project authoring, and runpodctl/runpod-mcp for infrastructure management. + GPUs/CPUs with `rp flash dev`, ship with `rp flash deploy`, and maintain or + migrate legacy `runpod-flash` projects (`from runpod_flash import Endpoint`). + Use runpod-build-template for explicit Dockerfile/template project authoring, + and runpodctl/runpod-mcp for infrastructure management. user-invocable: true metadata: author: runpod @@ -22,9 +22,9 @@ For an explicit Dockerfile/template project request, use deliverable with Flash unless the user chooses its code-first workflow. Write Python locally, run it on remote Runpod GPUs/CPUs with `rp flash dev`, then -`rp flash deploy` to ship. Flash is now part of the `runpod` package (2.0): one SDK, one -`rp` CLI, one login. The standalone `runpod-flash` package is deprecated on -**October 29, 2026** and gets no fixes after that date. +`rp flash deploy` to ship. Flash is part of the `runpod` package (2.0): one SDK, one +`rp` CLI, one login. The standalone `runpod-flash` package is deprecated and gets no fixes +after **October 29, 2026**. **`rp --help` and `rp flash --help` are authoritative for the command surface.** This skill is the mental model, the decision rules, and the gotchas that help output does not @@ -44,20 +44,10 @@ Check before writing code. The two APIs do not mix. Write new code on the SDK. For an existing `runpod-flash` project, propose migrating and check its *Known differences* first; don't migrate silently. -**If the SDK path fails, `runpod-flash` is the fallback.** A build, deploy or worker failure -can be an SDK bug rather than your code: a traceback inside +**If the SDK path fails on an SDK bug, `runpod-flash` is the fallback**: a traceback inside `runpod/apps/`, a worker that crashes before your function runs, or the same code working on -`runpod-flash`. Then: -1. Run `pip show runpod` and upgrade. Check the - [CHANGELOG](https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md) and the - [commit history for `runpod/apps`](https://github.com/runpod/runpod-python/commits/main/runpod/apps) - for a fix or a behavior change. The [runpod-flash history](https://github.com/runpod/flash/commits/main) - shows how the old package handled the same case. -2. Still broken: tell the user, and run that project on `runpod-flash` with the legacy reference - ([reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md)). Install it as an - isolated tool, `uv tool install runpod-flash --with "runpod<2"`, so the project's `runpod` - 2.x stays put. It gets no fixes after October 29, 2026, so move back to the SDK once the fix - ships. +`runpod-flash`. Upgrade `runpod` first; the steps to check for a fix and fall back are in +[reference/runpod-flash-legacy.md](reference/runpod-flash-legacy.md#fallback-from-the-sdk). **Worked examples first for anything multi-step.** Flash appears in end-to-end paths: [03 variant B (whisper endpoint)](../runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md), @@ -103,8 +93,8 @@ def main(): | `@app.task(...)` on a function | One ephemeral **Pod per call**, terminated when it returns | Training, fine-tuning, long jobs | Call styles on a queue or task handle: `fn.remote(...)` (blocks), `await fn.remote.aio(...)`, -`fn.spawn(...)` (returns a `Job`), `fn.stream(...)` (generator functions), `fn.local(...)` -(runs on your machine). API classes are called by route: `MyApi.post("/path", body)`. +`fn.spawn(...)` (returns a `Job`, or a `TaskJob` for a task), `fn.stream(...)` (queue generator +functions only), `fn.local(...)` (runs on your machine). API classes are called by route: `MyApi.post("/path", body)`. ## Dev vs deploy @@ -169,7 +159,8 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. ## Gotchas 1. **Queue arguments and results must be JSON-serializable.** Queue jobs are plain JSON in dev - and deploy alike; a non-JSON argument fails with `arguments must be json-serializable`. Pass + and deploy alike. A non-JSON argument fails in dev with `arguments must be json-serializable` + and against a deployed endpoint with Python's own JSON `TypeError`. Pass URLs or base64, not objects. Whole-number floats can arrive as integers (`1.0` → `1`), so cast numeric inputs explicitly (`torch.tensor(x, dtype=torch.float32)`). 2. **One file per app in dev.** A dev session ships only the module's source, so a sibling @@ -191,9 +182,9 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. keeps its Pod running after your script exits. 10. **Native CUDA libraries go in `dependencies` too**, e.g. faster-whisper needs `nvidia-cublas-cu12` + `nvidia-cudnn-cu12` or it silently falls back to CPU. -11. **Local Python must be 3.10–3.14.** Dev sessions and tasks ship code pickled by your - interpreter, so the worker matches your local version. `rp flash deploy --python-version` - sets it for deploys. +11. **Dev sessions and tasks need local Python 3.10–3.14.** They ship code pickled by your + interpreter, so the worker matches your local version. Deploy falls back to 3.12 with a + warning on other versions; `rp flash deploy --python-version` sets it explicitly. 12. **A set `RUNPOD_API_KEY` wins over `rp login`.** A stale exported key causes 401s while the saved login is fine. Check with `runpodctl user`, then `unset RUNPOD_API_KEY` or export a good key. @@ -209,8 +200,15 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. `AttributeError` on a `runpod.` name in a dev worker traceback means the image is behind your SDK; `rp flash deploy` runs your installed version. 15. **Teardown by app and environment:** `rp flash undeploy --app APP --env default -y` - removes one environment's endpoints; `rp flash app delete APP` removes every environment. - Then delete volumes you no longer need. + removes one environment's endpoints; `rp flash app delete APP --yes` removes every + environment (without `--yes` it prompts, and aborts with no terminal). Then delete volumes + you no longer need. +16. **Queue and API names need 3+ characters to deploy.** The name defaults to the function or + class name, and dev sessions don't check it, so `class SD` passes `rp flash dev` and fails + `rp flash deploy`. Rename it or pass `name=`. +17. **Every endpoint gets your account API key.** Deploy and dev put `RUNPOD_API_KEY` in each + endpoint's template env, whether or not its code calls another resource, so anyone who can + read the template can read the key. ## Resources diff --git a/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md b/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md index f27d6ef7..c61c31b2 100644 --- a/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md +++ b/plugins/runpod/skills/flash/evals/connect-existing-endpoint.eval.md @@ -18,8 +18,8 @@ The agent should: ## Assertions - Uses `runpod.Endpoint("abc123xyz")` -- Passes `{"prompt": "hello"}` as the input (the client wraps it in `{"input": ...}`; the agent - does NOT double-wrap it as `{"input": {"prompt": ...}}`) -- Sets a timeout above 60 seconds or uses `run()` + `output(timeout=...)` +- Passes `{"prompt": "hello"}` as the input (the client wraps it in `{"input": ...}`; a dict + that already has an `input` key is sent unchanged) +- Sets an explicit timeout that covers a cold start, or uses `run()` + `output(timeout=...)` - Does NOT create an `App` or decorate a function for this - Does NOT import from `runpod_flash` diff --git a/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md b/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md index 517bef47..f9f2de4f 100644 --- a/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md +++ b/plugins/runpod/skills/flash/evals/sdk-failure-fallback.eval.md @@ -14,7 +14,7 @@ The agent should: `pip show runpod`, upgrade, and look at the runpod-python CHANGELOG and the commit history for `runpod/apps` for a fix 2. If no fix is out, offer `runpod-flash` as the fallback for this project, installed as an - isolated tool with `uv tool install runpod-flash --with "runpod<2"` + environment of its own with the project's requirements and `runpod<2` 3. Deploy with `flash deploy` from the pre-migration code (the previous commit), and keep any working `runpod-flash` endpoints running 4. Tell the user `runpod-flash` gets no fixes after October 29, 2026, and to move back to the @@ -24,7 +24,7 @@ The agent should: - Checks the installed `runpod` version or upgrades before falling back - Points to the runpod-python CHANGELOG or commit history -- Installs `runpod-flash` isolated from the project's `runpod` 2.x (`uv tool install ... --with "runpod<2"` or a separate environment) +- Installs `runpod-flash` in a separate environment with `runpod<2` and the project's requirements, not into the project's `runpod` 2.x environment - Does NOT install `runpod-flash` into the project environment in a way that downgrades `runpod` - Does NOT delete working `runpod-flash` endpoints while the SDK version is broken - Mentions the October 29, 2026 no-fixes date diff --git a/plugins/runpod/skills/flash/examples/README.md b/plugins/runpod/skills/flash/examples/README.md index 419b0c02..48696536 100644 --- a/plugins/runpod/skills/flash/examples/README.md +++ b/plugins/runpod/skills/flash/examples/README.md @@ -9,6 +9,10 @@ rp flash deploy queue_gpu.py # persistent endpoints rp flash deploy queue_gpu.py --build-only # offline check: discovery + dependency resolution ``` +Live checks ran on `runpod` 2.0.0rc1, before the GPU-pinning and build changes in 2.0.1; +`queue_gpu.py` (float dtype) and `cpu_gpu_pipeline.py` (parallel CPU stage) changed slightly +since. Re-run them on 2.0.1. + | File | Shows | Live check (`runpod` 2.0.0rc1, `--once`) | |---|---|---| | [`queue_gpu.py`](queue_gpu.py) | `@app.queue` on a GPU list with `workers=(0, 5)` | ✅ 2026-10-16 | diff --git a/plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..3b50d6cd078feb0f4f770b25862a3f9c6fc09690 GIT binary patch literal 2796 zcmZuz&2Jn<7O(2=>FN2%*vUGv2??&4V0IMSvjk_eE1ca}AP7PVrpXQwCT_K7%ASsU zx|^z+*v?@!c#+~ryBi;ULCPNYW=>rB69xtp)Z&1|hm@O*CBkyrSJl(j#Bxb~u8;Tq ze(&{MC>_fqSZ5aF+dmlyeJ>B`%Jw$<4`I_rHWFw7+4xN?@GU$C+=tgWWNXtnxfS{? zeJ*#H10h?t^R{7Urm$^Z?&&y*Kr%awY-?IeB&K_UA_JC+M`+0nqXV9-jh{#5g1ZW) zXNe!W^~puocY_+OlWR9_%n?r69j``-Xt+YXbK;U_Sf@S-fr~_xk=2l`P&Sp%6Id)F zOTNn+q)zV;H}V|C?DX_ZMeAg8Qc50Xj{I=_NdA2ODr*H%SfBC&PY~Gi#Pn)|2H;F$ zop9kY!6(QfT?!c`z$x4n%86j^(vnx3AUw>A1~}|R)b|1^1=cz70@9?-kb&(Q3ppo@ ze%PX1@G0`9)Gnjolyff($a>f!HP`p^ENls>x&fQz2K=X_1~Kzj636OP21Dx;>V*dN zDD(oMQmLe9lZ1C5Z!*DC`RjKrLD$6OG7VyL_O)5(;~W>UP{W}mIRofU;D5=Gkr zWai@R4M@%#5LmA{lqwS_1ZM4DgiSkXEp!b>cobE!bj#ME-Hgk<;HxO;H3tNi0k!61 zsiur$D}}?|fwTyexdN&k8_SWnSk~f<0&r{s5}*;Tmvtt!GTB+NA;p2m;M8AIS6mtz zC-I*nzqRZIypmwI!dE=s=M~i&6}sj&BcJlhQT-7cvJK;6FYYOMk_qG^K-%a>tGs8T zlV>|;r*@V;f9I>AH#Uub=l02?bV>TWPk_weHmafp?N1*gREOI$ z=pEd^Hhve+qlz2`+GEKd8;fCm9WrFpy_$d!O$kI-#K#;*VO?H3jtY0I&7^~I(Q(`$ z2nE=8oOmb=p#SzX@bMuazqZj6q-8GdN%Eg*^^cu|huT4<;2_87KV+RZU^9kBA^s|w zOXGsObX5V;6H<9^Rq>pR_|UTP!caOUZS8W;QUqwJY70-v+eW`OnU_nths&ARj9A!= zL|kgRYfeB{9TBe3fR`~l2gYJUq9`t>J(7f7|rBrRt{s z&ETnReS7v3vol)xdgKqCftNbgOaJ2rXxO;&SNvPH(x9#{8c#KrOo>XkENGH&8)m^l z0ssdvH${ubTDbCG=pYphmlYXLA}|xJaVhmbIX?#+TxLOeKRB{|zB4e|u||6iWwID_ z$@lB!W3lBpf!mZI$~*G=(Sk|mNO*nNa{b=Rfa5HAj0-u-gAkt0q|;(UekR!OKozuS z$ORjREjusCq`W){sD%)`n&S$=yhS<6LHu|hu4~ly9S0mb0|F|zt137qKgxiYA&WPLNT8#2Y6|UD;M$`2IHUpf` z!ykVeNE`iNs_l7Xbmd eX*cn%I-f_H^$3rAfkyzv_>c1GnV;oOx$%EF?t`!Z literal 0 HcmV?d00001 diff --git a/plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..8110bb152d12c8fefbd729d4ee8221b1bbe5b983 GIT binary patch literal 1416 zcmZWpO>f&q5M5FdNl}(1yJ-s40klrhHmYo?K<&CQiq>fyq(%UnWpY&*np{hC$t6oJ zRm&&&8<~xmjE(~W-m36Kj6TBpt}-T3(x}M4u|i(c{96EZMLWaw(jlH z{qqa}@Ayl)Fi39T1nio zN}_gm8ppa&^!!8@Gowl~+QKweSM`EE(-2b4i7o`~-ZWmTOy~4s$|B`v`SS!|f5}dV zJG?*5U^6ZM&zT>&o7Ym}1e|%BZ3r(9^ip~vDXYTH^OMo9I%j^&M*rA!U1+1g^}EQ1 zA&UGsv`_^1Lf?ZX?n13g+CGMsZzK2s`!EdPo@+*JM(RL}kd5qWT~#4lK>}rxMk-kk zx~lppieit6zM3HOitv{h#mHy{F@wEgi4QH_6ZbxnR}r=Y-@yd#cA;Z0K?hrIY&*CGeauXeHiFMLpF!KP zA@&KhLSzzDSGNykW^9=`r#vuNP!(y$3<8?PriZEml{%&yqo_3=eR(4Z?ralh#5-*>YW7#}^+A z)_dMi`mC4XFza~lXz#@8@1O4f=3at@D<-vmwm$fRNflOm3dgiL^ZePxgV#^rEUpZt z)l2X-D_WZ4tUf-)gXVCy)+_v*nf)hU>Q~MxgN0%K)7}qnv-y+z{n}~m{NUoFH+QRl WesxkCs%w8|zj>bn*-{8uKmP$3ep|o* literal 0 HcmV?d00001 diff --git a/plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..d6b349595f0e26aad141869aec4bca0ed4154846 GIT binary patch literal 3078 zcma)8U2Gf25#GHY{v9RDc3{btAfIH(F{>X+wPeex8bGD&xG<{9XPZp}MZ}3*NvDu^ zm%XDUD=okl1!4mMtN>1{pe~{o0V)Ct;730;%}XBgK!J_bZ1Z3sees(d1t|PdboP#< zDT4{Rzzv79v$L}^-^^}bb5jJt_}4sJUX&2}n*XT|xx(y!4om@Q$VGEV!#~9?egn^d z_KTwnX~GEhrLOpfI1_$R10hY+q(Q97=LLUG3$!9F*ov|(BNb&={*IL)tw{^l)bK zJF2ZyJ)0$2bwM`=th_rIiKsBBtuADBXGzVFn`+w5rxxL}4Vz>QlMLFo)PVuZOp{3D z`VvtSGdKLL)O42UrlV48#ZgxbcL~g!8QuAj>Vn#{lH)8M#ToadlpI(@S8A=1n+e#3B4VE>Tx3 z`X-@{s=FXglflR|3sv1;3&Vj%(*q*|V=Ul17!#NXLKe*so0u75rVZkxd9o`BM*-kP z`1UD~0!pGeVeEGZWkA+}uHq%E;aQmR$80722=$++$^aZfokmSOTZ3A?Zap6tAgSUL zZj(tov@Af0o2O<>WKz~}Tqb3Vw9Dj$tfjj#%mND*&FQYgeeY&tkmp6ibm9;r+see9 zH;ru8iF@gdlT|%uXNePUkYg52rF7G@TnK0?MGt|s7WkZ(ffUexkc9i6a&qtb{hs$< z-{b5@Q`;|XlaJfK=o$5<_CXbSapVjiJceEz{{l?GFF9o6B%a~Y7e*3Q3vQ-bu5}t! z*xN!2RaUd&wS%hE|D9<%_@Do)T86j_Pq<(@7Np!i7M{{w{Swu4gatCx zvQ3=@bjQ7A6B_07ftz|ZPas3C#WNEVDR+Vic7_R=1&7->4b{a9nMG_$Y-**6<4_J0 z9$;r2NCABSxn*&&^)pWtL;jjXjk()-EKv|0!BNvmI>@`b37%?4-vY=P>XMf0~9Q0n= z3N|2$C*g=6Kq&toek1F&DPvLE2|J#BD7D2mzl&L2~6aa!;rsT`0#tI zl+z89j)A-rKIbbS1@yS7WxZ?t+D2C?+OgqmPHs9|Q>D(~2Z@J=A13~GxO6sAQl<)1 zUrFKH{rBY$qaQ>+dVRO|+)nSg2X47{qTKf5e@L(VJ3jK@>h@~6d91j))wOKBmiIiG;%7=Q3tA$rf&0}AliI*co#o*@fyW_h``;OASG5JX8ECvgSQe^1MfuXW8 zT)ei0-@SPE%AG44_`}c#p+80*MNbv46{bqcFn8hyMYnuxr08yme^`5O?f1Vbx;w{4 z9!aCmz6qj37oM@m&@-kCKV!#69Nri|Ioz7)lRg>jObkhXKhmEVl|GHMCB`Gr7I=qw z!ux(K%z~*@#!9DB5NPjVz{E-0W}*&6ek)J%ep382wPdZdo=w5~k>0W`o;@DYBJw(c zAL-CU&dTJo`Ph`zJ@+vW8WS4+zTK&(`P!tpX5`Bs2;#J-4x(LK)XHP=)&FL8ywZw21)xkmq<$L>Xu5Zc9UJGhN+ ObdFCQ|Aup(d;bH(^>a%A literal 0 HcmV?d00001 diff --git a/plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..d4d73a16baef25c8d157dc2e2dae6bd9ce0375a4 GIT binary patch literal 1832 zcmZ`(&2Jk;6rb5IyN>hKs7(TaOcGIH(bxu3Q7Q_FsA&00%t{21(kz?x#MySeJKdda zYfc7LYI{Hp5{Rnef`kOzdTi;T{S(p#snx)t5=baF2P1Li&H5v0RYuy`_ujmD^XC2D zZ(iu@%^(=Rt%y@+RD}MJN@o~tp?M92Ix-PQ%gDrUV~$ValGbtC`A;BInZYfct0&dc zo}IOhlt*TI0h@bFHIGg06*yB4Qy)gAF^s%{nU(=@9nQ$Mm0dqGHH^AGDHBhjT*iJM z4xJ;%OCOTe*mt-a_+*8yxqg}0KCxpSM2_vbeubQgX-oJFImhKOUEPLG97d zNzf!NqhaJmocg@Gow$%TV`LKYn2zG=WF|A`*GRBR_!=di^e)TmM#PPX?KqTid&Q#% z;ks~T$nw1OJw(n0;c19YJjx@vHN!lcuLfb2MCV-YtU;CdIZA!GRwXMn62**pwR~p& zvsC+Nq5E-_)pDv(!6H)akRn^A4C+XI$EA@YL2m<(WMfEJn{UBXM~i4#IeY=3GT6q@ z2Y3yecnMZoV@c^q2cPOl&<@H{x4jAqK&);_KVh7;y_iPA2q}+4ACl5!F@(PA1vVF2 z*>!jh3qxMf=8?>3QoP8#xB^HO06-R$3(;xU^P)lmd4Zm{tIVTOA?b`Z?jmbhw(ker z=G3yneh4!FUo;6*9sP@R;J|h=e|_w`ec!y>NaP=mwQNoBLPqaUB}U%Lok&Jl&DB34=*^?H;51#=|q7og4sg9Zk(gz==d#_|K zXV({hJo^378~ZP1H`BBAxd++7^+Vqt`Rd61q3OFr(?8jpL&fXQ-59*R@8-T+!?(w7 zj@^28qvynib|RDzJMeXfY(aT`lfzU;J5Bo}*x+5$7I3H33@z!MIE!5X#(@kbezegK ze~eHCt_;+G)eaOu5*6BB|Em?G@kUajN=qA=orf}wgVHvj*|w0Lg#}q0$&hNmsnA&n z4;^qq4dNCd+O^c=Qx;Bv?+AQR9;Q0_Gc$N4e>s0`d^3})zxlhi=hD=LcOLZgU5dXv za&_p}p0N#W;;OT$P2B4lOX?*VI8%JJD3teRgg$quSS;q0a4!VglK0Q0Mar_ufn!;a zQ9?hV&M_ubc-@4aaITf_fEU0MZNVcL){?h5N$qEpoYF!+{aQ6B#~yt>EI??5s4!fO%S%oLxAKVd2@gk=}#zv0<}sTau z?9AD7n{#I7d^0nBu`q%*w(8v+krDca{rF9469-=dQAH|pQ5vcE73|_SanhIi+AvNa zRXC4Z3|D+pOon=dZBi9g=|WpajjHl-qy~;7W5B@Y+ZwJ6ma#)DgR0W841ryk*>$Xf z5YxaRR*~bVqZ3l`B#OthI}pHYR4bXpGKof#R&JKKMcpKhPH*dsBvf}QhD-9)Duc$Y zieAx)Zsu&OWV-X=Fo8yGk~cJ`NOJmZ;xcy`<|MdtY#X|hIArs^XeOhXrscwp zW->Gi-Ui`wPC!>h|3Xo`g!g&>!83b&{Mc>%AAknpSu=vQ0DL^_^ALZvRfNdKDV${7 zr_FEdy`Xr3D2!5mJr@&i2fo7G@RS z5+Z-5OZ${RpB+re$zwjZJNojIC>2Z|@A7@BM=mEvd~SE_%76TGghJ>9QiG8BXjj9h zlH)$r9erz60ULILo@)QghM)Y<6K1hoVg=+$*-B1hCmnFBl+_Er`z>oHw!X-<5=lU<8ui=IrOT1`(dO)v11lBsKyGPw0*%XUj;PrUKcRSzJ6YGoau%;EY4 z-IL0?W_qDbom!a!)pDM)s%6*KHJbAzR)>^rh9{L=y-be+8CNJf?v(9;hu`w>h9{S; zoNjm`OyuE=CmELGIBY+s7z#t95vDBjDx}juv%s9cKvzZINCS_R*mk5Asb6hOe6q9^ zX(}_-#fM60Yw?a;$9F>T6CK=sx%Tq*_1g8uV&hgbIC4w-U|p^VJ_diXEvx_x{Qj{@dVC{nl@W8;c)~eKPT2U~;!V zzSAFX_Rnld50%JvtQLE(|Dh7~iO9B6Q{E5nhfv?>UKquW)%A~zUmLqa&+ZI8+Z>92 z{`}qNzPj|~rLV4hd8K*o>b=Odd(t(^uEotz)~=k(&YhV%%ODbd{sSUAU~!q}|Nldz zD)zvo2yiJ;1{z*~Z>mF2!4GAi5Em%B*gZK9@DGqlS*5k`lJ5iuJKjw&!4z~1@dS=| ztpDKZ9{>CJ_15@VWuO1^^s?=02P=eF037*TAgXA^zXt&MRW#dEJw6)rXkee$i+w$E zc*-2+p%)ur5d?lfTSpU9CIr?D417Ht7VOki>i`DZ*EF36CV|gkdR6po07VDuLk;D@ z;6yWW>a*8BKlAC&9z^DTFW=4oG4%(txp3`4cYCv7(`lXj{|qe1zauyd`F)hI(MkvH zRahw>u}`8Eb~t!D%pTUXgYGu&@gbz(Mv0-b9drdxou?hhq+YD`S6!7a9zm!8bqY5# zN_C!ha7PDVU4xqnuU9>HsnCw;#m!KD`t1BdoV-9b#rd=OwqWeVHu3n%a$M#G&Mxn= ziC;82|oGe_~;<9hYmdlY<3=<8_-qJ-(yF%S86NormIVjq|nys zdj7sN{z&QH?yL3HZ#SH#GFe?@l2__lL#)4XFFbxcKY(TdsVvP3%1xp7<5$<~&h`w|4mpAL< z&A`d3xEDZz{Lb*(!&~|4a1$Tu=J+k$S>PQoR=UaglDpMHS b<0N2|pzPw|9Xz}*$asQ@M*hJl9@@VF;+AUs literal 0 HcmV?d00001 diff --git a/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py index e177e00b..076fe50b 100644 --- a/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py +++ b/plugins/runpod/skills/flash/examples/cpu_gpu_pipeline.py @@ -33,5 +33,6 @@ def main(): print("one batch:", score.remote(preprocess.remote(rows))) batches = [[{"a": i, "b": i + 1}] for i in range(4)] - jobs = [score.spawn(preprocess.remote(b)) for b in batches] + pre = [preprocess.spawn(b) for b in batches] + jobs = [score.spawn(j.result()) for j in pre] print("fan-out:", [job.result() for job in jobs]) diff --git a/plugins/runpod/skills/flash/examples/queue_gpu.py b/plugins/runpod/skills/flash/examples/queue_gpu.py index df602e98..5b44af5a 100644 --- a/plugins/runpod/skills/flash/examples/queue_gpu.py +++ b/plugins/runpod/skills/flash/examples/queue_gpu.py @@ -17,7 +17,7 @@ def gpu_sum(values: list[float]) -> dict: # torch ships on the built-in GPU image; import heavy packages inside the function import torch - t = torch.tensor(values, device="cuda") + t = torch.tensor(values, dtype=torch.float32, device="cuda") return {"gpu": torch.cuda.get_device_name(0), "sum": t.sum().item()} diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index d827336a..d3b85f9a 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -59,10 +59,10 @@ On a queue or task handle: |---|---| | `fn.remote(*args, **kwargs)` | runs remotely, blocks for the result; on a queue, stops waiting after 300 s with `TimeoutError` | | `await fn.remote.aio(...)` | async form | -| `fn.spawn(...)` | submits, returns a job immediately | -| `fn.stream(...)` | yields chunks from a generator function | +| `fn.spawn(...)` | submits, returns a job immediately (a `TaskJob` for a task) | +| `fn.stream(...)` | queue only: yields chunks from a generator function | | `fn.local(...)` | runs the body in the local process | -| `fn.with_options(webhook=..., execution_timeout=..., ttl=..., low_priority=...)` | a copy of the handle with per-job options | +| `fn.with_options(webhook=..., execution_timeout=..., ttl=..., low_priority=...)` | queue only: a copy of the handle with per-job options | Queue jobs from `.spawn()`: `job.id`, `job.status()`, `job.result(timeout=...)`, `job.stream()`, `job.cancel()`, `job.retry()`; each has an `.aio` async form. diff --git a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md index fe83ee11..ca667491 100644 --- a/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md +++ b/plugins/runpod/skills/flash/reference/migrate-from-runpod-flash.md @@ -4,7 +4,7 @@ concepts: [flash] # Migrate a `runpod-flash` project to the Runpod SDK -`runpod-flash` is deprecated on October 29, 2026 and gets no fixes after that date. Its +`runpod-flash` is deprecated and gets no fixes after October 29, 2026. Its features live in the `runpod` package (2.0) as SDK apps with the `rp flash` CLI. Endpoints already deployed with `runpod-flash` keep running; migrating changes how you build and deploy, not what is running. @@ -14,11 +14,8 @@ convert, smoke-test with `rp flash dev --once`, deploy, then remove the old endp Keep a way back until the SDK version works: convert on a branch, and leave the `runpod-flash` endpoints running until the new ones pass. If the SDK fails on something the old -package handled, roll back to the previous commit and keep deploying with `flash`, installed with -`uv tool install runpod-flash --with "runpod<2"`. Check the runpod-python -[CHANGELOG](https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md) and -[commit history](https://github.com/runpod/runpod-python/commits/main/runpod/apps) for a fix -before retrying. +package handled, roll back to the previous commit and keep deploying with `flash` +([fallback steps](runpod-flash-legacy.md#fallback-from-the-sdk)). ## 1. Inventory @@ -36,7 +33,8 @@ CI step that calls `flash`. These patterns behave differently in the SDK. If the project relies on one, tell the user before converting, and test it carefully or keep the project on `runpod-flash`: -- Projects split across multiple local files. Discovery ships one module per app. +- Projects split across multiple local files. Deploy packages the whole project, but a dev + session ships only the entry module, so sibling imports fail under `rp flash dev`. - Classes as remote resources, other than `@app.api` classes. - Path and query parameters on HTTP routes. - Resources that reference an existing template or endpoint with `template=` or `id=`. @@ -47,7 +45,8 @@ before converting, and test it carefully or keep the project on `runpod-flash`: ## 3. Install and log in ```bash -pip uninstall runpod-flash && pip install "runpod>=2.0.1" +uv tool install runpod-flash --with "runpod<2" # keeps `flash` for cleanup and rollback +pip uninstall -y runpod-flash && pip install "runpod>=2.0.1" # uv: uv remove runpod-flash && uv add "runpod>=2.0.1" rp login # a key saved by `flash login` keeps working ``` @@ -58,7 +57,7 @@ rp login # a key saved by `flash login` keeps working |---|---| | `from runpod_flash import Endpoint, GpuGroup, GpuType` | `import runpod` | | project folder = app | `app = runpod.App("APP_NAME")` | -| `@Endpoint(name="x", gpu=..., ...)` on a function | `@app.queue(name="x", gpu=...)` | +| `@Endpoint(name="embed", gpu=..., ...)` on a function | `@app.queue(name="embed", gpu=...)` (queue and API names need 3+ characters to deploy) | | `api = Endpoint(name=..., ...)` + `@api.post("/p")` functions | `@app.api(...)` on a class with `@runpod.post("/p")` methods | | class `@Endpoint` with `__init__` loading a model | `@app.api` class with a `@runpod.init` method | | `await fn(x)` | `fn.remote(x)` or `await fn.remote.aio(x)` | @@ -74,7 +73,9 @@ rp login # a key saved by `flash login` keeps working | `env={"HF_TOKEN": "hf_..."}` | `env={"HF_TOKEN": runpod.Secret("hf-token")}` after `rp secret add hf-token` | | `Endpoint(id="ID")` + `await ep.runsync(...)` | `runpod.Endpoint("ID").run_sync({...}, timeout=120)` | | `Endpoint(name=..., image="IMG")` (image is a Serverless worker) | `image="IMG"` on a decorator (any Python 3 image); a prebuilt worker image goes through runpodctl/runpod-mcp + `runpod.Endpoint` | -| `idle_timeout`, `execution_timeout_ms`, `flashboot`, `scaler_type`, `scaler_value`, `accelerate_downloads`, `datacenter`, `dependencies`, `system_dependencies`, `gpu_count`, `max_concurrency` | same names on the decorator | +| `accelerate_downloads`, `datacenter`, `dependencies`, `system_dependencies`, `gpu_count` | same names on every decorator | +| `idle_timeout`, `execution_timeout_ms`, `flashboot`, `scaler_type`, `scaler_value` | same names on `@app.queue` and `@app.api` (a task runs one Pod per call) | +| `max_concurrency` | `@app.queue` only | Before and after for a queue function: @@ -142,9 +143,8 @@ class MyApi: - **GPU device names pin the exact device**, as runpod-flash's pool exclusions did, now using the platform's live pool membership, so pool members runpod-flash didn't know about (MIG slices) are excluded too. -- **Every resource gets `RUNPOD_API_KEY`** in its worker env. runpod-flash added it only to - endpoints that made remote calls, so after migrating, more endpoint templates hold the account - key. +- **Every resource gets `RUNPOD_API_KEY`** in its worker env, including resources that never + call another one. ## 6. Smoke-test, deploy, clean up diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index 9ae47014..a1f0bc79 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -40,7 +40,7 @@ app = runpod.App("sd") @app.api(gpu="5090", workers=(0, 3), idle_timeout=300, dependencies=["diffusers", "transformers"]) -class SD: +class StableDiffusion: @runpod.init def load(self): import torch diff --git a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md index 56936e67..eb968461 100644 --- a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md +++ b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md @@ -5,14 +5,15 @@ concepts: [flash, serverless-handler, job, container-disk] # Legacy `runpod-flash` reference Use this to maintain a project that still imports `runpod_flash`, or as the fallback when the -Runpod SDK path fails on an SDK bug ([../SKILL.md](../SKILL.md), *Which SDK is this project -on?*). `runpod-flash` is deprecated on October 29, 2026 and gets no fixes after that date; -propose [migrating](migrate-from-runpod-flash.md). New code goes on the Runpod SDK. `flash --help` is authoritative for the installed version. +Runpod SDK path fails on an SDK bug (see [Fallback from the SDK](#fallback-from-the-sdk)). +`runpod-flash` is deprecated and gets no fixes after October 29, 2026; propose +[migrating](migrate-from-runpod-flash.md). New code goes on the Runpod SDK. `flash --help` is +authoritative for the installed version. ## Setup and CLI ```bash -uv tool install runpod-flash --with "runpod<2" # Python 3.10-3.13; isolated, keeps runpod 1.x +pip install runpod-flash # Python 3.10-3.13; next to runpod 2.x, see "Fallback from the SDK" flash login # saves a key to ~/.runpod/config.toml (--no-open for headless) flash init my-project # scaffold (writes AGENTS.md + CLAUDE.md) flash dev # local server on :8888; functions run on remote workers @@ -26,6 +27,31 @@ flash update # upgrade the CLI 1.21.0 and later print a deprecation notice; `RUNPOD_FLASH_SILENCE_DEPRECATION=1` silences it. + +## Fallback from the SDK + +When the Runpod SDK fails on what looks like an SDK bug (a traceback inside `runpod/apps/`, a +worker that crashes before your function runs, or code that worked on `runpod-flash`): + +1. Run `pip show runpod` and upgrade. Check the runpod-python + [CHANGELOG](https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md) and the + [commit history for `runpod/apps`](https://github.com/runpod/runpod-python/commits/main/runpod/apps) + for a fix or a behavior change. The + [runpod-flash history](https://github.com/runpod/flash/commits/main) shows how the old + package handled the same case. +2. Still broken: tell the user, and run the project on `runpod-flash` from the pre-migration + code. Give it its own environment with the project's requirements and `runpod` 1.x, so the + SDK project keeps 2.x: + ```bash + uv venv .venv-flash + uv pip install --python .venv-flash runpod-flash "runpod<2" -r requirements.txt + .venv-flash/bin/flash deploy + ``` + `flash dev` and `flash deploy` import your modules locally, so that environment needs the + project's own imports. For cleanup only, `uv tool install runpod-flash --with "runpod<2"` + is enough. +3. It gets no fixes after October 29, 2026, so move back to the SDK once the fix ships. + ## Modes ```python diff --git a/plugins/runpod/skills/flash/reference/setup-and-cli.md b/plugins/runpod/skills/flash/reference/setup-and-cli.md index ebd6a792..7c931d68 100644 --- a/plugins/runpod/skills/flash/reference/setup-and-cli.md +++ b/plugins/runpod/skills/flash/reference/setup-and-cli.md @@ -92,7 +92,10 @@ Take exact flags from `rp --help`; the list above is the shape of the **1,500 MB**. On the built-in GPU images `torch`, `torchvision`, `torchaudio` and `triton` are left out automatically, because the image ships them. - `.runpodignore` and the project's `.gitignore` files decide what source goes in. `.env` - files, keys and credential files are always left out. + files, keys and credential files are always left out, and so are tests (`tests/`, `test/`, + `test_*.py`, `*_test.py`), virtualenvs (`.venv/`, `venv/`, `env/`), caches and `*.tar.gz`. + A helper module named like a test (`test_utils.py`) is dropped too and fails to import on + the worker. - `--build-only` resolves dependencies and packages without touching your account, which makes it a quick check that discovery and dependency resolution succeed. diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index 6dbe9d7b..dbe607e8 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -30,9 +30,9 @@ rules: - id: flash.runpod-flash-deprecated statement: > - The standalone `runpod-flash` package is deprecated on October 29, 2026 - and gets no fixes after that date. Endpoints it deployed keep running. - Write new code on the Runpod SDK and migrate existing projects. + The standalone `runpod-flash` package is deprecated and gets no fixes + after October 29, 2026. Endpoints it deployed keep running. Write new code + on the Runpod SDK and migrate existing projects. status: documented evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } @@ -42,15 +42,16 @@ rules: statement: > `runpod-flash` still installs alongside the Runpod SDK: its CLI is `flash`, separate from `rp`, and `runpod_flash` imports with `runpod` 2.x - installed. Install it as an isolated tool with `uv tool install - runpod-flash --with "runpod<2"` so it keeps its own `runpod` 1.x and the - project keeps 2.x. Use it as a fallback when an SDK build, deploy or - worker step fails in the SDK rather than in your code. It gets no fixes - after October 29, 2026. + installed. To fall back from the SDK, give it its own environment with the + project's requirements and `runpod<2`, so the SDK project keeps 2.x; + `flash dev` and `flash deploy` import the project's modules. For cleanup + alone, `uv tool install runpod-flash --with "runpod<2"` is enough. Use it + when an SDK build, deploy or worker step fails in the SDK rather than in + your code. It gets no fixes after October 29, 2026. status: verified see: [flash.runpod-flash-deprecated] evidence: - - { source: live-probe, ref: "one venv with runpod 2.0.0rc2 + runpod-flash 1.21.0: import runpod_flash ok, flash --help and rp --version both run; uv tool install runpod-flash --with 'runpod<2' resolved runpod 1.12.0 + runpod-flash 1.21.0 and flash --help exited 0", seen: "2026-10-22" } + - { source: live-probe, ref: "one venv with runpod 2.0.0rc2 + runpod-flash 1.21.0: import runpod_flash ok, flash --help and rp --version both run; uv tool install runpod-flash --with 'runpod<2' resolved runpod 1.12.0 + runpod-flash 1.21.0 and flash --help exited 0; uv venv + uv pip install runpod-flash 'runpod<2' -r requirements.txt (numpy) gave runpod 1.12.0, numpy importable, flash --help ok", seen: "2026-10-22" } - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } - id: flash.not-flashboot @@ -125,11 +126,13 @@ rules: statement: > `rp flash deploy` packages an artifact of up to 1500 MB. On the built-in GPU images it leaves out `torch`, `torchvision`, `torchaudio` and - `triton`. Local Python must be 3.10 to 3.14, and `--python-version` picks - the worker's Python for deploys. + `triton`. Dev sessions and tasks need local Python 3.10 to 3.14. Deploy + falls back to Python 3.12 with a warning on other local versions, and + `--python-version` picks the worker's Python for deploys. status: documented evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/deploy", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L190", seen: "2026-10-22", note: "On an unsupported local interpreter, deploy warns and builds for DEFAULT_PYTHON_VERSION." } - { source: skill, path: plugins/runpod/skills/flash/reference/setup-and-cli.md } - id: flash.sdk-defaults @@ -159,11 +162,24 @@ rules: statement: > Deploy and tear down through the CLI rather than raw REST or GraphQL calls: `rp flash deploy`, `rp flash undeploy --app APP --env ENV` for one - environment, and `rp flash app delete APP` for every environment. + environment, and `rp flash app delete APP --yes` for every environment. + Without `--yes` the delete prompts for confirmation and aborts when there + is no terminal. status: documented evidence: - { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" } - { source: skill, path: plugins/runpod/skills/flash/reference/setup-and-cli.md } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L718", seen: "2026-10-22", note: "app delete takes --yes/-y; otherwise click.confirm(..., abort=True)." } + + - id: flash.resource-name-min-length + statement: > + `rp flash deploy` rejects queue and API resources whose name is shorter + than 3 characters. The name defaults to the function or class name, and + dev sessions do not check it, so a short name passes `rp flash dev` and + fails at deploy. Pass `name=` or rename the function or class. + status: documented + evidence: + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L133", seen: "2026-10-22", note: queue and api names must be at least 3 characters } - id: flash.api-key-precedence statement: > diff --git a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md index c379c631..ae478689 100644 --- a/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md +++ b/plugins/runpod/skills/runpod/golden-paths/07-network-volume-handoff.md @@ -91,10 +91,11 @@ vol = runpod.NetworkVolume("", create=False) # attach EXISTING by id @app.queue(cpu="cpu3c-1-2", workers=(0, 1), datacenter="EU-RO-1", mounts={"/runpod-volume": vol}) def read(path: str = "hello.txt") -> dict: # input keys = parameter names - p = os.path.join("/runpod-volume", path) # /runpod-volume, not /workspace + root = str(vol.path) # /runpod-volume, not /workspace + p = os.path.join(root, path) return {"exists": os.path.exists(p), "content": open(p).read() if os.path.exists(p) else None, - "runpod_volume_listing": sorted(os.listdir("/runpod-volume"))} + "runpod_volume_listing": sorted(os.listdir(root))} @runpod.local_entrypoint def main(): diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 8ccfb861..62f30c58 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -14,7 +14,8 @@ GPU, dataset) — nothing structural changes. **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` -and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code, +and generated. Section 3 is ported to the Runpod SDK and not yet re-run as written: on +2026-10-17 a variant that loaded the base model through `VolumeCache` instead of from the mount, deployed with `rp flash deploy` on a pre-release `runpod` 2.0 build, loaded TinyLlama plus a LoRA adapter from a network volume and generated text (the adapter was an untrained one saved by a helper function, standing in for the pod training step). `peft` and `accelerate` were plain @@ -179,7 +180,7 @@ BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" _M = _T = None -@app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="EU-RO-1", +@app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="", mounts={"/runpod-volume": vol}, env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache dependencies=["transformers", "peft", "accelerate"]) @@ -237,6 +238,7 @@ parameters, non-empty input. For high-throughput serving, swap the transformers ## Cost & cleanup ```bash +rp flash app delete ft-serve --yes # SDK serve step; or delete the endpoint by id: runpodctl serverless delete # (undeploy+redeploy for fresh workers if a code change is stuck — see 07) runpodctl network-volume delete # pod already removed; deletes the adapter + cache ``` diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 17bcd697..7641264e 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -74,8 +74,8 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | 04 | [LoRA fine-tune (training run) on a pod](04-finetune-pod.md) | pod / batch job | runpodctl pod + volume | ✅ live-verified (training phase, via 08) | | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | -| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16) | -| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step via `rp flash deploy` (2026-10-17) | +| 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16, reading the literal `/runpod-volume` path; the file now reads `vol.path`) | +| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK serve step ported (the 2026-10-17 SDK run loaded the base model through `VolumeCache`) | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified | From 9451b6ee812ee131f70981cf044e9286553ae6b8 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 23 Oct 2026 13:07:05 -0400 Subject: [PATCH 14/18] docs(flash): address review round 2 Create the runpod-flash fallback environment outside the project so a deploy doesn't package it, reword the legacy install comment, and remove committed bytecode (with *.pyc ignored). --- .gitignore | 3 +++ .../api_model_service.cpython-314.pyc | Bin 2796 -> 0 bytes .../__pycache__/call_deployed.cpython-314.pyc | Bin 1416 -> 0 bytes .../__pycache__/cpu_gpu_pipeline.cpython-314.pyc | Bin 3078 -> 0 bytes .../__pycache__/queue_gpu.cpython-314.pyc | Bin 1832 -> 0 bytes .../__pycache__/train_then_serve.cpython-314.pyc | Bin 3145 -> 0 bytes .../flash/reference/runpod-flash-legacy.md | 12 ++++++------ .../skills/runpod-usage/concepts/flash.yaml | 5 ++++- 8 files changed, 13 insertions(+), 7 deletions(-) delete mode 100644 plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc delete mode 100644 plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc delete mode 100644 plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc delete mode 100644 plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc delete mode 100644 plugins/runpod/skills/flash/examples/__pycache__/train_then_serve.cpython-314.pyc diff --git a/.gitignore b/.gitignore index 0bb6ecf3..81cc5d31 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,6 @@ result-* # Concept graph tooling (ontology/) node_modules/ ontology/build/ + +# Python bytecode +*.pyc diff --git a/plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/api_model_service.cpython-314.pyc deleted file mode 100644 index 3b50d6cd078feb0f4f770b25862a3f9c6fc09690..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 2796 zcmZuz&2Jn<7O(2=>FN2%*vUGv2??&4V0IMSvjk_eE1ca}AP7PVrpXQwCT_K7%ASsU zx|^z+*v?@!c#+~ryBi;ULCPNYW=>rB69xtp)Z&1|hm@O*CBkyrSJl(j#Bxb~u8;Tq ze(&{MC>_fqSZ5aF+dmlyeJ>B`%Jw$<4`I_rHWFw7+4xN?@GU$C+=tgWWNXtnxfS{? zeJ*#H10h?t^R{7Urm$^Z?&&y*Kr%awY-?IeB&K_UA_JC+M`+0nqXV9-jh{#5g1ZW) zXNe!W^~puocY_+OlWR9_%n?r69j``-Xt+YXbK;U_Sf@S-fr~_xk=2l`P&Sp%6Id)F zOTNn+q)zV;H}V|C?DX_ZMeAg8Qc50Xj{I=_NdA2ODr*H%SfBC&PY~Gi#Pn)|2H;F$ zop9kY!6(QfT?!c`z$x4n%86j^(vnx3AUw>A1~}|R)b|1^1=cz70@9?-kb&(Q3ppo@ ze%PX1@G0`9)Gnjolyff($a>f!HP`p^ENls>x&fQz2K=X_1~Kzj636OP21Dx;>V*dN zDD(oMQmLe9lZ1C5Z!*DC`RjKrLD$6OG7VyL_O)5(;~W>UP{W}mIRofU;D5=Gkr zWai@R4M@%#5LmA{lqwS_1ZM4DgiSkXEp!b>cobE!bj#ME-Hgk<;HxO;H3tNi0k!61 zsiur$D}}?|fwTyexdN&k8_SWnSk~f<0&r{s5}*;Tmvtt!GTB+NA;p2m;M8AIS6mtz zC-I*nzqRZIypmwI!dE=s=M~i&6}sj&BcJlhQT-7cvJK;6FYYOMk_qG^K-%a>tGs8T zlV>|;r*@V;f9I>AH#Uub=l02?bV>TWPk_weHmafp?N1*gREOI$ z=pEd^Hhve+qlz2`+GEKd8;fCm9WrFpy_$d!O$kI-#K#;*VO?H3jtY0I&7^~I(Q(`$ z2nE=8oOmb=p#SzX@bMuazqZj6q-8GdN%Eg*^^cu|huT4<;2_87KV+RZU^9kBA^s|w zOXGsObX5V;6H<9^Rq>pR_|UTP!caOUZS8W;QUqwJY70-v+eW`OnU_nths&ARj9A!= zL|kgRYfeB{9TBe3fR`~l2gYJUq9`t>J(7f7|rBrRt{s z&ETnReS7v3vol)xdgKqCftNbgOaJ2rXxO;&SNvPH(x9#{8c#KrOo>XkENGH&8)m^l z0ssdvH${ubTDbCG=pYphmlYXLA}|xJaVhmbIX?#+TxLOeKRB{|zB4e|u||6iWwID_ z$@lB!W3lBpf!mZI$~*G=(Sk|mNO*nNa{b=Rfa5HAj0-u-gAkt0q|;(UekR!OKozuS z$ORjREjusCq`W){sD%)`n&S$=yhS<6LHu|hu4~ly9S0mb0|F|zt137qKgxiYA&WPLNT8#2Y6|UD;M$`2IHUpf` z!ykVeNE`iNs_l7Xbmd eX*cn%I-f_H^$3rAfkyzv_>c1GnV;oOx$%EF?t`!Z diff --git a/plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/call_deployed.cpython-314.pyc deleted file mode 100644 index 8110bb152d12c8fefbd729d4ee8221b1bbe5b983..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1416 zcmZWpO>f&q5M5FdNl}(1yJ-s40klrhHmYo?K<&CQiq>fyq(%UnWpY&*np{hC$t6oJ zRm&&&8<~xmjE(~W-m36Kj6TBpt}-T3(x}M4u|i(c{96EZMLWaw(jlH z{qqa}@Ayl)Fi39T1nio zN}_gm8ppa&^!!8@Gowl~+QKweSM`EE(-2b4i7o`~-ZWmTOy~4s$|B`v`SS!|f5}dV zJG?*5U^6ZM&zT>&o7Ym}1e|%BZ3r(9^ip~vDXYTH^OMo9I%j^&M*rA!U1+1g^}EQ1 zA&UGsv`_^1Lf?ZX?n13g+CGMsZzK2s`!EdPo@+*JM(RL}kd5qWT~#4lK>}rxMk-kk zx~lppieit6zM3HOitv{h#mHy{F@wEgi4QH_6ZbxnR}r=Y-@yd#cA;Z0K?hrIY&*CGeauXeHiFMLpF!KP zA@&KhLSzzDSGNykW^9=`r#vuNP!(y$3<8?PriZEml{%&yqo_3=eR(4Z?ralh#5-*>YW7#}^+A z)_dMi`mC4XFza~lXz#@8@1O4f=3at@D<-vmwm$fRNflOm3dgiL^ZePxgV#^rEUpZt z)l2X-D_WZ4tUf-)gXVCy)+_v*nf)hU>Q~MxgN0%K)7}qnv-y+z{n}~m{NUoFH+QRl WesxkCs%w8|zj>bn*-{8uKmP$3ep|o* diff --git a/plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/cpu_gpu_pipeline.cpython-314.pyc deleted file mode 100644 index d6b349595f0e26aad141869aec4bca0ed4154846..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3078 zcma)8U2Gf25#GHY{v9RDc3{btAfIH(F{>X+wPeex8bGD&xG<{9XPZp}MZ}3*NvDu^ zm%XDUD=okl1!4mMtN>1{pe~{o0V)Ct;730;%}XBgK!J_bZ1Z3sees(d1t|PdboP#< zDT4{Rzzv79v$L}^-^^}bb5jJt_}4sJUX&2}n*XT|xx(y!4om@Q$VGEV!#~9?egn^d z_KTwnX~GEhrLOpfI1_$R10hY+q(Q97=LLUG3$!9F*ov|(BNb&={*IL)tw{^l)bK zJF2ZyJ)0$2bwM`=th_rIiKsBBtuADBXGzVFn`+w5rxxL}4Vz>QlMLFo)PVuZOp{3D z`VvtSGdKLL)O42UrlV48#ZgxbcL~g!8QuAj>Vn#{lH)8M#ToadlpI(@S8A=1n+e#3B4VE>Tx3 z`X-@{s=FXglflR|3sv1;3&Vj%(*q*|V=Ul17!#NXLKe*so0u75rVZkxd9o`BM*-kP z`1UD~0!pGeVeEGZWkA+}uHq%E;aQmR$80722=$++$^aZfokmSOTZ3A?Zap6tAgSUL zZj(tov@Af0o2O<>WKz~}Tqb3Vw9Dj$tfjj#%mND*&FQYgeeY&tkmp6ibm9;r+see9 zH;ru8iF@gdlT|%uXNePUkYg52rF7G@TnK0?MGt|s7WkZ(ffUexkc9i6a&qtb{hs$< z-{b5@Q`;|XlaJfK=o$5<_CXbSapVjiJceEz{{l?GFF9o6B%a~Y7e*3Q3vQ-bu5}t! z*xN!2RaUd&wS%hE|D9<%_@Do)T86j_Pq<(@7Np!i7M{{w{Swu4gatCx zvQ3=@bjQ7A6B_07ftz|ZPas3C#WNEVDR+Vic7_R=1&7->4b{a9nMG_$Y-**6<4_J0 z9$;r2NCABSxn*&&^)pWtL;jjXjk()-EKv|0!BNvmI>@`b37%?4-vY=P>XMf0~9Q0n= z3N|2$C*g=6Kq&toek1F&DPvLE2|J#BD7D2mzl&L2~6aa!;rsT`0#tI zl+z89j)A-rKIbbS1@yS7WxZ?t+D2C?+OgqmPHs9|Q>D(~2Z@J=A13~GxO6sAQl<)1 zUrFKH{rBY$qaQ>+dVRO|+)nSg2X47{qTKf5e@L(VJ3jK@>h@~6d91j))wOKBmiIiG;%7=Q3tA$rf&0}AliI*co#o*@fyW_h``;OASG5JX8ECvgSQe^1MfuXW8 zT)ei0-@SPE%AG44_`}c#p+80*MNbv46{bqcFn8hyMYnuxr08yme^`5O?f1Vbx;w{4 z9!aCmz6qj37oM@m&@-kCKV!#69Nri|Ioz7)lRg>jObkhXKhmEVl|GHMCB`Gr7I=qw z!ux(K%z~*@#!9DB5NPjVz{E-0W}*&6ek)J%ep382wPdZdo=w5~k>0W`o;@DYBJw(c zAL-CU&dTJo`Ph`zJ@+vW8WS4+zTK&(`P!tpX5`Bs2;#J-4x(LK)XHP=)&FL8ywZw21)xkmq<$L>Xu5Zc9UJGhN+ ObdFCQ|Aup(d;bH(^>a%A diff --git a/plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc b/plugins/runpod/skills/flash/examples/__pycache__/queue_gpu.cpython-314.pyc deleted file mode 100644 index d4d73a16baef25c8d157dc2e2dae6bd9ce0375a4..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1832 zcmZ`(&2Jk;6rb5IyN>hKs7(TaOcGIH(bxu3Q7Q_FsA&00%t{21(kz?x#MySeJKdda zYfc7LYI{Hp5{Rnef`kOzdTi;T{S(p#snx)t5=baF2P1Li&H5v0RYuy`_ujmD^XC2D zZ(iu@%^(=Rt%y@+RD}MJN@o~tp?M92Ix-PQ%gDrUV~$ValGbtC`A;BInZYfct0&dc zo}IOhlt*TI0h@bFHIGg06*yB4Qy)gAF^s%{nU(=@9nQ$Mm0dqGHH^AGDHBhjT*iJM z4xJ;%OCOTe*mt-a_+*8yxqg}0KCxpSM2_vbeubQgX-oJFImhKOUEPLG97d zNzf!NqhaJmocg@Gow$%TV`LKYn2zG=WF|A`*GRBR_!=di^e)TmM#PPX?KqTid&Q#% z;ks~T$nw1OJw(n0;c19YJjx@vHN!lcuLfb2MCV-YtU;CdIZA!GRwXMn62**pwR~p& zvsC+Nq5E-_)pDv(!6H)akRn^A4C+XI$EA@YL2m<(WMfEJn{UBXM~i4#IeY=3GT6q@ z2Y3yecnMZoV@c^q2cPOl&<@H{x4jAqK&);_KVh7;y_iPA2q}+4ACl5!F@(PA1vVF2 z*>!jh3qxMf=8?>3QoP8#xB^HO06-R$3(;xU^P)lmd4Zm{tIVTOA?b`Z?jmbhw(ker z=G3yneh4!FUo;6*9sP@R;J|h=e|_w`ec!y>NaP=mwQNoBLPqaUB}U%Lok&Jl&DB34=*^?H;51#=|q7og4sg9Zk(gz==d#_|K zXV({hJo^378~ZP1H`BBAxd++7^+Vqt`Rd61q3OFr(?8jpL&fXQ-59*R@8-T+!?(w7 zj@^28qvynib|RDzJMeXfY(aT`lfzU;J5Bo}*x+5$7I3H33@z!MIE!5X#(@kbezegK ze~eHCt_;+G)eaOu5*6BB|Em?G@kUajN=qA=orf}wgVHvj*|w0Lg#}q0$&hNmsnA&n z4;^qq4dNCd+O^c=Qx;Bv?+AQR9;Q0_Gc$N4e>s0`d^3})zxlhi=hD=LcOLZgU5dXv za&_p}p0N#W;;OT$P2B4lOX?*VI8%JJD3teRgg$quSS;q0a4!VglK0Q0Mar_ufn!;a zQ9?hV&M_ubc-@4aaITf_fEU0MZNVcL){?h5N$qEpoYF!+{aQ6B#~yt>EI??5s4!fO%S%oLxAKVd2@gk=}#zv0<}sTau z?9AD7n{#I7d^0nBu`q%*w(8v+krDca{rF9469-=dQAH|pQ5vcE73|_SanhIi+AvNa zRXC4Z3|D+pOon=dZBi9g=|WpajjHl-qy~;7W5B@Y+ZwJ6ma#)DgR0W841ryk*>$Xf z5YxaRR*~bVqZ3l`B#OthI}pHYR4bXpGKof#R&JKKMcpKhPH*dsBvf}QhD-9)Duc$Y zieAx)Zsu&OWV-X=Fo8yGk~cJ`NOJmZ;xcy`<|MdtY#X|hIArs^XeOhXrscwp zW->Gi-Ui`wPC!>h|3Xo`g!g&>!83b&{Mc>%AAknpSu=vQ0DL^_^ALZvRfNdKDV${7 zr_FEdy`Xr3D2!5mJr@&i2fo7G@RS z5+Z-5OZ${RpB+re$zwjZJNojIC>2Z|@A7@BM=mEvd~SE_%76TGghJ>9QiG8BXjj9h zlH)$r9erz60ULILo@)QghM)Y<6K1hoVg=+$*-B1hCmnFBl+_Er`z>oHw!X-<5=lU<8ui=IrOT1`(dO)v11lBsKyGPw0*%XUj;PrUKcRSzJ6YGoau%;EY4 z-IL0?W_qDbom!a!)pDM)s%6*KHJbAzR)>^rh9{L=y-be+8CNJf?v(9;hu`w>h9{S; zoNjm`OyuE=CmELGIBY+s7z#t95vDBjDx}juv%s9cKvzZINCS_R*mk5Asb6hOe6q9^ zX(}_-#fM60Yw?a;$9F>T6CK=sx%Tq*_1g8uV&hgbIC4w-U|p^VJ_diXEvx_x{Qj{@dVC{nl@W8;c)~eKPT2U~;!V zzSAFX_Rnld50%JvtQLE(|Dh7~iO9B6Q{E5nhfv?>UKquW)%A~zUmLqa&+ZI8+Z>92 z{`}qNzPj|~rLV4hd8K*o>b=Odd(t(^uEotz)~=k(&YhV%%ODbd{sSUAU~!q}|Nldz zD)zvo2yiJ;1{z*~Z>mF2!4GAi5Em%B*gZK9@DGqlS*5k`lJ5iuJKjw&!4z~1@dS=| ztpDKZ9{>CJ_15@VWuO1^^s?=02P=eF037*TAgXA^zXt&MRW#dEJw6)rXkee$i+w$E zc*-2+p%)ur5d?lfTSpU9CIr?D417Ht7VOki>i`DZ*EF36CV|gkdR6po07VDuLk;D@ z;6yWW>a*8BKlAC&9z^DTFW=4oG4%(txp3`4cYCv7(`lXj{|qe1zauyd`F)hI(MkvH zRahw>u}`8Eb~t!D%pTUXgYGu&@gbz(Mv0-b9drdxou?hhq+YD`S6!7a9zm!8bqY5# zN_C!ha7PDVU4xqnuU9>HsnCw;#m!KD`t1BdoV-9b#rd=OwqWeVHu3n%a$M#G&Mxn= ziC;82|oGe_~;<9hYmdlY<3=<8_-qJ-(yF%S86NormIVjq|nys zdj7sN{z&QH?yL3HZ#SH#GFe?@l2__lL#)4XFFbxcKY(TdsVvP3%1xp7<5$<~&h`w|4mpAL< z&A`d3xEDZz{Lb*(!&~|4a1$Tu=J+k$S>PQoR=UaglDpMHS b<0N2|pzPw|9Xz}*$asQ@M*hJl9@@VF;+AUs diff --git a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md index eb968461..0e006ab7 100644 --- a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md +++ b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md @@ -13,7 +13,7 @@ authoritative for the installed version. ## Setup and CLI ```bash -pip install runpod-flash # Python 3.10-3.13; next to runpod 2.x, see "Fallback from the SDK" +pip install runpod-flash # Python 3.10-3.13; to run next to runpod 2.x, see "Fallback from the SDK" flash login # saves a key to ~/.runpod/config.toml (--no-open for headless) flash init my-project # scaffold (writes AGENTS.md + CLAUDE.md) flash dev # local server on :8888; functions run on remote workers @@ -27,7 +27,6 @@ flash update # upgrade the CLI 1.21.0 and later print a deprecation notice; `RUNPOD_FLASH_SILENCE_DEPRECATION=1` silences it. - ## Fallback from the SDK When the Runpod SDK fails on what looks like an SDK bug (a traceback inside `runpod/apps/`, a @@ -41,11 +40,12 @@ worker that crashes before your function runs, or code that worked on `runpod-fl package handled the same case. 2. Still broken: tell the user, and run the project on `runpod-flash` from the pre-migration code. Give it its own environment with the project's requirements and `runpod` 1.x, so the - SDK project keeps 2.x: + SDK project keeps 2.x. Create it outside the project directory so a deploy doesn't package + it (`rp flash deploy` skips only `.venv/`, `venv/` and `env/`). ```bash - uv venv .venv-flash - uv pip install --python .venv-flash runpod-flash "runpod<2" -r requirements.txt - .venv-flash/bin/flash deploy + uv venv ~/.venvs/flash-fallback + uv pip install --python ~/.venvs/flash-fallback runpod-flash "runpod<2" -r requirements.txt + ~/.venvs/flash-fallback/bin/flash deploy ``` `flash dev` and `flash deploy` import your modules locally, so that environment needs the project's own imports. For cleanup only, `uv tool install runpod-flash --with "runpod<2"` diff --git a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml index dbe607e8..508d765a 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/flash.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/flash.yaml @@ -44,7 +44,9 @@ rules: `flash`, separate from `rp`, and `runpod_flash` imports with `runpod` 2.x installed. To fall back from the SDK, give it its own environment with the project's requirements and `runpod<2`, so the SDK project keeps 2.x; - `flash dev` and `flash deploy` import the project's modules. For cleanup + `flash dev` and `flash deploy` import the project's modules. Create that + environment outside the project directory so a deploy does not package it; + `rp flash deploy` skips only `.venv/`, `venv/` and `env/`. For cleanup alone, `uv tool install runpod-flash --with "runpod<2"` is enough. Use it when an SDK build, deploy or worker step fails in the SDK rather than in your code. It gets no fixes after October 29, 2026. @@ -53,6 +55,7 @@ rules: evidence: - { source: live-probe, ref: "one venv with runpod 2.0.0rc2 + runpod-flash 1.21.0: import runpod_flash ok, flash --help and rp --version both run; uv tool install runpod-flash --with 'runpod<2' resolved runpod 1.12.0 + runpod-flash 1.21.0 and flash --help exited 0; uv venv + uv pip install runpod-flash 'runpod<2' -r requirements.txt (numpy) gave runpod 1.12.0, numpy importable, flash --help ok", seen: "2026-10-22" } - { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L35", seen: "2026-10-23", note: "DEFAULT_IGNORES skips .venv/, venv/ and env/, not other virtualenv names." } - id: flash.not-flashboot statement: > From 1d38d89a0b433741271240e0055775207e290f58 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 23 Oct 2026 13:24:39 -0400 Subject: [PATCH 15/18] docs(flash): one fallback venv per project --- .../runpod/skills/flash/reference/runpod-flash-legacy.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md index 0e006ab7..fe021154 100644 --- a/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md +++ b/plugins/runpod/skills/flash/reference/runpod-flash-legacy.md @@ -43,9 +43,9 @@ worker that crashes before your function runs, or code that worked on `runpod-fl SDK project keeps 2.x. Create it outside the project directory so a deploy doesn't package it (`rp flash deploy` skips only `.venv/`, `venv/` and `env/`). ```bash - uv venv ~/.venvs/flash-fallback - uv pip install --python ~/.venvs/flash-fallback runpod-flash "runpod<2" -r requirements.txt - ~/.venvs/flash-fallback/bin/flash deploy + uv venv ~/.venvs/flash- # one per project, so packages don't mix + uv pip install --python ~/.venvs/flash- runpod-flash "runpod<2" -r requirements.txt + ~/.venvs/flash-/bin/flash deploy ``` `flash dev` and `flash deploy` import your modules locally, so that environment needs the project's own imports. For cleanup only, `uv tool install runpod-flash --with "runpod<2"` From 1495679a4071dc0b7a77985e9e51561a14601c03 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 23 Oct 2026 14:16:01 -0400 Subject: [PATCH 16/18] docs(volume-cache): load volume-backed models through VolumeCache on the Runpod SDK (#90) * docs(volume-cache): load volume-backed models through VolumeCache on the Runpod SDK Stacked on the SDK 2.0 flash lane. VolumeCache: - Golden path 08's serve step restores the base model through VolumeCache. Live-verified: a later cold start restored 2.2 GB and loaded it in 7.2 s. - Golden path 09 keeps its model cache on local disk. - The Flash patterns add global volumes and the per-endpoint namespace in dev sessions. - model-caching.md gains a measured comparison. Qwen2.5-1.5B on fresh workers: VolumeCache 9.1 s, mount load 11 s, download 16-21 s, global-volume restore 30-55 s. * docs(volume-cache): global volumes need runpod>=2.0.1 Before 2.0.1, VolumeCache restores from a global volume copied whole files and could drop large weight shards (runpod-python #621), and the error was swallowed. The global-volume rule, model-caching.md and the Flash pattern now set the floor at 2.0.1 and mark the 2026-10-17 global-volume timings as measured before the parallel restore. * docs(volume-cache): cite SDK source instead of PRs; drop external benchmark link * docs(volume-cache): address review round 1 - Golden path 09: the template's MODEL_CACHE_DIR defaults to local disk, and the VolumeCache snippet uses the handler's own constants. - Golden path 08: the training step no longer claims the SDK serve step reuses its cache; the first SDK cold start downloads and fills the mirror. Status restored to verified, since this code is what ran. - restore-beats-mount-load names the model and says smaller models or a faster mount can change the result. * chore: remove committed bytecode from golden path 09's template --- plugins/runpod/skills/flash/SKILL.md | 2 + plugins/runpod/skills/flash/reference/api.md | 3 ++ .../runpod/skills/flash/reference/patterns.md | 14 +++++++ .../runpod-usage/concepts/volume-cache.yaml | 41 +++++++++++++++++++ .../golden-paths/08-finetune-to-serverless.md | 32 ++++++++++----- .../09-custom-serverless-dev-loop/README.md | 33 ++++++++++----- .../template/handler.py | 11 ++--- .../skills/runpod/golden-paths/README.md | 2 +- .../runpodctl/reference/model-caching.md | 27 +++++++++++- 9 files changed, 134 insertions(+), 31 deletions(-) diff --git a/plugins/runpod/skills/flash/SKILL.md b/plugins/runpod/skills/flash/SKILL.md index 7b21385d..3fce31a3 100644 --- a/plugins/runpod/skills/flash/SKILL.md +++ b/plugins/runpod/skills/flash/SKILL.md @@ -178,6 +178,8 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns. 8. **Volumes outlive everything.** Dev sessions and `rp flash undeploy` delete endpoints, not the `NetworkVolume`s they created, and those keep billing. Queue and API resources mount one volume at `/runpod-volume`; read `vol.path` inside the remote function, never at import time. + Load model weights through `VolumeCache`, not straight off the mount + ([reference/patterns.md](reference/patterns.md#loading-ml-models)). 9. **Tasks bill until they return.** A `.spawn()`ed task that you never `wait()` or `cancel()` keeps its Pod running after your script exits. 10. **Native CUDA libraries go in `dependencies` too**, e.g. faster-whisper needs diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index d3b85f9a..a1b77b9c 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -141,6 +141,9 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must - A new network volume is placed in a data center that has network storage and stock for every resource sharing it; resources using it are scheduled in that data center. - Read `models.path` **inside** the remote function. Volumes are not mounted locally. +- Load model weights through `VolumeCache(dirs=[...], volume_path=str(models.path))`, which + copies them to the worker's disk, rather than from the mount + ([patterns.md](patterns.md#loading-ml-models)). - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. ## Models and secrets diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index a1f0bc79..6a9cb881 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -110,6 +110,16 @@ start unless you cache. Pick one: `runpod.serverless.VolumeCache` is importable on the worker without listing `runpod` in `dependencies`. + The same code works on a `runpod.GlobalVolume` (GPU resources only): mount it at + `/runpod-volume` and `VolumeCache` mirrors there. Restores from a global volume measured + slower than from a network volume, so prefer a network volume when the endpoint stays in one + data center ([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)). + +**Don't load weights straight off the mount** (`HF_HOME=/runpod-volume/...` or +`from_pretrained("/runpod-volume/model")`). Every read then crosses the network; restoring with +`VolumeCache` and loading from local disk measured faster +([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)). + `VolumeCache` details that bite: - **Best-effort.** Cache errors are logged and swallowed, and the worker cold-starts normally, so a broken cache looks like a slow cold start. Use `VolumeCache(..., best_effort=False)` @@ -120,6 +130,10 @@ start unless you cache. Pick one: headroom; the default is 30 GB on GPU. - Let one worker populate the mirror first. Workers that cold-start together on an empty mirror each download and sync a full copy. +- The mirror is keyed by endpoint ID (`namespace` defaults to `RUNPOD_ENDPOINT_ID`). Every dev + session creates new endpoints, so each session starts from an empty mirror and leaves its copy + on the volume. Pass a fixed `namespace="qwen-7b"` to share one mirror across dev sessions and + deploys. Full behavior: [`concepts/volume-cache.yaml`](../../runpod-usage/concepts/volume-cache.yaml). diff --git a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml index 98feb6fa..df0a5eea 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml @@ -124,3 +124,44 @@ rules: see: [network-volume.concurrent-writes] evidence: - { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" } + + - id: volume-cache.restore-beats-mount-load + statement: > + With Qwen2.5-1.5B (3.1 GB), a cold start that restored the model with + `VolumeCache` and loaded it from local disk measured faster than loading + it straight from the network volume or downloading it: restore 5.2 s plus + load 3.9 s, against 11 s loading from the mount and 16-21 s downloading + from Hugging Face and loading. Smaller models or a faster mount can narrow + or flip the gap, so time your own model. + status: verified + evidence: + - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct on L4/RTX 4090 workers: fresh-worker VolumeCache hydrate 5.2 s + load 3.9 s (3.10 GB restored); load from /runpod-volume 10.7-11.6 s; Hugging Face download + load 16.3-21.4 s", seen: "2026-10-17" } + + - id: volume-cache.global-volume + statement: > + `VolumeCache` mirrors onto a global volume mounted at its `volume_path` + (`/runpod-volume` on a GPU endpoint). Use `runpod>=2.0.1`. Before + 2.0.0rc2, files of 256 KiB or more failed to copy onto a global volume. + Before 2.0.1, a restore from a global volume could drop large files. Both + failures are swallowed, so the worker downloads the model again. Restores + that copied whole files took 26-51 s for 3.1 GB, against 5.2 s from a + network volume. 2.0.1 restores files in parallel byte ranges and retries + failed reads. + status: verified + see: [volume-cache.best-effort-hides-failures] + evidence: + - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct, GlobalVolume mounted at /runpod-volume: fresh-worker hydrate restored all 3.10 GB in 26.2 s, 46.6 s and 51.4 s", seen: "2026-10-17" } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L292", seen: "2026-10-22", note: "copy2 copies mtime and mode after the data; the global volume rejects utime/chmod, so big files failed to copy. The fix copies the data and treats the metadata step as optional." } + - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L389", seen: "2026-10-22", note: "Hydrate copies big files as parallel byte ranges and retries ENOMEM, EIO and EAGAIN reads, which a global volume returns under whole-file parallel copies." } + + - id: volume-cache.namespace-per-endpoint + statement: > + The mirror is keyed by `namespace`, which defaults to the endpoint ID. + Each Runpod SDK dev session creates new endpoints, so it starts from an + empty mirror and leaves its copy on the volume. Pass a fixed `namespace=` + to share one mirror across dev sessions and deploys. + status: documented + see: [volume-cache.mirror-sizing, flash.dev-session] + evidence: + - { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L87", seen: "2026-10-15", note: namespace falls back to RUNPOD_ENDPOINT_ID. } + - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: Dev sessions create temporary endpoints and delete them when the session ends. } diff --git a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md index 62f30c58..272c0ad3 100644 --- a/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md +++ b/plugins/runpod/skills/runpod/golden-paths/08-finetune-to-serverless.md @@ -1,7 +1,7 @@ --- lanes: [runpodctl, ssh, flash, runpod-mcp] mcp: partial -concepts: [pod, network-volume, flash, serverless-endpoint, worker, serverless-handler, data-center, gpu-pool] +concepts: [pod, network-volume, flash, serverless-endpoint, worker, serverless-handler, data-center, gpu-pool, volume-cache] --- # Golden path 08 — fine-tune → serve (LoRA on a pod → serverless) @@ -14,13 +14,14 @@ GPU, dataset) — nothing structural changes. **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume` -and generated. Section 3 is ported to the Runpod SDK and not yet re-run as written: on -2026-10-17 a variant that loaded the base model through `VolumeCache` instead of from the mount, +and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code, deployed with `rp flash deploy` on a pre-release `runpod` 2.0 build, loaded TinyLlama plus a LoRA adapter from a network volume and generated text (the adapter was an untrained one saved by a helper function, standing in for the pod training step). `peft` and `accelerate` were plain -`dependencies` and the worker ran the image's `torch 2.9.1+cu128`. Run it with -`rp flash deploy` (flash SKILL gotcha 14 covers dev sessions on older runtime images). +`dependencies` and the worker ran the image's `torch 2.9.1+cu128`. The first cold start +downloaded the base model in 10.3 s; a later cold start restored 2.2 GB through `VolumeCache` +and loaded it in 7.2 s. Run it with `rp flash deploy` (flash SKILL gotcha 14 covers dev +sessions on older runtime images). The `runpod-flash` live run used a **minimal peft + transformers `Trainer`** script (below) — it reuses the pod template's torch (no dependency fights) and is the fastest small-model path; the **axolotl** flow is documented alongside it as the richer option for scaling. @@ -86,7 +87,7 @@ robust).** Reuses the template's torch, installs only `peft` + `datasets`, train on a 200-row slice of `mhenrichsen/alpaca_2k_test`. Whole thing (install + download + train) ran in **~5 min** on one 4090: ```bash -export HF_HOME=/workspace/hf-cache # cache base model on the VOLUME (reused at serve) +export HF_HOME=/workspace/hf-cache # cache base model on the VOLUME (the runpod-flash serve run reused it) pip install --break-system-packages -q peft datasets python3 - <<'PY' import torch @@ -163,8 +164,15 @@ Mind the two things from 07: mount path is `/runpod-volume`, and the worker call as `generate(**job_input)`, so the `input` keys are its parameter names (empty input is rejected). -**Dependencies:** `peft` and `accelerate` go in `dependencies` like any package; the build leaves -their `torch` dependency to the GPU image (flash SKILL gotcha 13). +**Model loading:** the adapter is small, so the worker reads it straight from the volume through +`vol.path`. The base model is the large part: keep its Hugging Face cache on the worker's own +disk and wrap the load in `VolumeCache`, so later cold starts copy it from the volume to local +disk instead of downloading it or reading it over the network mount (flash +[patterns](../../flash/reference/patterns.md#loading-ml-models)). The first SDK cold start +downloads the base model from Hugging Face even though training cached it on the volume; that +download fills the `VolumeCache` mirror, and later cold starts restore from it. `peft` and +`accelerate` go in `dependencies` like any package; the build leaves their `torch` dependency to +the GPU image (flash SKILL gotcha 13). The `runpod-flash` handler that ran used a function with a `global` cache and `**kwargs`. The SDK version keeps the same model code; the whole module ships, so the paths can be @@ -177,23 +185,25 @@ app = runpod.App("ft-serve") vol = runpod.NetworkVolume("", create=False) # the SAME volume, by id BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0" +HF_CACHE = "/opt/hf-cache" # on the worker's disk, not the volume _M = _T = None @app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="", - mounts={"/runpod-volume": vol}, - env={"HF_HOME": "/runpod-volume/hf-cache"}, # reuse training's base-model cache + mounts={"/runpod-volume": vol}, env={"HF_HOME": HF_CACHE}, dependencies=["transformers", "peft", "accelerate"]) def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict: import torch from peft import PeftModel + from runpod.serverless import VolumeCache from transformers import AutoModelForCausalLM, AutoTokenizer global _M, _T adapter = str(vol.path / "outputs" / "lora-out") # written by the pod's training run if _M is None: # load once per worker _T = AutoTokenizer.from_pretrained(adapter) - base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16).to("cuda") + with VolumeCache(dirs=[HF_CACHE]): # copy the base model's cache to local disk + base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16).to("cuda") _M = PeftModel.from_pretrained(base, adapter).eval() text = f"### Instruction:\n{prompt}\n\n### Response:\n" ids = _T(text, return_tensors="pt").to("cuda") diff --git a/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/README.md b/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/README.md index 93582cfb..f3aa9747 100644 --- a/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/README.md @@ -1,7 +1,7 @@ --- lanes: [runpodctl, docker, ssh, rest, runpod-mcp] mcp: partial -concepts: [pod, template, serverless-endpoint, serverless-handler, container-image, worker, network-volume, log-stream] +concepts: [pod, template, serverless-endpoint, serverless-handler, container-image, worker, network-volume, log-stream, volume-cache] --- # Golden path 09 — custom serverless the hard way (iterate in a pod, then flip to serverless) @@ -168,7 +168,7 @@ create the endpoint from that template, scale-to-zero: runpodctl template create --name whisper-dualmode-sl --serverless \ --image /whisper-dualmode:v1 --container-disk-in-gb 15 \ --env '{"MODE_TO_RUN":"serverless"}' # → template id, e.g. nnbpsuubpi - # heavy models on a volume? also add MODEL_CACHE_DIR=/runpod-volume/whisper-cache here + # heavy models on a volume? wrap the model load in VolumeCache (see below) # 2. endpoint from that template (gpu-id, not gpu-type; workers-min 0 = scale-to-zero) runpodctl serverless create --name whisper-custom \ @@ -209,14 +209,27 @@ result: the dual-mode invariant holds end to end. ## Heavy things → network volume (don't bake giant weights) Small models (whisper `base` ≈ 140 MB) are fine baked in. For **large** weights -(`large-v3`, LLMs, LoRA stacks) don't fatten the image — cache them on a **network -volume** and point the handler at it. That's golden path -[07](../07-network-volume-handoff.md)'s handoff: -- Pod mounts the volume at **`/workspace`**; serverless sees the **same** volume at - **`/runpod-volume`** (the template's `MODEL_CACHE_DIR` already defaults to - `/runpod-volume/...` when that path exists). -- Download once (on the pod, or the first worker), reuse across all workers — no - re-download per cold start. +(`large-v3`, LLMs, LoRA stacks) don't fatten the image — keep a copy on a **network +volume** and restore it to local disk on each cold start. The volume is the same one golden +path [07](../07-network-volume-handoff.md) hands off: the pod sees it at **`/workspace`**, +serverless at **`/runpod-volume`**. +- Keep the model cache on the container disk (the handler's default, `/app/whisper-cache`), + not under `/runpod-volume`. Loading straight off the mount reads every byte over the + network, which measured slower than restoring the files to local disk first. +- Wrap the load in `VolumeCache` (Runpod Python SDK `runpod>=1.12.0`) inside `handler.py`. + A cold worker copies the files an earlier worker saved, loads from local disk, and saves + anything new back to the volume. This step was not part of the 2026-07-10 live run. + ```python + from runpod.serverless import VolumeCache + + with VolumeCache(dirs=[MODEL_CACHE_DIR]): # replaces the plain _MODEL = WhisperModel(...) + _MODEL = WhisperModel( + MODEL_SIZE, device=_DEVICE, compute_type=_COMPUTE, download_root=MODEL_CACHE_DIR + ) + ``` +- Size `--container-disk-in-gb` for the whole model plus headroom; the restore writes there. + Full behavior and measured numbers: + [model-caching](../../../runpodctl/reference/model-caching.md#network-volume--load-from-local-disk-with-volumecache). ## Verify it works (all three passed live, 2026-07-10) 1. ✅ `python handler.py` on the pod printed a correct JFK transcription (`en`, 11.0 s). diff --git a/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/template/handler.py b/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/template/handler.py index f48257f8..d7f5b61d 100644 --- a/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/template/handler.py +++ b/plugins/runpod/skills/runpod/golden-paths/09-custom-serverless-dev-loop/template/handler.py @@ -25,13 +25,10 @@ MODE_TO_RUN = os.getenv("MODE_TO_RUN", "pod") MODEL_SIZE = os.getenv("WHISPER_MODEL", "base") # tiny|base|small|medium|large-v3 -# Cache the model weights on the network volume when one is mounted, so every -# serverless worker (and pod restart) reuses the same download instead of pulling -# it again. /runpod-volume is the serverless mount; fall back to local disk. -MODEL_CACHE_DIR = os.getenv( - "MODEL_CACHE_DIR", - "/runpod-volume/whisper-cache" if os.path.isdir("/runpod-volume") else "/app/whisper-cache", -) +# Keep the model cache on local disk. Loading weights straight off a network volume +# reads every byte over the network; to reuse downloads across workers, wrap the load +# below in VolumeCache (see the README's "Heavy things -> network volume"). +MODEL_CACHE_DIR = os.getenv("MODEL_CACHE_DIR", "/app/whisper-cache") # --- Load the model ONCE, at import (cold-start rule). Runs in BOTH modes. ------- print("------- BOOT -------") diff --git a/plugins/runpod/skills/runpod/golden-paths/README.md b/plugins/runpod/skills/runpod/golden-paths/README.md index 7641264e..8c5135f5 100644 --- a/plugins/runpod/skills/runpod/golden-paths/README.md +++ b/plugins/runpod/skills/runpod/golden-paths/README.md @@ -75,7 +75,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified | | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified | | 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16, reading the literal `/runpod-volume` path; the file now reads `vol.path`) | -| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK serve step ported (the 2026-10-17 SDK run loaded the base model through `VolumeCache`) | +| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step with `VolumeCache` via `rp flash deploy` (2026-10-17, pre-release `runpod` 2.0 build) | | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified | | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified | | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified | diff --git a/plugins/runpod/skills/runpodctl/reference/model-caching.md b/plugins/runpod/skills/runpodctl/reference/model-caching.md index e8eb63a6..10bc66e9 100644 --- a/plugins/runpod/skills/runpodctl/reference/model-caching.md +++ b/plugins/runpod/skills/runpodctl/reference/model-caching.md @@ -157,8 +157,31 @@ runpod.serverless.start({"handler": handler}) - **Populate with one worker.** Workers that cold-start together on an empty mirror each download and sync a full copy back. -Measure it like the rest of this page: time a cold start with and without it on your model, -in your data center. Exact rules: [`volume-cache`](../../runpod-usage/concepts/volume-cache.yaml). +- **A global volume works too.** Mount it where the network volume would go (`/runpod-volume` + on an endpoint; GPU compute only) and the same code mirrors there. Use `runpod>=2.0.1`: + older versions can skip or drop large files on a global volume and swallow the error, so the + worker downloads the model again. Use one when the endpoint spans data centers. + +### Measured cold starts + +What a fresh worker spent getting Qwen2.5-1.5B-Instruct (3.1 GB) from storage onto the GPU, +measured on Runpod Serverless (L4 and RTX 4090 workers, 2026-10-17): + +| Where the weights come from | Download or restore | Load | Total | +|---|---|---|---| +| Download from Hugging Face every cold start | — | — | 16–21 s | +| Load straight from a network volume | — | 11 s | 11 s | +| `VolumeCache` restore from a network volume | 5.2 s | 3.9 s | **9.1 s** | +| `VolumeCache` restore from a global volume (before 2.0.1) | 26–51 s | 2–4 s | 30–55 s | + +- **Copy first, then load.** The restore read the volume at about 600 MB/s, and loading from + local disk is quick. Loading straight off the mount reads every byte over the network. +- **Global volumes were the slowest source.** These restores copied whole files and were several + times slower than from a network volume. 2.0.1 restores in parallel byte ranges, so re-time a + global volume on 2.0.1 before ruling it out. +- Treat these as direction, not a spec: time your own model in your data center. + +Exact rules: [`volume-cache`](../../runpod-usage/concepts/volume-cache.yaml). ## HF model cache — `--model-reference` From 81c2d6994b693e1d17051008b13ea7365c0739eb Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 23 Oct 2026 15:02:00 -0400 Subject: [PATCH 17/18] docs(volume-cache): address review round 5 Global-volume advice in patterns.md says its timings predate 2.0.1's parallel restore; api.md shows VolumeCache's /runpod-volume default and when to pass volume_path; the global-volume load time is ~4 s, with the hydrate-plus-load totals in the evidence. --- plugins/runpod/skills/flash/reference/api.md | 7 ++++--- plugins/runpod/skills/flash/reference/patterns.md | 5 +++-- .../runpod/skills/runpod-usage/concepts/volume-cache.yaml | 2 +- plugins/runpod/skills/runpodctl/reference/model-caching.md | 2 +- 4 files changed, 9 insertions(+), 7 deletions(-) diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index a1b77b9c..26db9a50 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -141,9 +141,10 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must - A new network volume is placed in a data center that has network storage and stock for every resource sharing it; resources using it are scheduled in that data center. - Read `models.path` **inside** the remote function. Volumes are not mounted locally. -- Load model weights through `VolumeCache(dirs=[...], volume_path=str(models.path))`, which - copies them to the worker's disk, rather than from the mount - ([patterns.md](patterns.md#loading-ml-models)). +- Load model weights through `VolumeCache(dirs=[...])`, which copies them to the worker's disk, + rather than from the mount ([patterns.md](patterns.md#loading-ml-models)). Its `volume_path` + defaults to `/runpod-volume`; pass `volume_path=str(models.path)` when the volume mounts + elsewhere, as on a task. - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. ## Models and secrets diff --git a/plugins/runpod/skills/flash/reference/patterns.md b/plugins/runpod/skills/flash/reference/patterns.md index 6a9cb881..1066e1ef 100644 --- a/plugins/runpod/skills/flash/reference/patterns.md +++ b/plugins/runpod/skills/flash/reference/patterns.md @@ -112,8 +112,9 @@ start unless you cache. Pick one: The same code works on a `runpod.GlobalVolume` (GPU resources only): mount it at `/runpod-volume` and `VolumeCache` mirrors there. Restores from a global volume measured - slower than from a network volume, so prefer a network volume when the endpoint stays in one - data center ([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)). + slower than from a network volume before 2.0.1's parallel restore; re-time on your version, + and prefer a network volume when the endpoint stays in one data center + ([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)). **Don't load weights straight off the mount** (`HF_HOME=/runpod-volume/...` or `from_pretrained("/runpod-volume/model")`). Every read then crosses the network; restoring with diff --git a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml index df0a5eea..4df9753b 100644 --- a/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml +++ b/plugins/runpod/skills/runpod-usage/concepts/volume-cache.yaml @@ -150,7 +150,7 @@ rules: status: verified see: [volume-cache.best-effort-hides-failures] evidence: - - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct, GlobalVolume mounted at /runpod-volume: fresh-worker hydrate restored all 3.10 GB in 26.2 s, 46.6 s and 51.4 s", seen: "2026-10-17" } + - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct, GlobalVolume mounted at /runpod-volume: fresh-worker hydrate restored all 3.10 GB in 26.2 s, 46.6 s and 51.4 s; hydrate plus model load 30-55 s", seen: "2026-10-17" } - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L292", seen: "2026-10-22", note: "copy2 copies mtime and mode after the data; the global volume rejects utime/chmod, so big files failed to copy. The fix copies the data and treats the metadata step as optional." } - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L389", seen: "2026-10-22", note: "Hydrate copies big files as parallel byte ranges and retries ENOMEM, EIO and EAGAIN reads, which a global volume returns under whole-file parallel copies." } diff --git a/plugins/runpod/skills/runpodctl/reference/model-caching.md b/plugins/runpod/skills/runpodctl/reference/model-caching.md index 10bc66e9..d441ec69 100644 --- a/plugins/runpod/skills/runpodctl/reference/model-caching.md +++ b/plugins/runpod/skills/runpodctl/reference/model-caching.md @@ -172,7 +172,7 @@ measured on Runpod Serverless (L4 and RTX 4090 workers, 2026-10-17): | Download from Hugging Face every cold start | — | — | 16–21 s | | Load straight from a network volume | — | 11 s | 11 s | | `VolumeCache` restore from a network volume | 5.2 s | 3.9 s | **9.1 s** | -| `VolumeCache` restore from a global volume (before 2.0.1) | 26–51 s | 2–4 s | 30–55 s | +| `VolumeCache` restore from a global volume (before 2.0.1) | 26–51 s | ~4 s | 30–55 s | - **Copy first, then load.** The restore read the volume at about 600 MB/s, and loading from local disk is quick. Loading straight off the mount reads every byte over the network. From 4dd848eb60d229a096ec13bdb6fe95cb49f41486 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 23 Oct 2026 15:04:31 -0400 Subject: [PATCH 18/18] docs(flash): pass a VolumeCache namespace on tasks --- plugins/runpod/skills/flash/reference/api.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plugins/runpod/skills/flash/reference/api.md b/plugins/runpod/skills/flash/reference/api.md index 26db9a50..53c1ec0f 100644 --- a/plugins/runpod/skills/flash/reference/api.md +++ b/plugins/runpod/skills/flash/reference/api.md @@ -143,8 +143,9 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False) # must - Read `models.path` **inside** the remote function. Volumes are not mounted locally. - Load model weights through `VolumeCache(dirs=[...])`, which copies them to the worker's disk, rather than from the mount ([patterns.md](patterns.md#loading-ml-models)). Its `volume_path` - defaults to `/runpod-volume`; pass `volume_path=str(models.path)` when the volume mounts - elsewhere, as on a task. + defaults to `/runpod-volume`. On a task, pass `volume_path=str(models.path)` and a + `namespace=`: a task runs on a pod, which has no `RUNPOD_ENDPOINT_ID`, and without a namespace + the cache silently does nothing. - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing. ## Models and secrets