justinwlin · Oct 23, 2026 · Oct 20, 2026 · Oct 22, 2026 · Oct 22, 2026 · Oct 22, 2026
@@ -178,6 +178,8 @@ HuggingFace models stay in code (see [reference/patterns.md](reference/patterns.
 8. **Volumes outlive everything.** Dev sessions and `rp flash undeploy` delete endpoints, not
    the `NetworkVolume`s they created, and those keep billing. Queue and API resources mount one
    volume at `/runpod-volume`; read `vol.path` inside the remote function, never at import time.
+   Load model weights through `VolumeCache`, not straight off the mount
+   ([reference/patterns.md](reference/patterns.md#loading-ml-models)).
 9. **Tasks bill until they return.** A `.spawn()`ed task that you never `wait()` or `cancel()`
    keeps its Pod running after your script exits.
 10. **Native CUDA libraries go in `dependencies` too**, e.g. faster-whisper needs

@@ -141,6 +141,9 @@ existing = runpod.NetworkVolume("vol_id_or_name", create=False)          # must
 - A new network volume is placed in a data center that has network storage and stock for every
   resource sharing it; resources using it are scheduled in that data center.
 - Read `models.path` **inside** the remote function. Volumes are not mounted locally.
+- Load model weights through `VolumeCache(dirs=[...], volume_path=str(models.path))`, which
+  copies them to the worker's disk, rather than from the mount
+  ([patterns.md](patterns.md#loading-ml-models)).
 - Undeploying an app or ending a dev session leaves its volumes in place, and they keep billing.
 
 ## Models and secrets

@@ -110,6 +110,16 @@ start unless you cache. Pick one:
   `runpod.serverless.VolumeCache` is importable on the worker without listing `runpod` in
   `dependencies`.
 
+  The same code works on a `runpod.GlobalVolume` (GPU resources only): mount it at
+  `/runpod-volume` and `VolumeCache` mirrors there. Restores from a global volume measured
+  slower than from a network volume, so prefer a network volume when the endpoint stays in one
+  data center ([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)).
+
+**Don't load weights straight off the mount** (`HF_HOME=/runpod-volume/...` or
+`from_pretrained("/runpod-volume/model")`). Every read then crosses the network; restoring with
+`VolumeCache` and loading from local disk measured faster
+([model-caching](../../runpodctl/reference/model-caching.md#measured-cold-starts)).
+
 `VolumeCache` details that bite:
 - **Best-effort.** Cache errors are logged and swallowed, and the worker cold-starts normally,
   so a broken cache looks like a slow cold start. Use `VolumeCache(..., best_effort=False)`
@@ -120,6 +130,10 @@ start unless you cache. Pick one:
   headroom; the default is 30 GB on GPU.
 - Let one worker populate the mirror first. Workers that cold-start together on an empty mirror
   each download and sync a full copy.
+- The mirror is keyed by endpoint ID (`namespace` defaults to `RUNPOD_ENDPOINT_ID`). Every dev
+  session creates new endpoints, so each session starts from an empty mirror and leaves its copy
+  on the volume. Pass a fixed `namespace="qwen-7b"` to share one mirror across dev sessions and
+  deploys.
 
 Full behavior: [`concepts/volume-cache.yaml`](../../runpod-usage/concepts/volume-cache.yaml).
 

@@ -124,3 +124,44 @@ rules:
     see: [network-volume.concurrent-writes]
     evidence:
       - { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" }
+
+  - id: volume-cache.restore-beats-mount-load
+    statement: >
+      With Qwen2.5-1.5B (3.1 GB), a cold start that restored the model with
+      `VolumeCache` and loaded it from local disk measured faster than loading
+      it straight from the network volume or downloading it: restore 5.2 s plus
+      load 3.9 s, against 11 s loading from the mount and 16-21 s downloading
+      from Hugging Face and loading. Smaller models or a faster mount can narrow
+      or flip the gap, so time your own model.
+    status: verified
+    evidence:
+      - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct on L4/RTX 4090 workers: fresh-worker VolumeCache hydrate 5.2 s + load 3.9 s (3.10 GB restored); load from /runpod-volume 10.7-11.6 s; Hugging Face download + load 16.3-21.4 s", seen: "2026-10-17" }
+
+  - id: volume-cache.global-volume
+    statement: >
+      `VolumeCache` mirrors onto a global volume mounted at its `volume_path`
+      (`/runpod-volume` on a GPU endpoint). Use `runpod>=2.0.1`. Before
+      2.0.0rc2, files of 256 KiB or more failed to copy onto a global volume.
+      Before 2.0.1, a restore from a global volume could drop large files. Both
+      failures are swallowed, so the worker downloads the model again. Restores
+      that copied whole files took 26-51 s for 3.1 GB, against 5.2 s from a
+      network volume. 2.0.1 restores files in parallel byte ranges and retries
+      failed reads.
+    status: verified
+    see: [volume-cache.best-effort-hides-failures]
+    evidence:
+      - { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct, GlobalVolume mounted at /runpod-volume: fresh-worker hydrate restored all 3.10 GB in 26.2 s, 46.6 s and 51.4 s", seen: "2026-10-17" }
+      - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L292", seen: "2026-10-22", note: "copy2 copies mtime and mode after the data; the global volume rejects utime/chmod, so big files failed to copy. The fix copies the data and treats the metadata step as optional." }
+      - { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L389", seen: "2026-10-22", note: "Hydrate copies big files as parallel byte ranges and retries ENOMEM, EIO and EAGAIN reads, which a global volume returns under whole-file parallel copies." }
+
+  - id: volume-cache.namespace-per-endpoint
+    statement: >
+      The mirror is keyed by `namespace`, which defaults to the endpoint ID.
+      Each Runpod SDK dev session creates new endpoints, so it starts from an
+      empty mirror and leaves its copy on the volume. Pass a fixed `namespace=`
+      to share one mirror across dev sessions and deploys.
+    status: documented
+    see: [volume-cache.mirror-sizing, flash.dev-session]
+    evidence:
+      - { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L87", seen: "2026-10-15", note: namespace falls back to RUNPOD_ENDPOINT_ID. }
+      - { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: Dev sessions create temporary endpoints and delete them when the session ends. }
@@ -1,7 +1,7 @@
 ---
 lanes: [runpodctl, ssh, flash, runpod-mcp]
 mcp: partial
-concepts: [pod, network-volume, flash, serverless-endpoint, worker, serverless-handler, data-center, gpu-pool]
+concepts: [pod, network-volume, flash, serverless-endpoint, worker, serverless-handler, data-center, gpu-pool, volume-cache]
 ---
 
 # Golden path 08 — fine-tune → serve (LoRA on a pod → serverless)
@@ -14,13 +14,14 @@ GPU, dataset) — nothing structural changes.
 **Status:** COVERED — live-verified 2026-07-10 end to end. A pod LoRA-trained
 TinyLlama-1.1B (loss 2.07→1.68, ~5 min incl. install + download), wrote the adapter to the
 volume; a `runpod-flash` serverless worker then loaded base + adapter from `/runpod-volume`
-and generated. Section 3 is ported to the Runpod SDK and not yet re-run as written: on
-2026-10-17 a variant that loaded the base model through `VolumeCache` instead of from the mount,
+and generated. Section 3 is ported to the Runpod SDK: on 2026-10-17 the section 3 code,
 deployed with `rp flash deploy` on a pre-release `runpod` 2.0 build, loaded TinyLlama plus a
 LoRA adapter from a network volume and generated text (the adapter was an untrained one saved by
 a helper function, standing in for the pod training step). `peft` and `accelerate` were plain
-`dependencies` and the worker ran the image's `torch 2.9.1+cu128`. Run it with
-`rp flash deploy` (flash SKILL gotcha 14 covers dev sessions on older runtime images).
+`dependencies` and the worker ran the image's `torch 2.9.1+cu128`. The first cold start
+downloaded the base model in 10.3 s; a later cold start restored 2.2 GB through `VolumeCache`
+and loaded it in 7.2 s. Run it with `rp flash deploy` (flash SKILL gotcha 14 covers dev
+sessions on older runtime images).
 The `runpod-flash` live run used a **minimal peft + transformers `Trainer`** script (below) —
 it reuses the pod template's torch (no dependency fights) and is the fastest small-model
 path; the **axolotl** flow is documented alongside it as the richer option for scaling.
@@ -86,7 +87,7 @@ robust).** Reuses the template's torch, installs only `peft` + `datasets`, train
 on a 200-row slice of `mhenrichsen/alpaca_2k_test`. Whole thing (install + download + train)
 ran in **~5 min** on one 4090:
 ```bash
-export HF_HOME=/workspace/hf-cache                     # cache base model on the VOLUME (reused at serve)
+export HF_HOME=/workspace/hf-cache                     # cache base model on the VOLUME (the runpod-flash serve run reused it)
 pip install --break-system-packages -q peft datasets
 python3 - <<'PY'
 import torch
@@ -163,8 +164,15 @@ Mind the two things from 07: mount path is `/runpod-volume`, and the worker call
 as `generate(**job_input)`, so the `input` keys are its parameter names (empty input is
 rejected).
 
-**Dependencies:** `peft` and `accelerate` go in `dependencies` like any package; the build leaves
-their `torch` dependency to the GPU image (flash SKILL gotcha 13).
+**Model loading:** the adapter is small, so the worker reads it straight from the volume through
+`vol.path`. The base model is the large part: keep its Hugging Face cache on the worker's own
+disk and wrap the load in `VolumeCache`, so later cold starts copy it from the volume to local
+disk instead of downloading it or reading it over the network mount (flash
+[patterns](../../flash/reference/patterns.md#loading-ml-models)). The first SDK cold start
+downloads the base model from Hugging Face even though training cached it on the volume; that
+download fills the `VolumeCache` mirror, and later cold starts restore from it. `peft` and
+`accelerate` go in `dependencies` like any package; the build leaves their `torch` dependency to
+the GPU image (flash SKILL gotcha 13).
 
 The `runpod-flash` handler that ran used a function with a `global` cache and `**kwargs`.
 The SDK version keeps the same model code; the whole module ships, so the paths can be
@@ -177,23 +185,25 @@ app = runpod.App("ft-serve")
 
 vol = runpod.NetworkVolume("<vol-id>", create=False)        # the SAME volume, by id
 BASE = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
+HF_CACHE = "/opt/hf-cache"                                  # on the worker's disk, not the volume
 _M = _T = None
 
 
 @app.queue(gpu="ADA_24", workers=(0, 1), idle_timeout=60, datacenter="<dc>",
-           mounts={"/runpod-volume": vol},
-           env={"HF_HOME": "/runpod-volume/hf-cache"},       # reuse training's base-model cache
+           mounts={"/runpod-volume": vol}, env={"HF_HOME": HF_CACHE},
            dependencies=["transformers", "peft", "accelerate"])
 def generate(prompt: str = "Hello", max_new_tokens: int = 64) -> dict:
     import torch
     from peft import PeftModel
+    from runpod.serverless import VolumeCache
     from transformers import AutoModelForCausalLM, AutoTokenizer
 
     global _M, _T
     adapter = str(vol.path / "outputs" / "lora-out")        # written by the pod's training run
     if _M is None:                                           # load once per worker
         _T = AutoTokenizer.from_pretrained(adapter)
-        base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16).to("cuda")
+        with VolumeCache(dirs=[HF_CACHE]):                   # copy the base model's cache to local disk
+            base = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.float16).to("cuda")
         _M = PeftModel.from_pretrained(base, adapter).eval()
     text = f"### Instruction:\n{prompt}\n\n### Response:\n"
     ids = _T(text, return_tensors="pt").to("cuda")

@@ -1,7 +1,7 @@
 ---
 lanes: [runpodctl, docker, ssh, rest, runpod-mcp]
 mcp: partial
-concepts: [pod, template, serverless-endpoint, serverless-handler, container-image, worker, network-volume, log-stream]
+concepts: [pod, template, serverless-endpoint, serverless-handler, container-image, worker, network-volume, log-stream, volume-cache]
 ---
 
 # Golden path 09 — custom serverless the hard way (iterate in a pod, then flip to serverless)
@@ -168,7 +168,7 @@ create the endpoint from that template, scale-to-zero:
 runpodctl template create --name whisper-dualmode-sl --serverless \
   --image <you>/whisper-dualmode:v1 --container-disk-in-gb 15 \
   --env '{"MODE_TO_RUN":"serverless"}'                  # → template id, e.g. nnbpsuubpi
-  # heavy models on a volume? also add MODEL_CACHE_DIR=/runpod-volume/whisper-cache here
+  # heavy models on a volume? wrap the model load in VolumeCache (see below)
 
 # 2. endpoint from that template (gpu-id, not gpu-type; workers-min 0 = scale-to-zero)
 runpodctl serverless create --name whisper-custom \
@@ -209,14 +209,27 @@ result: the dual-mode invariant holds end to end.
 
 ## Heavy things → network volume (don't bake giant weights)
 Small models (whisper `base` ≈ 140 MB) are fine baked in. For **large** weights
-(`large-v3`, LLMs, LoRA stacks) don't fatten the image — cache them on a **network
-volume** and point the handler at it. That's golden path
-[07](../07-network-volume-handoff.md)'s handoff:
-- Pod mounts the volume at **`/workspace`**; serverless sees the **same** volume at
-  **`/runpod-volume`** (the template's `MODEL_CACHE_DIR` already defaults to
-  `/runpod-volume/...` when that path exists).
-- Download once (on the pod, or the first worker), reuse across all workers — no
-  re-download per cold start.
+(`large-v3`, LLMs, LoRA stacks) don't fatten the image — keep a copy on a **network
+volume** and restore it to local disk on each cold start. The volume is the same one golden
+path [07](../07-network-volume-handoff.md) hands off: the pod sees it at **`/workspace`**,
+serverless at **`/runpod-volume`**.
+- Keep the model cache on the container disk (the handler's default, `/app/whisper-cache`),
+  not under `/runpod-volume`. Loading straight off the mount reads every byte over the
+  network, which measured slower than restoring the files to local disk first.
+- Wrap the load in `VolumeCache` (Runpod Python SDK `runpod>=1.12.0`) inside `handler.py`.
+  A cold worker copies the files an earlier worker saved, loads from local disk, and saves
+  anything new back to the volume. This step was not part of the 2026-07-10 live run.
+  ```python
+  from runpod.serverless import VolumeCache
+
+  with VolumeCache(dirs=[MODEL_CACHE_DIR]):          # replaces the plain _MODEL = WhisperModel(...)
+      _MODEL = WhisperModel(
+          MODEL_SIZE, device=_DEVICE, compute_type=_COMPUTE, download_root=MODEL_CACHE_DIR
+      )
+  ```
+- Size `--container-disk-in-gb` for the whole model plus headroom; the restore writes there.
+  Full behavior and measured numbers:
+  [model-caching](../../../runpodctl/reference/model-caching.md#network-volume--load-from-local-disk-with-volumecache).
 
 ## Verify it works (all three passed live, 2026-07-10)
 1. ✅ `python handler.py` on the pod printed a correct JFK transcription (`en`, 11.0 s).

@@ -25,13 +25,10 @@
 MODE_TO_RUN = os.getenv("MODE_TO_RUN", "pod")
 MODEL_SIZE = os.getenv("WHISPER_MODEL", "base")  # tiny|base|small|medium|large-v3
 
-# Cache the model weights on the network volume when one is mounted, so every
-# serverless worker (and pod restart) reuses the same download instead of pulling
-# it again. /runpod-volume is the serverless mount; fall back to local disk.
-MODEL_CACHE_DIR = os.getenv(
-    "MODEL_CACHE_DIR",
-    "/runpod-volume/whisper-cache" if os.path.isdir("/runpod-volume") else "/app/whisper-cache",
-)
+# Keep the model cache on local disk. Loading weights straight off a network volume
+# reads every byte over the network; to reuse downloads across workers, wrap the load
+# below in VolumeCache (see the README's "Heavy things -> network volume").
+MODEL_CACHE_DIR = os.getenv("MODEL_CACHE_DIR", "/app/whisper-cache")
 
 # --- Load the model ONCE, at import (cold-start rule). Runs in BOTH modes. -------
 print("------- BOOT -------")

@@ -75,7 +75,7 @@ observed output) → Gotchas we hit → Cost & cleanup → Skill gaps folded bac
 | 05 | [Custom model → serverless endpoint](05-model-to-endpoint-pipeline.md) | cross-lane pipeline | hf → docker → runpodctl | ✅ live-verified |
 | 06 | [Interactive dev pod (SSH / VS Code)](06-dev-pod.md) | pod / interactive | runpodctl pod + volume | ✅ live-verified |
 | 07 | [Network-volume handoff (pod → volume → serverless)](07-network-volume-handoff.md) | pod + serverless | runpodctl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK volume handoff via `rp flash deploy` (2026-10-16, reading the literal `/runpod-volume` path; the file now reads `vol.path`) |
-| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; 🔄 SDK serve step ported (the 2026-10-17 SDK run loaded the base model through `VolumeCache`) |
+| 08 | [Fine-tune → serve (LoRA on a pod → serverless)](08-finetune-to-serverless.md) | train → serve loop | runpodctl + peft/axolotl + Runpod SDK (`rp flash`) | ✅ live-verified with `runpod-flash`; ✅ SDK serve step with `VolumeCache` via `rp flash deploy` (2026-10-17, pre-release `runpod` 2.0 build) |
 | 09 | [Custom serverless dev loop (iterate in a pod → dual-mode image → serverless)](09-custom-serverless-dev-loop/README.md) | custom image / escape hatch | runpodctl + docker (dual-mode `MODE_TO_RUN`) | ✅ live-verified |
 | 10 | [Multi-region HA serverless (multi-volume + data sync)](10-multi-region-ha-serverless.md) | serverless / availability | runpodctl ≥v2.4.0 (`--network-volume-ids`) + S3 API (aws) | ✅ live-verified |
 | 11 | [Public Endpoints (call a ready hosted model)](11-public-endpoints.md) | serverless / hosted | Runpod Public Endpoint API (native + OpenAI-compatible) | ✅ live-verified |