[justinwlin]

Flash → SDK 2.0 fix stack

Status as of 2026-10-23. 1–5 are merged, and 10 is merged into 9. Hover a link for its status.

#	PR	Fixes	Needs
1	runpod/runpod-python#627	✅ merged: CI e2e fixed on main	—
2	runpod/runpod-python#623	✅ merged: torch pulled in by peft/accelerate no longer bundled	—
3	runpod/runpod-python#624	✅ merged: requested GPU devices are pinned, not whole pools	—
4	runpod/runpod-python#625	✅ merged: rp flash dev deletes its endpoints when stopped by a signal	—
5	runpod/runpod-python#626	✅ merged: vol.path works on deployed workers	—
6	runpod/runpod-python#628	❌ closed as a duplicate of #641, which was also closed unmerged. The API key is still copied into every endpoint	—
7	—	Release runpod 2.0.1 with 2–5 (release PR #594). 2.0.0rc2 (2026-10-20) predates them	2–5 merged ✅
8	runpod-workers/sdk-runtime#1	Rebuild runtime images (dev sessions run an old SDK)	independent
9	#89	Plugin docs for the fixed SDK (flash lane → SDK 2.0)	2.0.1 on PyPI
10	👉 #90 (this PR)	✅ merged into #89: VolumeCache guidance (global volumes, measured numbers, golden paths 08/09)	—

Split out of #89 so the SDK 2.0 port and the VolumeCache guidance can be reviewed separately. Stacked on #89: the base is docs/flash-to-runpod-sdk-2, and the code uses vol.path and runpod>=2.0.1 from it. Retarget to main after #89 merges.

main already recommends VolumeCache for network volumes (#86), and #89 ports that guidance to SDK syntax. This PR adds the rest: global volumes, measured numbers, and the two golden paths that still loaded weights straight off the mount.

What changes

• Golden path 08: the serve step restores the base model through VolumeCache and loads it from local disk, not HF_HOME=/runpod-volume/hf-cache. The small LoRA adapter is still read from the volume.
• Golden path 09: keeps MODEL_CACHE_DIR on container disk and wraps the load in VolumeCache.
• Flash patterns.md, api.md and SKILL gotcha 8: load weights through VolumeCache, not from the mount. They also cover the same code on a runpod.GlobalVolume and the per-endpoint mirror namespace (each dev session starts from an empty mirror unless you pass a fixed one).
• model-caching.md: a global-volume bullet and a Measured cold starts table.
• volume-cache concept: three new rules: restore-beats-mount-load, global-volume (needs runpod>=2.0.1: #620 fixed large-file copies onto a global volume and perf(serverless): range-parallel VolumeCache hydrate runpod-python#621 fixed restores that dropped large files; the 2026-10-17 global-volume timings predate #621 and are labeled so) and namespace-per-endpoint.

Live verification

2026-10-17, a local runpod build with runpod/runpod-python#623–#628:

Item	Result
Golden path 08 serve step with VolumeCache	first cold start downloaded the base model in 10.3 s; a later cold start restored 2.2 GB and loaded in 7.2 s
Storage comparison, Qwen2.5-1.5B (3.1 GB), fresh workers only	VolumeCache from network volume 9.1 s (5.2 restore + 3.9 load), load from mount 11 s, download 16–21 s, VolumeCache from global volume 30–55 s with complete restores

A trial counted only on a fresh worker (worker id + uptime), because redeploys and 150 s waits did not recycle idle workers.

Repo checks: ontology validate --strict (309 rules, 0 errors), check:guides, links, branding, frontmatter, versions, marketplace, CLI absence claims, migrate checks.

[OneAsterisk]

Claude review, round 1

Verdict: Ready after fixes. Gate passed; approve when marked ready. One Medium in golden path 09, where the shipped template still does what the new text says not to do.
Approval gate: 0 Critical, 0 High, 3 Medium+Low → gate passed, but the PR is a draft, so this is a comment, not an approval.

Existing feedback

None. No reviews or comments on the PR yet.

PR description vs code

Claim	Status	Where
GP 08 restores the base model through VolumeCache and loads from local disk; adapter still read from the volume	✅	08-finetune-to-serverless.md:186-205
GP 09 keeps MODEL_CACHE_DIR on container disk and wraps the load in VolumeCache	⚠️ README says so, but template/handler.py:31-34 still defaults to /runpod-volume/whisper-cache and the template create step does not set the variable	09-.../README.md:171,216-226
Flash patterns.md, api.md, SKILL gotcha 8 load through VolumeCache; cover GlobalVolume and namespace	✅	patterns.md:113-136, api.md:144-146, SKILL.md:190-191
model-caching.md adds a global-volume bullet and a Measured cold starts table	✅ numbers match the concept evidence (3.1 GB / 5.2 s ≈ 600 MB/s checks out)	model-caching.md:160-184
Three new concept rules	✅ ids, see: targets, and anchors all resolve	volume-cache.yaml:128-166
#620 fixed large-file copies onto a global volume; #621 fixed restores	✅ 1d7ae99 (#620) is in v2.0.0rc2; 29704a4 (#621) is on main after rc2, so not yet released	runpod-python history
CI	✅ all checks pass (2 skipped by path filter)

Findings

Medium

• GP 09 template contradicts the new guidance. 09-custom-serverless-dev-loop/README.md:216. Details inline.

Low

• GP 08 training step still says its base-model cache is "reused at serve". 08-finetune-to-serverless.md:90 (anchored at :186). The SDK serve step no longer reads /runpod-volume/hf-cache. That copy is now unused billed storage. Details inline.
• The restore-beats-mount-load rule states one measurement as a general fact. volume-cache.yaml:131. Details inline.

Nits

• 09-.../README.md:219 runs to about 150 characters and breaks the wrapping used in the rest of the list.

Verified clean

• VolumeCache signature at runpod-python main: namespace defaults to RUNPOD_ENDPOINT_ID, volume_path defaults to /runpod-volume, so GP 08 omitting volume_path is correct for its mount.
• GlobalVolume on CPU raises "global volumes require gpu compute" (runpod/apps/volume.py:360), which matches "GPU resources only".
• The 256 KiB threshold matches _SMALL_FILE_THRESHOLD in v1.12.0. copy2 was replaced by _copy_metadata in rc2, and ranged hydrate landed after rc2. The pinned-commit evidence links point at the right functions.
• New anchors resolve: #loading-ml-models, #measured-cold-starts, #network-volume--load-from-local-disk-with-volumecache.
• GP 08 and 09 frontmatter both add volume-cache to concepts. The golden-paths README status row for 08 matches the PR body's live run.
• GP 08 leaves Hub access on, so HF snapshot symlinks re-link after hydrate (patterns.md gotcha).

Questions for the author

• Will runpod 2.0.1 be cut from a main that includes 29704a4 (#621)? The global-volume rule and the "before 2.0.1" table row depend on it.
• For GP 08, is it intended that the first SDK cold start re-downloads the base model even though training cached it on the same volume? If yes, a line saying so would stop readers expecting a warm first start.

---

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 24 in / ~11k out / ~0.80M cache read / ~72k cache write ≈ 0.88M. Estimated cost: ~$2.2 using the rate table in the skill's sum_usage.py (not re-checked against a live rate card this run; 5-minute cache-write TTL assumed). Round 1 of this PR. This is automated assistance; a human owns the final call.}

[OneAsterisk]

Claude review, round 2

Verdict: Ready after fixes. Gate passed; approve when marked ready. All round 1 findings are fixed. The fix commit added one stray .pyc file.
Approval gate: 0 Critical, 0 High, 1 Medium+Low → gate passed, but the PR is a draft, so this is a comment, not an approval.

The branch was rebased onto the updated #89 (5925d7d), so I compared this PR's own diff (5925d7d...4c48081) against round 1's (6411449...4450ffc). The only change is 4c48081.

Status of prior findings

#	Finding	Severity	Status	Evidence
R1-1	GP 09 template still defaulted MODEL_CACHE_DIR to the volume; snippet could KeyError	Medium	Fixed	template/handler.py:28-31 defaults to /app/whisper-cache; README.md:216-229 names that default and the snippet uses the handler's own MODEL_CACHE_DIR, MODEL_SIZE, _DEVICE, _COMPUTE, 4c48081
R1-2	GP 08 training step said its cache is "reused at serve"	Low	Fixed	08-finetune-to-serverless.md:90 now scopes that to the runpod-flash run; :171-173 says the first SDK cold start downloads, 4c48081
R1-3	restore-beats-mount-load stated one measurement as a general fact	Low	Fixed	volume-cache.yaml:130-135 names the model, says "measured", and adds "time your own model", 4c48081
R1-nit	Long line in GP 09 list	Nit	Fixed	README.md:216-221 rewrapped

Existing feedback

• No human or bot reviews or comments on the PR. My three round 1 threads were already resolved; I replied "Fixed" in each.

PR description vs code

Claim	Status	Where
GP 08 restores the base model through VolumeCache; adapter read from the volume	✅	08-finetune-to-serverless.md:167-205
GP 09 keeps MODEL_CACHE_DIR on container disk and wraps the load in VolumeCache	✅ now true for both README and template	template/handler.py:31, README.md:216-229
Flash docs, model-caching.md, three concept rules	✅ unchanged since round 1
CI	✅ all checks pass (2 skipped by path filter)

Findings

Low

• Compiled bytecode committed. 09-custom-serverless-dev-loop/template/__pycache__/handler.cpython-314.pyc (added in 4c48081). Details in the file comment.

Verified clean

• VolumeCache on the pod path is a no-op: available needs a namespace and an existing /runpod-volume (rp_volume_cache.py:119,167), so wrapping the import-time load does not break MODE_TO_RUN=pod.
• The step 2 scp and the Dockerfile COPY both name files explicitly, so the stray .pyc does not reach the pod or the image.
• The handler comment's pointer to "Heavy things → network volume" matches the README heading at :210.
• GP 08 status now says the section 3 code itself ran on 2026-10-17. The 10.3 s / 7.2 s numbers match the golden-paths README row and the PR body.

Questions for the author

• (Carried from round 1) PyPI still shows runpod 1.12.0 latest and 2.0.0rc2 as the newest 2.x. Will 2.0.1 be cut from a main that includes #621? The global-volume rule depends on it.
• Out of scope here: base PR #89 (5925d7d) also adds five flash/examples/__pycache__/*.pyc files. Worth removing there too.

---

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: ~30 in / ~7k out / ~0.45M cache read / ~45k cache write ≈ 0.50M (estimated from this session; no sub-agents). Estimated cost: ~$1.3 at Opus list rates assumed ($5/M in, $25/M out, $0.50/M cache read, $6.25/M 5-minute cache write; not re-checked against a live rate card this run). Round 2 of this PR. This is automated assistance; a human owns the final call.}

[OneAsterisk]

Claude review, round 3

Verdict: Approved. The only new commit removes the stray .pyc from round 2, and the PR's content is otherwise unchanged.
Approval gate: 0 Critical, 0 High, 0 Medium+Low → approved.

The branch was rebased onto the updated #89 (9451b6e). git range-diff shows this PR's four earlier commits are identical to the round 2 commits (463af47..4c48081 = 867d035..b4921ac). The only new commit is f2e41e3, so this round reviews that commit and re-checks the round 2 finding.

Status of prior findings

#	Finding	Severity	Status	Evidence
R1-1	GP 09 template defaulted MODEL_CACHE_DIR to the volume	Medium	Fixed (round 2)	unchanged at head
R1-2	GP 08 training step said its cache is "reused at serve"	Low	Fixed (round 2)	unchanged at head
R1-3	restore-beats-mount-load generalized one measurement	Low	Fixed (round 2)	unchanged at head
R2-1	Compiled bytecode template/__pycache__/handler.cpython-314.pyc committed	Low	Fixed	f2e41e3 deletes it. Base 9451b6e adds *.pyc to .gitignore, so it cannot come back.

Existing feedback

• No human or bot reviews or comments on the PR. All four of my threads are resolved. I replied "Fixed" on the R2-1 thread.

PR description vs code

Claim	Status	Where
GP 08, GP 09, Flash docs, model-caching.md, three concept rules	✅ content unchanged since round 2, verified there
Stacked on #89, retarget to main after it merges	✅ base is docs/flash-to-runpod-sdk-2 at 9451b6e
CI	✅ all checks pass (pytorch-pod-lifecycle and template-examples skipped by path filter)

Findings

None.

Verified clean

• f2e41e3 touches only the .pyc. No .pyc files remain in the tree. The one __pycache__ path left is testdata/runpod-migrate/skipped-dirs/__pycache__/legacy.py, which is intentional test data.
• The base update (5925d7d..9451b6e) changes .gitignore, runpod-flash-legacy.md, flash.yaml and removes the five flash/examples/__pycache__ files. None of these overlap this PR's nine files. This also settles round 2's out-of-scope note about those files.

Questions for the author

• (Carried from rounds 1 and 2) PyPI still lists runpod 1.12.0 as latest, with 2.0.0rc2 as the newest 2.x (checked 2026-10-23). The global-volume rule and the vol.path code depend on 2.0.1. Will this PR wait for 2.0.1 to be published, as the stack table says for #89?

---

_{Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 10 in / ~2.6k out / ~215k cache read / ~43k cache write ≈ 260k as of the usage check before writing this review (no sub-agents). Estimated cost: ~$1.0 at Opus list rates assumed ($5/M in, $25/M out, $0.50/M cache read, $6.25/M 5-minute cache write; not re-checked against a live rate card this run). Round 3 of this PR. This is automated assistance; a human owns the final call.}