Repository navigation
docs(storage): /workspace can be network-backed; install on the container disk - #98
Conversation
…iner disk Live probes on GPU pods in EU-RO-1, EU-SE-1 and US-MO-2 showed /workspace (the volume disk, no network volume attached) as a fuse mount of the data center network filesystem, including in data centers that list no network-volume support, and the container disk as a local overlay. Direct writes: /workspace 443-713 MB/s, container disk 2.4-4.1 GB/s. - pod-volume-disk: the volume disk is tied to its machine, not a disk inside it; new rule network-filesystem-backed. - container-disk: new rule fastest-local-storage (installs, builds and scratch go here). - machine.host-local-storage and storage.md updated to match. - New eval fast-disk-for-installs.
…ge cluster Probes in OC-AU-1, US-TX-4 and US-PA-1, whose data centers list no storage cluster, showed /workspace as a local xfs Docker volume, as fast as the container disk. The signal is a storage cluster, not network-volume support; findmnt /workspace tells them apart.
Describe where /workspace is network-backed by whether the data center offers network volumes and what findmnt shows, and drop host mount paths and API field names from the probe evidence.
OneAsterisk
left a comment
There was a problem hiding this comment.
Claude review, round 1
Verdict: Approved. The guidance is correct and backed by a clear probe. The four items below are wording and consistency fixes.
Approval gate: 0 Critical, 0 High, 4 Medium+Low → approved.
Existing feedback
No prior reviews or comments. CI: all required checks pass. pytorch-pod-lifecycle and template-examples were skipped (path-filtered, unrelated to concept docs).
PR description vs code
| Claim | Status | Where |
|---|---|---|
pod-volume-disk summary and machine-level-not-data-center-level say "tied to its machine" |
✅ | concepts/pod-volume-disk.yaml:5,31 |
New rule network-filesystem-backed with findmnt check |
✅ | concepts/pod-volume-disk.yaml:42-58 |
New rule container-disk.fastest-local-storage |
✅ | concepts/container-disk.yaml:36-48 |
machine.host-local-storage and reference/storage.md updated to match |
✅ | concepts/machine.yaml:51, reference/storage.md:31-50,160 |
| New eval | ✅ | evals/fast-disk-for-installs.eval.md |
| Other docs agree with the new rules | golden-paths/25-bake-vs-mount/README.md:43-45 still disagrees (Medium below) |
Findings
Medium
- Golden path 25 now contradicts the new container-disk rule.
plugins/runpod/skills/runpod/golden-paths/25-bake-vs-mount/README.md:43-45says that in a network-volume region "the only thing truly on local disk is the image and what's baked into it."container-disk.fastest-local-storagesays runtime writes to/,/root,/optand/tmpare local, and the probe backs that. An agent that reads the golden path will tell users that runtimepip installoutside/workspaceis not local. Reword that callout to "the image and anything written outside the mount path (the container disk) are local; everything under/workspaceis on the network mount." Contributor rule 12 asks for this re-read. (Inline note oncontainer-disk.yaml:39, since the golden path is outside the diff.)
Low
- "Always local, in every data center" generalizes from six data centers.
container-disk.yaml:39,pod-volume-disk.yaml:52,storage.md:49and eval line 14 state it as a universal fact understatus: verified. The probe covered 6 data centers and Secure Cloud GPU pods only. Say "local in every data center probed" or drop "in every data center". Thefindmntcheck already gives agents a way to confirm on the pod. - "Fastest" overclaims. The rule id
container-disk.fastest-local-storage,storage.md:160"(fastest; …)" and eval line 13 "the fastest storage a pod has" all say fastest. The PR's own probe has/workspacefaster on US-PA-1 (578 vs 559 MB/s) and near-equal on OC-AU-1 and US-TX-4. "Always local" is the supported claim. Suggestcontainer-disk.always-localand "as fast as or faster than/workspace." - Single-run throughput figures in a rule statement and a guide.
container-disk.yaml:40-41andstorage.md:33quote 2.4-4.1 GB/s and 443-713 MB/s.concepts/README.md(Writing style) says to avoid figures that change often. One 512 MBddper pod varies with hardware and load. Keep the numbers in the evidenceref(already there) and say "several times faster" in the statement, whichpod-volume-disk.yaml:47already does.
Nits
pod-volume-disk.yaml:51-52"a block device such asxfs":xfsis a filesystem.storage.md:48and the eval say "xfson a device", which is accurate. Match them.evals/fast-disk-for-installs.eval.md:30asserts "Notes the container disk is wiped on stop or restart", but Expected behavior (lines 13-21) never lists it. Add it as step 5 so the two sections match.
Simplification opportunities
None worth a change. The new rules link with see: instead of restating each other.
Verified clean
--container-disk-in-gbexists intestdata/runpodctl/command-surface.json.- Evidence entries carry
source,refandseenasconcepts/README.mdrequires forlive-probe. No time-bound wording, no internal names. applies_tofields match the ones already used on each concept (field:disk,field:mounts.persistent.size).- The
see:targets resolve (container-disk.what-survives, cross-links between the two new rules). - The "tied to its machine" rewording keeps the existing public-docs evidence valid while it no longer claims a local disk.
building-images.md:140-143andrunpodctl/reference/model-caching.mdstill agree with the new rules.
Questions for the author
docs.runpod.io/pods/storage/types(the Comparison table) lists the volume disk as "Fast (local)". The probe contradicts that. Should the docs team get a fix request?pod-volume-disk.machine-level-not-data-center-levelstill cites that page.machine.host-local-storagekeeps "is lost if the machine fails". Does that hold when/workspaceis afusemount on network storage, or is it now true only for localxfsvolume disks?
Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 22 in / 6,774 out / 624,799 cache read / 70,571 cache write = 702,166 (main session, no sub-agents; diff under 300 lines). Estimated cost: $1.91 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Round 1 of this PR. This is automated assistance; a human owns the final call.
Reword golden path 25 so runtime writes outside /workspace count as local, rename container-disk.fastest-local-storage to always-local, drop the single-run throughput figures from statements, fix the xfs wording and list the wiped-on-stop step in the eval.
OneAsterisk
left a comment
There was a problem hiding this comment.
Claude review, round 2
Verdict: Approved. Every round 1 finding is fixed at 4d796a2, and the new commit adds no new problems.
Approval gate: 0 Critical, 0 High, 0 Medium+Low → approved.
Status of prior findings
| # | Finding | Severity | Status | Evidence |
|---|---|---|---|---|
| R1-1 | Golden path 25 callout contradicts the container-disk rule | Medium | Fixed | golden-paths/25-bake-vs-mount/README.md:43-46 now says writes outside the mount path are local, 4d796a2 |
| R1-2 | "Always local, in every data center" generalizes from six data centers | Low | Fixed | "in every data center" removed from container-disk.yaml:39 and the eval; evidence ref names the six data centers, 4d796a2 |
| R1-3 | "Fastest" overclaims | Low | Fixed | Rule renamed to container-disk.always-local (container-disk.yaml:36); see: updated in pod-volume-disk.yaml:55; storage.md:160 and eval line 13 no longer say fastest. No reference to the old id remains in the repo, 4d796a2 |
| R1-4 | Single-run throughput figures in a statement and a guide | Low | Fixed | container-disk.yaml:39-42 and storage.md:32-33 now say "several times faster"; figures stay in evidence ref, 4d796a2 |
| R1-5 | Nit: xfs called a block device |
Nit | Fixed | pod-volume-disk.yaml:51 says "an xfs filesystem on a device", 4d796a2 |
| R1-6 | Nit: eval assertion missing from Expected behavior | Nit | Fixed | Step 5 added at fast-disk-for-installs.eval.md:22-23, 4d796a2 |
Existing feedback
No comments from other reviewers or bots. CI: all checks pass at 4d796a2, including nix-checks. pytorch-pod-lifecycle and template-examples were skipped (path-filtered, unrelated to concept docs).
PR description vs code
| Claim | Status | Where |
|---|---|---|
pod-volume-disk says "tied to its machine" and adds network-filesystem-backed with a findmnt check |
✅ | concepts/pod-volume-disk.yaml:5,31,42-58 |
New rule container-disk.always-local |
✅ | concepts/container-disk.yaml:36-49 |
machine.host-local-storage, reference/storage.md and golden path 25 updated |
✅ | concepts/machine.yaml:51, reference/storage.md:31-50,160, golden-paths/25-bake-vs-mount/README.md:43-46 |
| Golden path 25 no longer says the image is the only local storage | ✅ | golden-paths/25-bake-vs-mount/README.md:43-46 |
| New eval | ✅ | evals/fast-disk-for-installs.eval.md |
Findings
Nits
golden-paths/25-bake-vs-mount/README.md:45-46"Everything under/workspaceis on the network mount" repeats the callout's first sentence. Suggestion inline.
Simplification opportunities
None.
Verified clean
- The new
always-localstatement matches its evidence: local overlay on all six probed pods, "several times faster" only where/workspaceisfuse, "about the same" where it is localxfs. storage.md, the eval,pod-volume-disk.network-filesystem-backedand golden path 25 now use the samefindmntwording (fuse= network-backed,xfson a device = local).- Eval Expected behavior (steps 1-5) and Assertions now match one to one.
Questions for the author
Both round 1 questions are still open:
docs.runpod.io/pods/storage/typeslists the volume disk as "Fast (local)", and the probe contradicts that. Should the docs team get a fix request?machine.host-local-storagekeeps "is lost if the machine fails". Does that hold when/workspaceis afusemount on network storage?
Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 14 in / 3,322 out / 322,538 cache read / 48,045 cache write = 373,919 (main session, no sub-agents; incremental diff under 300 lines). Estimated cost: $1.21 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Round 2 of this PR. This is automated assistance; a human owns the final call.
…ne-failure wording The volume disk can sit on network storage, so a machine failure makes it unavailable rather than necessarily destroying it.
OneAsterisk
left a comment
There was a problem hiding this comment.
Claude review, round 3
Verdict: Approved. The round 2 nit is fixed, and the machine-failure wording now answers round 2 question 2. One Low consistency gap remains.
Approval gate: 0 Critical, 0 High, 1 Medium+Low → approved.
Status of prior findings
| # | Finding | Severity | Status | Evidence |
|---|---|---|---|---|
| R1-1 | Golden path 25 callout contradicts the container-disk rule | Medium | Fixed (round 2) | golden-paths/25-bake-vs-mount/README.md:43-45, 4d796a2 |
| R1-2 | "Always local, in every data center" generalizes from six data centers | Low | Fixed (round 2) | container-disk.yaml:39, 4d796a2 |
| R1-3 | "Fastest" overclaims | Low | Fixed (round 2) | container-disk.always-local, 4d796a2 |
| R1-4 | Single-run throughput figures in a statement and a guide | Low | Fixed (round 2) | container-disk.yaml:39-42, storage.md:32-33, 4d796a2 |
| R1-5 | Nit: xfs called a block device |
Nit | Fixed (round 2) | pod-volume-disk.yaml:51, 4d796a2 |
| R1-6 | Nit: eval assertion missing from Expected behavior | Nit | Fixed (round 2) | fast-disk-for-installs.eval.md:22-23, 4d796a2 |
| R2-1 | Nit: golden path 25 callout repeats its first sentence | Nit | Fixed | golden-paths/25-bake-vs-mount/README.md:43-45 drops the last sentence, 6f96601 |
Existing feedback
No comments from other reviewers or bots. CI: all checks pass at 6f96601, including nix-checks. pytorch-pod-lifecycle and template-examples were skipped (path-filtered).
PR description vs code
| Claim | Status | Where |
|---|---|---|
pod-volume-disk says "tied to its machine" and adds network-filesystem-backed with a findmnt check |
✅ | concepts/pod-volume-disk.yaml:5,31,42-58 |
New rule container-disk.always-local |
✅ | concepts/container-disk.yaml:36-49 |
machine.host-local-storage, reference/storage.md and golden path 25 updated |
✅ | concepts/machine.yaml:49-55, reference/storage.md, golden-paths/25-bake-vs-mount/README.md:43-45 |
| Golden path 25 no longer says the image is the only local storage | ✅ | golden-paths/25-bake-vs-mount/README.md:43-45 |
| New eval | ✅ | evals/fast-disk-for-installs.eval.md |
Findings
Low
- Machine-failure wording now differs across four places (
concepts/machine.yaml:52).machine.host-local-storagenow says the volume disk "can become unavailable or be lost if the machine fails". Three other places still say it "is lost":pod-volume-disk.yaml:74(deprecated-prefer-network, a file this PR already edits),pod.yaml:149(persistent-mount-limits) andreference/pods-and-machines.md:42. Use the new wording in all four, or at least inpod-volume-disk.yaml:74.
Simplification opportunities
None.
Verified clean
- The golden path 25 callout reads correctly after the cut:
/workspaceis on the network mount, the image and the container disk are local. - The new
machine.host-local-storagewording fits thenetwork-filesystem-backedrule. Afuse-backed/workspacemay become unreachable instead of destroyed. The billing evidence still supports "not billed while the machine is unavailable". - No other file changed since round 2.
Questions for the author
- (Still open from rounds 1 and 2.)
docs.runpod.io/pods/storage/typeslists the volume disk as "Fast (local)", and the probe contradicts that. Should the docs team get a fix request?
Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 16 in / 3,138 out / 409,065 cache read / 57,156 cache write = 469,375 (main session, no sub-agents; 9-line incremental diff). Estimated cost: $1.40 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Usage counted before posting. Round 3 of this PR. This is automated assistance; a human owns the final call.
| the machine is unavailable. Use a network volume for data that must | ||
| outlive the machine. | ||
| The pod's volume disk is tied to its machine. The data stays while the pod | ||
| is stopped, can become unavailable or be lost if the machine fails, and is |
There was a problem hiding this comment.
Low: This wording is better, but three places still say the volume disk "is lost" when the host fails: concepts/pod-volume-disk.yaml:74 (deprecated-prefer-network, already edited in this PR), concepts/pod.yaml:149 (persistent-mount-limits) and reference/pods-and-machines.md:42. Use the same "can become unavailable or be lost" wording there, so an agent does not get two answers to the same question.
Why
The skills said the pod volume disk (
/workspace) is always a local disk. That isn't true everywhere. In data centers that offer network volumes,/workspaceis network-backed and several times slower than the container disk. Data centers without network volumes vary: some give a local/workspace, others don't. The container disk is local in every data center, so packages installed outside/workspaceare fast wherever the pod runs.What changes
pod-volume-disk: the volume disk is tied to its machine (summary andmachine-level-not-data-center-level). New rulenetwork-filesystem-backed: network-backed where network volumes are offered, either way elsewhere, andfindmnt /workspacetells them apart.container-disk: new rulealways-local. Put installs, builds and scratch on the container disk and size it for them, and keep only what must survive a restart in/workspace.machine.host-local-storage,reference/storage.mdand golden path 25's callout updated to match. Golden path 25 no longer says the image is the only local storage in a network-volume region.runpod-usage/evals/fast-disk-for-installs.eval.md.Live probe (2026-10-26)
ubuntu:22.04GPU pods with a 10 GB volume disk and no network volume, runningfindmnt,df -hTanddd if=/dev/zero bs=1M count=512 oflag=direct. All probe pods were deleted./workspace/workspacewritefuse)fuse)fuse)xfsxfsxfsThe root filesystem (container disk) was a local overlay on every pod.
Repo checks: ontology validate
--strict, check:guides, links, branding, frontmatter, versions, marketplace, CLI absence claims, migrate checks.