Skip to content

docs(storage): /workspace can be network-backed; install on the container disk - #98

Merged
justinwlin merged 6 commits into
mainfrom
docs/workspace-on-network-fs
Oct 26, 2026
Merged

justinwlin merged 6 commits into
mainfrom
docs/workspace-on-network-fs

Conversation

@justinwlin

@justinwlin justinwlin commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Why

The skills said the pod volume disk (/workspace) is always a local disk. That isn't true everywhere. In data centers that offer network volumes, /workspace is network-backed and several times slower than the container disk. Data centers without network volumes vary: some give a local /workspace, others don't. The container disk is local in every data center, so packages installed outside /workspace are fast wherever the pod runs.

What changes

  • pod-volume-disk: the volume disk is tied to its machine (summary and machine-level-not-data-center-level). New rule network-filesystem-backed: network-backed where network volumes are offered, either way elsewhere, and findmnt /workspace tells them apart.
  • container-disk: new rule always-local. Put installs, builds and scratch on the container disk and size it for them, and keep only what must survive a restart in /workspace.
  • machine.host-local-storage, reference/storage.md and golden path 25's callout updated to match. Golden path 25 no longer says the image is the only local storage in a network-volume region.
  • New eval runpod-usage/evals/fast-disk-for-installs.eval.md.
Live probe (2026-10-26)

ubuntu:22.04 GPU pods with a 10 GB volume disk and no network volume, running findmnt, df -hT and dd if=/dev/zero bs=1M count=512 oflag=direct. All probe pods were deleted.

Pod Offers network volumes /workspace /workspace write Container disk write
RTX 2000 Ada, EU-RO-1 yes network-backed (fuse) 713 MB/s 2.8 GB/s
A40, EU-SE-1 no network-backed (fuse) 443 MB/s 4.1 GB/s
L4, US-MO-2 no network-backed (fuse) 568 MB/s 2.4 GB/s
L40S, OC-AU-1 no local xfs 5.0 GB/s 5.4 GB/s
L40S, US-TX-4 no local xfs 4.3 GB/s 4.3 GB/s
RTX PRO 6000 MIG, US-PA-1 no local xfs 578 MB/s 559 MB/s

The root filesystem (container disk) was a local overlay on every pod.

Repo checks: ontology validate --strict, check:guides, links, branding, frontmatter, versions, marketplace, CLI absence claims, migrate checks.

…iner disk

Live probes on GPU pods in EU-RO-1, EU-SE-1 and US-MO-2 showed /workspace (the volume disk, no network volume attached) as a fuse mount of the data center network filesystem, including in data centers that list no network-volume support, and the container disk as a local overlay. Direct writes: /workspace 443-713 MB/s, container disk 2.4-4.1 GB/s.

- pod-volume-disk: the volume disk is tied to its machine, not a disk inside it; new rule network-filesystem-backed.
- container-disk: new rule fastest-local-storage (installs, builds and scratch go here).
- machine.host-local-storage and storage.md updated to match.
- New eval fast-disk-for-installs.
…ge cluster

Probes in OC-AU-1, US-TX-4 and US-PA-1, whose data centers list no storage cluster, showed /workspace as a local xfs Docker volume, as fast as the container disk. The signal is a storage cluster, not network-volume support; findmnt /workspace tells them apart.
@justinwlin justinwlin changed the title docs(storage): /workspace is network-backed; do installs on the container disk docs(storage): /workspace is network-backed where the data center runs a storage cluster Oct 26, 2026
Describe where /workspace is network-backed by whether the data center
offers network volumes and what findmnt shows, and drop host mount paths
and API field names from the probe evidence.
@justinwlin justinwlin changed the title docs(storage): /workspace is network-backed where the data center runs a storage cluster docs(storage): /workspace can be network-backed; install on the container disk Oct 26, 2026

@OneAsterisk OneAsterisk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review, round 1

Verdict: Approved. The guidance is correct and backed by a clear probe. The four items below are wording and consistency fixes.
Approval gate: 0 Critical, 0 High, 4 Medium+Low → approved.

Existing feedback

No prior reviews or comments. CI: all required checks pass. pytorch-pod-lifecycle and template-examples were skipped (path-filtered, unrelated to concept docs).

PR description vs code

Claim Status Where
pod-volume-disk summary and machine-level-not-data-center-level say "tied to its machine" ✅ concepts/pod-volume-disk.yaml:5,31
New rule network-filesystem-backed with findmnt check ✅ concepts/pod-volume-disk.yaml:42-58
New rule container-disk.fastest-local-storage ✅ concepts/container-disk.yaml:36-48
machine.host-local-storage and reference/storage.md updated to match ✅ concepts/machine.yaml:51, reference/storage.md:31-50,160
New eval ✅ evals/fast-disk-for-installs.eval.md
Other docs agree with the new rules ⚠️ golden-paths/25-bake-vs-mount/README.md:43-45 still disagrees (Medium below)

Findings

Medium

  • Golden path 25 now contradicts the new container-disk rule. plugins/runpod/skills/runpod/golden-paths/25-bake-vs-mount/README.md:43-45 says that in a network-volume region "the only thing truly on local disk is the image and what's baked into it." container-disk.fastest-local-storage says runtime writes to /, /root, /opt and /tmp are local, and the probe backs that. An agent that reads the golden path will tell users that runtime pip install outside /workspace is not local. Reword that callout to "the image and anything written outside the mount path (the container disk) are local; everything under /workspace is on the network mount." Contributor rule 12 asks for this re-read. (Inline note on container-disk.yaml:39, since the golden path is outside the diff.)

Low

  • "Always local, in every data center" generalizes from six data centers. container-disk.yaml:39, pod-volume-disk.yaml:52, storage.md:49 and eval line 14 state it as a universal fact under status: verified. The probe covered 6 data centers and Secure Cloud GPU pods only. Say "local in every data center probed" or drop "in every data center". The findmnt check already gives agents a way to confirm on the pod.
  • "Fastest" overclaims. The rule id container-disk.fastest-local-storage, storage.md:160 "(fastest; …)" and eval line 13 "the fastest storage a pod has" all say fastest. The PR's own probe has /workspace faster on US-PA-1 (578 vs 559 MB/s) and near-equal on OC-AU-1 and US-TX-4. "Always local" is the supported claim. Suggest container-disk.always-local and "as fast as or faster than /workspace."
  • Single-run throughput figures in a rule statement and a guide. container-disk.yaml:40-41 and storage.md:33 quote 2.4-4.1 GB/s and 443-713 MB/s. concepts/README.md (Writing style) says to avoid figures that change often. One 512 MB dd per pod varies with hardware and load. Keep the numbers in the evidence ref (already there) and say "several times faster" in the statement, which pod-volume-disk.yaml:47 already does.

Nits

  • pod-volume-disk.yaml:51-52 "a block device such as xfs": xfs is a filesystem. storage.md:48 and the eval say "xfs on a device", which is accurate. Match them.
  • evals/fast-disk-for-installs.eval.md:30 asserts "Notes the container disk is wiped on stop or restart", but Expected behavior (lines 13-21) never lists it. Add it as step 5 so the two sections match.

Simplification opportunities

None worth a change. The new rules link with see: instead of restating each other.

Verified clean

  • --container-disk-in-gb exists in testdata/runpodctl/command-surface.json.
  • Evidence entries carry source, ref and seen as concepts/README.md requires for live-probe. No time-bound wording, no internal names.
  • applies_to fields match the ones already used on each concept (field:disk, field:mounts.persistent.size).
  • The see: targets resolve (container-disk.what-survives, cross-links between the two new rules).
  • The "tied to its machine" rewording keeps the existing public-docs evidence valid while it no longer claims a local disk.
  • building-images.md:140-143 and runpodctl/reference/model-caching.md still agree with the new rules.

Questions for the author

  1. docs.runpod.io/pods/storage/types (the Comparison table) lists the volume disk as "Fast (local)". The probe contradicts that. Should the docs team get a fix request? pod-volume-disk.machine-level-not-data-center-level still cites that page.
  2. machine.host-local-storage keeps "is lost if the machine fails". Does that hold when /workspace is a fuse mount on network storage, or is it now true only for local xfs volume disks?

Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 22 in / 6,774 out / 624,799 cache read / 70,571 cache write = 702,166 (main session, no sub-agents; diff under 300 lines). Estimated cost: $1.91 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Round 1 of this PR. This is automated assistance; a human owns the final call.

Reword golden path 25 so runtime writes outside /workspace count as local,
rename container-disk.fastest-local-storage to always-local, drop the
single-run throughput figures from statements, fix the xfs wording and
list the wiped-on-stop step in the eval.

@OneAsterisk OneAsterisk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review, round 2

Verdict: Approved. Every round 1 finding is fixed at 4d796a2, and the new commit adds no new problems.
Approval gate: 0 Critical, 0 High, 0 Medium+Low → approved.

Status of prior findings

# Finding Severity Status Evidence
R1-1 Golden path 25 callout contradicts the container-disk rule Medium Fixed golden-paths/25-bake-vs-mount/README.md:43-46 now says writes outside the mount path are local, 4d796a2
R1-2 "Always local, in every data center" generalizes from six data centers Low Fixed "in every data center" removed from container-disk.yaml:39 and the eval; evidence ref names the six data centers, 4d796a2
R1-3 "Fastest" overclaims Low Fixed Rule renamed to container-disk.always-local (container-disk.yaml:36); see: updated in pod-volume-disk.yaml:55; storage.md:160 and eval line 13 no longer say fastest. No reference to the old id remains in the repo, 4d796a2
R1-4 Single-run throughput figures in a statement and a guide Low Fixed container-disk.yaml:39-42 and storage.md:32-33 now say "several times faster"; figures stay in evidence ref, 4d796a2
R1-5 Nit: xfs called a block device Nit Fixed pod-volume-disk.yaml:51 says "an xfs filesystem on a device", 4d796a2
R1-6 Nit: eval assertion missing from Expected behavior Nit Fixed Step 5 added at fast-disk-for-installs.eval.md:22-23, 4d796a2

Existing feedback

No comments from other reviewers or bots. CI: all checks pass at 4d796a2, including nix-checks. pytorch-pod-lifecycle and template-examples were skipped (path-filtered, unrelated to concept docs).

PR description vs code

Claim Status Where
pod-volume-disk says "tied to its machine" and adds network-filesystem-backed with a findmnt check ✅ concepts/pod-volume-disk.yaml:5,31,42-58
New rule container-disk.always-local ✅ concepts/container-disk.yaml:36-49
machine.host-local-storage, reference/storage.md and golden path 25 updated ✅ concepts/machine.yaml:51, reference/storage.md:31-50,160, golden-paths/25-bake-vs-mount/README.md:43-46
Golden path 25 no longer says the image is the only local storage ✅ golden-paths/25-bake-vs-mount/README.md:43-46
New eval ✅ evals/fast-disk-for-installs.eval.md

Findings

Nits

  • golden-paths/25-bake-vs-mount/README.md:45-46 "Everything under /workspace is on the network mount" repeats the callout's first sentence. Suggestion inline.

Simplification opportunities

None.

Verified clean

  • The new always-local statement matches its evidence: local overlay on all six probed pods, "several times faster" only where /workspace is fuse, "about the same" where it is local xfs.
  • storage.md, the eval, pod-volume-disk.network-filesystem-backed and golden path 25 now use the same findmnt wording (fuse = network-backed, xfs on a device = local).
  • Eval Expected behavior (steps 1-5) and Assertions now match one to one.

Questions for the author

Both round 1 questions are still open:

  1. docs.runpod.io/pods/storage/types lists the volume disk as "Fast (local)", and the probe contradicts that. Should the docs team get a fix request?
  2. machine.host-local-storage keeps "is lost if the machine fails". Does that hold when /workspace is a fuse mount on network storage?

Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 14 in / 3,322 out / 322,538 cache read / 48,045 cache write = 373,919 (main session, no sub-agents; incremental diff under 300 lines). Estimated cost: $1.21 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Round 2 of this PR. This is automated assistance; a human owns the final call.

…ne-failure wording

The volume disk can sit on network storage, so a machine failure makes it
unavailable rather than necessarily destroying it.
@justinwlin
justinwlin merged commit 242709b into main Oct 26, 2026
8 checks passed

@OneAsterisk OneAsterisk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review, round 3

Verdict: Approved. The round 2 nit is fixed, and the machine-failure wording now answers round 2 question 2. One Low consistency gap remains.
Approval gate: 0 Critical, 0 High, 1 Medium+Low → approved.

Status of prior findings

# Finding Severity Status Evidence
R1-1 Golden path 25 callout contradicts the container-disk rule Medium Fixed (round 2) golden-paths/25-bake-vs-mount/README.md:43-45, 4d796a2
R1-2 "Always local, in every data center" generalizes from six data centers Low Fixed (round 2) container-disk.yaml:39, 4d796a2
R1-3 "Fastest" overclaims Low Fixed (round 2) container-disk.always-local, 4d796a2
R1-4 Single-run throughput figures in a statement and a guide Low Fixed (round 2) container-disk.yaml:39-42, storage.md:32-33, 4d796a2
R1-5 Nit: xfs called a block device Nit Fixed (round 2) pod-volume-disk.yaml:51, 4d796a2
R1-6 Nit: eval assertion missing from Expected behavior Nit Fixed (round 2) fast-disk-for-installs.eval.md:22-23, 4d796a2
R2-1 Nit: golden path 25 callout repeats its first sentence Nit Fixed golden-paths/25-bake-vs-mount/README.md:43-45 drops the last sentence, 6f96601

Existing feedback

No comments from other reviewers or bots. CI: all checks pass at 6f96601, including nix-checks. pytorch-pod-lifecycle and template-examples were skipped (path-filtered).

PR description vs code

Claim Status Where
pod-volume-disk says "tied to its machine" and adds network-filesystem-backed with a findmnt check ✅ concepts/pod-volume-disk.yaml:5,31,42-58
New rule container-disk.always-local ✅ concepts/container-disk.yaml:36-49
machine.host-local-storage, reference/storage.md and golden path 25 updated ✅ concepts/machine.yaml:49-55, reference/storage.md, golden-paths/25-bake-vs-mount/README.md:43-45
Golden path 25 no longer says the image is the only local storage ✅ golden-paths/25-bake-vs-mount/README.md:43-45
New eval ✅ evals/fast-disk-for-installs.eval.md

Findings

Low

  • Machine-failure wording now differs across four places (concepts/machine.yaml:52). machine.host-local-storage now says the volume disk "can become unavailable or be lost if the machine fails". Three other places still say it "is lost": pod-volume-disk.yaml:74 (deprecated-prefer-network, a file this PR already edits), pod.yaml:149 (persistent-mount-limits) and reference/pods-and-machines.md:42. Use the new wording in all four, or at least in pod-volume-disk.yaml:74.

Simplification opportunities

None.

Verified clean

  • The golden path 25 callout reads correctly after the cut: /workspace is on the network mount, the image and the container disk are local.
  • The new machine.host-local-storage wording fits the network-filesystem-backed rule. A fuse-backed /workspace may become unreachable instead of destroyed. The billing evidence still supports "not billed while the machine is unavailable".
  • No other file changed since round 2.

Questions for the author

  1. (Still open from rounds 1 and 2.) docs.runpod.io/pods/storage/types lists the volume disk as "Fast (local)", and the probe contradicts that. Should the docs team get a fix request?

Reviewed by Claude (claude-opus-5-5) via the pr-review skill. Tokens: 16 in / 3,138 out / 409,065 cache read / 57,156 cache write = 469,375 (main session, no sub-agents; 9-line incremental diff). Estimated cost: $1.40 from the bundled sum_usage.py rate card (rates not re-checked this run; cache writes assumed at the 5-minute TTL rate). Usage counted before posting. Round 3 of this PR. This is automated assistance; a human owns the final call.

the machine is unavailable. Use a network volume for data that must
outlive the machine.
The pod's volume disk is tied to its machine. The data stays while the pod
is stopped, can become unavailable or be lost if the machine fails, and is

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: This wording is better, but three places still say the volume disk "is lost" when the host fails: concepts/pod-volume-disk.yaml:74 (deprecated-prefer-network, already edited in this PR), concepts/pod.yaml:149 (persistent-mount-limits) and reference/pods-and-machines.md:42. Use the same "can become unavailable or be lost" wording there, so an agent does not get two answers to the same question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants