Track sandbox live share bugs: the time ABI bind hides or breaks shares at /run targets #486

Summary

A focused bug hunt covered NVX's sandbox live shares (filesystem and single-file mappings from #459 and #475). It reviewed how nvx.py sandbox validates and lowers --mount targets in scripts/nvx_tools/sandbox.py, how the guest agent (guest/common/nvx-init-agent) mounts them, and how the container runtime (guest/alpine/nvx-container-launch and nvx-container-enter) enters the workload afterwards. It also skimmed the host telemetry added in #479 and the CI change classifier changed in #483. No open bug-hunt issue covers this component. The closed hunts tracked other components: #464 the managed sandbox lifecycle, #462 the egress policy compiler, and #374 release source collection.

The hunt found one reproducible bug. Line references are permalinks to dev at c29f3d1. No repository files were changed. The reproductions ran against a git archive export of that commit.

• 1. Reserve /run, /run/nvx, and the /run/nvx/time tree as sandbox share targets, in nvx.py and in the guest agent

1. The time ABI state bind hides or breaks live shares at /run targets (Medium)

Where

• nvx.py sandbox rejects only the targets that the container runtime claimed when the reserved list was introduced in 13d2722 (2026-09-30). The list holds the /proc, /sys, /dev, /.nvx-agent, and /etc/machine-id trees, plus /etc exactly (sandbox.py L43-L48, validate_mount_target L105-L126).
• The guest agent applies the same list. Its comment states the invariant that this bug breaks: "The container runtime mounts or binds these paths after the share, so a share there would be shadowed or would expose host data to runtime writes" (nvx-init-agent L151-L169).
• The agent mounts the shares (L548) and only then starts the container runtime. The one-shot path runs it at L577-L579. The managed agent runs it for every exec (nvx-managed-agent.c L1210).
• 149dc68 (2026-10-04) added a runtime-owned bind to the runtime but did not add it to either list. The runtime creates each of run, nvx, and time below the container root with mkdir -p, then bind-mounts /run/nvx/time over the result read-only, under set -eu (nvx-container-enter L49-L60).
• The documentation promises that only the listed paths are reserved, and that "any validation or mount failure aborts the sandbox with status 125 instead of starting the workload without its shares" (doc/run.md L702-L711).
• OpenVMM does not catch these targets either. At the pinned nanvix/openvmm@2130f24, validate_microvm_guest_mount_target checks only the syntax: an absolute, canonical path without whitespace, \, or =.

Trigger

sandbox run or sandbox provision, followed by start and exec, with a --mount that matches either case:

• The target is /run or /run/nvx.
• The target is /run/nvx/time or a path below it, such as /run/nvx/time/x.

Other /run targets, such as /run/secrets or /run/nvx/other, are unaffected.

Expected

These targets are rejected before boot, as /etc and /etc/machine-id are. /etc is reserved exactly because the runtime writes /etc/machine-id inside it. A target that slips through should fail in the guest with status 125, rather than start the workload without its share.

Actual

nvx.py accepts the targets, and so do OpenVMM and the guest agent. The agent reports the share as mounted. The runtime then acts as follows:

Reproduction

1. Host side, on Windows 11 (10.0.26300) with Python 3.14.7. The commands ran in a git archive export of c29f3d1, with empty placeholder files for openvmm.exe, build/vmlinux, build/initramfs.cpio.gz, the layer, and the scratch image. sandbox run --dry-run accepts all three /run targets and rejects the reserved controls:

   > python <export>\scripts\nvx.py sandbox run --dry-run --hypervisor whp --layer distro,distro.erofs,11111111-1111-1111-1111-111111111111 --scratch scratch.ext4 --mount /run/nvx/time,share,ro --entrypoint /bin/ls --arg /run/nvx/time
   >> <export>\openvmm\target\release\openvmm.exe --machine microvm ... --mount /run/nvx/time,share,ro --microvm-lifecycle one-shot ... --cmdline "nvx_sandbox=1 ... nvx_arg=/run/nvx/time"
   exit=0
   (same for --mount /run,share,ro and --mount /run/nvx,share,ro: exit=0)
   --mount /proc,share,ro -> error: sandbox mount target /proc overlaps the reserved /proc tree (exit=1)
   --mount /etc,share,ro  -> error: sandbox mount target /etc is reserved (exit=1)
   

2. Guest side, in a privileged alpine:3.24 container (Alpine 3.24.2; Docker Engine 29.7.2 on WSL2 kernel 6.18.33.2). This is the guest's Alpine branch. The script below runs these parts verbatim from the export:

   • mount_live_shares and its helpers from guest/common/nvx-init-agent
   • the one-shot launch sequence from nvx-init-agent
   • guest/alpine/nvx-container-launch and nvx-container-enter

   Stand-ins replace the parts that need a VM:

   • A bind mount of a host directory, holding HOST_MARKER, replaces the virtio-fs mount.
   • A tmpfs with the Alpine userland replaces the overlay.
   • A /run/nvx/time with state and daemon.pid replaces the time ABI state.

   SandboxLaunch.kernel_command_line() and mounts_command_line_fragment() from the export generated each scenario's kernel command line. The /srv/data control shows that the harness exposes a share correctly. Scenarios A to C use one share, which the agent mounts with mount_live_share. Scenario D uses two shares, which the agent binds as aggregate children with bind_live_share.

   ================ scenario control: virtfs_dir=/srv/data virtfs_tag=microvm virtfs_mode=ro
   NVX-SANDBOX-SHARE: target=/srv/data mode=ro tag=microvm
   workload: /bin/ls -a /srv/data
   .
   ..
   HOST_MARKER
   workload exit status: 0
   host directory after the run: ./HOST_MARKER
   ================ scenario A: virtfs_dir=/run/nvx/time virtfs_tag=microvm virtfs_mode=ro
   NVX-SANDBOX-SHARE: target=/run/nvx/time mode=ro tag=microvm
   workload: /bin/ls -a /run/nvx/time
   .
   ..
   daemon.pid
   state
   workload exit status: 0
   host directory after the run: ./HOST_MARKER
   ================ scenario B: virtfs_dir=/run virtfs_tag=microvm virtfs_mode=ro
   NVX-SANDBOX-SHARE: target=/run mode=ro tag=microvm
   workload: /bin/ls -a /run
   mkdir: can't create directory '/run/nvx/rootfs/run/nvx': Read-only file system
   workload exit status: 1
   host directory after the run: ./HOST_MARKER
   ================ scenario C: virtfs_dir=/run virtfs_tag=microvm virtfs_mode=rw
   NVX-SANDBOX-SHARE: target=/run mode=rw tag=microvm
   workload: /bin/ls -a /run
   .
   ..
   HOST_MARKER
   nvx
   workload exit status: 0
   host directory after the run: ./HOST_MARKER ./nvx ./nvx/time
   ================ scenario D: virtfs_dir=/run/nvx/shares virtfs_tag=microvm virtfs_mode=ro
   NVX-SANDBOX-SHARE: target=/srv/data mode=ro child=0-96b53e47e3fc7c29db4c5a4c6977decf
   NVX-SANDBOX-SHARE: target=/run/nvx/time mode=ro child=1-aabbcbdff808cbd31c52ff745fe29f64
   workload: /bin/ls -a /srv/data /run/nvx/time
   /run/nvx/time:
   .
   ..
   daemon.pid
   state
   
   /srv/data:
   .
   ..
   HOST_MARKER
   workload exit status: 0
   host directory after the run: ./0-96b53e47e3fc7c29db4c5a4c6977decf ./0-96b53e47e3fc7c29db4c5a4c6977decf/HOST_MARKER ./1-aabbcbdff808cbd31c52ff745fe29f64 ./1-aabbcbdff808cbd31c52ff745fe29f64/HOST_MARKER ./HOST_MARKER
   

   Reproduction script (docker run --rm --privileged -v "$PWD:/repro:ro" alpine:3.24 sh /repro/repro.sh, with the export in ./export and cmdline-{control,A,B,C,D}.txt beside it)

   #!/bin/sh
   set -eu
   exec 2>&1
   apk add --no-cache -q util-linux setpriv libcap-ng libeconf >/dev/null
   [ -e /bin/setpriv ] || ln -s "$(command -v setpriv)" /bin/setpriv
   src=/repro/export/guest
   install -m 0755 "$src/alpine/nvx-container-launch" /sbin/nvx-container-launch
   install -m 0755 "$src/alpine/nvx-container-enter" /sbin/nvx-container-enter
   awk '/^(unmount_live_shares|cmdline_value|validate_share_target|make_share_mount_point|record_live_share|claim_share_target|mount_live_share|parse_share_child|bind_live_share|mount_live_shares)\(\) \{/,/^\}/' \
       "$src/common/nvx-init-agent" >/tmp/agent-functions.sh
   
   # The time ABI state that nvx-time keeps in the guest at boot.
   mkdir -p /run/nvx/time
   echo "guest time ABI state" >/run/nvx/time/state
   echo 1 >/run/nvx/time/daemon.pid
   
   scenario() {
       name=$1
       HOST_SHARE=/host/share-$name
       mkdir -p "$HOST_SHARE"
       echo "host data" >"$HOST_SHARE/HOST_MARKER"
       # An aggregate's root lists one directory per child.
       for token in $(cat "/repro/cmdline-$name.txt"); do
           case "$token" in nvx_share=*) child=${token#*=}; child=${child%%,*}; mkdir -p "$HOST_SHARE/$child"; echo "host data" >"$HOST_SHARE/$child/HOST_MARKER" ;; esac
       done
       echo "================ scenario $name: $(grep -o 'virtfs_dir=[^ ]* virtfs_tag=[^ ]* virtfs_mode=[^ ]*' "/repro/cmdline-$name.txt")"
       HOST_SHARE=$HOST_SHARE NAME=$name unshare --mount --propagation private sh -eu -c '
           . /tmp/agent-functions.sh
           # As nvx-init-agent, except that the VM would power off instead.
           fatal() { echo "NVX-SANDBOX-ERROR: $*" >&2; unmount_live_shares || true; exit 125; }
           # Stand-in for the virtio-fs device: bind the host directory instead.
           mount() {
               if [ "$1" = -t ] && [ "$2" = virtiofs ]; then
                   command mount --bind "$HOST_SHARE" "$6" &&
                       command mount -o "remount,bind,$4" "$6"
               else
                   command mount "$@"
               fi
           }
           runtime=/run/nvx
           rootfs=$runtime/rootfs
           share_mountpoints=
           cmdline=$(cat "/repro/cmdline-$NAME.txt")
           # Stand-in for the container overlay: a tmpfs with the Alpine userland.
           mkdir -p "$rootfs"
           command mount -t tmpfs tmpfs "$rootfs"
           cp -a /bin /sbin /lib /usr /etc "$rootfs/"
           tr -d - </proc/sys/kernel/random/uuid >"$runtime/workload-machine-id"
           mount_live_shares
           # The one-shot launch of nvx-init-agent, without the cgroup placement.
           set -- "$(cmdline_value nvx_entrypoint)"
           set -f
           for token in $cmdline; do
               case "$token" in nvx_arg=*) set -- "$@" "${token#*=}" ;; esac
           done
           set +f
           echo "workload: $*"
           barrier=$runtime/container-start
           mkfifo "$barrier"
           /sbin/nvx-container-launch "$barrier" "$rootfs" nvx-sandbox 65534 65534 nobody / "$@" &
           container_pid=$!
           printf "start\n" >"$barrier"
           rm -f "$barrier"
           status=0
           wait "$container_pid" || status=$?
           echo "workload exit status: $status"
           unmount_live_shares
       ' || echo "agent exit status: $?"
       echo "host directory after the run: $(cd "$HOST_SHARE" && find . -mindepth 1 | sort | tr '\n' ' ')"
   }
   
   for name in control A B C D; do
       scenario "$name"
   done

   This script generated each cmdline-<name>.txt from the export, and its output matches the command lines that nvx.py sandbox run --dry-run printed in step 1, plus the virtfs_* tokens that OpenVMM appends:

   from pathlib import Path
   from nvx_tools.sandbox import SandboxLaunch, SandboxLayer, SandboxMount, mounts_command_line_fragment
   for name, target, mode in [("control", "/srv/data", "ro"), ("A", "/run/nvx/time", "ro"), ("B", "/run", "ro"), ("C", "/run", "rw")]:
       mount = SandboxMount(guest_target=target, host_path=Path("share"), access=mode)
       launch = SandboxLaunch(layers=(SandboxLayer("distro", Path("distro.erofs"), "11111111-1111-1111-1111-111111111111"),),
                              scratch=Path("scratch.ext4"), entrypoint="/bin/ls", args=("-a", target), mounts=(mount,))
       Path(f"cmdline-{name}.txt").write_text(launch.kernel_command_line() + mounts_command_line_fragment((mount,)) + "\n", newline="\n")
   # D: an aggregate of two shares.
   mounts = (SandboxMount(guest_target="/srv/data", host_path=Path("share-a"), access="ro"),
             SandboxMount(guest_target="/run/nvx/time", host_path=Path("share-b"), access="ro"))
   launch = SandboxLaunch(layers=(SandboxLayer("distro", Path("distro.erofs"), "11111111-1111-1111-1111-111111111111"),),
                          scratch=Path("scratch.ext4"), entrypoint="/bin/ls", args=("-a", "/srv/data", "/run/nvx/time"), mounts=mounts)
   Path("cmdline-D.txt").write_text(launch.kernel_command_line() + mounts_command_line_fragment(mounts) + "\n", newline="\n")

Suggested fix

• In scripts/nvx_tools/sandbox.py, add /run/nvx/time to RESERVED_MOUNT_TARGETS (a tree), and /run and /run/nvx to RESERVED_EXACT_MOUNT_TARGETS. This mirrors /etc and /etc/machine-id, because the runtime creates directories inside them. Extend the comment above the lists to mention the time state.
• In guest/common/nvx-init-agent, add /run | /run/nvx | /run/nvx/time | /run/nvx/time/* to the reserved case in validate_share_target. A configuration from an older host then fails with status 125 instead of starting without its share.
• In doc/run.md, add the new paths to the reserved list at L702-L706.
• Extend test_mount_target_validation_rejects_unsafe_and_reserved_targets (test_nvx_tools.py L11878-L11911). Cover /run, /run/nvx, /run/nvx/time, and /run/nvx/time/x as reserved, and /run/secrets and /run/nvx/other as accepted.
• Optional defense in depth: before each runtime bind, nvx-container-enter could exit 125 when the path or one of its ancestors below the container root is already a mount point. A runtime bind added later then could not hide a share silently.

Explored

The hunt read the following components:

• Sandbox live shares, from filesystem: map host paths from any volume, each with its own access #459 and filesystem: share single host files without their directories #475:
  • scripts/nvx_tools/sandbox.py
  • the run and sandbox --mount wiring in scripts/nvx.py
  • the share config formats 2 to 6 in scripts/nvx_tools/sandbox_lifecycle.py
  • guest/common/nvx-hostmount
  • guest/common/nvx-init-agent
  • guest/alpine/nvx-container-launch and nvx-container-enter
  • guest/common/nvx-virtio-restore-probe
  • aci_edge_sandboxes/src/openvmm/filesystem.rs
  • OpenVMM's --mount and --mount-child parsers and guest-target validation at the pinned revision
• scripts/nvx_tools/host_telemetry.py, from ci: record benchmark host provenance and storage telemetry #479
• classify-ci-changes --null, from ci: classify both rename paths and unquoted filenames #483

Checked, not filed

• aci_edge_sandboxes on Windows. plan() refuses a read-only file inside a read-write directory because Windows opens names case-insensitively. It does not refuse a read-only directory there, although the same aliasing applies. The crate's README documents that the guest's bind alone enforces a read-only path inside a read-write mapping, so this is a documented limitation.
• Commas in aci_edge_sandboxes host paths. The crate never rejects them in --mount-child. OpenVMM's MicrovmMountChildCli parses with split_once and rsplit_once, and the crate always emits an explicit mode, so such paths round-trip correctly.
• nvx-virtio-restore-probe. It derives its I/O directory from the first nvx_share token as if the token named a directory child; a ,file suffix would yield TARGET,MODE. The probe is a test helper used only with directory shares, so the impact is low.
• host_telemetry.py. The SYSTEM_PERFORMANCE_INFORMATION and x64 SYSTEM_PROCESS_INFORMATION offsets match the NT layouts. The windowing (nearest samples and 32-bit wrap) and the busy, latency, and queue formulas are consistent.
• classify-ci-changes --null. The classifier drops the empty entry that the trailing NUL of git diff -z produces. The regex anchors err toward running more suites.
• Managed share configuration. Format selection and reload are symmetric, and relative policy paths are joined to the share again at launch.
• nvx.py run aggregate targets. Their validation and nvx-hostmount token parsing agree with the Python producer.

Validation limits

• No microVM booted. The guest side ran in a privileged Linux container rather than a VM, with three substitutions:
  • A read-only or read-write bind replaced the virtio-fs mount. OpenVMM's host-side ro enforcement also fails writes with EROFS.
  • A tmpfs replaced the EROFS and scratch overlay.
  • The cgroup placement and the agent's earlier boot steps were skipped. They do not affect the order of the share mount and the runtime bind.
• The managed lifecycle (sandbox start and exec) is affected by code reading of nvx-managed-agent.c. The hunt did not execute it.
• Only the Alpine sandbox runtime was reproduced. guest/common/nvx-container-enter-azurelinux does not bind the time state.
• No hypervisor tests (test-microvm, filesystem-shares) ran.