[Bug]: Linux server freezes for minutes when a network filesystem stalls, because spawn() waits for the child's chdir and PATH lookup #30183

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

child_process.spawn() on Linux returns only after the child has run chdir(cwd) and the PATH lookup, and the event loop is blocked for that time. If the project directory, or a PATH directory searched before the binary, is on a network filesystem that stops answering, the whole server stops for as long as the filesystem does. This is separate from the fork cost in #21289. libuv 1.53's switch to posix_spawn won't help here: glibc's posix_spawn uses CLONE_VFORK, so the caller is still suspended until the child execs (see step 4).

A FUSE filesystem where every lookup takes 5 s stands in for an NFS server that has stopped answering:

1. sudo apt install python3-fusepy, then save this as slowfs.py and run DELAY=5 python3 slowfs.py /tmp/slow &:

   slowfs.py

   import errno, os, stat, sys, time
   from fusepy import FUSE, FuseOSError, Operations
   
   DELAY = float(os.environ.get("DELAY", "5"))
   DIRS = {"/", "/repo", "/bin"}
   
   class SlowFS(Operations):
       def getattr(self, path, fh=None):
           time.sleep(DELAY)
           if path in DIRS:
               now = time.time()
               return dict(st_mode=stat.S_IFDIR | 0o755, st_nlink=2, st_uid=os.getuid(), st_gid=os.getgid(),
                           st_size=0, st_atime=now, st_mtime=now, st_ctime=now)
           raise FuseOSError(errno.ENOENT)
       def readdir(self, path, fh):
           return [".", ".."] + ([d[1:] for d in DIRS - {"/"}] if path == "/" else [])
       def access(self, path, mode):
           return 0
   
   FUSE(SlowFS(), sys.argv[1], foreground=True, attr_timeout=0, entry_timeout=0, negative_timeout=0)

2. Run node spawn-stall.mjs /tmp/slow:

   spawn-stall.mjs

   import { spawn } from "node:child_process";
   import { performance } from "node:perf_hooks";
   
   const mnt = process.argv[2];
   const cases = [
     ["baseline: cwd=/tmp, command 'true'", "true", { cwd: "/tmp" }],
     ["cwd on slow fs, command 'true'", "true", { cwd: `${mnt}/repo` }],
     ["slow dir first on PATH, command 'true'", "true", { cwd: "/tmp", env: { ...process.env, PATH: `${mnt}/bin:/usr/bin:/bin` } }],
     ["slow dir last on PATH, missing command 'muse'", "muse", { cwd: "/tmp", env: { ...process.env, PATH: `/usr/bin:/bin:${mnt}/bin` } }],
   ];
   for (const [label, cmd, opts] of cases) {
     let last = performance.now(), maxLag = 0;
     const tick = setInterval(() => { const n = performance.now(); maxLag = Math.max(maxLag, n - last - 10); last = n; }, 10);
     await new Promise((r) => setTimeout(r, 100));
     const t0 = performance.now();
     const child = spawn(cmd, [], { ...opts, stdio: "ignore" });
     const inSpawn = performance.now() - t0;
     await new Promise((r) => { child.on("error", r); child.on("exit", r); });
     await new Promise((r) => setTimeout(r, 100));
     clearInterval(tick);
     console.log(`${label.padEnd(48)} spawn() ${inSpawn.toFixed(0)} ms, max event-loop lag ${maxLag.toFixed(0)} ms`);
   }

3. On the server, put a project on such a mount (for us, NFSv4 hard,timeo=600) and wait for the filesystem to stall. Pull request sweeps, VCS status and provider and CLI probes all start processes in the project directory or look them up by name on PATH.

4. Optional: compile this against glibc to see the posix_spawn behaviour that libuv 1.53 will use:

   posix-spawn-chdir.c

   #define _GNU_SOURCE
   #include <spawn.h>
   #include <stdio.h>
   #include <sys/wait.h>
   #include <time.h>
   extern char **environ;
   int main(int argc, char **argv) {
     posix_spawn_file_actions_t fa; posix_spawn_file_actions_init(&fa);
     posix_spawn_file_actions_addchdir_np(&fa, argv[1]);
     char *args[] = {"/bin/true", NULL}; pid_t pid; struct timespec a, b;
     clock_gettime(CLOCK_MONOTONIC, &a);
     int err = posix_spawn(&pid, "/bin/true", &fa, NULL, args, environ);
     clock_gettime(CLOCK_MONOTONIC, &b);
     printf("posix_spawn(cwd=%s) blocked the caller for %.0f ms\n", argv[1],
            (b.tv_sec - a.tv_sec) * 1e3 + (b.tv_nsec - a.tv_nsec) / 1e6);
     if (!err) waitpid(pid, NULL, 0);
   }

Expected behavior

A hung filesystem under one project only delays git work for that project. The rest of the server keeps answering, and clients stay connected.

Actual behavior

Repro output (Ubuntu 24.04, Node 24.22.1, libuv 1.52.1, the same libuv as Node 26.12.1, which T3 Code 0.1.6 bundles):

baseline: cwd=/tmp, command 'true'               spawn() 7 ms, max event-loop lag 6 ms
cwd on slow fs, command 'true'                   spawn() 5005 ms, max event-loop lag 5004 ms
slow dir first on PATH, command 'true'           spawn() 10004 ms, max event-loop lag 10001 ms
slow dir last on PATH, missing command 'muse'    spawn() 10003 ms, max event-loop lag 10004 ms
posix_spawn(cwd=/tmp) blocked the caller for 0 ms
posix_spawn(cwd=/tmp/slow/repo) blocked the caller for 5001 ms

On our server (a university workstation), the project and part of PATH are on NFS. The file server stops answering a few times a day. Each time, every client shows " did not respond" until it recovers. The longest stalls in event-loop-stalls.ndjson match the kernel's NFS messages:

The kernel logs "not responding" only after timeo (60 s), so each outage starts before its message. Where server.trace.ndjson still covered a stall, one spawn span starts and ends with it:

• 01:16:41, 250 s: PullRequestWatchReactor.sweep → RepositoryIdentityResolver.resolveCacheKey → processRunner.runProcessCore (git), 252 s, with cwd on the project's NFS mount.
• 12:42:59, 20.6 s: UsageService.resolveTranscriptDirs → MuseUsageHome.initialize → createMuseSdkHostEffect, 20.7 s. muse is not installed, so the lookup searched every PATH directory, including the NFS ones.

In the 22 h of trace we have, the server started git 1,019 times, lsof 455 times and gh 191 times. It also started each missing forge CLI (glab, az, tea, fj) and tailscale about 20 times. Any of those starts can be the one that lands on a stall.

Impact

Major degradation or frequent failure

Version or commit

0.1.6 (server)

Environment

Server: Ubuntu 22.04, kernel 6.8, T3 Code 0.1.6 (Node 26.12.1, libuv 1.52.1), project on NFSv4. Repro: Ubuntu 24.04, Node 24.22.1, glibc 2.39. Clients connect over T3 Connect.

Workaround

None that fixes it. Putting local directories first on PATH avoids the lookup part, but the project directory has to stay on NFS. #29721 and #26794 reduce how often git and tailscale are started, which narrows the window without removing it.

Starting children from a separate long-lived helper process (as in the closed #18219) would confine a hung chdir to that helper and leave the event loop running. Related: #21289, #25245.