[zumbrunn]

Refs #42

Problem

One synchronous loop blocks the only thread serving requests, so every URL stops answering at once, with nothing in the logs, until someone restarts the server. A blocked thread can't log, time out or shut down, so the check has to run elsewhere.

Change

• src/runtime/watchdog.ts: a Worker expects a main-thread heartbeat every second. After 15 s of silence (DUNE_WATCHDOG_TIMEOUT_MS) it writes a report to <site>/.dune/hangs/ (in-flight requests, with token-sized path segments masked; plus a native stack sample on macOS), logs the key lines to stderr, and SIGKILLs the process so a supervisor (systemd Restart=, the dune dev wrapper exiting 137) sees a dead process. The newest 20 reports are kept.
• DUNE_WATCHDOG=exit (default) | warn (report, don't kill; chosen automatically under --inspect/--inspect-brk) | off.
• Startup limit: 2 minutes (DUNE_WATCHDOG_STARTUP_TIMEOUT_MS) until the server calls ready(), so a big site's initial indexing can't put a supervised process into a kill/restart loop.
• Not killing what shouldn't be killed: the stall clock is monotonic (performance.now()), and a gap in the Worker's own ticks (laptop sleep, Ctrl-Z, SIGSTOP) resets it instead of firing. I confirmed the original version did kill a process after a 4 s SIGSTOP/SIGCONT; there is now a regression test.
• Legitimately synchronous work: new subpath @dune/core/watchdog with watchdogProgress() (a rate-limited heartbeat for long synchronous loops; only protects loops that call it, never put it in an unbounded one) and watchdogGrace(ms, reason) (a declared busy window: capped at 10 min, auto-expires, nests, logged). Dune's own loops needed neither: on a 4,474-page site, startup indexing and the search build never held the event loop for over a second.
• The kill always happens: writing the report is raced against a 5 s timeout; a failed Deno.kill is logged and the watchdog carries on in warn mode instead of going silent.
• Event-loop-lag warning when the main thread's own timer ran over a second late.
• Started in dune dev and dune serve right after the root is resolved; in-flight request tracking is one middleware in createDuneApp (no-op without a watchdog).

Things found while building it

• Deno.exit() from a Worker only ends the Worker; a spinning main thread kept running at ~97% CPU. Hence SIGKILL.
• A Worker can't be unref'd and keeps the process alive. serve's two graceful-shutdown paths call stop(); a test covers it.
• On a real dune dev: the re-exec wrapper exits 137 and the esbuild service child exits on its own when its parent dies (no orphan). A test mirrors reexec.ts.
• In Deno 2.9 Deno.kill on your own pid needs no --allow-run, so I could not produce the kill-refused branch; the guard is defensive and untested.

Verification

• deno lint src/; deno test -A --unstable-kv tests/runtime tests/cli: 390 passed
• tests/runtime/watchdog_test.ts (16 tests, real child processes): stuck loop SIGKILLed with a report naming the stuck request and masking token segments; warn leaves the process alone; sub-limit block logs lag only; exits on its own after stop(); off; SIGSTOP/SIGCONT pause survives; startup vs. ready limits; progress keeps a synchronous loop alive; grace protects, expires, and nests; inspector flips to warn; a hung sample doesn't stop the kill (macOS); old reports pruned; wrapper exits 137 and a stdin-reading child doesn't outlive the kill.
• Live: scaffolded site with a page that spins forever, DUNE_WATCHDOG_TIMEOUT_MS=5000 dune dev: killed after 5 s with GET /hang/ (5s) in the log, report and sample written, wrapper exited, nothing left running.

Not done

• No stack sample on Linux (the report says so and names eu-stack/gdb).
• The original hang that prompted this is still unreproduced; this makes the next occurrence diagnosable and non-fatal for a supervisor, it doesn't find the cause.
• Rebuild-on-change timing on a large site wasn't measured (only startup).
• Docs: duneorg/dune-docs (new page, linked below). CHANGELOG entry is under [Unreleased].

🤖 Generated with Claude Code