A blocked event loop takes the whole server down silently: add a watchdog, an instance registry and hang-aware dune ps #42

Problem

Reported on a real site (inline edit / edit view, pages with tables and blockquotes): the server stopped answering every URL, logged nothing, and only a restart cleared it. Not reproduced on a clean copy, so the trigger is still unknown.

That symptom means the event loop is blocked (a synchronous loop, e.g. catastrophic regex backtracking or a conversion that never terminates) or wedged. Deno runs all requests on one thread, so one bad request takes the site down for everybody, and a blocked thread can't log, time out, or shut down gracefully. Fixing this one bug won't prevent the next one; the failure mode needs containment.

Related, from the earlier dune ps work (#37): a machine ended up with ~45 stale dune processes and dune ps could not tell a healthy server from a stuck one. It finds processes by parsing command lines and lsof, which is heuristic and says nothing about whether a process is responsive.

What we know (spikes)

• Deno.exit() called from a Worker only ends the Worker; a main thread spinning in a loop keeps running (process stayed at ~97% CPU). Killing a stuck process from inside needs SIGKILL.
• A Worker cannot be unref'd and keeps the process alive until terminated, so every normal exit path must stop it explicitly (same class of bug as the idle dune serve processes).
• macOS sample gives only native frames with no JS names, so it can't say which request hung. In-flight request tracking can.

Plan (stacked PRs)

1. Watchdog (done, branch feat/event-loop-watchdog): a Worker expects a main-thread heartbeat; after 15 s (DUNE_WATCHDOG_TIMEOUT_MS) of silence it writes a report to <site>/.dune/hangs/ (in-flight requests, plus a native stack sample on macOS) and SIGKILLs the process so a supervisor (systemd Restart=, the dune dev wrapper) sees a dead process. DUNE_WATCHDOG=warn|off. The main thread also logs when its own timer ran over a second late. Applies to dune dev and dune serve.
2. Instance registry: serve/dev write ~/.dune/instances/<pid>.json (pid, process start time, mode, root, title, port, version) on listen and remove it on exit; stale entries are verified by pid + start time. The watchdog can mark an instance hung there, since its Worker survives a blocked main thread.
3. dune ps reads the registry (keeping the current command-line heuristics as a fallback for older versions and orphans) and shows listening / not listening / hung. Rebased on a trimmed feat(cli): dune ps finds the CLI and task wrappers and says "not listening"; new dune stop #37 (which lands first as the base); stop/idle/SIGKILL semantics to be settled there (SIGKILL as opt-in --force).

Later, decided after the original hang is root-caused: Worker isolation with terminate() + timeout for CPU work on user-controlled input (markdown/MDX compile, editor markdown/document conversion, TSX).

Out of scope / known limits

• Cannot make a hang non-fatal in-process; this is detection + recovery.
• No Linux stack sample yet (the report says so and names eu-stack/gdb).
• The original hang's root cause is still open. Next time it happens, before restarting: record CPU % of the wrapper and child (dune ps, ps -o pcpu) and run sample <pid> 5.

Acceptance

• A stuck main thread is killed within the configured timeout and leaves a report naming the in-flight requests.
• Graceful serve shutdown is not delayed by the watchdog.
• dune ps can tell a hung instance from a healthy one without parsing command lines.