fix(lighthouse): resolve per-run TEMP to absolute — unbreaks all Web Vitals on Windows

[hugo-ccabral]

Problem

Every Lighthouse measurement fails on Windows with Unable to connect to Chrome, so the entire Web Vitals section of benchmark reports comes back empty. On a real farmrio run: 8/8 measurements errored, all 3 retries each, and report.json contained only {"error": "lighthouse exit 1: Unable to connect to Chrome"} for every page.

Root cause

measureLighthouseOnce builds a per-run temp dir and hands it to the spawned lighthouse process as TEMP/TMP:

const runTemp = join(opts.outDir, `.tmp-${opts.id}-a${attempt}`);

outDir descends from --output, which defaults to the relative ./parity-output — so TEMP was relative. chrome-launcher feeds os.tmpdir() straight into Chrome's --user-data-dir, and Chrome cannot resolve a relative profile dir. It dies during launch, and chrome-launcher surfaces the misleading Unable to connect to Chrome — which points at connectivity, not at a path.

How it was isolated

Replicated the exact spawn() options in a standalone probe and flipped only the path form:

runTemp	Result
relative	✗ Unable to connect to Chrome, no report
absolute	✓ report written

Ruled out along the way: shell (all runs were Git Bash), Lighthouse concurrency (3 concurrent standalone runs all succeeded), and Playwright contention (a standalone run succeeded while the benchmark's own Lighthouse was active).

Why this hides so well: running the same command from a terminal exports TEMP through MSYS/cmd path normalization, which makes it absolute before Chrome ever sees it. Only the programmatic spawn preserves the relative form — so it reproduces from the tool and never from the shell you'd debug it in.

Changes

Two independent commits, reviewable separately:

157ee9c fix(lighthouse) — the actual bug. resolve() instead of join() for runTemp. Fixes Lighthouse for every consumer (benchmark, vitals, checks), not just the benchmark path.

05de19d perf(benchmark) — independent measurement-validity issue found while debugging. The benchmark ran home/PLP/PDP through Promise.all, so each side had 3 CPU-throttled Lighthouse passes measuring simultaneously, both sides overlapping, with Playwright's browsers still open. That inflates each other's LCP/TBT and undercuts the prod-vs-cand comparison the report exists to make. The vitals command already accounts for this (--lighthouse-concurrency defaults to cores/2 capped at 2); the benchmark path never got it. Costs vitals-phase wall-clock. Drop this commit if speed matters more than precision.

Verification

• Typecheck clean (tsc --noEmit)
• 1368 tests pass; the 16 failures are pre-existing Windows path / vi.stubGlobal issues, untouched by this change
• Live farmrio benchmark on the fixed build: cand-mobile-home-a1.json written at 1.5MB on attempt a1 — first try, no retries. Same run pre-fix: 3 empty retry dirs, 0 reports.

🤖 Generated with Claude Code

---

Summary by cubic

Fixes Windows Lighthouse failures by resolving the per-run TEMP/TMP directory to an absolute path, restoring Web Vitals. Also runs the three benchmark Lighthouse pages sequentially to avoid inflated metrics and launch failures.

• Old: TEMP/TMP derived from a relative --output path caused chrome-launcher to pass a relative --user-data-dir to Chrome, which failed to launch with "Unable to connect to Chrome" and produced empty reports. New: use resolve() to make the per-run temp path absolute.
• Old: benchmark measured home/PLP/PDP in parallel; new: run them sequentially. Side effect: longer vitals phase, but more stable LCP/TBT and fewer handshake failures while Playwright browsers are open.
• Affects all consumers of the shared lighthouse path (benchmark, vitals, checks). No configuration changes or migrations required.

^{Written for commit 05de19d. Summary will update on new commits.}

[hugo-ccabral]

Verdict: COMMENT — the resolve() fix is correct and I independently confirmed the root cause from the on-disk artifacts; the sequential commit is safe but the comment it adds bakes in a causal story the PR body itself disproves.

I verified the diagnosis rather than taking it on faith. From the failing run in parity-output/runs/2026-08-25T00-06-36-076Z/:

• every .tmp-*-aN/lighthouse.XXXXXXXX/ contains chrome.pid + empty chrome-out.log/chrome-err.log, and zero DevToolsActivePort files across the whole run (find ... -name DevToolsActivePort | wc -l → 0)
• no profile artifacts (Default/, Local State) anywhere in those dirs

So Chrome spawned fine and wrote nothing — it discarded the relative --user-data-dir, fell back to the default profile, and never wrote the port file where chrome-launcher polls. That is exactly "Unable to connect to Chrome", and it rules out the OOM/contention stories. The post-fix run 2026-08-25T00-19-59-304Z/ has zero orphan lighthouse.* dirs (chrome-launcher's own destroyTmp ran, i.e. full launch/connect/teardown succeeded) and six *-a1.json reports. Diagnosis confirmed.

resolve(opts.outDir, ...) is also strictly safe: identical to join when outDir is already absolute (commands/check.ts:120 passes mkdtempSync(tmpdir())), identical normalization for .., correct for UNC, and the second segment is .tmp--prefixed so it can never absolutize away the base.

Worth fixing

• engine/benchmark.ts:1450-1454 — the new comment claims the parallel Lighthouse passes "starve Chrome's launch handshake ... which fails all three attempts with Unable to connect to Chrome". Your own PR body rules that out ("Lighthouse concurrency (3 concurrent standalone runs all succeeded)"), and commit 157ee9c proves the failure was the relative path. The comment is the artifact that outlives the PR, and this one points the next debugger at concurrency instead of at TEMP. Keep the real justification (CPU-throttled passes inflate each other's LCP/TBT) and drop the launch-failure claim.

• commands/benchmark.ts:206-211 — prod and cand still run through Promise.all, and each side now reaches its vitals block at roughly the same time, so two throttled Lighthouse passes still overlap. This PR takes effective concurrency 6 → 2, not → 1. That happens to land on the same number as the vitals default (commands/vitals.ts:143-145, min(2, cores/2)), so the outcome is fine — but the comment right above at :206, "Lighthouse runs after each side closes its own context, so they never fight over the CPU", is now demonstrably false and is worth correcting in the same commit.

• engine/lighthouse.ts:214 — only TEMP/TMP are set. On macOS/Linux os.tmpdir() reads TMPDIR first, and macOS always sets it, so both this fix and the original per-run-writable-TEMP hardening are no-ops off Windows. TMPDIR: runTemp alongside the other two makes the isolation actually hold everywhere.

• engine/lighthouse.ts:181 — runTemp is never removed. Measured on this repo's output: runs/<id>/lighthouse/ is 103 MB across 36 .tmp-* dirs, ~2.9 MB each, and the content is a node-compile-cache that gets discarded and rebuilt from scratch on every attempt because TEMP is unique per attempt. Pre-existing — the failing runs leaked the same — but this fix is what makes these runs start completing at scale. Attempts inside measureLighthouse are sequential, so there is no contention to guard against between them: keying the dir per page instead of per attempt (drop -a${attempt}) both bounds the growth and lets the V8 compile cache actually hit on retries. An rmSync(runTemp, { recursive: true, force: true }) after the readFileSync would do it too.

• engine/lighthouse.ts:210 — new Promise((resolve) => ...) now shadows the node:path resolve you just imported, inside that executor. Arity mismatch means tsc catches misuse today, so this is only a trap for later; renaming the executor param to done costs nothing.

Checked, fine

• outPath does not need resolving. It stays relative but is consistent: the child inherits cwd from spawn (no cwd: override), and readFileSync(outPath) resolves against the same cwd. I grepped the whole package for process.chdir — zero hits — and the reports empirically land in the run dir. Not a latent twin of the runTemp bug.
• Sequential does not change error semantics. measureLighthouse returns { error } for every modeled failure (non-zero exit, parse failure, runtimeError, missing LCP) and the retry loop swallows nothing. The only paths that actually throw are the two mkdirSync calls and report.audits being undefined on a parseable-but-non-LHR JSON (lighthouse.ts:244-245); those escape measureLighthouse uncaught, land outside the try/catch that ends at benchmark.ts:1404, and kill the whole run identically under Promise.all and under sequential awaits. Sequential is marginally better here — it no longer orphans two live lighthouse child processes when the first one throws.
• Other consumers. commands/vitals.ts:157 gets the fix for free (its lhDir descends from the same relative ./parity-output) and its runWithConcurrency cap is untouched. commands/check.ts never calls measureLighthouse at all — only vitals.ts and engine/benchmark.ts do — so the "fixes checks too" line in the description is overstated, harmlessly.
• No hidden timeout blown. The 3-minute watchdog is commands/e2e.ts:185 only; nothing bounds the benchmark path. Worst case for vitals does roughly triple (3 pages x 3 attempts x --max-wait-for-load=90000 = up to ~13.5 min per side per viewport), which is a cost, not a break.
• spawn with a path arg exists at exactly one site in the package; the only other spawn (e2e.ts:185) passes an inline -e script with no paths. No twin of this bug elsewhere.