test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf)

[siyabendoezdemir]

Test-harness follow-ups from PR #10's review: F2, F4, F5, F6 and F7. Test code and scripts only; nothing in the app changes. It is stacked on PR #17, which is stacked on PR #16, and all three contain main with settings (PR #15).

What changes

• F2: tests no longer make DNS lookups (49c8b6a).
  • The dead proxy stops every request, but Chromium looks up a link's host before anyone clicks it. PK-04's link to meet.google.com sent queries upstream.
  • Orbit now launches with --host-resolver-rules=MAP * ~NOTFOUND, EXCLUDE localhost, EXCLUDE 127.0.0.1. This applies in the e2e harness, the capture script and the perf script. The perf script also had no dead proxy, for Orbit or its Chromium baseline; it now has both.
  • A test's own --host-resolver-rules go first, so a later lane can still map a host or make one fail (pages lane, NV-13). Chromium checks every EXCLUDE before any MAP and looks an excluded host up for real, so a test's own EXCLUDE may only name a loopback host; offlineArgs() throws otherwise (4345e72, review F3).
  • Onboarding's first-run launches (launchFirstRun) get the same rules (4345e72, review F1). A note says the rules don't stop an mDNS lookup, which nothing asks for (F4).
• F4: evaluateHandle is deferred too, and method shorthands work (162d58d).
  • The harness already runs app.evaluate in a task of its own, so it never lands part-way through main's code. app.evaluateHandle still ran by interrupt.
  • The wrapper threw on a method shorthand (({ m(_e, a) {…} }).m), which Playwright's own evaluate accepts.
  • The relaunch test's two calls through app.browserWindow(page) (a handle, whose own evaluate can't be deferred) now use app.evaluate.
• F5: the perf script's timings (c8957f6).
  • The tab switch times the command alone, without the test hook's deferral hop.
  • The direct ctx.tabs.sleep call starts between tasks.
  • A new --port option sets the fixture server's first port.
• F6: SV-26 never passes without seeing the grow-in (eba76a7).
  • When the renderer was starved, both shape checks were skipped and the test passed on the end state alone.
  • Now that round's pane is closed and a new one grows in, up to three rounds (as for timing budgets, D46). If no round's frames came in time, the test fails and names each round's frame counts.
  • After the review (4ead39e):
    • A retry waits for the last close to finish before adding its pane (M1). Until the closed pane is gone and its ghost has faded, the neighbour may not reach the new pane's edge, and the new pane appears at full size.
    • The gate asks for 3 frames by 300 ms, not 8: a frame by 100 ms and two while the pane grows are what the shape checks need (M2).
• F7: SV-25's sidebar test checks that its drag is live (eba76a7), so it can't pass because no drag started. Since 4ead39e it checks again just before its count, after the 900 ms wait (review F9).

Left for later: F3 (waitOffline checks sessions only at launch). A check of sessions made mid-test needs a way for a test to opt out when it sets a proxy on purpose. No session leaked in the traced run.

Tests that fail without the change

• F2: the new smoke test resolved example.com and localhost in every profile's session. Without the rules: Expected substring "ERR_NAME_NOT_RESOLVED", Received "2606:4700:10::6814:179a … 104.20.23.154 172.66.147.243".
  • Since 4345e72 it probes 127.0.0.2 instead (review F5). An address needs no lookup, so only the rules refuse it, and a missing rule can't make the test itself send a query. With the rules missing only that address: Received string "127.0.0.2".
• F4: the new TB-22 test keeps main busy in timers and asks five times through each path.
  • Without the evaluateHandle wrapper: Expected false, Received true (the handle's code ran inside a timer).
  • Without the shorthand change: SyntaxError: Unexpected token '{'.
• F6 and F7 (test changes): checked with two scratch edits, taking the press out of SV-25's drag and making the sampler miss every frame before 300 ms. The old tests both passed. The new ones fail:
  • SV-25: Expected pattern /\bis-dragging-tab\b/, Received string "".
  • SV-26: "frames never came in time to show the grow-in (round 1: 0 frames by 100 ms, 0 by 300 ms; …)".
• Review M1 and M2 (scratch copies of SV-26, at load 25–43):
  • M1, rounds 1 and 2 forced to retry, 10 runs each:
    • without the wait, 6 of the 20 panes added right after a close entered at full size, and 2 runs failed (Expected: < 389.6, Received: 487);
    • with it, 0 of 20 entered at full size, and 10 of 10 runs passed.
  • M2, the shell renderer kept busy 45 ms per frame:
    • gated at 8 frames, 3 of 3 runs failed with "frames never came in time", though every round grew from width 0 with 4–6 frames by 300 ms;
    • gated at 3, 3 of 3 passed, with both shape checks run.
• The EXCLUDE guard (scratch spec):
  • example.com, host.example and 127.0.0.2 are refused;
  • localhost, x.localhost and ::1 pass through, in order.
• smoke, onboarding and split, with the guard's scratch: 53 passed.
• F5: one perf run, offline, on ports 3390+: exit 0; tab switch 5.7 ms median in main and 25 ms until painted.

Merging

• main (60da889, settings) was merged into PR test(e2e): ask Orbit to quit from a task of its own #16's branch, then up the stack: no conflicts. Each lower head passes typecheck and lint.
• The gate ran once, on this PR's head, which contains test(e2e): ask Orbit to quit from a task of its own #16's and test(commandbar): judge CB-22's budgets on the best of up to three rounds #17's heads and main (D50); see below. After the review, test(e2e): ask Orbit to quit from a task of its own #16, test(commandbar): judge CB-22's budgets on the best of up to three rounds #17 and this PR merge in that order, each retargeted to main after the one below it lands.

Merge gate (D42, D50)

Run locally on the Linux VPS on this PR's head as it was, 42c9819. It contains PR #16's head 2019597, PR #17's head 728fc45 and main (60da889, with settings):

npm run build; npm run typecheck; npm run lint; ./node_modules/.bin/vitest run
xvfb-run -a -s "-screen 0 1440x900x24" ./node_modules/.bin/playwright test --workers 1 --retries 0 --reporter=line

• Build, typecheck and lint: exit 0.
• E2E at 1440×900, one worker, no retries: 305 passed, 0 failed, 2 skipped, exit 0, in 19.3 min (07:50–08:10 UTC, load average 16–43).
  • The two skipped are media's test.fixme frame tests (MD-01, MD-09).
  • The run includes settings' 36 tests under the new resolver rules.
  • Main logged no uncaught exception, and no quit was killed.
• Unit: 434 of 435. The failure was PR test(commandbar): judge CB-22's budgets on the best of up to three rounds #17's CB-22 ranking bench. At a load average of 39, beside vitest's other workers, all three of its runs missed the 30 ms p95 in wall time: 54.5, 30.3 and 51.6 ms. Alone it passed 4 of 4.
  • PR test(commandbar): judge CB-22's budgets on the best of up to three rounds #17 now times that bench in its thread's CPU time (9d9c314, D51), with a positive control that still fails it. It was merged here as 7be3b2b.
• On the new head 7be3b2b (08:15 UTC, load 17–21): build, typecheck and lint exit 0; unit 435 of 435 in 49 files. In that run the bench's wall p95 was 36.3 ms, which the old test would have failed, and its CPU p95 was 8.4 ms.
  • git diff --stat 42c9819 7be3b2b shows tests/unit/commandbar.test.ts only (+24/−8). The e2e suite loads no unit test, so the e2e result above carries over (D50).
• macOS: unverified. The macOS e2e run needs a Mac. The change is test code and scripts only.

Merge gate after the review (D42, D50)

Run locally on the Linux VPS on this PR's head 4345e72. It contains PR #16's head f49d66a, PR #17's head 2c9e366 and main (60da889):

npm run build; npm run typecheck; npm run lint; ./node_modules/.bin/vitest run
xvfb-run -a -s "-screen 0 1440x900x24" ./node_modules/.bin/playwright test --workers 1 --retries 0 --reporter=line

• Build, typecheck and lint: exit 0. Unit: 435 of 435 in 49 files.
• E2E at 1440×900, one worker, no retries: 306 passed, 0 failed, 2 skipped, exit 0, in 16.2 min (10:25–10:41 UTC, load average 9–34).
  • The two skipped are media's test.fixme frame tests (MD-01, MD-09).
  • The one test more than the last gate is PR test(e2e): ask Orbit to quit from a task of its own #16's cancelled-quit test. It failed, as its test.fail() expects.
  • CB-22's e2e test met its budgets in round 1 of its first launch: p95 11.5 ms to results and 33.2 ms painted, open median 24.4 ms.
  • Main logged no uncaught exception, and no quit was killed outside the cancelled-quit test.
• macOS: unverified. The macOS e2e run needs a Mac. The change is test code and scripts only.

Review

• Independent review (reviews/lead-harness2.md): approve with 2 must-fixes, both in SV-26. The lead fixed both, with failing-first evidence above:
  • M1: a retry added its pane while the last close was still animating, so the pane appeared at full size;
  • M2: the gate asked for 8 frames by 300 ms, more than its checks need.
• Follow-ups:
  • Done here: F1, F3, F4, F5 and F9.
  • Routed to the finish-views lane: F10, Orbit's entryRect fallback, which skips the grow-in when no old pane overlaps.
  • Open: F2 ([::1] and *.localhost fixtures; none is used), F6 (functionSource's error message), F7 (the scripts' remaining interrupts), F8 (a note on the perf script) and F11 (macOS SV-26; waitOffline mid-test).
• The fix check for all three PRs runs on this head, 4345e72.

🤖 Generated with Claude Code

---

Note

Low Risk
Changes are limited to e2e helpers, test specs, and capture/perf scripts; production app code is untouched.

Overview
Test harness and tooling follow-ups from PR #10 review — no application code changes.

Offline isolation is tightened with Chromium --host-resolver-rules so hostnames (except localhost) fail to resolve, blocking DNS lookups before link clicks. Shared via new offlineArgs() in e2e, plus capture and perf scripts; perf also applies the dead proxy to Orbit and the Chromium baseline. A smoke test asserts resolver behavior across profile sessions.

Main-process Playwright hooks defer evaluateHandle like evaluate, support method shorthand via functionSource, and add TB-22; relaunch tests use app.evaluate instead of handle evaluate.

Perf harness adds --port, defers inspector-driven work with setImmediate for sleep/tab-switch timings, and times tab activation through ctx.commands.run directly.

Split e2e hardening: SV-25 asserts an active tab drag during the wait; SV-26 retries grow-in sampling up to three rounds so the test cannot pass on end state alone when frames are starved.

^{Reviewed by Cursor Bugbot for commit 4345e72. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 382eec85-0109-4b59-9266-30cc80670b77)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 79704018-729a-452a-adfb-950a9f6f4c5c)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 2b658f71-c8b3-4d63-84f1-d7e615b386f0)