[siyabendoezdemir]

CB-22's budget tests fail full runs on the shared test machine often enough to need a rerun in most merge gates. This PR judges them on the best of up to three measuring rounds (D46), and times the ranking bench in its thread's CPU time (D51). The budgets themselves don't change. It's stacked on PR #16 and changes test code only.

Why

• Gate runs share a 16-core machine with up to three agents' test runs, at load averages of 25–50. Other work there can only add delay.
• CB-22 failed the full runs of chrome's and settings' fix rounds and several of the lead's runs, and each time passed alone. The media lane's review saw main fail it 7 of 9 times on this machine.
• In this PR's own check, at load 28, round 1 missed the paint budget (p95 64.8 ms against 50) and round 2 met it. The old test would have failed that run.

What

• commandbar.spec.ts, "[CB-22][CB-07][FL-17] at 500 tabs and 10,000 history entries…":
  • The test measures in rounds. A round is four opens of the bar, with a query typed each time, as the whole test did before.
  • A round that misses any budget (results ≤ 30 ms per keystroke, painted ≤ 50 ms, open ≤ 50 ms) is measured again, up to three rounds. The test passes on the first round that meets them all; if none does, it fails on the last round's numbers.
  • Each retry runs in a fresh launch of the same seeded profile (2c9e366, review M1). The test closes Orbit, launches it again on the same 500 tabs and 10,000 history entries, and waits for history and the idle warm-up. Every round is then the bar's first use after a launch, as a user meets it. The test's timeout is 120 s.
  • The bar keeps only its last 100 timings, so each round clears them before it starts. Each round's numbers are logged.
• commandbar.test.ts, "[CB-07][CB-22] ranks 500 tabs and 10,000 history entries in ≤ 30 ms per keystroke":
  • A run that misses the budget is measured again, up to three runs. The first run's warm-up round stays.
  • Each keystroke is timed in the thread's CPU time (process.threadCpuUsage()), and the same samples' wall p95 is logged beside it (D51, 9d9c314). Ranking never waits, so its CPU time is its whole cost; wall time on a loaded machine also counts other processes' turns. The e2e test still times the app in wall time.
  • On a Node before 23.9, which lacks threadCpuUsage, the bench uses the process's CPU time instead (engines allows Node 22). Its comment notes what CPU time can't see: a blocking wait, or work on another thread. It also notes that the clock moves in 1 ms ticks here (review F3–F5).

Test

Scratch variants (removed) kept main busy 40 ms in every 50:

Variant	Result	p95 keystroke → results (budget 30 ms)
Old e2e test, main busy during its one measurement	fails	48.2 ms
First version (retries in the same app), main busy in round 1 only	passes	46.6, then 11.2 ms
New e2e test, main busy in every round	fails after 3 rounds	48.1, 45.7, 48.0 ms

Review M1. The review found that a retry in the same app measured a warmer app. With nothing injected, round 1 was slower than round 2 in 6 of 6 runs, so a slowdown confined to the first use after launch passed on round 2. The positive control for the fix keeps main busy 40 ms in every 50 during the first round after each launch:

Variant	Result	p95 keystroke → results
Same-app retries	passes on round 2	52.3, then 13.4 ms
Fresh-launch retries (2c9e366)	fails after 3 rounds	50.3, 51.8, 47.0 ms

Without anything injected, at load 26–37, the fixed test passed 3 of 3. One run needed a retry: round 1 painted at 50.3 ms against 50, and round 2 passed in its fresh launch at 41.7 ms.

• Before D51, the unit test passed with a slow first run and failed after three slow runs (p95 43.6, 40.4, 39.7 ms).
• D51 (9d9c314):
  • The stack's gate on 42c9819 failed the wall-time bench in all three runs at load 39 (p95 54.5, 30.3 and 51.6 ms). Alone it then passed 4 of 4.
  • Timed in CPU time, it passed alone and in four full unit runs at load 17–31: CPU p95 6.3–8.8 ms, while the same samples' wall p95 was 15.7–36.3 ms.
  • With a scratch 35 ms of CPU added to rank() for queries of 14 or more characters, all three runs fail at a CPU p95 of 39.3 ms.
• commandbar.spec.ts: 36 passed. commandbar.test.ts: 30 passed. Lint and typecheck: exit 0.

Merge gate (D42)

Run locally on the Linux VPS on this PR's head, 581dbb1. It contains PR #16's head e4486e4, and merging main after #16 lands won't change its tree:

npm run build; npm run typecheck; npm run lint; ./node_modules/.bin/vitest run
xvfb-run -a -s "-screen 0 1440x900x24" ./node_modules/.bin/playwright test --workers 1 --retries 0 --reporter=line

• Build, typecheck and lint: exit 0. Unit: 375 of 375 in 39 files.

• E2E at 1440×900, one worker, no retries: 267 passed, 0 failed, 2 skipped, exit 0, in 12.6 min (06:52–07:04 UTC, load average 9–27). The two skipped are media's test.fixme frame tests (MD-01, MD-09). CB-22's test met its budgets in round 1 (p95 10.6 ms to results, 34.8 ms painted, open median 19.8 ms).

• Main logged no uncaught exception, and no quit was killed.

• macOS: unverified. The macOS e2e run needs a Mac. The change is test code only.

• Update, 07:50 UTC: main moved to 60da889 (settings, PR Settings window, appearance, onboarding and updates #15). It was merged into PR test(e2e): ask Orbit to quit from a task of its own #16's branch, and that into this one, with no conflicts (head 728fc45), then into PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19.

• Update, 08:16 UTC: under D50 the gate reran once, on PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19's head 42c9819, which contained 728fc45:

  • e2e: 305 passed, 0 failed, 2 skipped; CB-22's e2e test passed;
  • unit: 434 of 435, the one failure being this PR's ranking bench at load 39.

  That bench now runs in CPU time (9d9c314, D51). This head is now 9d9c314, and PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19's is 7be3b2b. 7be3b2b differs from the gated 42c9819 only in tests/unit/commandbar.test.ts, so build, typecheck, lint and unit reran on it (exit 0; 435 of 435), and the e2e result carries over (D50). Details are in PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19.

• Update after the review: this branch's fix (2c9e366) changes the e2e test, so the gate reran in full on PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19's new head 4345e72, which contains it: build, typecheck and lint exit 0; unit 435 of 435; e2e 306 passed, 0 failed, 2 skipped, exit 0, in 16.2 min (10:25–10:41 UTC, load average 9–34). CB-22's e2e test met its budgets in round 1 of its first launch (p95 11.5 ms to results, 33.2 ms painted, open median 24.4 ms). Details are in PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19.

Review

• Independent review (reviews/lead-budgets.md): approve with 1 must-fix.
  • M1: a retry measured a warmer app.
  • Fixed in 2c9e366, with F3–F5. D46 and D51 were updated to match.
• Routed: F2 (CB-22's open median never judges the slowest opens) and F6 (other wall-time budgets that a loaded machine can trip) went to the feel lane's brief. F1 is inherent to best-of-three.
• The fix check runs on PR test: harness follow-ups from PR #10's review (no DNS lookups, deferred evaluateHandle, SV-25/SV-26, perf) #19's head 4345e72, which contains this branch.

🤖 Generated with Claude Code

---

Note

Low Risk
Test-only changes to measurement and retry logic; no production command bar or ranking behavior is modified.

Overview
Stabilizes CB-22 performance gates on shared CI without changing the 30 ms / 50 ms budgets.

The e2e [CB-22][CB-07][FL-17] test now runs up to three measurement rounds (four bar opens + typing each). A round that misses any budget is retried after a fresh Orbit launch on the same seeded profile (500 tabs, 10k history), with launch / settle helpers and per-round clearing of __orbitCommandBar timings. The test passes on the first round that meets all budgets, otherwise fails on the last round; timeout is raised to 120s.

The unit ranking bench for 500 tabs / 10k history now asserts on thread CPU time (process.threadCpuUsage, with process.cpuUsage fallback) instead of wall clock, logs CPU and wall p95 per run, and re-runs up to three times if CPU p95 exceeds 30 ms.

^{Reviewed by Cursor Bugbot for commit 2c9e366. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 46cd1ffa-fc6e-4d5d-92bb-b2af6d881fcd)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 679936bc-9b75-4efe-99a6-315c59b6cfc7)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 3ecb2667-75b3-4eef-a3b1-84249970ea57)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: a74761b5-f428-41c2-a8c0-2191cd499786)