Repository navigation
PRD-550: the merge queue reviews the narrowed plan - #476
Conversation
Evidence from 253 CI runs on 2026-10-08/09: merge groups fail 62%, jobs wait ~50 min for runners, tree reuse never fires, and native legs up to 42 min sit on the merge path.
Add scripts/ci-merge-latency.mjs (last push -> merge, enqueue -> merge, attempts, first failing job, queue wait, from gh) and scripts/ci-speed-loop.mjs (ledger + self-contained HTML dashboard with a chart per goal). Record the 2026-10-08 baseline as iteration 1. Tick the PRD-550 phase 3 box that this proves.
A `merge_group` on develop reviews the exact tree a merge would produce on the base branch, so it earns the same narrowed selection a pull request into develop earns; the reduced native tier follows the same rule. An unresolved queue entry still owes every row, because it has no candidate identity to narrow by. The exhaustive board moves to the event that promotes a tree: a develop push now selects `full` instead of the `warm` cache-only lane, which keeps both cache publishers (`build-artifacts`, `test-native`) inside a board that qualifies the tree rather than verifying nothing. PRD-550 phase 2 is IN PROGRESS: the gating/qualifying split, the `test-unit` expansion out of UNPROVEN_REUSE_BOARDS and the ci-janitor revert PR are still open, and both phase-2 boxes need a CI run id. Verified: pnpm exec vitest run the six CI contract specs, 411 of 411 pass. Typecheck is unverified here: this worktree has no package dists, so the root tsc reports TS2307 for every workspace import, at HEAD and before it.
Iteration #2 rides with 90dee0c (the queue stops running the exhaustive board). 0 PRs merged since 2026-10-09, so push->merge, p90 and first-attempt are n/a and the two measuring metrics show the board this change removes: merge-group board wall 119.1 min (goal 15) and queue wait p90 50.9 min (goal 3), with `integration / velocity` red twice. Decision `provisional`. The page also stops claiming a reload refreshes it: it is a static render of docs/ci-speed/ledger.json, so it names the command that regenerates it. A window that merged no PR keeps the last window that did in the PR table, instead of blanking the audit rows.
… phase 1) Both red merge groups of 2026-10-09 (37874560157, 37883642740) fail first on `integration / velocity` with `TN_PLAYTEST_RESOURCE_ASSERTION_FAILED ... 'p95NoiseExcessMs'` (`scripts/verify-velocity-history.ts:189`). That arm prices actual frame submission against split-control noise, so a runner twenty other jobs share reports their load as render cost: it is a measurement, not a correctness gate. `scripts/verify-velocity-history.ts` takes `--variant=<history|cost|both>` (default `both`). The integration lane runs `--variant=history`, so the merge path keeps the authored motion/control variants, the pixel comparison, the `velocity.tsconfig.json` type check and the unit specs that cover the cost code. A new scheduled `velocity-cost` job in `performance-regression.yml` runs `--variant=cost` on a lane that is not on the merge path, and `summary.json` records which arm produced it. Verified: pnpm typecheck exit 0; biome clean; 397 CI contract tests pass (ci-structure, ci-needs, ci-integration-scope, ci-qualification, ci-template-selection, ci-efficiency); both arms run locally under Xvfb — `--variant=history` exit 0 with 13 variants and no cost arm, `--variant=cost` exit 0 with both cost arms and no history variants; an unknown `--variant` exits 1 with the usage error. PRD-550 phase 1 is IN PROGRESS: the auto-exposure frame-rate budget and the `test-unit` classification are open, and the phase boxes need CI run ids.
…k (PRD-550 phase 1)
A triage of three failing `test-unit` shards (jobs 113669815836, 113540990094,
113540990242) found seven failing items, all wall-clock expiries or a
native-host wait, and no code red. The compile specs are the ones whose budget
does not fit: 193.7 s and 215.0 s for 22 and 18 cases locally, 25-45 s per
template against the shared 60 s unit-test default, so a merge-group runner
two to three times slower than an idle machine turned a compile that was
merely concurrent into `Test timed out in 60000ms`.
Both halves now set `{ timeout: 180_000 }` on the suite, following the
`check-native-shims.spec.ts` precedent. `world-gpu-scene.spec.ts` (70 cases in
60.7 s) and `check-capability-docs.spec.ts` (15 in 13.9 s) are fast when the
machine is not contended, so their reds are the shared-runner class and stay
with phase 3 capacity instead of getting a wider budget.
Verified: the four named specs pass on an idle machine, 125 cases, exit 0,
218 s; biome clean.
Iteration 3 measures the 2026-10-09 window again: 0 PRs merged, 9 merge-group runs, board median 113.3 min, queue wait p90 49.9 min. Integration / velocity was the first failing job in 2 of 4 red merge groups. The change under review is PRD-550 phase 1 (481824f, a478510). The iteration stays provisional until that change merges and the window moves. Re-run before this push: - scripts/__tests__/ci- contract specs: 18 files, 546 tests pass. - template-assets-compile-1 and -2: 40 of 40 cases pass in 220 s. - scripts/__tests__/ci-speed-loop.spec.ts: 15 of 15 pass after the ledger update.
Same 2026-10-09 window: 0 PRs merged, 12 merge-group runs. PR 476 head 6ec34f5 has 69 checks passing and 12 skipped, with merge state CLEAN. integration / velocity and the iOS simulator job pass on that head.
… dry run dies" This reverts commit 8257404.
A null exit status from the dry run now reports the spawn error code and the signal. Only the dry-run assertion changes; the 120000 ms ceiling and every other assertion stay as they were. Verified: conformance-runner.test.mjs 76 of 76 pass, 29 s.
…prd-550-merge-in-30
readAttemptJobs now makes 5 attempts on connection errors, with 1, 2, 4 and 8 s back-off. Refs: CI run 38018731032, head bafcac5.
|
Published candidate Local checks on the merged candidate: lint, typecheck, docs/link checks and 240 tests across five focused CI/dashboard specs passed. Typecheck required rebuilding stale ignored core declarations; no source repair was needed. Pre-push drift checks passed. The earlier full The previous run 38025989864 attempt 2 now reports success, but it qualified the old The local HTML dashboard refreshes every 10 minutes. Its 18 iterations still describe the unmerged develop baseline; projected planner savings remain projections, not measured PR impact. Latency and runner-queue acceptance remain open. |
|
Dashboard refreshed to iteration 19 at 2026-10-10T07:20:48Z. This is still an unmerged develop baseline, now covering 8 merged PRs, 44 merge-group runs and 3,035 queue observations.
The sample changed because two other PRs merged. The lower median is not evidence of PR476 impact. Candidate |
|
Fresh PR candidate CI is green: run 38033595736, attempt 1 completed successfully at 2026-10-10T07:56:51Z on
The existing PR is ready for João's review. No merge performed. PRD550's latency, queue and repeated-run acceptance remain open; this green run does not close them. |
|
Runner capacity repair is live and committed in The balancer cached its own last routing write, so deleting a routing variable outside the loop left idle local runners unused. It now reconciles both variables against GitHub on each poll, preserves two-poll idle debounce and stop-file protection, and corrects wrong labels. Only the controller was replaced; runner containers and native coverage remain unchanged. Both expected labels were restored at 2026-10-10T13:34:48Z. This controller pickup is temporary across service/boot until the deployed source receives the fix.
The observed older merge-group tail was hosted templates, not native jobs. Hosted template placement is intentional and remains unchanged; no native jobs were removed. The earlier 65→41 template-kit projection is reduced work, not a measured 37% wall-time improvement. PRD phases and latency acceptance remain open. PR476 remains unmerged. |
release-reports only checks out, installs and assembles JSON reports from downloaded parity artifacts - no compile or GPU leg - yet it selected the heavy TN_RUNNER pool and queued behind builds while light slots sat idle. It now uses the TN_RUNNER_LIGHT switch; its condition, timeout, steps, receipts and every native/parity build label are unchanged. The light-lane contract in ci-structure.spec.ts now covers release-reports (red before the workflow change, green after). PRD-550 phase 3 records the placement; live queue-wait effect unverified.
|
CI checkpoint on
Auto-merge was enabled at the owner's request and remains blocked by required checks. No direct/admin merge or receipt bypass. |
|
Stability checkpoint: Prior full attempt finished in 39.18 minutes. Its only runtime failure was this test exceeding 15 seconds; native coverage, other unit shards and movement playtest passed. Why that build slowed in CI remains unproved. A failed-job-only attempt appeared afterward; I did not initiate it and will not use it to claim fresh receipt coverage. Fresh candidate qualification is pending. Auto-merge remains enabled and blocked by required checks. Stability, repeated merge-queue behavior and the 30-minute latency target remain unproved; no phase or acceptance box was closed. |
|
Fresh qualification is green on 6e49570: https://github.com/ThreeNativeHQ/threenative-engine/actions/runs/38073264985 69 jobs passed, including all four unit shards, native parity, Integration and ci-required. The only skipped job was the conditional Android V8 publishing job. The exposure import smoke test now passes in CI after moving capture-only imports into main; its existing 5-second limit and assertions remain unchanged. Native collector evidence aggregation also passed on the light runner lane. The full board took approximately 48 minutes from run creation to completion. This is below the recorded 95.1-minute historical median, but above the 30-minute target; it is one fresh green candidate, not proof of sustained stability or causal speedup. PRD-550 latency and repeated-stability acceptance remain open. Protected auto-merge has advanced this PR into merge queue, position 1, awaiting fresh merge-group checks. No admin merge or check bypass was used. |
|
The fresh merge-group qualification also passed: https://github.com/ThreeNativeHQ/threenative-engine/actions/runs/38076471863 69 jobs passed, including ci-required, on merge candidate 51271e0. Runtime was approximately 50 minutes. Combined with the fresh PR board, the reported regression has now passed both qualification contexts. This still does not establish sustained stability or the 30-minute target. PR476 remains open. GitHub's timeline records a manual queue removal by jonit-dev at 2026-10-10T18:37:49Z, before this merge-group run completed. The mergeQueueEntry and autoMergeRequest are now null. I have not overridden that manual removal; the owner has been asked whether to restore protected auto-merge. No direct/admin merge or bypass was used. |
1 participant
PRD-550 phase 2, first half: the board the queue runs.
What changes
merge_groupon develop selects the same narrowed plan a pull request into develop earns, and takes the reduced native tier under the same rule. An unresolved queue entry still owes every row, because it has no candidate identity to narrow by.fullinstead of thewarmcache-only lane. The exhaustive board moves to the event that promotes a tree; both cache publishers (build-artifacts,test-native) stay inside it, so the base-branch caches every pull request reads keep being published.AGENTS.mdstates the new rule (pnpm sync:agentsmirror regenerated).Dashboard
docs/ci-speed/index.htmlgains iteration #2 of the speed loop (window since 2026-10-09): merge-group board wall 119.1 min (goal 15), queue wait p90 50.9 min (goal 3),integration / velocityred twice. 0 PRs merged in that window, so the merge-derived metrics aren/aand the decision isprovisional; iteration #1 stays the baseline. The page now names the command that regenerates it instead of claiming a reload does, and a window that merged no PR keeps the last window that did in the PR table.Verified here
pnpm typecheck— exit 0. This worktree had never been built, so its first run reported 168 unresolved-workspace-import errors atf0e8e76afand at HEAD alike; building@threenative/assets,@threenative/physics,create-threenative,@threenative/procedural-animalsand@threenative/uimade it green. That first, identical failure is why 90dee0c records typecheck as unverified.pnpm exec vitest runoverci-structure,ci-needs,ci-integration-scope,ci-qualification,ci-template-selection,ci-efficiency,ci-speed-loop— 411 + 15 pass.pnpm exec biome checkon the touched scripts — no new findings (the 4 complexity warnings onci-change-scope.mjsare unchanged fromf0e8e76af).pnpm sync:agents --check— 23 mirrors in sync. Pre-pushci:fast— agents, docs, lint, drift pass.Open
test-unitleavingUNPROVEN_REUSE_BOARDS, theci-janitorrevert PR, and the ≤ 15 min wall all need a CI run id.