Repository navigation
🐛 fix(ci): stop nightly unit-test shard hangs and singular-error false passes - #24672
Conversation
…e passes - MissionBrowser.test.tsx: return stable useToast/useNavigate/useLocation values and mock useAuth/useClusterContext. A fresh showToast per render re-fired useMissionTree's root-node effect (setTreeNodes) forever, hanging nightly shard 26. - useUpgradeStateMachine.test.tsx: fake only setTimeout/setInterval instead of vitest 5's fake-everything default (shard 34 hang/OOM). - unit-test.sh: match singular 'Errors 1 error' so worker OOMs fail. - nightly-test-suite.yml: pin runner to ubuntu-24.04 (same as #24667). Refs #24572 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
✅ Deploy Preview for kubestellarconsole ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
👋 Hey @hivecommons-hive[bot] — thanks for opening this PR!
This is an automated message. |
✅ Test Coverage CheckAll new source files in this PR have corresponding test files. Checked |
|
hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88 |
♿ Accessibility Audit (WCAG 2.1 AA)✅ No WCAG 2.1 AA violations detected in audited routes. Powered by axe-core. Target: WCAG 2.1 AA compliance. |
bc6f865
ℹ️ Sentinel notice — informational; author is trusted (hive agent), merge is not blockedHive flagged this PR (author @hivecommons-hive[bot], head
Hive did not add the |
There was a problem hiding this comment.
Verified the diagnosis against the tree at bc6f865; this looks correct to me.
Shard 26 (MissionBrowser.test.tsx): confirmed the mechanism. useMissionTree.ts:219 lists showToast in the root-node effect's deps, and that effect unconditionally calls setTreeNodes(rootNodes) (new array) and setExpandedNodes(new Set()) (useMissionTree.ts:215-218). The old mock's useToast: () => ({ showToast: vi.fn() }) returned a fresh function every render, so the effect re-fired on every render it caused — an unbounded loop. The hoisted stable mocks break it. Also checked the new mock specifiers: MissionBrowser.tsx:8 imports useAuth from ../../lib/auth and :10 imports useClusterContext from ../../hooks/useClusterContext, exactly matching the vi.mock paths, so both mocks will actually take effect.
Shard 34 (useUpgradeStateMachine.test.tsx): useUpgradeStateMachine.ts uses only setInterval/clearInterval (retry + TTL loops) plus promise microtasks, so the narrowed toFake: ['setTimeout','clearTimeout','setInterval','clearInterval'] covers everything the hook schedules. beforeEach/afterEach both call vi.useRealTimers(), so the narrowing can't leak across tests. The PR body is candid that this one is "most likely" rather than log-pinned, which seems like the right epistemic framing.
unit-test.sh: errors? now matches vitest's singular Errors 1 error; [1-9][0-9]* still excludes 0 errors. Applying the same check to the per-shard "hung after completion → pass" path closes the same hole where a worker SIGABRT inside a timed-out shard was treated as pass.
Runner pin: mirrors the coverage-hourly pin (#24667) with a revisit note; the body correctly notes shards 26/34 also failed on ubuntu-24.04, so the pin is defense-in-depth, not claimed as root cause.
No findings. Only CI (the next nightly) can confirm shard 34, which is why Refs #24572 staying open until it passes is the right call.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. Check out what's new:
Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey |
|
Post-merge build verification passed ✅ Both Go and frontend builds compiled successfully against merge commit |
✅ Post-Merge Verification: passedCommit: |
Refs #24572 — left open until the next nightly passes.
What's actually failing
The nightly job log only shows passing tests. The real failures are in
suite-unit-test.login thenightly-test-logs-<date>artifact. All three recent runs fail on the same two shards:TIMED OUT after 900sTIMED OUT after 900sTIMED OUT after 900sTIMED OUT after 900sTIMED OUT after 900sFATAL ERROR: Ineffective mark-compacts near heap limit→Worker exited unexpectedly with signal SIGABRT … useUpgradeStateMachine.test.tsx,Errors 1 errorShards 26 and 34 also fail on the 10-26 ubuntu-24.04 run, so the runner image is not the root cause. #24608 and #24637 did not touch either of them.
Root causes and fixes
src/components/missions/MissionBrowser.test.tsx. I reproduced vitest's shard split (sha1 of the path, even split over 2805 files). It matches all 58 files of shard 26 exactly. Files run largest first, and the hang starts at the first file that never reports, which is MissionBrowser.test.tsx.useToastmock, which returned{ showToast: vi.fn() }, soshowToastwas a new function on every render.useMissionTree's root-node effect hasshowToastin its deps and callssetTreeNodes(newArray)/setExpandedNodes(new Set())every time it runs. So the effect re-runs on every render and never stops.act()this one runs on microtasks. That blocks the event loop, so the reporter never flushes and the shard hits the 900s cap.__tests__/MissionBrowser.test.tsxrenders the same component with a stableshowToast, and it passes in the same run.useAuthanduseClusterContextare mocked too, like the sibling test. Without that mock,useClusterContextpulls in five live MCP data hooks.src/hooks/__tests__/useUpgradeStateMachine.test.tsx. It stalls right after the last real-timer test ("fetches versions … in parallel on connect") in all three runs, and in coverage-hourly run 40376902665 too. The next tests are the firstvi.useFakeTimers()tests.useFakeTimers()fakes every timer-like global except nextTick/queueMicrotask, includingsetImmediate,performance,Date,Intland rAF.setTimeout/setInterval(and their clears). That is all the hook uses, and it matches the existingtoFakepattern insrc/lib/unified/__tests__/registerHooks-*.test.ts.scripts/unit-test.shreported worker crashes as passing. The checkErrors +[1-9][0-9]* errorsdid not match vitest's singularErrors 1 error(shard 34 on 10-28). The regex is nowerrors?, and it is also applied to the per-shard "hung after completion → pass" path.runs-on: ubuntu-latestbecame ubuntu-26.04 on 10-27 (see the runner image line in both runs). The nightly job is now pinned toubuntu-24.04, the same way 🐛 fix(ci): pin coverage-hourly shard job to ubuntu-24.04 #24667 pinned coverage-hourly.Not changed
unit-testreally failed (summary.failed > 0). The job only goes red whennightly-compare.shsees a new regression compared with the previous night, which it doesn't for a suite that was already failing.— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88