Skip to content

🐛 fix(ci): stop nightly unit-test shard hangs and singular-error false passes - #24672

Merged
hivecommons-hive[bot] merged 1 commit into
mainfrom
scanner/fix-24572
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
mainfrom
scanner/fix-24572

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Refs #24572 — left open until the next nightly passes.

What's actually failing

The nightly job log only shows passing tests. The real failures are in suite-unit-test.log in the nightly-test-logs-<date> artifact. All three recent runs fail on the same two shards:

Run Image Shard 19 Shard 26 Shard 34
40277752379 (10-26) ubuntu-24.04 auth.test OOM (fixed by #24637) TIMED OUT after 900s TIMED OUT after 900s
40458881935 (10-27) ubuntu-26.04 2 auth.test timeouts (flaky, passed 10-28) TIMED OUT after 900s TIMED OUT after 900s
40641476424 (10-28) ubuntu-26.04 pass TIMED OUT after 900s FATAL ERROR: Ineffective mark-compacts near heap limit → Worker exited unexpectedly with signal SIGABRT … useUpgradeStateMachine.test.tsx, Errors 1 error

Shards 26 and 34 also fail on the 10-26 ubuntu-24.04 run, so the runner image is not the root cause. #24608 and #24637 did not touch either of them.

Root causes and fixes

  1. Shard 26 is src/components/missions/MissionBrowser.test.tsx. I reproduced vitest's shard split (sha1 of the path, even split over 2805 files). It matches all 58 files of shard 26 exactly. Files run largest first, and the hang starts at the first file that never reports, which is MissionBrowser.test.tsx.
    • The cause is its useToast mock, which returned { showToast: vi.fn() }, so showToast was a new function on every render.
    • useMissionTree's root-node effect has showToast in its deps and calls setTreeNodes(newArray) / setExpandedNodes(new Set()) every time it runs. So the effect re-runs on every render and never stops.
    • React does not stop passive-effect loops, and inside act() this one runs on microtasks. That blocks the event loop, so the reporter never flushes and the shard hits the 900s cap.
    • The sibling test __tests__/MissionBrowser.test.tsx renders the same component with a stable showToast, and it passes in the same run.
    • Fix: the mocks now return stable hoisted values for toast, navigate and location. useAuth and useClusterContext are mocked too, like the sibling test. Without that mock, useClusterContext pulls in five live MCP data hooks.
  2. Shard 34 is src/hooks/__tests__/useUpgradeStateMachine.test.tsx. It stalls right after the last real-timer test ("fetches versions … in parallel on connect") in all three runs, and in coverage-hourly run 40376902665 too. The next tests are the first vi.useFakeTimers() tests.
    • Vitest 5's bare useFakeTimers() fakes every timer-like global except nextTick/queueMicrotask, including setImmediate, performance, Date, Intl and rAF.
    • Fix: fake only setTimeout/setInterval (and their clears). That is all the hook uses, and it matches the existing toFake pattern in src/lib/unified/__tests__/registerHooks-*.test.ts.
    • ⚠️ Unlike shard 26, I could not pin this one to a specific line from the logs, so treat it as the most likely cause rather than a confirmed one.
  3. scripts/unit-test.sh reported worker crashes as passing. The check Errors +[1-9][0-9]* errors did not match vitest's singular Errors 1 error (shard 34 on 10-28). The regex is now errors?, and it is also applied to the per-shard "hung after completion → pass" path.
  4. Runner pin. runs-on: ubuntu-latest became ubuntu-26.04 on 10-27 (see the runner image line in both runs). The nightly job is now pinned to ubuntu-24.04, the same way 🐛 fix(ci): pin coverage-hourly shard job to ubuntu-24.04 #24667 pinned coverage-hourly.

Not changed

  • The "Regression recurred" comment on a run whose job shows success is not a false positive. unit-test really failed (summary.failed > 0). The job only goes red when nightly-compare.sh sees a new regression compared with the previous night, which it doesn't for a suite that was already failing.
  • Nothing was run locally. The diagnosis comes only from CI logs and artifacts, and CI is the verdict.

— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

…e passes

- MissionBrowser.test.tsx: return stable useToast/useNavigate/useLocation
  values and mock useAuth/useClusterContext. A fresh showToast per render
  re-fired useMissionTree's root-node effect (setTreeNodes) forever, hanging
  nightly shard 26.
- useUpgradeStateMachine.test.tsx: fake only setTimeout/setInterval instead
  of vitest 5's fake-everything default (shard 34 hang/OOM).
- unit-test.sh: match singular 'Errors  1 error' so worker OOMs fail.
- nightly-test-suite.yml: pin runner to ubuntu-24.04 (same as #24667).

Refs #24572

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@netlify

netlify Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for kubestellarconsole ready!

Name Link
🔨 Latest commit bc6f865
🔍 Latest deploy log https://app.netlify.com/projects/kubestellarconsole/deploys/6ae1b945ff3d5a8300dad9f7
😎 Deploy Preview https://deploy-preview-24672.console-deploy-preview.kubestellar.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions github-actions Bot added the ai-generated Pull request generated by AI label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hey @hivecommons-hive[bot] — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.


This is an automated message.

@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ Test Coverage Check

All new source files in this PR have corresponding test files.

Checked web/src/hooks/ and web/src/components/ against origin/main.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: b74e25a32bb0c6a94d9b01f358f7f8251d321fc9

— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

♿ Accessibility Audit (WCAG 2.1 AA)

✅ No WCAG 2.1 AA violations detected in audited routes.


Powered by axe-core. Target: WCAG 2.1 AA compliance.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
bc6f865

ℹ️ Sentinel notice — informational; author is trusted (hive agent), merge is not blocked

Hive flagged this PR (author @hivecommons-hive[bot], head bc6f86529985) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/nightly-test-suite.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the diagnosis against the tree at bc6f865; this looks correct to me.

Shard 26 (MissionBrowser.test.tsx): confirmed the mechanism. useMissionTree.ts:219 lists showToast in the root-node effect's deps, and that effect unconditionally calls setTreeNodes(rootNodes) (new array) and setExpandedNodes(new Set()) (useMissionTree.ts:215-218). The old mock's useToast: () => ({ showToast: vi.fn() }) returned a fresh function every render, so the effect re-fired on every render it caused — an unbounded loop. The hoisted stable mocks break it. Also checked the new mock specifiers: MissionBrowser.tsx:8 imports useAuth from ../../lib/auth and :10 imports useClusterContext from ../../hooks/useClusterContext, exactly matching the vi.mock paths, so both mocks will actually take effect.

Shard 34 (useUpgradeStateMachine.test.tsx): useUpgradeStateMachine.ts uses only setInterval/clearInterval (retry + TTL loops) plus promise microtasks, so the narrowed toFake: ['setTimeout','clearTimeout','setInterval','clearInterval'] covers everything the hook schedules. beforeEach/afterEach both call vi.useRealTimers(), so the narrowing can't leak across tests. The PR body is candid that this one is "most likely" rather than log-pinned, which seems like the right epistemic framing.

unit-test.sh: errors? now matches vitest's singular Errors 1 error; [1-9][0-9]* still excludes 0 errors. Applying the same check to the per-shard "hung after completion → pass" path closes the same hole where a worker SIGABRT inside a timed-out shard was treated as pass.

Runner pin: mirrors the coverage-hourly pin (#24667) with a revisit note; the body correctly notes shards 26/34 also failed on ubuntu-24.04, so the pin is defense-in-depth, not claimed as root cause.

No findings. Only CI (the next nightly) can confirm shard 34, which is why Refs #24572 staying open until it passes is the right call.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 9b9c5a7 into main Oct 28, 2026
58 of 59 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-24572 branch October 28, 2026 09:45
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#24672bc6f865` sha256:b457c3220d1ef1103901560d6e1d2d25a7725df56cfb41ae49ea719df71c9e68 — download from the dashboard or `hivectl review evidence #24672`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Post-merge build verification passed ✅

Both Go and frontend builds compiled successfully against merge commit 9b9c5a7facc88fa1206b2031b4a985c4ff81f987.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Post-Merge Verification: passed

Commit: 9b9c5a7facc88fa1206b2031b4a985c4ff81f987
Specs run: Missions.spec.ts mission-import.spec.ts smoke.spec.ts
Report: https://github.com/kubestellar/console/actions/runs/40666692594

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated Pull request generated by AI author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tier/3-restricted

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants