ci: improve backend coverage reliability and reduce redundant work

[ctkm-aelf]

Result

Improve CI reliability while preserving the full test suite, feature checks and coverage gates. This PR targets main. Final validation of head e1585ea9 includes main 14d76f4a (#1716), tested together as merge commit 15cf1bf7. It changes CI workflows/helpers and fixes one observed race in the test harness; production behavior is unchanged.

Changes

• Establish at least 8 GiB total swap on the four heavy backend runners, allocating only the deficit and retaining 8 GiB free disk. Fail setup if the reserve cannot be established. Compiler and test concurrency are unchanged.
• Pin Rust 1.98.1, cargo-nextest 0.9.146, cargo-llvm-cov 0.9.1 and the MongoDB 8 image digest. Cache informational base reports only for the exact source SHA, measurement recipe and actual runner image/architecture; missing identity forces a fresh measurement.
• Bound dependency installation to five minutes, with noninteractive apt, 20-second acquisition timeouts and two acquisition retries. Update/install failures fail the job. Package signatures remain enforced; tests are never retried.
• Give the full backend suite and both backend coverage jobs a 90-minute outer limit; billing smoke retains 45 minutes. An earlier 45-minute coverage limit cut off a progressing suite, and a later passing backend job took 37m14s. These are failure limits, not expected runtimes.
• Record memory/swap, disk, CPU/load/pressure, process/cgroup observations, tested commit/tree and tool versions every 30 seconds, with 14-day diagnostic artifacts. Preserve command errors and cancellation; optional telemetry cannot turn failures into success.
• Fix the demonstrated NyxBot direct-group test race: a callback returns 202 before background processing completes, but the test previously relied on a 100 ms sleep. A test-only completion observer now waits for each real callback, including silent messages, before checking exact results. Registrations are isolated by channel/message and removed on timeout or cancellation. All original message cases and assertions remain; the sole handler hook is under cfg(test).
• Remove unused initial coverage-report passes with --no-report, retaining instrumented execution, exports and threshold enforcement. Cancel superseded PR CodeQL scans while preserving all four languages. Include Mobile failures/cancellations in the aggregate CI gate.

Coverage contract

Keep the full backend and service-adapter suite, standalone billing smoke, production gcp-kms build and embedded-input guard, AWS/GCP/combined feature builds, CLI, frontend, mobile, SDK, Oracle, wizard and plugin checks. Gates remain 73% backend / 64% CLI / 15% frontend. No test exclusions, additional ignores/skips, retries, looser assertions or reduced feature matrices are introduced. Head coverage is always freshly measured.

Evidence and validation

Final head e1585ea9 passes CI, CodeQL and Release. Both independent backend coverage jobs and all four CodeQL language scans passed. Release publishing/build steps are intentionally skipped for a PR.

Coverage	Base	Head	Unchanged gate
Backend	87.44%	87.44%	73%
CLI	72.45%	72.45%	64%
Frontend	71.00%	71.00%	15%

• Full backend nextest: 6,873 passed / 2 skipped. Standalone billing smoke: 1 passed, retaining its separate Cargo feature-unification check.
• Backend coverage head/base each report 6,858 passed / 2 ignored / 0 filtered. Comparing the logs confirms exactly the same 6,860 named tests and outcomes, including should-panic cases.
• Both backend reports measure the same 509 source files. Instrumented lines are 379,717 base / 379,718 head; every per-file denominator matches except one added measured line for the cfg(test) callback-completion hook in nyxbot.rs. No measured source scope was removed. Covered lines are 332,009 base / 332,023 head; this small execution variation is not claimed as a CI-induced coverage improvement.
• CLI head/base each pass 1,377 tests. Frontend head/base each pass 3,990 tests in 391 files. All feature builds, production backend image/input guard, formatting, Clippy, mobile, SDK, Oracle, wizard, plugin and aggregate checks passed.
• All four heavy backend commands exited 0 and had at least 8.002 GiB total swap. Sampled peak swap usage: full suite 2.915 GiB, billing smoke 2.765 GiB, head coverage 3.259 GiB, base coverage 3.310 GiB. Both coverage jobs exceeded the former 3 GiB capacity. Resource logs retain actual runner image identity; head and base used different hosted image revisions, so their runtimes are not a controlled speed comparison.
• Nineteen focused helper tests, Actionlint, ShellCheck, Rust formatting and diff checks pass. The focused direct-group regression passes against real MongoDB locally (1 passed, no retries) and in the final hosted suites. Independent review found no blocking issue in failure propagation, cache identity, resource provisioning, timeout margins or callback synchronization. The reviewer checked cargo-llvm-cov 0.9.1's implementation: --no-report skips reporting after normal instrumented test execution.

Earlier validation exposed issues that were corrected before this final run. A 45-minute coverage limit interrupted a progressing suite; a subsequent passing full backend job took 37m14s, so both full-test and coverage bounds now allow 90 minutes. In run 36741290868, the required gate passed but informational base coverage failed the direct-group test from #1712 (6,857 passed / 1 failed / 2 ignored), revealing the asynchronous test race fixed here. That failed base produced no complete coverage report and is not counted as all-green validation. The final run above passed both head and base.

The September 30 audit found 16 compilation shutdowns and one assertion failure among 17 inspected failed coverage jobs; none failed the percentage gate. The shutdown signature predates the parallel split. Resource observations support more capacity and better diagnostics, but do not establish OOM or exhaustion of GitHub-wide compute as the root cause. A passing pipeline cannot establish zero future flakes or guaranteed speedup. Safe savings here come from removing unused reports and obsolete scans; database semantics and useful parallelism are retained. Historical audit and implementation details: docs/plans/ci-robustness-and-latency.md.

[github-actions]

📊 Code coverage

Component	Lines	Threshold	Status	Δ vs base
Backend (nyxid)	87.44%	73%	✅	— 0.00
CLI (nyxid-cli)	72.45%	64%	✅	— 0.00
Frontend (vitest)	71.00%	15%	✅	— 0.00

Gate: line coverage must stay at or above the threshold. Ratchet plan (W21): Backend → 55%, CLI → 50%, Frontend → 30% by quarter end.