Repository navigation
CI flake: coverage job fails intermittently in vendor / in_process_edge_cases #1416
Description
The coverage job (the Linux leg of test, which feeds ci-ok) failed twice in about 3 hours on commits where the same SHA also passed. Each time a different test binary failed:
| Run | PR / branch | SHA | Failed target | Same SHA passed in |
|---|---|---|---|---|
| 38184196576 (job 114603173378) | agent/fix-go-selected-version-gate | 3add9b2 | -p socket-patch-cli --test vendor |
38184171663 |
| 38194961803 attempt 1 (job 114635755196) | #1410 | e1b04a6 | -p socket-patch-cli --test in_process_edge_cases |
attempt 2 of the same run |
Neither PR touches the failing binaries: #1410 changes only tests/repair/*, tests/scan/* and tests/cli/shared_helper_copies.rs. Nothing else failed in the last 48h of CI except one main push, where a Depot runner lost DNS to static.rust-lang.org for 2+ minutes and rustup-retry.sh ran out of attempts (38182319501). That one was infrastructure.
What the janitor could and couldn't check (2026-10-12 run):
- The routine sandbox can't fetch Actions log blobs (
*.blob.core.windows.netis denied), and the MCP log tool returns only the last 5,000 of ~16,000 lines. The panic message and test name for both failures are in the part it couldn't read. - No local repro: 1,260 runs of the
in_process_edge_casesbinary (16 concurrent workers on 4 cores) and 260 runs of thevendorbinary (10 concurrent workers) all passed, without llvm-cov instrumentation.
Next step: someone with log access should run gh run view 38194961803 --attempt 1 --log-failed and gh run view 38184196576 --log-failed, grep for ---- / panicked, and cluster by test name. Under the merge queue, a flake like this evicts the entry and costs a full rebuild.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 12, 2026
[agent] Triaged as p3 (CI-only). I hit the same wall as the janitor: the job log endpoint redirects to
*.blob.core.windows.net, which this sandbox gets a 403 for, so I could not read the failing test names for jobs 114635755196 / 114603173378 either. Labeledagent:needs-humanuntil someone with log access posts the---- <test> ----/panickedlines; with those, an agent can root-cause it.Generated by Claude Code
mikolalysenko commentedon Oct 12, 2026
[agent] Another occurrence:
coverageon #1421 (SHA ab0ffcb, job 114727527547) failed in-p socket-patch-cli --test in_process_edge_cases. #1421 only touches the Composer crawler's global-home lookup. The panic text is again outside the last 5,000 lines the log tool returns. I've re-run the job once.Generated by Claude Code
mikolalysenko commentedon Oct 12, 2026
[ci-janitor] #1418 should fix this. It reproduced
in_process_edge_casesfailing 8 of 60 runs locally with--test-threads=16. Both failures were in thecache_envscrub self-tests (ambient_bun_scrub_removes_only_matching_parent_vars/ambient_vlt_scrub_removes_only_matching_parent_vars, "SOCKET_OFFLINE ... not scrubbed"). The cause is a race with in-process tests thatset_var("SOCKET_OFFLINE"). After the fix, 200 of 200 runs passed.common/cache_env.rsis linked into about 120 test binaries, so the same race also explains the--test vendoroccurrence.#1418 is green (
ci-okandclippypass) and is waiting on human review. This run (2026-10-12 11:40 UTC) found no other flake in the last ~36h of CI. The macOS nightlygo_work_that_does_not_use_the_project_makes_the_replace_inertfailure is covered by #1411, which is also green and waiting on review.Generated by Claude Code
Port #1418 cache_env scrub snapshot fix
mikolalysenko commentedon Oct 12, 2026
[ci-janitor] Status 2026-10-12 17:55 UTC: #1418 merged at 16:10 (95e880e). Every
in_process_edge_casesfailure since this issue opened ran on a base older than that commit:All six merge_group runs after the merge passed
coverage: #1407, #1413, #1430, #1421, #1428 (rebuilt on 3b12cb7) and #1433. Nocoveragefailure has been recorded since 16:01 UTC. Locally, 440 runs of the binary at main 1bd3da5 passed (200 sequential, 240 under 4× contention without llvm-cov), so I have no local repro either way.I think this can be closed once a few more hours of merge-queue runs stay green. #1438 (failure annotations for
coverage) is still open and would make any recurrence diagnosable without log-blob access.Generated by Claude Code
mikolalysenko commentedon Oct 12, 2026
[ci-janitor] Status 2026-10-12 19:50 UTC: closing as fixed by #1418.
coveragejob has failed on a base that includes it. The only failures in the last 24h ran on older bases (listed above) or were real defects on PR branches. For example, Fix Cargo source replacement being ignored (#338, #455) #1420'sformat_models_are_purearchitecture check was fixed before it merged.in_process_edge_caseswith--test-threads=16all passed.coveragefailure happens again, its panic will be posted as a check-run annotation, so a new issue can name the test.Generated by Claude Code