fix: stop the Windows CI job timing out on integration tests - #34
Merged
Merged
Conversation
The Windows job has been failing intermittently on main while Ubuntu passes, on a different handful of tests each run: three from the locked-protocol guards on 602640d, eight across those and the dashboard tests on f7242fb. Every one failed with "Test timed out in 5000ms" rather than on an assertion, which is a scheduling problem, not a regression. 5s is Vitest's default and was never chosen for this suite. The tests that hit it do real work rather than compute: the locked-protocol guards initialise scratch git repositories and run the CLIs as child processes, and the dashboard tests build the site and hash artifacts. Measured on a developer machine they peak around 2.1s each, and Windows runners are several times slower at process spawning and file I/O, which puts the slowest of them close enough to the ceiling to cross it some runs and not others. testTimeout and hookTimeout are now 20s. That is headroom rather than permission to be slow: a genuinely hung process still fails the run, just later, and any test approaching this ceiling is doing something worth looking at. Nothing about the tests themselves changes. Full suite 552 passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The badge was stale, and the failure was flaky
Main is green: the run on
0f0c5be, which includes all four merged PRs, passed on both Windows and Ubuntu. The README badge was still showing the earlier failed run on602640de.That earlier failure was not caused by those merges. Each of the four branches passed CI on its own, and main had already failed the same way on
f7242fbdon 8 September, before any of that work existed.What is actually wrong
The Windows job fails intermittently while Ubuntu passes, on a different handful of tests each time:
602640degate4b.test.tsf7242fbdgate4b.test.tsanddashboard.test.tsEvery one is
Test timed out in 5000msrather than a failed assertion. A different subset failing each run, on one operating system, with no assertion involved, is a scheduling problem.5s is Vitest's default and was never chosen for this suite. The tests that hit it do real work rather than compute: the locked-protocol guards initialise scratch git repositories and run the CLIs as child processes, and the dashboard tests build the site and hash artifacts. Measured locally on Windows they peak at 2129ms for a single test, and GitHub's Windows runners are several times slower at process spawning and file I/O. That puts the slowest tests close enough to the ceiling to cross it on some runs and not others.
The change
testTimeoutandhookTimeoutset to 20s, with the reasoning recorded in the config so the next person does not have to re-derive it.This is headroom, not permission to be slow. A genuinely hung process still fails the run, just later, and any test that actually approaches this ceiling is doing something worth looking at. Nothing about the tests themselves changes.
Full suite 552 passing, typecheck clean.
Worth knowing
This does not make the suite faster, and a slow test is still a real cost on every run. If the Windows job stays close to the edge after this, the better fix is to stop those guards spawning a fresh git repository per test and share one across the describe block, which would cut most of the 14.8s that file currently takes. That is a larger change than a flaky-CI fix should carry, so I have left it.
Note
Low Risk
Overview
This PR addresses intermittent Windows CI failures in integration tests by modifying
vitest.config.ts(adding 17 lines) to configure test timeouts and/or parallelism settings appropriate for the Windows runner. The change ensures the Windows job can complete before GitHub Actions' default job timeout, since the prior failures were caused by tests exceeding the time limit rather than actual test regressions. The fix is justified because main was passing on the relevant commits and the Windows-specific timeout behavior had been flaky across multiple unrelated runs.Written by Gitzilla for commit a875660. This will update automatically on new runs. Configure in the Gitzilla dashboard.