Skip to content

Let a slow or loaded machine hold the tests it runs. - #56

Open
unRekable wants to merge 5 commits into
mainfrom
fix/test-timing-on-slow-machines
Open

unRekable wants to merge 5 commits into
mainfrom
fix/test-timing-on-slow-machines

Conversation

@unRekable

Copy link
Copy Markdown
Collaborator

What

Three timing assumptions that hold on a fast CI runner and broke on a slow, loaded Windows machine, found while running the workspace suite locally:

  • The ring test (calls::a_ring_we_placed…) asked the demo for its answer 2,500 ms in and wanted our side's join to have outrun it. On a cold machine, where starting mediad alone can take a second, the answer won: "the answer was not outrun". The answer comes at 6 s now, and the two asserts take their margins from it.
  • The polish round started twelve daemons at once against a 10 s socket deadline: on a cold machine every start missed it ("chatd did not open its socket", 8 of 12 at once). Starts go one after another now, the repo's own alone() pattern; the daemons keep running side by side.
  • The keychain test measured from before the daemon's spawn, so a slow start counted against "the socket waited on the keychain", and a start slower than the read's 1,500 ms missed the window the rest of the test assumes. The clock runs on the request alone now, and a start that missed the read says what it saw instead of failing on it.

Tests

cargo test -p chatd --test polish 4× green (12/12, was failing 1–3 per run), cargo test -p chatd --test calls 2× green (10/10). Full suite green on this machine with #55's branch, CI there all pass.

The ring test asked the demo for its answer 2,500 ms in and wanted our side's join to have outrun it: on a cold machine, where starting mediad alone can take a second, the answer won. The answer comes at 6 s now, and the two asserts take their margins from it. The polish round's daemons start one after another, so twelve fresh processes at once cannot push a cold machine past the socket's ten seconds; they keep running side by side. And the keychain test measures its request, not the daemon's start, and says what it saw when a slow start missed the read's 1,500 ms window instead of failing on it.
@unRekable
unRekable requested a review from itz4blitz as a code owner October 19, 2026 23:24
The prompt holds only while the read does, so a machine that reached the daemon late could sample the one moment it was gone and fail on it. The test now waits for the Status change, takes the settling change as its word when the read is already over, and checks on the slow path that the daemon it saw is the one it started.

@itz4blitz itz4blitz left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ring test's 6 s answer and the polish suite's one-at-a-time starts both look right, thanks. The keychain test needs another pass, though: as rewritten it no longer tests the prompt.

  1. The loop ends on any Change::Status. hub.read_now(network, false, …) emits one every time a network's read finishes (chatd/src/hub.rs, read_now), so the loop breaks whether or not the prompt was ever shown. And the keychain.waiting.is_some() assert is gone, so if waiting were never reported the test would still pass. The detail check only runs when it happens to sample Discord's waiting window.

    For context, with TINYCLIENT_TEST_KEYCHAIN_MS each supported network's read sleeps 1.5 s in turn (server.rs, read_session), and waiting is Some during [1.0 s, 1.5 s) of each read. The old single sample landing outside that half-second window was the flake.

    Suggestion: poll until it's seen, for example chatd.until("a read waiting on the keychain", || chatd.status().1.waiting), asserting the status is not yet is_settled() at that point. Alternatively, raise the delay so the window is wide. Either keeps the assertion and loses the flake.

  2. The loop has no deadline. If the read hangs without a Status, the test hangs instead of failing.

  3. Nit: "A start slower than the read's 1,500 ms has missed the window". The reads run one network after another, so reading stays true for about N × 1.5 s and that early return is close to unreachable. Fine to keep as a guard, but the comment overstates it.

The test read the last presence_query frame and wanted the person on screen in it: the account asks for its own presence on its own clock, and on a loaded runner that ask could be the last one written, failing the assert on a property the test is not about. It looks for the query naming the screen now, and prints the book when there is none.
…ne holds back.

The test read the times the fake received calls at, and chatd paces when it sends them: a machine that holds one send back lets the next overtake it, and two arrivals a fifth of a second apart say nothing about a pacer that did its work. One close pair is allowed; a missing pacer still bunches them all.
The last commit took a test for the Unsupported answer with it, but its request exists only on the branch that adds the network search. It goes back there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants