Skip to content

Slice 127: a test that times out says so - #105

Merged
HackTuah merged 2 commits into
mainfrom
slice/127-test-timeout-says-so
Sep 26, 2026
Merged

HackTuah merged 2 commits into
mainfrom
slice/127-test-timeout-says-so

Conversation

@HackTuah

@HackTuah HackTuah commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Trinity.SessionCase.collect/3 returns the same value whether it found what it was waiting for or gave up waiting:

after
  remaining -> Enum.reverse(acc)     # a timeout and a match are the same value

So every assertion downstream reports a missing value rather than a timeout, and a reader goes looking for a logic defect that is not there.

Where it came from

main's push run for the slice 125 merge failed on the FIPS leg while the pull_request run of the same commit passed:

code:  assert {:forked, child_id} = Enum.find(events, &match?({:forked, _}, &1))
right: nil

Reproduced locally byte for byte by shortening that test's deadline from 15,000 ms to 300 ms. That is the whole proof it is a deadline the slower runner missed and not R26: no Exqlite.Error, no Database busy, no write path involved.

Why this was worth a slice

The same trap cost a working day earlier in this session. Slice 126 was opened on a test ending collect(id, &match?({:state, :idle}, &1), 15_000), where an {:state, :idle} broadcast from before the message under test was already sitting in the mailbox. collect returned at once, every assertion ran mid-turn, and the result was a confident report of a product defect in the approval handshake that did not exist. The tell was in the output the whole time: Finished in 0.2 seconds for a test nominally waiting fifteen.

The fix

await_event/3 raises, naming the deadline and every event it did see:

before:   right: nil

after:    waited 200ms for a "01a0..." event matching the given condition and it never arrived.
          Events seen, in order: [state: :thinking, state: :idle]
          An empty list means the session emitted nothing at all, which usually means it was never
          started, the test never subscribed, or the thing that should have triggered it did not
          run. A non-empty list means the session was working and did not reach this condition in time.

"Waited 15s and saw thinking then idle" and "waited 15s and saw nothing at all" send a reader to opposite places, which is why the events are in the message.

20 asserting call sites migrated, and the count came from the census rather than from reading.

A census makes the old shape unwritable. The result of collect/3 may only be discarded. It enumerates from git ls-files, skips """ blocks and # comments (without which it reported its own explanation of itself), and asserts it enumerated more than 40 files so it cannot go blind. Demonstrated against a planted violation:

test/trinity/tools/runner_test.exs:48: events = collect(id, &match?({:state, :idle}, &1), 10_000)
Result: 5/6 passed

The deadline is measured, not guessed

Five instrumented runs of the fork wait, then reverted:

4786  4764  4771  4761  4754   (ms)

All three waits in that file: compacting = 0 ms, idle = 2-3 ms, fork = 4745-4774 ms. Only the fork wait is real — the other two return from the mailbox immediately and were left alone rather than raised on a hunch.

15,000 was 3.1× local and the leg still missed it, so 60,000. The reasoning is in the comment beside it: a generous deadline costs nothing on the passing path, because await_event/3 returns the instant the condition matches. The only thing it buys is how long a genuine hang takes to report, so the trade is "long enough that a slow runner never trips it", not "long enough that the test usually passes".

Naming

await_event/3, not await/3 as the slice specified. Trinity.MCP.ServersUnderTest already exports await/2, and a default argument makes await/3 export an await/2 too, so every MCP test importing both failed to compile. The longer name is also the better one.

Nothing under lib/ changed. This is entirely test infrastructure.

mix gate ; echo "EXIT=$?"
prod_check: PASS
3745 mods/funs, found no issues.
trinity.names: OK over 661 tracked files
Result: 910 passed (20 properties, 890 tests), 18 excluded
trinity.coverage: 042 80.31% vs 005 80.31%: OK
EXIT=0

collect/3 returns the same thing whether it found what it was waiting for or
gave up waiting:

    after
      remaining -> Enum.reverse(acc)

So every assertion downstream reports a missing value rather than a timeout, and
a reader goes looking for a logic defect that is not there.

The failure on main's FIPS leg (run 36253217084) was exactly this:

    code:  assert {:forked, child_id} = Enum.find(events, &match?({:forked, _}, &1))
    right: nil

Reproduced locally byte for byte by shortening that test's deadline from 15,000
to 300 ms, which is the whole proof that it is a deadline the slower runner
missed and not R26: no Exqlite error, no Database busy, no write path involved.

This trap cost a working day earlier in this session. Slice 126 was opened on a
test ending collect(id, &match?({:state, :idle}, &1), 15_000), where an idle
broadcast from before the message under test was already in the mailbox; collect
returned at once and every assertion ran mid-turn, producing a confident report
of a product defect in the approval handshake that did not exist.

await/3 does not exist yet, so these fail undefined.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
collect/3 returned the same value whether it found what it was waiting for or
gave up waiting:

    after
      remaining -> Enum.reverse(acc)

So every assertion downstream reported a missing value rather than a timeout,
and a reader goes looking for a logic defect that is not there.

Opened from main's FIPS leg failing the slice 125 merge push (run 36253217084)
while the pull_request run of the same commit passed. Not R26: no Exqlite error,
no Database busy, no write path. Reproduced locally byte for byte by shortening
one deadline from 15,000 ms to 300 ms.

This trap cost a working day earlier in the same session. Slice 126 was opened
on a test ending collect(id, &match?({:state, :idle}, &1), 15_000), where an
idle broadcast from before the message under test was already in the mailbox;
collect returned at once, every assertion ran mid-turn, and the result was a
confident report of a product defect in the approval handshake that did not
exist.

await_event/3 raises, naming the deadline and every event it did see, because
"waited 15s and saw thinking then idle" and "waited 15s and saw nothing at all"
send a reader to opposite places. 20 asserting call sites migrated; the count
came from the census rather than from reading.

The census makes the old shape unwritable: the result of collect/3 may only be
discarded. It enumerates from git ls-files, skips triple-quoted blocks and
comments (without which it reported its own explanation of itself), asserts it
enumerated more than 40 files so it cannot go blind, and was demonstrated
against a planted violation.

The deadline was sized from a measurement, not a hunch. Five instrumented runs
gave 4,745 to 4,786 ms for the fork wait; the file's other two waits return from
the mailbox in 0 to 3 ms and were left alone rather than raised on a hunch.
15,000 was 3.1x local and the leg still missed it, so 60,000. A generous
deadline costs nothing on the passing path, because await_event/3 returns the
instant the condition matches; the only thing it buys is how long a genuine hang
takes to report.

Named await_event/3 rather than await/3: Trinity.MCP.ServersUnderTest already
exports await/2, and a default argument makes await/3 export an await/2 too, so
every MCP test importing both failed to compile.

Nothing under lib/ changed.

Signed-off-by: Ayla Croft <aylacroft@proton.me>
@HackTuah
HackTuah merged commit a53012f into main Sep 26, 2026
8 checks passed
@HackTuah
HackTuah deleted the slice/127-test-timeout-says-so branch September 26, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant