flake: parked_idle_parity::resumed_connection_keeps_registry_identity resets under full-suite load (2 of 7 complete runs)

[TinDang97]

Summary

parked_idle_parity::resumed_connection_keeps_registry_identity intermittently fails on x86_64 Linux under monoio with ECONNRESET, only when the full test suite runs. It passes reliably in isolation. It is not diagnosed.

It took the ci-gcp.sh monoio leg red on PR #806, which is how it surfaced.

Failure

thread 'resumed_connection_keeps_registry_identity' panicked at tests/parked_idle_parity.rs:98:41:
read reply: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

Line 98 is stream.read(&mut chunk).expect("read reply") inside command_reply — a connection was reset mid-reply. The test attaches two connections, names one wakekeeper, then twice sleeps PARK_WAIT (4600 ms, against --conn-park-secs 2) and data-wakes it, checking CLIENT LIST identity across the park.

What is measured

All on GCE c3-standard-8 (Xeon 8481C, 8 vCPU), debug binary, --shards 1.

condition	result
the single test, alone, idle host	15 / 15 pass
the whole parked_idle_parity binary (7 tests), alone	7/7, twice
the single test under 8 yes spinners on 8 vCPU	8 / 8 pass
complete full-suite runs (265 binaries, doctests, RC captured)	2 fail / 7

The two failures both occurred while three concurrent agent builds shared the host. The five most recent complete runs — after the host quieted — are green, including four consecutive runs in one hunt.

Pure CPU load does not reproduce it. That is the one clear negative result: 8-way saturation on 8 vCPU left it 8/8 green, which rules out "the runner was just slow" and rules out widening a timeout as the fix.

Hypothesis, untested against a failure

moon's client listeners use SO_REUSEPORT, so a second server on a taken port binds successfully — both processes stay alive and the kernel splits connections, with no bind error, no log line and no panic (moon#489; moon#365's ConnectionReset in crash_matrix_cross_plane has the same signature).

The port flock (common::reserve_port -> claim_across_processes -> dir_lock::acquire on $TMPDIR/moon-test-ports/<port>) is held by the test binary, not the moon child. If a test binary is SIGKILLed — OOM under concurrent builds, or a harness stop — the flock releases while the orphaned server keeps listening. The next binary to draw that port binds successfully and its client can be reset.

A confirmed instance of the precondition was found on a different host: one orphaned moon server, 1h23m old, surviving an OOM kill of its parent test binary.

This was not confirmed on the failing host. A sampler recording every port held by more than one moon PID, once per second, ran across four full-suite runs and observed zero duplicates — but all four of those runs were green, so it never sampled a failure. The hypothesis is neither confirmed nor refuted.

Suggested next steps

1. Keep the port-duplicate sampler attached and run the suite under concurrent build load, which is the condition both failures shared, rather than under CPU spinners.
2. Independently: make the port lock outlive the test binary, or have reserve_port verify no live listener holds the port before handing it out. SO_REUSEPORT means a successful bind proves nothing.
3. tests/parked_idle_parity.rs's connect_retry already tolerates load-induced resets at connect with a 10 s deadline and a comment explaining why; command_reply has no equivalent tolerance and uses a bare .expect("read reply"). That asymmetry is worth revisiting only after the cause is known — retrying a reset that indicates a real port collision would convert a diagnosable failure into a silent one.

Related

• moon#489 — SO_REUSEPORT makes a duplicate bind succeed silently
• moon#365 — ConnectionReset in crash_matrix_cross_plane, same signature
• moon#713 (CLOSED) — integration tests orphaning their server when an assert fires
• moon#284 / moon#18 — the port reservation helper this relies on