Flaky: cluster/router and root tests race on embedded-etcd ports (the #509 defect, two more copies) #575

What

cluster/router and the root package start embedded etcds on ports chosen by a
probe-then-close-then-rebind helper. Under a multi-package run the port can be taken between the
probe and etcd's bind, and any test in those packages fails at setup.

This is the same defect #509/#513 fix in cluster/etcd, in two copies that PR does not touch.

Observed

First seen as --- FAIL: TestHedgeOwnersSpanAllAbsent (0.00s) during a
go test ./engine/ ./cluster/... . run; it passed on rerun. It does not reproduce in isolation
(160 runs of the test alone, 18 runs of the whole package, and runs under 40 spinning CPU hogs all
pass). It reproduces from go test ./cluster/... -count=2, which starts enough embedded etcds
concurrently to collide — third attempt, in the same helper:

--- FAIL: TestSeriesNarrowsAndFailsOver (0.00s)
    read_test.go:190:
        Error Trace: cluster/router/router_test.go:53
                     cluster/router/read_test.go:134
                     cluster/router/read_test.go:190
        Error:       Received unexpected error:
                     listen tcp 127.0.0.1:41131: bind: address already in use

router_test.go:53 is the require.NoError after embed.StartEtcd. Any test reaching
startEtcd is a candidate victim, which is why the name in the failure varies — the two seen so
far are TestHedgeOwnersSpanAllAbsent and TestSeriesNarrowsAndFailsOver. The 0.00s duration is
the tell: the test never ran, setup failed.

Mechanism

cluster/router/router_test.go:24 and cluster_test.go:42:

func freeAddr(t *testing.T) string {
	l, err := lc.Listen(t.Context(), "tcp", "127.0.0.1:0")
	require.NoError(t, err)
	defer func() { _ = l.Close() }()

	return l.Addr().String()
}

Two independent problems, both named in #513:

1. The probe window. The listener is closed before embed.StartEtcd binds, so anything else on
   the machine — including a sibling test package's etcd — can take the port in between.
   embed.StartEtcd takes URLs and will not accept an already-bound listener, so the window cannot
   be closed; test(cluster/etcd): absorb the port race, and stop asserting a legal rejoin #513's answer is to re-pick and retry, bounded.
2. Two calls do not guarantee two ports. Both call sites are

   lc := url.URL{Scheme: httpScheme, Host: freeAddr(t)}
   lp := url.URL{Scheme: httpScheme, Host: freeAddr(t)}

   The first listener is closed before the second opens, so the OS may hand the same port back and
   etcd is asked to bind its client and peer URLs to one address. test(cluster/etcd): absorb the port race, and stop asserting a legal rejoin #513 fixes this with
   freeAddrs(t, n), holding every listener open until all n are chosen.

Fix

Lift #513's freeAddrs(t, n) + startEtcdOnFreshPorts (bounded retry, non-bind errors reported on
the first attempt) into a shared test helper and use it from all three places, rather than making a
third copy of the same three lines. #513 is still open; this should land on top of it or be folded
into it.

Affected:

• cluster/router/router_test.go:24,40-41 (and :190, a single free port for a peer server)
• cluster_test.go:42,88-89