fix(dist): rotate the GC sampling phase so stepping cannot lock the sweep cursor — fixes #300

[ImDanXie]

Summary

When the cleaner sweeps with stepHint > 1, the co-proc inspects every step-th key beneath a per-range cursor. The cleaner resumes each session at the key where the previous one stopped, and when a session wraps it stops at its own start key — so the cursor is pinned and the same modulo class of the route table is inspected in every session. Routes in the remaining classes are never scanned and dead routes in them are never collected.

Root cause

• DistWorkerCoProc.gc() samples every stepUsed-th key and, on wrap, terminates when it returns to sessionStartKey; the recorded nextStartKey is therefore the session's own start key.
• DistWorkerCleaner seeds the next session from that key with the same step, so the sampled residue class is fixed forever.

Measured on a live broker under subscription churn: the same ~50% of routes stayed un-inspected across 400/900 consecutive GC rounds (a stable gap, not random).

Impact

Dead routes in the skipped classes are never collected; the route table (and the memory/cache working set) grows monotonically with subscribe/unsubscribe churn — the exact scenario the GC exists to bound.

Fix

Rotate a per-session phase offset (0..step-1) before the first inspection, so consecutive sessions visit every residue class:

int phase = stepUsed > 1 ? Math.floorMod(gcSessionSeq.getAndIncrement(), stepUsed) : 0;
while (phase-- > 0) {
    itr.next();
    if (!itr.isValid()) {
        break; // tail reached; the scan loop below wraps to the first key
    }
}

No wire or protocol change — the phase is derived from a per-range session counter inside the co-proc. Worst case a key waits at most step sessions to be inspected, which is acceptable for a sampling sweep.

Test evidence

New regression test (gcSessionsRotatePhaseSoAllKeysGetInspected): 12 keys, stepHint = 2, successive sessions fed by nextStartKey; asserts every key is inspected within 8 sessions.

• Control experiment: fails on the unfixed code (only 6/12 keys ever inspected), passes with the fix (12/12).
• Existing GC expectations unchanged (7/7).

Production validation

Deployed in a rolling 6-node cluster (wholesale lib/ replacement, one node at a time, zero rollbacks); post-deploy: cluster mesh fully re-established, all 13 live client connections reconnected, cross-node delivery verified end-to-end. Route-table size and dist.gc.* metrics are being tracked against a captured baseline.

Fixes #300