fix(kv-server): complete the restore session future on synchronous start failure — fixes #303

[ImDanXie]

Summary

KVRangeRestorer.restoreFrom() wraps snapshot-install setup in try { … } catch (Throwable t) { log.error(…) } and returns the session's doneFuture. If startRestore() — or the subsequent messenger.send() — throws synchronously, the future is never completed.

Root cause & impact

• There is no timeout on this path (the receiver-driven path has an idle timeout; the synchronous-throw path has nothing), so the caller waits forever.
• Worse: the session stays cached in currentSession. A retry with the same snapshot from the same leader hits the reuse branch — existingSession.snapshot.equals(rangeSnapshot) && !doneFuture.isDone() && same leader — and returns the same dead future. The range is permanently stuck: it never applies anything and never retries.

Snapshot install is mandatory on expansion/rebalance, so any transient synchronous failure here is a hard wedge.

Fix

Complete the future exceptionally in the catch block; the caller observes the failure and a retry starts a fresh session:

} catch (Throwable t) {
    log.error("Unexpected error", t);
    onDone.completeExceptionally(new KVRangeStoreException("Snapshot restore failed to start", t));
}

Test evidence

New restoreFromStartFailureCompletesFutureExceptionally: startRestore throws → the future must be completed exceptionally, and a retry with the same snapshot must start a fresh session (not reuse the dead one).

• Control experiment: fails on the unfixed code (future never completes), passes with the fix.
• Existing restore expectations unchanged (9/9).

Production validation

Deployed in a rolling 6-node cluster (wholesale lib/ replacement, one node at a time, zero rollbacks); cluster mesh, all 13 live client connections and cross-node delivery verified post-deploy. No wedged range (a range stuck in snapshot restore) observed before or after; the fix closes the permanent-wedge path should a synchronous start failure ever occur.

Fixes #303