Skip to content

Fix RequestVote freshness after full log compaction - #668

Merged
greensky00 merged 1 commit into
eBay:masterfrom
thweetkomputer:fix/snapshot-vote-freshness
Oct 8, 2026
Merged

greensky00 merged 1 commit into
eBay:masterfrom
thweetkomputer:fix/snapshot-vote-freshness

Conversation

@thweetkomputer

@thweetkomputer thweetkomputer commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

When a follower compacts its entire log into a snapshot, handle_vote_req() can grant a vote to a candidate whose log is older than the follower's committed snapshot. The log_store::last_entry() contract requires an empty store to return a dummy entry with term zero, but the vote freshness comparison treats that dummy term as the follower's last log term.

For example, a follower with snapshot (last_log_term=1, last_log_idx=5) and no retained entries compares itself as (0, 5). It then accepts a candidate advertising (1, 3), even though that candidate lacks committed entries. This can allow an outdated candidate to become leader, violating leader completeness.

Use term_for_log(next_slot() - 1) for the receiver's last log term, matching the existing outgoing RequestVote path. This helper reads the retained log when present and otherwise reads the snapshot boundary. Use the same resolved term and index in the vote diagnostic.

Four independent tests in leader_election_test cover retained logs, fully compacted logs, reconstruction of the Raft server with the same snapshot/state/log store, and logs appended after the snapshot. Each checks rejection of older candidates, acceptance of equally or more up-to-date candidates, and the persisted voted_for state. They use NuRaft's existing fake network, state machine and in-memory log store. The reconstruction test does not simulate a disk or process restart.

Validation on Linux arm64 with Clang 18, Debug, and ENABLE_RAFT_STATS=1:

  • Before the fix, both fully compacted cases fail by granting the stale vote; the retained-log and post-snapshot-log cases pass.
  • After the fix: leader_election_test 13/13, snapshot_test 8/8, learner_new_joiner_test 4/4, and failure_test 7/7 pass with --abort-on-failure.
  • The same four complete suites pass with ADDRESS_SANITIZER=1 and no sanitizer findings.
  • The full runtests.sh run stops at the existing full consensus test assertion in raft_server_test.cxx:1898 (expected committed index 15, actual 5). An isolated build of unmodified upstream 0b01b185b46b8e482f6c384b636c35605121a41e reproduces the same failure with the same build settings. The full-consensus implementation and that test are unchanged.

Additional suites run separately after the early exit: timer_test 3/3, asio_service_test 37/37, req_resp_meta_test 7/7, priority_based_election_test 3/3, stream_transport_layer_test 5/5, and raft_stream_mode_test 6/6 pass. The remaining extended append_entries API test also passes.

custom_quorum_test initially stops at stop, term increment, restart test (custom_quorum_test.cxx:525), where S2 is expected to become leader within 500 ms. The log shows S2 becoming leader approximately 50 ms after that assertion. The same case passes when run individually on both unmodified upstream and this branch. Consequently, these results do not claim a clean full-suite run.

@greensky00

Copy link
Copy Markdown
Contributor

@thweetkomputer
Thanks for catching this. The fix itself looks good, but I think the test is a bit overly long. I'll go ahead and merge this first, then simplify the test separately.

@greensky00
greensky00 merged commit 14448e0 into eBay:master Oct 8, 2026
1 check passed
greensky00 added a commit to greensky00/NuRaft that referenced this pull request Oct 8, 2026
* Replace the four vote log freshness tests with a single test.

* Only the fully compacted case reproduces the bug fixed in eBay#668;
  the other three pass with or without the fix.

* Drop unused setup: extra log appends, the stale follower log,
  and unrelated raft params.

* Keep three vote checks: reject a shorter log, reject a lower
  term, and grant an equally up-to-date log.
@thweetkomputer
thweetkomputer deleted the fix/snapshot-vote-freshness branch October 8, 2026 05:27
greensky00 added a commit that referenced this pull request Oct 8, 2026
* Replace the four vote log freshness tests with a single test.

* Only the fully compacted case reproduces the bug fixed in #668;
  the other three pass with or without the fix.

* Drop unused setup: extra log appends, the stale follower log,
  and unrelated raft params.

* Keep three vote checks: reject a shorter log, reject a lower
  term, and grant an equally up-to-date log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants