Skip to content

fix: prevent iterator invalidation in handle_append_entries_resp during peer leave timeout - #675

Open
hardikkaurani wants to merge 4 commits into
eBay:masterfrom
hardikkaurani:fix/handle-append-entries-resp-iterator-invalidation
Open

hardikkaurani wants to merge 4 commits into
eBay:masterfrom
hardikkaurani:fix/handle-append-entries-resp-iterator-invalidation

Conversation

@hardikkaurani

Copy link
Copy Markdown

Summary

This PR fixes a critical iterator invalidation and potential use-after-free defect in raft_server::handle_append_entries_resp when handling append entries responses during server step-down/removal timeout (fixes #653).

Problem Description

In handle_append_entries_resp, peers_.find(resp.get_src()) was executed prior to check_srv_to_leave_timeout():

peer_itor it = peers_.find(resp.get_src());
...
check_srv_to_leave_timeout();
...
ptr<peer> p = it->second;

When a leaving node (srv_to_leave_) exceeds its timeout window during catch-up:

  1. check_srv_to_leave_timeout() triggers remove_peer_from_peers(srv_to_leave_).
  2. This erases the peer entry from peers_, which immediately invalidates iterator it if the response being handled belongs to the leaving peer.
  3. Accessing it->second subsequently results in dereferencing an invalidated iterator, causing segmentation faults or undefined behavior in production environments.

In comparison, other handlers such as handle_hb_timeout (src/handle_timeout.cxx) and handle_join_leave (src/handle_join_leave.cxx) consistently invoke check_srv_to_leave_timeout() before performing peer lookups.

Key Changes

  1. Reorder check before iterator lookup: Moved check_srv_to_leave_timeout() to the start of handle_append_entries_resp() before peers_.find(resp.get_src()). If the leaving peer timed out and was removed, peers_.find() safely returns peers_.end(), logging and returning cleanly without crashes.
  2. Immediate pointer extraction: Extracted ptr<peer> p = it->second; immediately upon confirming it != peers_.end(), preventing any subsequent dangling iterator accesses.
  3. Regression Test: Added remove_srv_timeout_append_entries_resp_test in tests/unit/raft_server_test.cxx and registered it in the test suite to verify late response handling for removed/timed-out servers.

…sponse

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
…n test suite

Signed-off-by: Hardik Kaurani <hardikkaurani1@gmail.com>
Copilot AI balanced review requested due to automatic review settings October 27, 2026 10:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@hardikkaurani

Copy link
Copy Markdown
Author

Hi @greensky00,

I opened this pull request to resolve the iterator invalidation issue reported in #653 within raft_server::handle_append_entries_resp.

Summary of changes:

  1. Lookup Ordering: Moved check_srv_to_leave_timeout() to execute prior to peers_.find(resp.get_src()). When a leaving server times out during catch-up, remove_peer_from_peers() erases it from peers_. Calling this check before lookup ensures peers_.find() returns peers_.end() cleanly and avoids dereferencing an invalidated iterator. This also aligns with the ordering already used in handle_hb_timeout and handle_join_leave.
  2. Safe Pointer Extraction: Captured ptr<peer> p = it->second; immediately following iterator validation to guard against dangling iterator references.
  3. Regression Test: Added remove_srv_timeout_append_entries_resp_test in tests/unit/raft_server_test.cxx verifying graceful handling when responses arrive after node removal/timeout.

The change is structured across 4 atomic signed-off commits. Whenever convenient, please approve the workflow run and let me know if any adjustments are needed. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Segfault: check_srv_to_leave_timeout() can invalidate an iterator in handle_append_entries_resp()

2 participants