Skip to content

State sync can leave committed partial state when a sync fails late #775

Closed

Description

@QuantumExplorer

Summary

MultiStateSyncSession commits subtree batches incrementally, so any failure discovered after the first flush leaves earlier subtrees persisted on disk. There is no marker distinguishing a half-synced database from a complete one.

This is pre-existing and already acknowledged in the code — grovedb/src/replication/state_sync_session.rs (~line 250):

TODO: This check is not fully atomic. apply_chunk() flushes completed subtree batches via set_new_transaction()/commit_transaction(), so on mismatch only the last transaction is rolled back while earlier subtrees remain on disk. A full fix requires staging all subtree commits and only persisting them after root hash verification passes.

Filing it so it is tracked rather than living only as a code comment. Surfaced by a security scan of #657 (finding 13).

Affected failure paths

Every late failure, not just one:

Why it is not fixed in #657

#657 adds indexed trees, which state sync rejects. That adds one more trigger, but the destination only learns a subtree is indexed when it walks that subtree's metadata, which is necessarily after earlier subtrees have been flushed. There is no signal at session start to check, so the rejection cannot be made early enough to dodge the problem. Fixing it properly means restructuring the transaction lifecycle, which does not belong in a feature PR for a different subsystem.

Suggested fix

As the TODO says: stage all subtree commits and persist only after root-hash verification passes. Failing that, a cheaper mitigation is a persisted "sync in progress" marker written before the first flush and cleared only on successful completion, so an interrupted database is detectable on open rather than silently readable as complete.

Either approach wants tests covering the non-indexed failure paths too — root-hash mismatch and mid-restore chunk failure — since those are reachable today.

Activity

  1. QuantumExplorer commented on Sep 7, 2026

    @QuantumExplorer
    MemberAuthor

    This is Claude, closing on behalf of the maintainer.

    Resolved by #840 (merged as 6ec8fee). The default RestoreCommitMode::Atomic keeps every restore write inside one transaction until commit_session has verified the root hash, so a failure at any point rolls the destination back entirely; there is exactly one commit site and it comes after the check. The opt-in Incremental mode is now the only path that commits early: it stamps the has_incomplete_restore() marker in the same transaction as its first intermediate commit, clears it only in the verified final commit, and every session constructor refuses to start over a marked directory, so a half-synced database is both distinguishable and unusable.

    Tests: failed_chunk_after_batched_sync_rolls_back_all_restored_subtrees, incremental_mode_still_refuses_a_root_hash_mismatch, abandoned_incremental_restore_marks_the_destination, a_marked_destination_refuses_to_start_another_restore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions