Skip to content

[Sync] Apply batched pulls sequentially and notify observers once per pull - #1984

Open
joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/sync-sequential-batches
Open

joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/sync-sequential-batches

Conversation

@joker-777

@joker-777 joker-777 commented Nov 2, 2026 •

Copy link
Copy Markdown

With _unsafeBatchPerCollection, pulled changes are applied in batches of up to 5000 records. This PR commits those batches one after another and sends observers one notification per pull.

Changes

  • Batches are now prepared and committed one after another, instead of all at once with Promise.all. A failed batch stops the apply: earlier batches stay committed, and later ones are never prepared, so no record is left with a pending prepared state
  • The whole apply runs inside database.experimentalBatchNotifications(), so observers fire once per pull instead of once per table and batch
  • experimentalBatchNotifications() now merges everything changed inside it into one notification: one call per experimentalSubscribe subscriber and one collection.changes emission per table. Before, it only deferred the per-batch notifications and replayed them one by one
  • Docs (Sync/Frontend.md and the SyncArgs comments) now say that the option shrinks the batch payload sent to the native side, not JS memory, and that it isn't transactional

Tests

  • A batched pull of 10001 records (5 batches) sends one notification
  • An adapter that fails the second batch shows that batches never overlap, that only the first 5000 records land, and that later records aren't in the cache
  • A Database test covers merging notifications
  • All three fail on master: 5 notifications, 3 batches called, 3 subscriber calls

Verification

🤖 Generated with Claude Code

* Apply batched pulls sequentially and notify observers once per pull

With _unsafeBatchPerCollection, all chunks were prepared up front and
committed with Promise.all, so a failed chunk left an arbitrary subset
applied (and the rest prepared in the cache), and observers re-fetched once
per table and chunk. Chunks are now prepared and committed one after
another, and the whole apply runs inside experimentalBatchNotifications.

experimentalBatchNotifications now merges the changes made within it into a
single notification instead of replaying one per batch.

The docs now say that the option shrinks the native batch payload, not JS
memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Clean up deleted records before applying changes, not concurrently

With Promise.all, a failing destroyDeletedRecords rejected the apply (and
let the sync Writer finish) while batches were still running. Those batches
could then commit and notify outside the Writer and the notification scope.
The steps now run one after another, in both the batched and the single-batch
path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@StasDoskalenko

Copy link
Copy Markdown

Hi @joker-777, thank you for all the sync work, these PRs are really well explained!

I maintain NitromelonDB, a maintained fork of WatermelonDB (New Architecture, Nitro Modules). This one applies there too: _unsafeBatchPerCollection still commits batches with Promise.all, and experimentalBatchNotifications() still replays notifications one by one. It would also fit nicely with our plans for paginated sync (StasDoskalenko#50).

Our fork was rewritten in TypeScript (strict mode) to reduce the chance of bugs like this one slipping through, so I think the community would really benefit if you opened your PRs against it too. The code lives in src/sync/impl/applyRemote.ts there, and the logic should carry over as is.

If you don't have the time or would rather not, no worries at all, I'm happy to port it myself. Just let me know. Thanks a lot!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants