Skip to content

[Sync] Make sync bookkeeping linear-time - #1987

Open
joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/sync-linear-bookkeeping
Open

joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/sync-linear-bookkeeping

Conversation

@joker-777

@joker-777 joker-777 commented Nov 2, 2026 •

Copy link
Copy Markdown

Two sync steps were quadratic in the number of local changes. This makes them linear. No behaviour change.

Changes

  • markLocalChangesAsSynced looked up each pushed record with affectedRecords.find(). Affected records are now indexed by table and id in a Map
  • Applying a pull checked every created and updated record with locallyDeletedIds.includes(). Locally deleted ids are now a Set, built once per table
  • The rest of src/sync/ already uses Sets and Maps for its lookups. Not changed: addToRawSet (per record, bounded by the column count) and LokiJS adapter internals

Tests

Two new Jest tests count array element reads through a Proxy instead of timing, so they aren't flaky: 2000 pushed records, and 1000 pulled updates to locally deleted records. Master needs 2,001,000 and 501,500 reads; this branch stays under 4000 and 3000.

Verification

🤖 Generated with Claude Code

markLocalChangesAsSynced looked up each pushed record with
affectedRecords.find(), and applying pulled changes checked every created and
updated record with locallyDeletedIds.includes(). Both were O(n²) in the number
of local changes. Records are now indexed by table and id in a Map, and the
locally deleted ids are kept in a Set.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@StasDoskalenko

Copy link
Copy Markdown

Hi @joker-777, thank you, and I really like the Proxy trick for testing this without relying on timing!

I maintain NitromelonDB, a maintained fork of WatermelonDB (New Architecture, Nitro Modules). We've done a fair bit of sync performance work there, but both of these are still quadratic: affectedRecords.find() in src/sync/impl/markAsSynced.ts and locallyDeletedIds.includes() in src/sync/impl/applyRemote.ts.

Our fork was rewritten in TypeScript (strict mode) to reduce the chance of bugs slipping through, so I think the community would really benefit if you opened your PRs against it too.

If you don't have the time or would rather not, no worries at all, I'm happy to port it myself. Just let me know. Thanks a lot!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants