Skip to content

[LokiJS] Report persistence failures and allow refusing in-memory fallback - #1979

Open
joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/loki-persistence-errors
Open

joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/loki-persistence-errors

Conversation

@joker-777

@joker-777 joker-777 commented Oct 31, 2026 •

Copy link
Copy Markdown

Problem

On the web, LokiJSAdapter can lose data without the app ever finding out:

  • Save failures are swallowed. With autosave, Loki saves without a callback, so when an IndexedDB save fails (quota exceeded, aborted transaction, connection closing), nothing reaches WatermelonDB. onQuotaExceededError only fires when the pre-open checker database fails, which a full disk does not trigger. database.write() resolving therefore does not mean the data is on disk (see Save database from in-memory loki db to IndexedDB manually on web #1329).
  • The in-memory fallback is silent. When IndexedDB is unavailable or fails to open, the adapter falls back to LokiMemoryAdapter with only a log line. Setup succeeds, onSetUpError is not called, and the app cannot tell that everything will be lost when the page is closed.
  • A deleted database comes back. When another tab deletes the database (e.g. on logout), IncrementalIndexedDBAdapter closes its connection on versionchange. The next autosave in this tab silently re-opens and re-creates the database from this tab's memory. Related multi-tab issue: Schema migration issue - multiple tabs #1865.

Changes

All new options are additive. Defaults keep today's behaviour, except for the deleted-database case.

  • onPersistenceError(error) is wired to Loki's autosaveCallback. It is called once per failure, not on every 500 ms autosave retry, and again if saving fails after it succeeded in the meantime. Raw IDB Events are unwrapped to event.target.error. QuotaExceededErrors are also passed to onQuotaExceededError. Save failures are now always logged, not only in the adapter's debug mode.
  • allowInMemoryFallback defaults to true. With false, a missing or unusable IndexedDB fails setup, so onSetUpError is called. The error's cause is the original IDB error.
  • onInMemoryFallback(reason) lets an app that keeps the default learn about the fallback, for example to warn the user. It is called once per adapter, even though setting up an empty database opens it twice.
  • versionchange: Loki saving is disabled through the existing lokiFatalError path instead of re-creating the database. The driver is put in the broken state, so further changes are rejected while reads still work. onPersistenceError is called with a VersionChangeError, and the app's own onversionchange is still called afterwards. Breaking the driver was factored out of DatabaseDriver._fatalError into _breakDriver, which does not depend on experimentalAllowsFatalError.
  • Synchronous access and open errors: if reading indexedDB or calling indexedDB.open throws (e.g. SecurityError when storage access is denied, or in a sandboxed iframe), it is handled like other open failures. It falls back to memory by default, where setup used to fail, or fails setup with allowInMemoryFallback: false.
  • Errors thrown by the app's callbacks on these paths are logged, so one failing callback can't skip the others or change the fallback policy.
  • Flow types, index.d.ts, the web adapter section of the Setup docs, and the changelog are updated.

Remaining multi-tab caveats (documented, not addressed here)

  • Two tabs writing concurrently still overwrite each other's saves, and onDidOverwrite is called. Coordinating writers, for example with Web Locks, is out of scope.
  • After versionchange, the tab must be reloaded. There is no in-place recovery.
  • The non-incremental LokiIndexedAdapter has no versionchange handling.
  • The callbacks only work with useWebWorker: false, like the existing ones.

Usage

const adapter = new LokiJSAdapter({
  schema,
  useWebWorker: false,
  useIncrementalIndexedDB: true,
  onSetUpError: (error) => { /* now also called when IndexedDB is unusable */ },
  allowInMemoryFallback: false,
  onPersistenceError: (error) => { /* warn that recent changes are not saved */ },
})

Tests

src/adapters/lokijs/persistence.test.js uses a fake incremental persistence adapter through _testLokiAdapter and a stubbed indexedDB global. It covers:

  • reporting once per failure, unwrapping the event, routing quota errors, and reporting again after a recovery;
  • versionchange stopping saves and rejecting writes;
  • the fallback with and without allowInMemoryFallback, and the checker open failure.

Jest for src/adapters/lokijs and src/adapters/__tests__ passes, with 150 tests. ESLint, TSLint and Flow are clean.

Validation against a real IndexedDB implementation

The unit tests use a fake persistence adapter. In addition, the cross-tab deletion path was checked once with the real LokiJSAdapter and lokijs IncrementalIndexedDBAdapter on top of fake-indexeddb 6.0.1, a spec-compliant IndexedDB implementation. That check is not committed, because adding a dev dependency would grow the diff.

  • Deleted after a save: another connection calls indexedDB.deleteDatabase(). The deletion is not blocked, onversionchange and onPersistenceError are each called once, later writes are rejected, and after several autosave intervals the database still does not exist.
  • Deleted while a save is pending: a write is made, then the database is deleted before autosave runs. The database is still not re-created.
  • Control: with this PR's versionchange handling disabled, the same scenario re-creates the deleted database.

Jest's sandbox can't simulate a global indexedDB getter that throws. That case was checked with a plain Node script that loads the adapter through Babel. With the fix, the default setup falls back to memory and allowInMemoryFallback: false fails setup, both with the original error as cause. Before the fix, setup failed in both modes.

Validation in a real browser with multiple tabs

This PR's code was transpiled with the repo's Babel config, bundled with esbuild into a test page, and served from localhost. Each tab of Chromium 152 ran its own LokiJSAdapter on real IndexedDB, with useIncrementalIndexedDB: true, useWebWorker: false and allowInMemoryFallback: false. The test page is not committed.

Scenario Result
Tab B deletes the database after tab A saved The deletion is not blocked. Tab A gets onPersistenceError with a VersionChangeError, then onversionchange. Tab A rejects batch and unsafeExecute, and 3 s later the database still does not exist.
Tab B deletes the database while tab A's save is in flight, coordinated over a BroadcastChannel so the delete starts 1 ms after the save Tab A gets versionchange mid-save and reports once, with no second error for the aborted save. The database is still gone 4 s later.
Logout: tab A resets the database while tab B has it open with 5 records Tab B is broken and rejects writes. A third tab opening the database finds only tab A's one post-reset record.
Control for the logout scenario, with this PR's versionchange handling disabled Tab B's next autosave writes its records back: 7 records, including the 5 logged-out ones. Tab A's new record is overwritten.
Tab B deletes the database while tab A is setting up, at delays from 0 to 80 ms At 0 ms the delete runs before the database exists, and setup creates a fresh one. At 2 and 5 ms, setup fails through onSetUpError after a VersionChangeError. From 10 to 80 ms, setup succeeds, then the driver is broken. In every case where the database existed, it stayed deleted.

The result does not depend on event order. Saving is disabled synchronously in the versionchange handler, and closing the connection lets an in-flight transaction finish before the deletion proceeds. So whichever finishes first, nothing writes to the database after it has been deleted.

Independent of #1980 (unsafeResetDatabase() during an autosave); the two touch lokiExtensions.js in different places and merge cleanly in either order.

🤖 Generated with Claude Code

…lback

- Add `onPersistenceError(error)`, wired to Loki's `autosaveCallback`: called once per save failure (not on every autosave retry), and again after a successful save. Raw IDB events are unwrapped, quota errors also go to `onQuotaExceededError`, and save failures are always logged.
- Add `allowInMemoryFallback` (default `true`): with `false`, unusable IndexedDB (missing, failing to open, or throwing on access/open) fails setup, so `onSetUpError` is called, with the original error as `cause`.
- Add `onInMemoryFallback(reason)`, called once per adapter.
- On `versionchange` (database deleted or upgraded in another tab), stop saving instead of re-creating the database from memory, break the driver so changes (`batch`, local storage, `unsafeExecute`, `unsafeResetDatabase`) are rejected, and report it via `onPersistenceError` before calling the app's `onversionchange`.
- Errors thrown by app callbacks on these paths are logged, so one failing callback can't skip the others or change the fallback policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@StasDoskalenko

Copy link
Copy Markdown

Hi @joker-777, thank you for such a careful investigation, especially the multi-tab browser tests!

I maintain NitromelonDB, a maintained fork of WatermelonDB (New Architecture, Nitro Modules). Something that might interest you: on the web, our SQLiteAdapter now runs SQLite compiled to WASM (wa-sqlite) in a web worker, stored in IndexedDB. It's the same import and adapter as on native, so most of what this PR works around goes away:

  • writes are real SQLite transactions on IndexedDB, not a periodic autosave of an in-memory copy
  • there's no silent fallback to memory: if the worker, WASM or IndexedDB isn't available, setup fails
  • tabs share one database and coordinate through Web Locks, instead of each tab saving its own copy over the others

More details are in the wa-sqlite adapter docs. One caveat: existing LokiJS data in the browser is not migrated automatically.

If you need to stay on LokiJS for now, the fork still ships the LokiJS adapter with the same code, so a PR there would be very welcome too. Our fork was rewritten in TypeScript (strict mode) to reduce the chance of bugs like this one slipping through, so I think the community would really benefit if you opened your PRs against it too.

If you don't have the time or would rather not, no worries at all, I'm happy to port it myself. Just let me know. Thanks a lot!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants