Skip to content

[LokiJS] Web migrations: destroyTable, unsafeExecuteLoki, warn on ignored SQL - #1988

Open
joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/loki-web-migrations
Open

joker-777 wants to merge 1 commit into
Nozbe:masterfrom
kenhub:fix/loki-web-migrations

Conversation

@joker-777

@joker-777 joker-777 commented Nov 2, 2026 •

Copy link
Copy Markdown

Migrations on the web (LokiJS) had no way to drop a table or change data: unsafeExecuteSql() steps are silently ignored there. A table dropped with SQL stays in IndexedDB and is loaded and parsed on every launch.

Changes

  • New destroyTable({ table }) migration step for both adapters. SQLite runs DROP TABLE IF EXISTS, so it's a no-op for a table that was already dropped with unsafeExecuteSql(). LokiJS removes the collection and deletes its chunks from IndexedDB, including leftover chunks of a collection that's no longer in Loki
  • The chunks are deleted in the same IndexedDB transaction that saves the Loki metadata without the collection. The persisted database stays consistent if the page closes mid-way, and a table that is destroyed and created again in one migration keeps its new data
  • New unsafeExecuteLoki(loki => {}) step, the LokiJS counterpart of unsafeExecuteSql(). It runs against the Loki instance during the migration and is ignored on SQLite. It can't be sent to a web worker, so LokiJSAdapter throws a clear error when it's used with useWebWorker: true
  • LokiJS logs a warning for each ignored unsafeExecuteSql() step, with the first 200 characters of the SQL
  • A failed LokiJS migration is no longer autosaved. Steps that ran before the failure changed the database in memory; now it's not saved, so the persisted database stays at the old version and the steps don't run twice next time
  • Migration syncs (getSyncChanges) accept the new steps, and skip tables and columns destroyed later in the range
  • destroyTable() rejects local_storage in any letter case

Loki's own removeCollection() throws for collections that IncrementalIndexedDBAdapter loaded lazily, so the driver removes the collection from loki.collections directly. Deleting the chunks relies on internals of the IncrementalIndexedDBAdapter in the lokijs version this repo pins. If they're missing, a warning is logged and the chunks are kept.

Tests

  • New src/adapters/lokijs/migrations.test.js, using fake-indexeddb and the real IncrementalIndexedDBAdapter. It checks the persisted IndexedDB keys after each migration: the destroyed table's chunks are gone; leftover chunks of a collection already missing from Loki are deleted; destroy plus create in one migration keeps the new data across later saves; and a failed save is retried with the old chunks intact until it succeeds. It also covers the SQL warning and unsafeExecuteLoki()
  • New shared adapter test destroys tables in migrations, which runs on LokiJS and SQLite
  • Unit tests for encodeMigrationSteps, getSyncChanges and the step constructors
  • Adds fake-indexeddb@^6.2.5 as a dev dependency

Verification

🤖 Generated with Claude Code

…ored SQL

* [LokiJS] Web migrations: destroyTable, unsafeExecuteLoki, warn on ignored SQL

- New destroyTable({ table }) migration step. SQLite runs DROP TABLE IF
  EXISTS. LokiJS removes the collection and deletes its chunks from
  IndexedDB in the same transaction that saves the updated metadata, so
  dropped tables are no longer loaded and parsed on every launch.
- New unsafeExecuteLoki(fn) step, the LokiJS counterpart of
  unsafeExecuteSql. It's ignored on SQLite.
- LokiJS logs a warning for each ignored unsafeExecuteSql step.
- Migration syncs skip tables and columns destroyed later in the range.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject reserved table names case-insensitively, and unsafeExecuteLoki on web workers

destroyTable() now rejects local_storage in any letter case, since SQLite
table names are case-insensitive. LokiJSAdapter throws a clear error in
development when unsafeExecuteLoki() steps are used with
useWebWorker: true, instead of failing with DataCloneError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Reject unsafeExecuteLoki with web workers in production too, test failed-save retry

The useWebWorker check now runs in production as well, so setup fails
with a clear error instead of DataCloneError. A new test aborts the
saves of a destroy + recreate migration, checks the old chunks are
intact, then lets the retry succeed and checks only the new data
remains after reopening.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Don't save changes made by a LokiJS migration that fails

If a migration step throws, Loki is stopped from saving before the
error is propagated, so the persisted database stays at the old version
and steps aren't applied twice when the migration runs again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Warn when a destroyed table's persisted data can't be deleted

destroyCollection() relies on IncrementalIndexedDBAdapter internals of
the pinned lokijs version. If the persistence adapter is incremental but
doesn't have them, it now logs a warning instead of silently keeping the
chunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@StasDoskalenko

Copy link
Copy Markdown

Hi @joker-777, thank you for this!

I maintain NitromelonDB, a maintained fork of WatermelonDB (New Architecture, Nitro Modules). Something that might interest you: on the web, our SQLiteAdapter now runs SQLite compiled to WASM (wa-sqlite) in a web worker, stored in IndexedDB. It's the same import and adapter as on native. Migrations run the same SQL on the web as on device, so unsafeExecuteSql() steps (including DROP TABLE) just work there, without a Loki-specific escape hatch. More details are in the wa-sqlite adapter docs. One caveat: existing LokiJS data in the browser is not migrated automatically.

A destroyTable migration step would still be lovely to have on all adapters, so a PR against our repo would be very welcome. Our fork was rewritten in TypeScript (strict mode) to reduce the chance of bugs slipping through, so I think the community would really benefit if you opened your PRs against it too.

If you don't have the time or would rather not, no worries at all, I'm happy to port it myself. Just let me know. Thanks a lot!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants