Skip to content

fix(fts): drop the tokenize macro with the index - #124

Open
zahariash wants to merge 2 commits into
LadybugDB:mainfrom
zahariash:fix/fts-drop-leftovers
Open

zahariash wants to merge 2 commits into
LadybugDB:mainfrom
zahariash:fix/fts-drop-leftovers

Conversation

@zahariash

@zahariash zahariash commented Oct 14, 2026 •

Copy link
Copy Markdown

This work was produced with the help of language models.

Fixes #119.

DROP_FTS_INDEX left behind the index's tokenize macro. Recreating an index under the same name reused the old macro, so rows built by the CREATE kept the old tokenization while rows inserted later and queries used the new one. The other half of #119, the CREATE failing on a leftover stopwords copy, is fixed on main by #104.

Fix. DROP_FTS_INDEX drops the macro after the index tables and now returns Index <name> has been dropped.; the NOTE #104 left there is gone, since LadybugDB/ladybug#1165 fixed DROP MACRO's WAL replay (LadybugDB/ladybug#1163). CREATE_FTS_INDEX replaces an existing tokenize macro, including the one schema.cypher creates during IMPORT DATABASE and one left by an interrupted CREATE. The macro is dropped only when it exists, because DROP MACRO IF EXISTS throws for a missing macro. Rebased onto main after #104; this overlaps #125 in create_fts_index.cpp, and whichever lands second gets rebased.

Tests. drop_recreate.test recreates an index with a new ignore_pattern and over a leftover macro, and drops an index with a stopwords table, reopens without a checkpoint so the drop is replayed, and recreates it with a new ignore_pattern, which finds 12345; these three fail on main. Its stopwords table and stopwords file cases already pass on main since #104 and stay as regression tests. multi_index.test no longer drops the macros before IMPORT DATABASE. FTS extension tests: no new failures.

zahariash and others added 2 commits October 14, 2026 18:25
DROP_FTS_INDEX left the index's tokenize macro behind, so an index
created again under the same name reused it: rows built by the CREATE
were tokenized with the old ignore_pattern or tokenizer while rows
inserted later used the new one. DROP MACRO's WAL replay is fixed
(LadybugDB/ladybug#1165), so DROP_FTS_INDEX now drops the macro and
reports "Index ... has been dropped.", and CREATE_FTS_INDEX replaces a
tokenize macro that already exists, from IMPORT DATABASE, an interrupted
CREATE, or a drop by an earlier version. The macro is dropped only when
it exists, because DROP MACRO IF EXISTS throws for a missing macro.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DROP_FTS_INDEX now also drops the tokenize macro and the stopwords
copy. The new case drops an index with a stopwords table, reopens
without a checkpoint so the drop is replayed, then recreates the index
with a new ignore_pattern and finds the row it now indexes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zahariash
zahariash force-pushed the fix/fts-drop-leftovers branch from 8fbc806 to 286c12d Compare October 14, 2026 16:31
@zahariash zahariash changed the title fix(fts): drop the tokenize macro and stopwords copy with the index fix(fts): drop the tokenize macro with the index Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Recreating a dropped FTS index uses the old tokenizer or fails on its stopwords table

1 participant