Repository navigation
fix(fts): tolerate interrupted CREATE/DROP residue (fixes #77) - #104
Merged
adsharma merged 2 commits intoOct 14, 2026
Merged
Conversation
…ixes LadybugDB#77) CREATE_FTS_INDEX / DROP_FTS_INDEX run as sequential statements outside a single transaction, so an interruption left internal tables behind with no in-place recovery (next CREATE failed 'already exists'; a torn DROP could leave states no command accepts; a torn index failed the whole DB open at LOAD). - CREATE self-heals: when the index entry is absent, internal-only residue tables (appears_in/info, docs, terms, per-index stopwords) are dropped first via DROP TABLE IF EXISTS, so a failed CREATE is simply retryable. Genuine user tables with colliding names still raise; the shared default stopwords table is never dropped. The tokenize macro is dropped + recreated so a retry always matches the current config. - DROP is guarded: all drops use IF EXISTS in referential order and now also cover appears_info, the per-index stopwords copy, and the tokenize macro, making an interrupted DROP retryable. - LOAD is tolerant: an index whose internal tables are incomplete is skipped instead of throwing, so the database still opens; repair with DROP_FTS_INDEX + CREATE_FTS_INDEX (queries fail cleanly with 'not loaded yet' until repaired).
…ATE (fixes LadybugDB#77) Follow-up hardening from CI feedback (3 failures): - InternalTableNamingConflict: the user/internal catalog split alone cannot tell residue from user tables, because useInternalCatalogEntry leaks as true when a statement throws (ClientContext resets it only on the success path), misfiling later user DDL into the internal set. Two safeguards: (a) save/restore the flag around residue cleanup so a failed CREATE no longer misfiles subsequent user tables; (b) drop internal-set candidates only if their schema exactly matches the FTS internal schema (docs/terms/appears_info/stopwords node shapes incl. PK, appears_in rel shape incl. endpoints when resolvable) —Fail safe: anything else still raises 'already exists'. - MultiFTSIndex / DropFTSCommitRecovery: stop emitting DROP MACRO. Macro-drop WAL replay is broken engine-side (getScalarMacroCatalogEntry looks in the functions set instead of macros), so any un-checkpointed DROP MACRO breaks the next open, and DROP_FTS_INDEX removing the macro broke explicit DROP MACRO test steps. Macros are left behind as before (CREATE reuses via skip-if-exists); revisit once the engine bug is fixed.
This was referenced Oct 14, 2026
|
LadybugDB/ladybug#1165 is on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #77.
CREATE_FTS_INDEX/DROP_FTS_INDEXexecute as sequential statements outside a single transaction (COPY can't run in a manual transaction), so anything failing mid-way left residue with no in-place recovery: next CREATE failedalready exists, a torn DROP could reach a state no command accepts, and a torn index made the whole database fail to open atLOAD.Changes (all localized to
create_fts_index.cpp/drop_fts_index.cpp/fts_extension.cpp):appears_in/appears_info/docs/terms, per-index stopwords copy) is dropped first viaDROP TABLE IF EXISTS, so a failed CREATE is simply retryable. Two safeguards so user data is never dropped: (1) tables visible in the regular catalog still raise; (2) internal-set candidates are dropped only if their schema exactly matches the FTS internal schema (node shapes incl. PK, rel shape incl. endpoints when resolvable) — anything else raises. The shareddefault_english_stopwordstable is never dropped.useInternalCatalogEntryis save/restored around cleanup becauseClientContextresets it only on the success path — without this, a failed CREATE misfiles later user DDL into the internal set (candidate engine follow-up in LadybugDB/ladybug).IF EXISTSin referential order (rel first) and now also coverappears_infoand the per-index stopwords copy (resolved from index aux info at rewrite time), making an interrupted DROP retryable. An entry-gone/tables-remain state is recoverable via re-CREATE.initFTSEntriespre-checks all four internal tables + stopwords; an incomplete index is skipped instead of throwing, so the database still opens. Repair withDROP_FTS_INDEX+CREATE_FTS_INDEX.DROP MACROtriggers a pre-existing engine bug (macro-drop WAL replay looks up the wrong catalog set and breaks un-checkpointed reopen — reproducible with plainCREATE/DROP MACROSQL, no FTS involved). Macros keep the old skip-if-exists reuse behavior; revisit once the engine bug is fixed.Verification: release build; full local FTS e2e suite green (except env-dependent
show_loaded_extensions, an artifact of the partial local build); real kill -9 mid-CREATE (300k docs) → reopen fine → retry CREATE succeeds → queries correct. Remaining CI matrix left to CI.