Skip to content

fix(fts): tolerate interrupted CREATE/DROP residue (fixes #77) - #104

Merged
adsharma merged 2 commits into
LadybugDB:mainfrom
adsharma:fix/ext77-fts-lifecycle-self-heal
Oct 14, 2026
Merged

adsharma merged 2 commits into
LadybugDB:mainfrom
adsharma:fix/ext77-fts-lifecycle-self-heal

Conversation

@adsharma

@adsharma adsharma commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #77.

CREATE_FTS_INDEX / DROP_FTS_INDEX execute as sequential statements outside a single transaction (COPY can't run in a manual transaction), so anything failing mid-way left residue with no in-place recovery: next CREATE failed already exists, a torn DROP could reach a state no command accepts, and a torn index made the whole database fail to open at LOAD.

Changes (all localized to create_fts_index.cpp / drop_fts_index.cpp / fts_extension.cpp):

  • Self-healing CREATE: when the index entry is absent, schema-confirmed residue (appears_in/appears_info/docs/terms, per-index stopwords copy) is dropped first via DROP TABLE IF EXISTS, so a failed CREATE is simply retryable. Two safeguards so user data is never dropped: (1) tables visible in the regular catalog still raise; (2) internal-set candidates are dropped only if their schema exactly matches the FTS internal schema (node shapes incl. PK, rel shape incl. endpoints when resolvable) — anything else raises. The shared default_english_stopwords table is never dropped.
  • Flag-leak guard: useInternalCatalogEntry is save/restored around cleanup because ClientContext resets it only on the success path — without this, a failed CREATE misfiles later user DDL into the internal set (candidate engine follow-up in LadybugDB/ladybug).
  • Guarded DROP: all table drops use IF EXISTS in referential order (rel first) and now also cover appears_info and the per-index stopwords copy (resolved from index aux info at rewrite time), making an interrupted DROP retryable. An entry-gone/tables-remain state is recoverable via re-CREATE.
  • Tolerant LOAD: initFTSEntries pre-checks all four internal tables + stopwords; an incomplete index is skipped instead of throwing, so the database still opens. Repair with DROP_FTS_INDEX + CREATE_FTS_INDEX.
  • Out of scope: tokenize-macro cleanup. Emitting DROP MACRO triggers a pre-existing engine bug (macro-drop WAL replay looks up the wrong catalog set and breaks un-checkpointed reopen — reproducible with plain CREATE/DROP MACRO SQL, no FTS involved). Macros keep the old skip-if-exists reuse behavior; revisit once the engine bug is fixed.

Verification: release build; full local FTS e2e suite green (except env-dependent show_loaded_extensions, an artifact of the partial local build); real kill -9 mid-CREATE (300k docs) → reopen fine → retry CREATE succeeds → queries correct. Remaining CI matrix left to CI.

…ixes LadybugDB#77)

CREATE_FTS_INDEX / DROP_FTS_INDEX run as sequential statements outside a
single transaction, so an interruption left internal tables behind with
no in-place recovery (next CREATE failed 'already exists'; a torn DROP
could leave states no command accepts; a torn index failed the whole DB
open at LOAD).

- CREATE self-heals: when the index entry is absent, internal-only
  residue tables (appears_in/info, docs, terms, per-index stopwords) are
  dropped first via DROP TABLE IF EXISTS, so a failed CREATE is simply
  retryable. Genuine user tables with colliding names still raise; the
  shared default stopwords table is never dropped. The tokenize macro is
  dropped + recreated so a retry always matches the current config.
- DROP is guarded: all drops use IF EXISTS in referential order and now
  also cover appears_info, the per-index stopwords copy, and the
  tokenize macro, making an interrupted DROP retryable.
- LOAD is tolerant: an index whose internal tables are incomplete is
  skipped instead of throwing, so the database still opens; repair with
  DROP_FTS_INDEX + CREATE_FTS_INDEX (queries fail cleanly with
  'not loaded yet' until repaired).
…ATE (fixes LadybugDB#77)

Follow-up hardening from CI feedback (3 failures):

- InternalTableNamingConflict: the user/internal catalog split alone
  cannot tell residue from user tables, because
  useInternalCatalogEntry leaks as true when a statement throws
  (ClientContext resets it only on the success path), misfiling later
  user DDL into the internal set. Two safeguards:
  (a) save/restore the flag around residue cleanup so a failed CREATE
  no longer misfiles subsequent user tables;
  (b) drop internal-set candidates only if their schema exactly matches
  the FTS internal schema (docs/terms/appears_info/stopwords node
  shapes incl. PK, appears_in rel shape incl. endpoints when
  resolvable) —Fail safe: anything else still raises 'already exists'.
- MultiFTSIndex / DropFTSCommitRecovery: stop emitting DROP MACRO.
  Macro-drop WAL replay is broken engine-side
  (getScalarMacroCatalogEntry looks in the functions set instead of
  macros), so any un-checkpointed DROP MACRO breaks the next open, and
  DROP_FTS_INDEX removing the macro broke explicit DROP MACRO test
  steps. Macros are left behind as before (CREATE reuses via
  skip-if-exists); revisit once the engine bug is fixed.
@zahariash

Copy link
Copy Markdown

LadybugDB/ladybug#1165 is on main, so the macro cleanup deferred here is unblocked. #124 does it (drops the macro in DROP_FTS_INDEX, replaces it in CREATE_FTS_INDEX, tests in drop_recreate.test) and gets rebased onto this once it lands, which also removes the NOTE in drop_fts_index.cpp; its own stopwords handling is redundant with this PR's and goes away in that rebase.

@adsharma
adsharma merged commit d422aed into LadybugDB:main Oct 14, 2026
2 checks passed
@adsharma
adsharma deleted the fix/ext77-fts-lifecycle-self-heal branch October 14, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants