Skip to content

fix(fts): skip rows COPY rejects with ignore_errors - #123

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
zahariash:fix/fts-copy-ignore-errors
Oct 14, 2026
Merged

adsharma merged 1 commit into
LadybugDB:mainfrom
zahariash:fix/fts-copy-ignore-errors

Conversation

@zahariash

Copy link
Copy Markdown

This work was produced with the help of language models.

Fixes #118.

COPY ... (ignore_errors=true) into a table with an FTS index mishandled a row rejected for a duplicate primary key. COPY indexes its rows in FTSIndex::finalize, but the rejected row was deleted before that, so the index looked for a document it had never seen and the COPY failed. A rejected row without text instead became a phantom document with the previous row's text, skewing BM25 scores.

Fix. delete_ skips a row at or past the indexed prefix (checkpointed rows plus this transaction's inserts) when the docs table has no document for it, so an indexed row is still deleted. finalize checks lookup() and skips deleted rows instead of indexing the previous row's vectors. This also makes rows a crash left outside the index deletable again, until a later insert moves the prefix past them; they stay unsearchable, tracked by LadybugDB/ladybug#1241.

Tests. copy_ignore_errors.test copies a duplicate with text and checks the other row is indexed and deletable, then one without text and checks the score equals a rebuilt index's, and one where the rejected row is the last in the file, followed by an insert and deletes. The new cases fail on main and pass here. FTS extension tests: no new failures.

COPY indexes its rows in FTSIndex::finalize, after the whole file is
read, but a row rejected for a duplicate primary key is deleted earlier,
through NodeTable::delete_. FTSIndex::delete_ then looked for a document
that was never indexed and threw "FTS index is inconsistent", failing the
whole COPY. delete_ now skips a row past the index's indexed prefix when
the docs table has no document for it, and finalize skips rows whose
lookup fails instead of indexing the previous row's values under the
deleted offset, which added a phantom document and skewed BM25 scores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---- ok
-STATEMENT CALL QUERY_FTS_INDEX('Doc', 'idx', 'banana') RETURN node.id, score;
---- 1
2|0.353634

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zahariash the exact score match may be a bit brittle and require frequent updates due to code/data changes.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 38c08a2: the pinned score is gone. The test now builds one index before the COPY (maintained through it, so it would pick up the phantom document) and one after (which can't), and checks both score the document the same. Still fails with 2|False without the fix.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

38c08a2 went onto the branch after the merge, so it never reached main. The same change is #126.

@adsharma
adsharma merged commit d0b6896 into LadybugDB:main Oct 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: COPY with ignore_errors into an FTS-indexed table fails or indexes a phantom document

2 participants