fix(lancedb): disable FTS with_position to avoid optimize crash#336
Merged
Conversation
FTS indexes built with with_position=True crash lance's optimize/compaction on lancedb >= 0.32 when merging an unindexed tail (Max offset exceeds length of values; upstream lance-format/lance#7653). The crash aborts optimize() including version cleanup, so the index dir grows unbounded until the disk fills. everos recall is OR-mode BM25 and never does phrase queries, so positions are never read -- disabling is lossless. - base: default with_position=False - infra: migrate_fts_indexes() rebuilds pre-fix indexes once at startup + reclaims orphans - cascade worker: count consecutive optimize failures, escalate warning->error Fixes #335. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gloryfromca
force-pushed
the
fix/fts-with-position
branch
from
02a170a to
e8ff5ef
Compare
cyfyifanchen
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix the FTS
with_positionoptimize crash, plus the disk-bloat and silent-failure amplifiers it caused.Fixes #335.
Root cause
On lancedb ≥ 0.32, FTS indexes built with
with_position=Truecrash lance'soptimize()/ compaction when merging an unindexed tail —Max offset exceeds length of valuesin the RepDef offset restoration (upstream regression,lance-encodingv4.0.0 → v6.0.0, reported atlance-format/lance#7653). The crash abortsoptimize()including version cleanup, so the index directory grows unbounded until the disk fills.EverOS recall is OR-mode BM25 (
MatchQuery) and never does phrase queries, so the positionswith_position=Truestores are never read.Changes (3)
core/persistence/lancedb/base.py— defaultwith_position=Falsefor FTS indexes (lossless; no phrase queries anywhere in recall).infra/persistence/lancedb/__init__.py—migrate_fts_indexes(): a one-time, marker-guarded startup migration that rebuilds pre-fix (with_position=True) indexes and reclaims the orphaned fragments left behind, so existing deployments self-heal on upgrade.memory/cascade/worker.py— count consecutiveoptimize()failures and escalatewarning → errorpast a threshold, instead of swallowing them silently (the amplifier that let a single index bug grow the disk unbounded before anyone noticed).Test