fix(search-index): index every page in full - #420
Merged
Merged
Conversation
upload.sh cut each page's cleaned text at 6000 characters (bytes on the
ubuntu runner), so keyword search never saw the rest of a long page: 129 of
279 zh pages and 144 of 281 en pages were cut, losing 44% and 47% of their
text. Each Meilisearch document now holds the whole cleaned page; ids are
unchanged, so the next sync upserts in place.
The content reaches jq through stdin: as a --arg, a page over 128 KiB would
exceed Linux's limit on a single command-line argument.
The index embedder must bound its own input, e.g. a documentTemplate of
"A document named {{doc.title}} with content {{doc.content | truncate: 6000}}":
Meilisearch 1.39 sends the full rendered template to a REST embedder
regardless of documentTemplateMaxBytes, and text-embedding-v4 rejects a long
page, which fails the whole indexing batch.
The comment claimed the embedder reads only the first documentTemplateMaxBytes of a document. Meilisearch does not apply that setting to REST embedders; the cut happens in the index's documentTemplate with the truncate filter.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
scripts/upload.shcut each page's cleaned text at 6000 characters before indexing it. The CI runner is Ubuntu, where GNUcut -ccounts bytes, so a Chinese page kept about 2000 characters:Keyword search could not match anything past the cut, and answers that live in the second half of a long page were unreachable.
What changes
jqthrough stdin. As a--arg, a page over 128 KiB would exceed Linux's limit on a single command-line argument.Required before this reaches
mainThe index's embedder must bound its own input:
Meilisearch 1.39 sends the full rendered template to a REST embedder regardless of
documentTemplateMaxBytes, and text-embedding-v4 rejects input over its length limit (Range of input length should be [1, 16000]). One long page fails the whole indexing batch (indexedDocuments: 0). With thetruncatefilter the batch indexes cleanly: a test index built by this branch's script holds 562 documents with 562 embeddings, the largest page at ~72 KB.Rollback
Revert this commit and run the workflow once with
workflow_dispatch; the ids are the same, so the old truncated documents overwrite the full ones in place.