feat: add HNSW RaBitQ with rotated storage and staged search - #1814
Draft
CLiqing wants to merge 6 commits into
Draft
feat: add HNSW RaBitQ with rotated storage and staged search#1814CLiqing wants to merge 6 commits into
CLiqing wants to merge 6 commits into
Conversation
Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
Collaborator
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: CLiqing The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Collaborator
Author
|
/kind feature |
Move RaBitQ dispatch and staged evaluation out of generic HNSW code. Reuse the refinement predicate and count full refinements in search statistics. Reject non-RaBitQ payloads before replacing a live index, retain filtered iterator bridges, and add targeted and SQ/PQ regression coverage. Remove unrelated serialization formatting changes. Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
Remove the separate native RaBitQ traversal. Keep search state, queues and stopping rules in the existing Knowhere searcher, with RaBitQ-specific staged distance evaluation and qb4 SIMD/prefetch batching. Preserve candidate-order refinement and request-local query parameters. Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
Signed-off-by: ChenLiqing <23721160+CLiqing@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
issue: #1747
Summary
Add
HNSW_RABITQto Knowhere with L2, inner product and cosine support.rbq_bits=1..9(default 1) and request-localrbq_bits_query=0..8(default 4) for search, range search and iteration.qb=4estimates with query-plane reuse and prefetch.refine_k.Faiss relationship and scope
This selectively adapts the standard RaBitQ layout and staged-search approach from facebookresearch/faiss#5526, reference
d8a85956060c7567ff40dc1eb0a77578533d8bad, to Knowhere's bundled Faiss. FP32 graph construction, rotation, cosine, filtering, refinement and request-local query parameters are Knowhere integration responsibilities.New index/adaptor classes are under
faiss/cppcontrib/knowhere. Existing core RaBitQ changes add a batched estimate interface and optimize SIMD kernels and unaligned-load safety.Current boundaries