Make Elasticsearch use a linear retriever for hybrid search

[jpountz]

I believe that the goal of the hybrid search benchmark is to compute top-10 hits for both the lexical and semantic search, before combining hits by summing up scores.

This is not exactly what the Elasticsearch _search call does, as it puts the vector query as a SHOULD clause of the bool query. So vector scores are computed first, and then summed up with lexical scores before top-10 hits are selected based on the summed up scores (which is harder on dynamic pruning). Switching to a linear retriever should hopefully fix this and make the comparison with Vespa a bit fairer.

[radu-gheorghe]

Hey, @jpountz, thanks for looking into this!

If I get linear retriever right, it takes rank_window_size out of both sub-queries (lexical and vector) and combines results based on both. Like in this example.

As opposed to computing the hybrid search score for all hits and getting the resulting top N, which is what we do now. Correct?

If so, this is an option to consider - though it does imply an accuracy trade-off. If I got something wrong, please correct me.

Either way, I think it's worth noting that the current comparison is fair, because Vespa does compute the hybrid score for all hits, too (first phase). We could aggregate only top N from each in a second or global phase. IIUC, the linear retriever runs on the coordinating node, so Vespa's equivalent should be the global phase. Not that it matters too much, in this test we have one node and one shard for both Vespa and ES.

[jpountz]

As opposed to computing the hybrid search score for all hits and getting the resulting top N, which is what we do now. Correct?

Almost, the top 10 vectors are computed first. Then Elasticsearch retrieves the top-10 docs by BM25 score + vector score, only taking the vector score into account if the vector was part of the top-10. (If you find it cumbersome or awkward, I won't disagree. :))

Vespa does compute the hybrid score for all hits

I wonder if this is actually true. I see the expression configured in something called "first-phase", and "phase 1" in Vespa docs looks like a reranker to me, so something that applies only to the top-k hits of some other query? If this applied to all hits, I would expect it to be defined as a function like the bm25 profile does?

Based on my extremely limited understanding of Vespa so far, I would expect the hybrid query to retrieve the top 100 (=targetHits) hits for vector search and BM25, then do a linear combination, and finally only keep the best 10 (=hits) hits?

But I may be completely off the hook, I still need to get a better mental model of how things work in Vespa!

[radu-gheorghe]

Thanks for clarifying the retriever part. It was confusing at first, but then it made sense 🙂

Regarding first phase, I'm sorry for the confusing docs. Another thing to fix 🙈 . Phase 1 ranking is simply scoring all matching hits, much like in Elasticsearch/Lucene, not re-ranking. The rank profile supports functions like in a programming language (i.e. for syntax sugar): the actual computation will be the same either way.

As for the "hybrid search" definition, I think the blame is on us (i.e. the search community as a whole) for the vague buzzwords 🙈 . In Vespa, like in Elasticsearch, you can combine scores from different sources in different ways (e.g. first phase ranking or later re-ranking in phase 2 or 3). Which one is "correct" is up for grabs 🤷 . The way we did it in the benchmark is more accurate, but slower than what you're suggesting. Which is consistent to other choices, like picking the original float values for vectors instead of binarizing them.

Thanks for getting into this - I really appreciate it. I know these benchmarks tend to be a delicate topic and I feel like there's a lot to learn for everyone (e.g. how I just learned about how the linear retriever works in ES).

[jpountz]

I just learned about how the linear retriever works in ES

Just to be clear, what I described is what happens when putting a knn query as a SHOULD clause of a bool query. The linear retriever is more like a first-stage reranker that only looks at top hits from the semantic and lexical queries.

Phase 1 ranking is simply scoring all matching hits, much like in Elasticsearch/Lucene, not re-ranking

To my knowledge, there are no good ways to perform pruning when sorting hits by lexical+semantic score. Are you suggesting that Vespa does a linear scan for this hybrid query? If not, can you describe what it does?

[radu-gheorghe]

Ahhh... I think I finally understand the pruning piece you're bringing up.

To answer the question about Vespa, my understanding is:

• first phase ranks all "matches" of the query
• in the benchmark, we have two OR clauses, each returning their (configurable) targetHits docs. So in the worse case scenario, we'll rank lexicalTargetHits + semanticTargetHits docs. Best case: they overlap and we rank max(lexicalTargetHits, semanticTargetHits).

With this in mind, I can see how Lucene does it differently (though I don't know exactly how) and it may not be apples to apples. And I see how if we use linear retriever in ES (now that we have it - I believe it wasn't there when we did the original benchmark), it would be closer in terms of how the hybrid ranking is achieved.

[jpountz]

in the benchmark, we have two OR clauses, each returning their (configurable) targetHits docs

This is where the main difference is. Lucene will consider numCandidates hits for the vector query, but all matches for the lexical query.

And I see how if we use linear retriever in ES (now that we have it - I believe it wasn't there when we did the original benchmark), it would be closer in terms of how the hybrid ranking is achieved.

++