Repository navigation
Performance issue list based on LDBC benchmark #122
Description
I'm creating this issue to uncover some deeper performance issues I've noticed in the LDBC SNB-SF1 benchmark summary table - it's clear that there are more fundamental reasons why lance-graph is ~10x slower than other alternatives for nearly all the queries.
Why this dataset?
LDBC SNB SF1 is a reasonably large and complex dataset with several wide tables as opposed to the earlier benchmark on an artificially generated social network dataset, which had fewer, narrower tables. This means that we're uncovering issues with the LDBC benchmark that are more likely to surface in users' real-world workloads, so it's worth investigating deeper from a query parsing and planning perspective.
Summary of issues (high-level)
-
The previous version of lance-graph used the
executeAPI, which rebuilt the catalog each time it was called (since the dataset was passed as an input parameter). In v0.5.1, lance-graph introduces a newCypherEngineAPI, which caches the catalog, so we can compare the pure query execution times with other engines in the benchmark repo. -
Currently, there doesn't exist a short‑circuit for existence checks due to a parser limitation in lance-graph. The parser rejects
COUNT(DISTINCT x) > 0, which blocks simple boolean returns as required by Q20-30 of the LDBC benchmark, and this forces full counts + Python post‑processing overheads. That removes the chance to stop early onLIMIT 1/EXISTS, which other engines can exploit this pattern. This is addressed by Parser rejects boolean comparison afterCOUNT(DISTINCT ...)during projection #123. -
There are query optimizer limitations around multi‑
MATCH+WHERE. Per Multi-match query with WHERE predicate downstream filter/projection fails #117, we must express many queries as a singleMATCHfollowed by theWHEREclause in order to be able to filter, which restricts the flexibility with which users can write their own queries using these patterns. From a performance perspective, it makes join‑order/alias‑pruning inefficient and risks huge intermediate joins if predicate pushdown or reorder isn't considered during planning. In general, from a DevEx perspective, the user should be able to write all kinds of patterns (including multi-match +WHEREqueries) without facing a performance penalty. -
No parameter binding / plan cache. Parameters are manually inlined (string replace), so each query is parsed and planned from scratch and can’t be cached as a prepared statement. File: query.py (apply_params + execute_query). This would be fixed by feat: Properly support parameter placeholders in Python/Rust #103.
-
Self-referencing alias in a Cypher query pattern doesn't work, meaning that the most expensive query in the LDBC benchmark (Q30) cannot be run against other tools. This is addressed by Self-pointing Cypher pattern causes lance-graph to hang indefinitely for high cardinality datasets #111.
-
LDBC queries typically involve large fan‑out traversals, which can become expensive quickly. In the DataFusion engine, graph traversal is implemented as joins over edge tables. If the engine doesn’t reorder joins aggressively, high‑degree steps (e.g., Comment ↔ Post ↔ Tag) can become expensive.
Note
Some of the above may not be query planning issues, but I've been explaining and inspecting the query plans with GPT 5.2 Codex to the best of my ability to see what could explain the performance gap. Experienced human eyes are needed to look through the plans and identify performance gaps with other query plans from Kuzu/Ladybug and Neo4j.
We can expand/contract this list as new issues become apparent, and link them to existing sub-issues in this repo.
Activity
[-]List of query planner performance issues based on LDBC benchmark[/-][+]List of parser/planner performance issues based on LDBC benchmark[/+]COUNT(DISTINCT ...)during projection #123[-]List of parser/planner performance issues based on LDBC benchmark[/-][+]Performance issue list based on LDBC benchmark[/+]ChunxuTang commentedon Feb 4, 2026
Hi @prrao87, thanks for proposing and running the benchmark!
I checked your latest benchmark results. Through profiling, I found that catalog rebuilding on every query execution could be the major bottleneck. Unlike Neo4j and KuzuDB, which cache catalogs in memory, calling lance-graph
query.execute()will rebuild the catalog every time. The catalog rebuilding begins to dominate the query execution time when the dataset schema becomes more complicated. Cc @beinanTo address that, I exposed a
CypherEngineAPI from this #124 to reuse the catalog. It's now included in the latest version v0.5.1. Could you try the new API and re-run the benchmark?prrao87 commentedon Feb 4, 2026
Hi @ChunxuTang, many thanks! I rebuilt 0.5.1 and reran the benchmark with the latest version, it's barely any improvement. The table of results is below. I suspected something else other than caching is the problem, because we'd have seen some sign of the cache missing in my first benchmark (with the simpler schema). I took a look at my code in much more detail over the last week, and I can't think of what else I'm doing wrong, so I think we should work our way through each of the other issues listed above. I think LDBC is very a high cardinality scenario, exposing other issues in the query plan that the previous benchmark didn't.
I also think yours and @beinan's eyes on the query plan for just Q1 of LDBC would help. This query doesn't materialize any unnecessary results, and doesn't do any downstream processing in Python either. I can't understand why it would be 4x slower than Neo4j, even though I created a scalar index on the
idcolumn in lance-graph (just like I created a uniqueness constraint in Neo4j). For such a simple query, I'd expect that lance-graph would do really well.Here's the latest results from the LDBC repo.
ChunxuTang commentedon Feb 5, 2026
@prrao87 You'll need to use the new CypherEngine API in the v0.5.1 to avoid duplicate catalog building when executing the queries. We're actually using lance-graph's
catalog building time + query execution timeto compare withpure query execution timesof Neo4j and Kuzu/Ladybug. That's why lance-graph looks much slower than others.Let me create a PR to your benchmark repo.
prrao87 commentedon Feb 5, 2026
Ah, I misunderstood your earlier statement about how to use the new Cypher engine. Looking forward to your PR. Thanks!
prrao87 commentedon Feb 5, 2026
Seems to help! I tried the new engine approach on the first benchmark (artificial dataset, 9 queries), and it's much faster 😁. Will update that repo and hopefully we can see a significant improvement here, too!
ChunxuTang commentedon Feb 5, 2026
@prrao87 I've just sent a PR to your repo. Would you mind taking a look? I think this would significantly improve the performance.
prrao87 commentedon Feb 5, 2026
Thanks a lot @ChunxuTang! Indeed, this significantly improves performance 😁. I just reran the benchmark, and lance-graph is now the fastest in quite a few of the queries. Q10 and Q27 ares interesting, because lance-graph is the slowest there, and Q20 is also interesting, because lance-graph is way faster than anything else 😅. Hopefully the benchmark is helpful to you and Beinan too as you optimize the planner going forward! 🚀
ChunxuTang commentedon Feb 5, 2026
@prrao87 Thanks for re-running the benchmark!
For your questions on Query 10 and Query 20:
In summary, the benchmark result demonstrates the advantages of lance-graph for GraphRAG scenarios (most queries with base property filtering and only shallow traversal).
@beinan and I are also exploring some hybrid queries (graph + vector) for benchmarking, which are very common in GraphRAG. That would also be useful for the evaluation.
ChunxuTang commentedon Feb 5, 2026
@prrao87 BTW, for your benchmark results, could you show the data rounded to one decimal place (e.g., 3.1 ms instead of 3ms)?
prrao87 commentedon Feb 5, 2026
Sounds good, will update the numbers and show them with 1 decimal rounding. I agree that these two benchmarks are analytical/traversal queries only, but they're important to see if the low-hanging fruit is addressed (which I think it is). Any further gains would require some special optimizations, that can be a longer term goal. It's still admirable how well Lance does at these kinds of queries that are traditionally the realm of pure graph DBs, so this was all in all, a very illuminating exercise for me.
I think more benchmarks that study Lance's vector/multimodal capabilities would be very helpful. I'd love to see your benchmarks with graph + vector queries (lance-graph will definitely be at an advantage in those). Do let me know if there are datasets and benchmarks you have in mind! Feel free to tag me on those issues as and when you create them. Happy to help you with creating new benchmarks too.
ChunxuTang commentedon Feb 10, 2026
Hi @prrao87, the last query (query 30) should be unblocked now by the latest release (0.5.2). Would you like to retry the query 30?
Just the query needs to be rewritten to
since we don't support the self-reference query yet.
prrao87 commentedon Feb 10, 2026
Hi @ChunxuTang, this is awesome! It not only works, but lance-graph is now the fastest solution among them all for Q30 😎.
Full table of updated results here 👇🏽
https://github.com/prrao87/graph-benchmark-ldbc?tab=readme-ov-file#high-level-results
Thanks a lot!