Skip to content

Rerun benchmark with CypherEngine in lance-graph #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
prrao87 merged 1 commit into from
Feb 5, 2026
Merged

Rerun benchmark with CypherEngine in lance-graph #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 34 additions & 32 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,39 +93,41 @@ each of the 30 queries run in the benchmark.

## High-level results

Query | neo4j-2025.12.1 (ms) | kuzu-0.11.3 (ms) | ladybug-0.14.1 (ms) | lance-graph-0.5.1 (ms) |
| Query | neo4j-2025.12.1 (ms) | kuzu-0.11.3 (ms) | ladybug-0.14.1 (ms) | lance-graph-0.5.1 (ms) |
| --- | --- | --- | --- | --- |
| q1 | 6ms | 2ms (2.4x) | 2ms (2.7x) | 24ms (0.2x) |
| q2 | 6ms | 1ms (4.4x) | 2ms (4.2x) | 25ms (0.3x) |
| q3 | 3ms | 1ms (2.1x) | 1ms (2.2x) | 26ms (0.1x) |
| q4 | 5ms | 1ms (4.6x) | 1ms (4.3x) | 25ms (0.2x) |
| q5 | 5ms | 4ms (1.3x) | 4ms (1.4x) | 24ms (0.2x) |
| q6 | 4ms | 1ms (5.1x) | 1ms (4.6x) | 23ms (0.2x) |
| q7 | 2ms | 31ms (0.1x) | 31ms (0.1x) | 35ms (0.1x) |
| q8 | 16ms | 3ms (5.6x) | 3ms (5.5x) | 23ms (0.7x) |
| q9 | 3ms | 2ms (1.3x) | 2ms (1.2x) | 25ms (0.1x) |
| q10 | 5ms | 2ms (2.7x) | 2ms (2.6x) | 48ms (0.1x) |
| q11 | 13ms | 8ms (1.6x) | 8ms (1.5x) | 26ms (0.5x) |
| q12 | 5ms | 18ms (0.3x) | 18ms (0.3x) | 40ms (0.1x) |
| q13 | 9ms | 50ms (0.2x) | 52ms (0.2x) | 31ms (0.3x) |
| q14 | 2ms | 2ms (1.0x) | 2ms (0.9x) | 25ms (0.1x) |
| q15 | 3ms | 3ms (1.2x) | 3ms (1.2x) | 25ms (0.1x) |
| q16 | 2ms | 2ms (1.0x) | 2ms (1.0x) | 27ms (0.1x) |
| q17 | 5ms | 3ms (1.8x) | 3ms (1.8x) | 25ms (0.2x) |
| q18 | 4ms | 2ms (2.3x) | 2ms (2.3x) | 25ms (0.2x) |
| q19 | 6ms | 13ms (0.5x) | 14ms (0.5x) | 42ms (0.1x) |
| q20 | 422ms | 12ms (35.2x) | 11ms (38.1x) | 25ms (16.9x) |
| q21 | 2ms | 1ms (2.7x) | 1ms (2.7x) | 24ms (0.1x) |
| q22 | 4ms | 20ms (0.2x) | 21ms (0.2x) | 35ms (0.1x) |
| q23 | 4ms | 1ms (2.9x) | 1ms (2.8x) | 25ms (0.2x) |
| q24 | 2ms | 2ms (1.0x) | 2ms (1.0x) | 25ms (0.1x) |
| q25 | 3ms | 2ms (1.9x) | 2ms (1.8x) | 24ms (0.1x) |
| q26 | 2ms | 4ms (0.5x) | 4ms (0.5x) | 26ms (0.1x) |
| q27 | 3ms | 13ms (0.3x) | 16ms (0.2x) | 46ms (0.1x) |
| q28 | 4ms | 2ms (2.6x) | 2ms (2.3x) | 26ms (0.2x) |
| q29 | 3ms | 1ms (2.4x) | 1ms (2.4x) | 26ms (0.1x) |

Q30 is the most expensive query of all, and was not run for this stage of the benchmark, because lance-graph's query planner doesn't yet support it. Stay tuned as more updates come along!
| q1 | 6ms | 2ms (2.4x) | 2ms (2.7x) | 1ms (3.9x) |
| q2 | 6ms | 1ms (4.4x) | 2ms (4.2x) | 2ms (2.8x) |
| q3 | 3ms | 1ms (2.1x) | 1ms (2.2x) | 3ms (0.8x) |
| q4 | 5ms | 1ms (4.6x) | 1ms (4.3x) | 3ms (1.5x) |
| q5 | 5ms | 4ms (1.3x) | 4ms (1.4x) | 2ms (2.4x) |
| q6 | 4ms | 1ms (5.1x) | 1ms (4.6x) | 1ms (5.0x) |
| q7 | 2ms | 31ms (0.1x) | 31ms (0.1x) | 13ms (0.2x) |
| q8 | 16ms | 3ms (5.6x) | 3ms (5.5x) | 1ms (12.4x) |
| q9 | 3ms | 2ms (1.3x) | 2ms (1.2x) | 2ms (1.2x) |
| q10 | 5ms | 2ms (2.7x) | 2ms (2.6x) | 25ms (0.2x) |
| q11 | 13ms | 8ms (1.6x) | 8ms (1.5x) | 3ms (4.0x) |
| q12 | 5ms | 18ms (0.3x) | 18ms (0.3x) | 18ms (0.3x) |
| q13 | 9ms | 50ms (0.2x) | 52ms (0.2x) | 9ms (1.0x) |
| q14 | 2ms | 2ms (1.0x) | 2ms (0.9x) | 3ms (0.6x) |
| q15 | 3ms | 3ms (1.2x) | 3ms (1.2x) | 2ms (1.4x) |
| q16 | 2ms | 2ms (1.0x) | 2ms (1.0x) | 4ms (0.5x) |
| q17 | 5ms | 3ms (1.8x) | 3ms (1.8x) | 3ms (2.0x) |
| q18 | 4ms | 2ms (2.3x) | 2ms (2.3x) | 2ms (2.0x) |
| q19 | 6ms | 13ms (0.5x) | 14ms (0.5x) | 19ms (0.3x) |
| q20 | 422ms | 12ms (35.2x) | 11ms (38.1x) | 2ms (175.8x) |
| q21 | 2ms | 1ms (2.7x) | 1ms (2.7x) | 2ms (0.9x) |
| q22 | 4ms | 20ms (0.2x) | 21ms (0.2x) | 12ms (0.3x) |
| q23 | 4ms | 1ms (2.9x) | 1ms (2.8x) | 3ms (1.5x) |
| q24 | 2ms | 2ms (1.0x) | 2ms (1.0x) | 2ms (0.8x) |
| q25 | 3ms | 2ms (1.9x) | 2ms (1.8x) | 1ms (2.1x) |
| q26 | 2ms | 4ms (0.5x) | 4ms (0.5x) | 3ms (0.6x) |
| q27 | 3ms | 13ms (0.3x) | 16ms (0.2x) | 23ms (0.1x) |
| q28 | 4ms | 2ms (2.6x) | 2ms (2.3x) | 3ms (1.5x) |
| q29 | 3ms | 1ms (2.4x) | 1ms (2.4x) | 3ms (1.2x) |

The fastest and slowest systems for each query can be visualized via the following heatmap.

![](./results/benchmark_heatmap.png)

> [!NOTE]
> The lance-graph query processor is undergoing significant optimization and a new version of the query planner is on the way soon. The numbers here are artificially large because several queries had to be rewritten to materialize results prematurely, with aggregations done externally so as to bypass (as yet) unsupported Cypher syntax in lance-graph. As a result, these results should be treated as a moving target -- follow this repo and check back in soon!
80 changes: 40 additions & 40 deletions lance_graph/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,54 +41,54 @@ uv run query.py "1,2,6"

```bash
❯ uv run pytest benchmark_query.py --benchmark-min-rounds=5 --benchmark-warmup-iterations=5 --benchmark-disable-gc --benchmark-sort=fullname
======================================= test session starts =======================================
============================================ test session starts =============================================
platform darwin -- Python 3.13.7, pytest-9.0.2, pluggy-1.6.0
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=True min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=5)
rootdir: /Users/prrao/code/graph-benchmark-ldbc-snb
configfile: pyproject.toml
plugins: anyio-4.12.1, benchmark-5.2.3, asyncio-1.3.0, Faker-40.1.2
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 29 items

benchmark_query.py ............................. [100%]


---------------------------------------------------------------------------------- benchmark: 29 tests -----------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_query1 23.0999 (1.03) 31.7585 (1.35) 23.9868 (1.05) 1.8277 (10.09) 23.5428 (1.03) 0.3856 (2.23) 1;3 41.6897 (0.95) 21 1
test_benchmark_query10 47.1176 (2.09) 48.2161 (2.06) 47.6572 (2.08) 0.3497 (1.93) 47.5551 (2.08) 0.6016 (3.47) 8;0 20.9832 (0.48) 21 1
test_benchmark_query11 25.3632 (1.13) 26.5225 (1.13) 25.8498 (1.13) 0.2844 (1.57) 25.8189 (1.13) 0.3637 (2.10) 13;0 38.6850 (0.88) 37 1
test_benchmark_query12 39.7474 (1.77) 40.7568 (1.74) 40.1831 (1.76) 0.2553 (1.41) 40.1617 (1.76) 0.3421 (1.97) 8;0 24.8861 (0.57) 24 1
test_benchmark_query13 30.6525 (1.36) 31.6718 (1.35) 31.1661 (1.36) 0.2219 (1.22) 31.1685 (1.37) 0.2735 (1.58) 9;0 32.0862 (0.73) 32 1
test_benchmark_query14 24.9189 (1.11) 25.7928 (1.10) 25.2736 (1.11) 0.2192 (1.21) 25.2234 (1.10) 0.2905 (1.68) 14;0 39.5670 (0.90) 40 1
test_benchmark_query15 24.4846 (1.09) 25.9639 (1.11) 24.8480 (1.09) 0.3059 (1.69) 24.7728 (1.09) 0.2651 (1.53) 7;4 40.2447 (0.92) 40 1
test_benchmark_query16 26.4125 (1.17) 28.7315 (1.22) 26.9644 (1.18) 0.4993 (2.76) 26.8477 (1.18) 0.4448 (2.57) 10;2 37.0859 (0.85) 38 1
test_benchmark_query17 24.8586 (1.11) 25.9273 (1.11) 25.1584 (1.10) 0.2070 (1.14) 25.1088 (1.10) 0.2497 (1.44) 10;1 39.7482 (0.91) 40 1
test_benchmark_query18 24.1813 (1.08) 25.1143 (1.07) 24.5338 (1.07) 0.2151 (1.19) 24.5376 (1.07) 0.3178 (1.83) 13;0 40.7601 (0.93) 40 1
test_benchmark_query19 40.9274 (1.82) 42.1138 (1.80) 41.5658 (1.82) 0.2597 (1.43) 41.5231 (1.82) 0.3043 (1.76) 6;1 24.0582 (0.55) 24 1
test_benchmark_query2 24.0586 (1.07) 25.6033 (1.09) 24.5185 (1.07) 0.3828 (2.11) 24.3870 (1.07) 0.4703 (2.71) 11;1 40.7855 (0.93) 40 1
test_benchmark_query20 24.7223 (1.10) 25.5913 (1.09) 25.0012 (1.09) 0.2271 (1.25) 24.9686 (1.09) 0.3116 (1.80) 12;1 39.9980 (0.91) 39 1
test_benchmark_query21 23.9809 (1.07) 24.8590 (1.06) 24.3109 (1.06) 0.2306 (1.27) 24.2767 (1.06) 0.4152 (2.40) 18;0 41.1339 (0.94) 40 1
test_benchmark_query22 34.5507 (1.54) 35.4196 (1.51) 35.0123 (1.53) 0.1942 (1.07) 35.0202 (1.53) 0.1733 (1.0) 10;3 28.5614 (0.65) 29 1
test_benchmark_query23 25.0542 (1.11) 25.7579 (1.10) 25.3481 (1.11) 0.1861 (1.03) 25.3184 (1.11) 0.3040 (1.75) 12;0 39.4507 (0.90) 39 1
test_benchmark_query24 24.2572 (1.08) 24.9892 (1.07) 24.6704 (1.08) 0.1812 (1.0) 24.6422 (1.08) 0.2831 (1.63) 14;0 40.5345 (0.93) 40 1
test_benchmark_query25 23.6439 (1.05) 24.3974 (1.04) 24.0185 (1.05) 0.1917 (1.06) 24.0060 (1.05) 0.3119 (1.80) 12;0 41.6346 (0.95) 41 1
test_benchmark_query26 25.3782 (1.13) 26.7081 (1.14) 25.8477 (1.13) 0.2269 (1.25) 25.8677 (1.13) 0.2506 (1.45) 8;1 38.6881 (0.88) 38 1
test_benchmark_query27 45.2601 (2.01) 47.5899 (2.03) 46.1629 (2.02) 0.7840 (4.33) 45.8066 (2.01) 1.2243 (7.07) 6;0 21.6624 (0.50) 22 1
test_benchmark_query28 25.3215 (1.13) 26.6050 (1.13) 25.9303 (1.13) 0.3192 (1.76) 25.9396 (1.14) 0.5335 (3.08) 13;0 38.5650 (0.88) 37 1
test_benchmark_query29 25.0946 (1.12) 26.3178 (1.12) 25.5235 (1.12) 0.2618 (1.44) 25.4866 (1.12) 0.2401 (1.39) 11;4 39.1795 (0.90) 39 1
test_benchmark_query3 25.0039 (1.11) 26.0560 (1.11) 25.5028 (1.12) 0.2508 (1.38) 25.5228 (1.12) 0.4286 (2.47) 12;0 39.2114 (0.90) 38 1
test_benchmark_query4 24.7615 (1.10) 25.7094 (1.10) 25.1573 (1.10) 0.1994 (1.10) 25.1555 (1.10) 0.2443 (1.41) 11;1 39.7500 (0.91) 40 1
test_benchmark_query5 23.8402 (1.06) 25.2271 (1.08) 24.4019 (1.07) 0.3319 (1.83) 24.3533 (1.07) 0.4441 (2.56) 13;0 40.9805 (0.94) 41 1
test_benchmark_query6 22.4930 (1.0) 23.4594 (1.0) 22.8720 (1.0) 0.2377 (1.31) 22.8306 (1.0) 0.3382 (1.95) 14;0 43.7215 (1.0) 43 1
test_benchmark_query7 34.7297 (1.54) 35.7101 (1.52) 35.3026 (1.54) 0.2547 (1.41) 35.2882 (1.55) 0.3002 (1.73) 9;0 28.3265 (0.65) 28 1
test_benchmark_query8 23.0910 (1.03) 23.9041 (1.02) 23.4985 (1.03) 0.2386 (1.32) 23.4764 (1.03) 0.4099 (2.37) 15;0 42.5560 (0.97) 42 1
test_benchmark_query9 24.1168 (1.07) 32.9910 (1.41) 24.7383 (1.08) 1.3370 (7.38) 24.5148 (1.07) 0.3144 (1.81) 1;1 40.4232 (0.92) 41 1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
collected 29 items

benchmark_query.py ............................. [100%]


------------------------------------------------------------------------------------ benchmark: 29 tests ------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_query1 1.3767 (1.95) 1.7772 (1.68) 1.4724 (1.77) 0.0525 (1.37) 1.4636 (1.77) 0.0596 (1.18) 54;6 679.1737 (0.56) 232 1
test_benchmark_query10 24.5189 (34.65) 26.1044 (24.74) 25.2219 (30.36) 0.3681 (9.62) 25.2228 (30.42) 0.5567 (11.05) 13;0 39.6482 (0.03) 39 1
test_benchmark_query11 2.7414 (3.87) 4.0430 (3.83) 3.2230 (3.88) 0.2305 (6.02) 3.2217 (3.89) 0.2930 (5.82) 75;2 310.2685 (0.26) 229 1
test_benchmark_query12 17.1755 (24.27) 18.3755 (17.41) 17.7318 (21.35) 0.2483 (6.49) 17.6685 (21.31) 0.2672 (5.30) 14;4 56.3959 (0.05) 58 1
test_benchmark_query13 8.3278 (11.77) 9.9391 (9.42) 8.6597 (10.43) 0.2016 (5.27) 8.6350 (10.41) 0.2352 (4.67) 22;1 115.4775 (0.10) 110 1
test_benchmark_query14 2.5917 (3.66) 3.2937 (3.12) 2.7956 (3.37) 0.1052 (2.75) 2.7737 (3.34) 0.1003 (1.99) 61;17 357.7025 (0.30) 318 1
test_benchmark_query15 2.1405 (3.02) 2.8080 (2.66) 2.2776 (2.74) 0.0736 (1.92) 2.2651 (2.73) 0.0787 (1.56) 84;13 439.0494 (0.36) 403 1
test_benchmark_query16 3.9124 (5.53) 5.2454 (4.97) 4.2132 (5.07) 0.1204 (3.14) 4.2151 (5.08) 0.1292 (2.56) 45;3 237.3504 (0.20) 234 1
test_benchmark_query17 2.4369 (3.44) 2.9189 (2.77) 2.5873 (3.11) 0.0704 (1.84) 2.5801 (3.11) 0.0785 (1.56) 82;13 386.5047 (0.32) 339 1
test_benchmark_query18 1.8815 (2.66) 2.3139 (2.19) 2.0325 (2.45) 0.0625 (1.63) 2.0295 (2.45) 0.0839 (1.67) 131;7 492.0153 (0.41) 455 1
test_benchmark_query19 18.4428 (26.06) 19.6378 (18.61) 18.9444 (22.81) 0.2700 (7.05) 18.9392 (22.84) 0.3506 (6.96) 16;1 52.7861 (0.04) 52 1
test_benchmark_query2 2.0713 (2.93) 2.5650 (2.43) 2.3162 (2.79) 0.0817 (2.13) 2.3036 (2.78) 0.1143 (2.27) 121;2 431.7420 (0.36) 389 1
test_benchmark_query20 2.1316 (3.01) 2.9786 (2.82) 2.3992 (2.89) 0.1500 (3.92) 2.3583 (2.84) 0.1600 (3.18) 85;26 416.8106 (0.35) 368 1
test_benchmark_query21 1.5456 (2.18) 1.9687 (1.87) 1.6820 (2.02) 0.0697 (1.82) 1.6773 (2.02) 0.0838 (1.66) 103;14 594.5466 (0.49) 398 1
test_benchmark_query22 12.1362 (17.15) 12.8211 (12.15) 12.4648 (15.01) 0.1629 (4.26) 12.4526 (15.02) 0.2373 (4.71) 26;0 80.2257 (0.07) 78 1
test_benchmark_query23 2.4816 (3.51) 3.0308 (2.87) 2.6765 (3.22) 0.0765 (2.00) 2.6667 (3.22) 0.0908 (1.80) 96;7 373.6270 (0.31) 332 1
test_benchmark_query24 1.8614 (2.63) 2.2366 (2.12) 1.9925 (2.40) 0.0576 (1.50) 1.9885 (2.40) 0.0692 (1.37) 124;9 501.8797 (0.42) 440 1
test_benchmark_query25 1.3479 (1.90) 1.8455 (1.75) 1.4648 (1.76) 0.0465 (1.22) 1.4623 (1.76) 0.0584 (1.16) 175;4 682.6643 (0.57) 611 1
test_benchmark_query26 2.8840 (4.08) 3.4320 (3.25) 3.1116 (3.75) 0.0938 (2.45) 3.1040 (3.74) 0.1305 (2.59) 93;3 321.3729 (0.27) 316 1
test_benchmark_query27 22.4221 (31.69) 23.7178 (22.48) 23.0234 (27.72) 0.3001 (7.84) 23.0482 (27.79) 0.4085 (8.11) 14;0 43.4341 (0.04) 42 1
test_benchmark_query28 2.6678 (3.77) 3.3802 (3.20) 2.8639 (3.45) 0.1074 (2.80) 2.8417 (3.43) 0.1035 (2.05) 62;16 349.1685 (0.29) 301 1
test_benchmark_query29 2.4007 (3.39) 3.0182 (2.86) 2.6757 (3.22) 0.0928 (2.42) 2.6743 (3.23) 0.1226 (2.43) 118;5 373.7399 (0.31) 383 1
test_benchmark_query3 3.1942 (4.51) 3.8568 (3.65) 3.3852 (4.08) 0.0895 (2.34) 3.3731 (4.07) 0.1102 (2.19) 75;4 295.4012 (0.25) 255 1
test_benchmark_query4 2.8370 (4.01) 3.6791 (3.49) 3.1119 (3.75) 0.1136 (2.97) 3.1145 (3.76) 0.1467 (2.91) 96;2 321.3500 (0.27) 309 1
test_benchmark_query5 1.8461 (2.61) 2.3278 (2.21) 2.1083 (2.54) 0.0858 (2.24) 2.1098 (2.54) 0.1243 (2.47) 128;3 474.3254 (0.39) 403 1
test_benchmark_query6 0.7076 (1.0) 1.0553 (1.0) 0.8306 (1.0) 0.0383 (1.0) 0.8292 (1.0) 0.0504 (1.0) 273;10 1,203.8883 (1.0) 944 1
test_benchmark_query7 12.7920 (18.08) 13.8115 (13.09) 13.2379 (15.94) 0.2134 (5.57) 13.1735 (15.89) 0.2816 (5.59) 21;1 75.5406 (0.06) 73 1
test_benchmark_query8 1.1148 (1.58) 1.4808 (1.40) 1.2611 (1.52) 0.0517 (1.35) 1.2631 (1.52) 0.0723 (1.44) 191;4 792.9300 (0.66) 602 1
test_benchmark_query9 2.0159 (2.85) 2.5005 (2.37) 2.1874 (2.63) 0.0663 (1.73) 2.1839 (2.63) 0.0880 (1.75) 127;3 457.1621 (0.38) 423 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
======================================= 29 passed in 31.56s =======================================
============================================ 29 passed in 27.64s =============================================
```
Binary file modified results/benchmark_heatmap.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading