Repository navigation
fix(planner): cap a lookup join by its flat outer rows - #1370
Merged
Merged
Conversation
The lookup join checked LOOKUP_JOIN_MAX_OUTER_CARDINALITY against the outer side's factorized estimate, but it flattens that side and probes once per flat row. An extend leaves its neighbours unflat, so a side that fans out passed the cap far above it: on LDBC SF1 complex query 5 the hasMember side was estimated at 73 rows before flattening and 11,895 after, and ran 1,569,282 Forum lookups in place of one scan and hash join. The cap, the fan-out discount and the lookup's cost and cardinality now use the flattened estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Thanks for getting to the bottom of this and this honest assessment. The planner is less than perfect, but the flat vs unflat error is the higher order bit. |
This was referenced Oct 25, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This work was produced with the help of language models.
Fixes #1369.
LDBC SF1 complex query 5 (
query_complex14.py 5) took 1.14 s on0.22.0.dev20261017and 2.34 s ondev20261021. ItsMATCH (friend)<-[membership:hasMember]-(forum:Forum)now fetchesforum.titleandforum.IDwith 1,569,282 lookups by node ID, where it used to scan Forum once and hash-join it.Fix. The lookup join from #1285 compared
LOOKUP_JOIN_MAX_OUTER_CARDINALITY(1024) with the outer side's estimate before flattening it, but it probes once per flat row. An extend keeps its neighbours unflat, so that estimate was 73 here and the flattened one 11,895. The cap, the fan-out discount, the lookup's cost and its reported cardinality now use the flattened estimate. The unflattened check stays in front as a pre-filter.Plans. Of 50 LDBC SF1 queries (30 short, 6 heavy, IC1-14), only IC5, IC2 and IC10 change: IC5 and IC2 go back to hash joins, and IC10 keeps one of its two lookup joins. The other 47 plans are identical, including the lookup joins in q1, q3, q7, q17 and IC7, and results are identical on all 50.
mainmainIC5 is back to the 10-17 nightly's 43.0G, and IC2 gives back a 2.7% instruction win. Wall times are left out because the machine was under heavy load.
Tests.
LookupJoinCountsFlatOuterRowsgives every outer node 20 neighbours. 100 nodes (2,000 probes into 5,000 rows) and 50 nodes (1,000 probes into 500 rows) got lookup joins onmainand get hash joins here. 10 nodes keep theirs, now reported as 200 rows instead of 10. Reverting the cap, the discount or the cardinality alone fails its case; the cost change has no case of its own. Full ctest: 3181/3181.#1307's runtime fallback is still worth having: IC5's flattened estimate is about 130x below its actual probes, because the friend set is estimated at 730 instead of 9,892 and the
joinDatefilter at 10% instead of 97%.