Repository navigation
perf(processor): keep Intersect's per-tuple scratch buffers on the operator - #1372
Merged
Merged
Conversation
Intersect allocated on every probed row: probeHTs built a vector of vectors it never read. Each intersected tuple allocated five more buffers: the tuple pointers, the lists, the list order, the left selection vector (up to 16 KB) and the list of selection vectors to slice. These are now members sized once in initLocalStateInternal, so probing does not allocate. initLocalStateInternal also resets the carry state (probedFlatTuples, tupleIdxPerBuildSide, carryBuildSideIdx), which only the constructor set, so a reused operator starts clean as its comment says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adsharma
reviewed
Oct 24, 2026
| sliceSelVectors(selVectorsForIntersectedLists, lSelVector); | ||
| lSelVector.setToUnfiltered(); | ||
| sliceSelVectors(selVectorsForIntersectedLists, leftSelVector); | ||
| leftSelVector.setToUnfiltered(); |
Contributor
There was a problem hiding this comment.
The new code reuses one leftSelVector across intersect rounds:
twoWayIntersect(..., leftSelVector, ...); // shrinks leftSelVector to the survivors
sliceSelVectors(..., leftSelVector);
leftSelVector.setToUnfiltered(); // <- keep the shrunken size, just unfilter it for next roundThe delta between the two variants of setToUnfiltered() is subtle and easy to miss.
Contributor
Author
There was a problem hiding this comment.
Agreed, the overload hides it. This merged before I got to it, so #1382 passes the size explicitly, leftSelVector.setToUnfiltered(leftSelVector.getSelSize()), with a comment that twoWayIntersect leaves the surviving left IDs at the front of outKeyVector. Resetting the size to the first list's length instead fails acc_hj, cyclic/single_label, cyclic/multi_label and cyclic/large_lists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This work was produced with the help of language models.
Fixes #1371.
Intersectallocated while probing: an unused vector for every probed row and five buffers for every intersected tuple. That is most of the allocation calls of LDBC SNB SF1 q30 and of the forum-member query below.Fix. The unused vector is gone. The five buffers are members sized once in
initLocalStateInternal; the selection vector holdsDEFAULT_VECTOR_CAPACITY, so each task clone allocates one 16 KB buffer instead of up to one per tuple. Init also resetsprobedFlatTuples,tupleIdxPerBuildSideandcarryBuildSideIdx, which only the constructor set.¹
MATCH (f:Forum)-[:hasMember]->(p:Person)-[:likePost]->(:Post)<-[:containerOf]-(f) RETURN count(*). Wheel-like builds against main b9db312, same rows; wall is the median of 3 interleaved rounds. q1 and a two-hopknowscount, without an INTERSECT, stay within run-to-run noise.Tests.
cyclic/large_lists.testgives a node 3,000 out-neighbours, so its list spans two build tuples, and runs a triangle and a three-build-side 4-clique over it, both pinned withHINT; expected values come from main. Full ctest: 3165/3165.Related to #1291; these allocations were per row, not per execution.