Repository navigation
Conversation
When one side of an inner equi-join on a single node ID is a bare single-node scan of that node (pure payload fetch, e.g. LDBC Q1's Person scan) and the other side already binds the node ID with a small estimated cardinality (<= LOOKUP_JOIN_MAX_OUTER_CARDINALITY), replace the full scan + hash join with per-tuple primary-key lookups (LogicalQueryPrimaryKeyLookup, inner semantics: non-matches dropped). Storage: NodeTable::lookupPK resolves INTERNAL_ID keys directly to offsets (table-ID + bounds + visibility checks, local-storage aware) without probing the HASH/ART primary-key index. Q1: top HASH_JOIN(p._ID) + full Person scan + semi-masker becomes a 0.07ms lookup over the small outer (was ~1ms of the ~2.4ms exec).
LogicalQueryPrimaryKeyLookup assumed nodeID/properties are new, but lookup joins use the already-bound join node ID as the key (same unique name), tripping the schema duplicate-insert assert and breaking shell tests. Use insertToGroupAndScopeMayRepeat (same as hash joins on node IDs) and skip properties already bound by the outer plan. Also fix clang-format layout flagged by CI.
The third CTRL_P was commented 'move up line' but on Linux CTRL_P navigates history instead of moving a row (row-up-first applies only on Windows and for the up-arrow escape on all platforms). With just this test's own entries in history.txt the extra history-prev clamps to a no-op, leaving the cursor on the bottom row, so the two Downs walk off the recalled entry onto the empty line and ENTER submits nothing until the 30s expect times out. The test only passed when older shared-history entries happened to absorb the overshoot, hence flaky across selections/orders. Send up-arrow for the row-up step so the sequence is independent of history depth.
…t results Planner (join order): - Lookup join no longer prunes hash alternatives. Both lookup orientations are added alongside the hash candidates, each maxCost-gated, and the cheapest plan wins. - Cost via CostModel::computeLookupJoinCost (outer cost + per-tuple probes). - Eligibility is semantic: bare single-node inner scan, single-table node, PK index present, key evaluable, payload properties that exist, outer within probe budget and smaller than the inner table (probes must save more scan than they cost; this also keeps full-coverage joins on the hash path so COUNT_REL_TABLE/degree/reachability rewrites still match). - encodeJoin renders QUERY_PRIMARY_KEY_LOOKUP as LKP(key) so plan tests can pin the new shape; SingleNodeTwoHopJoins expectations updated (results verified identical with the optimizer on/off). Storage: - ArrowNodeTable/IceDiskNodeTable resolve INTERNAL_ID keys directly to offsets in lookupPK. Parquet writer: - readFromFT bulk reads into fresh flat single-value vectors, which can only ever hold the first tuple (FactorizedTable::lookup asserts !(isFlat && numTuples > 1); release builds silently wrote corrupt files, surfacing as TProtocolException on import). Always read one tuple here; the bulk path could never be correct.
a19e5a4 to
82ffc3a
Compare
|
This is a fundamental change vs kuzu hash join algorithms. Could use more testing. Cc: @zahariash |
|
Tested 82ffc3a against its base, Correctness: no wrong results. I added checks to our workload simulator for the shapes that plan Performance: the heavy IC queries regress. 23 of the 44 plans change. Execute plus drain, median of 9 runs, the lower of two alternating rounds per build:
The outer side is far larger than its estimate. In IC5 the lookup on Arrow tables never take the lookup. On tables from Separately, |
This branch has not been deployed
2 participants
What
Replaces full node-table scan + hash join with per-tuple primary-key lookups when one side of an inner equi-join on a single node ID is a bare single-node scan of that node (pure payload fetch) and the other side already binds the node ID with small estimated cardinality (gated by new knob
PlannerKnobs::LOOKUP_JOIN_MAX_OUTER_CARDINALITY = 1024, INLJ-style pruning precedent).Storage:
NodeTable::lookupPKresolvesINTERNAL_IDkeys directly to offsets (table-ID + bounds + visibility, local-storage aware for open write transactions) without probing the HASH/ART PK index.Motivation (LDBC Q1)
The top
HASH_JOIN(p._ID)+ full 9892-row Person scan + semi-masker becomes aQUERY_PRIMARY_KEY_LOOKUP[p._ID]over the small outer: Person fetch ~1ms -> ~0.07ms; Q1 exec ~2.4ms -> ~2.0ms (remainder is the Tag-rooted 841-row chain + fixed overhead, tracked separately).Validation
QUERY_PRIMARY_KEY_LOOKUP[table=3 key=p._ID].e2e_testoptimizer suite: 4/4 pass.AnonymousParquetDeleteReload, which fails identically on basemain(pre-existing).DeleteLocalNode+RecursiveJoinManualTransaction, both green now).Notes