Repository navigation
Run joins on the local engine - #91
Merged
Merged
Conversation
With clojask.engine=local, inner, left, right, outer and rolling joins no longer start Onyx, including their group-by phase. engine/start-join evaluates a's rows on the pool and does the lookup in b's groups on the calling thread, in a's order; engine/start-outer works one group per pool task, first a's groups with their matches, then b's unmatched groups. Both take the same arguments as the Onyx drivers they replace. Local joins read the dataframe's batches of rows instead of Onyx's segments per batch. join/join-func returns the lookup as a function instead of def'ing it, so the global source and the rolling join's memo and res are now locals; defn-join stays as the Onyx sink's wrapper. engine-test runs every join kind on disk and in memory on both engines, with and without :select, and checks that a failing join surfaces as an ExecutionException and leaves no join files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 3 of replacing Onyx: with
clojask.engine=local(orCLOJASK_ENGINE=local), every join runs without starting Onyx, including its group-by phase. The default engine is still Onyx.Changes
engine/start-join(inner, left, right, rolling forward/backward): a's rows are evaluated on the pool; the lookup in b's groups runs on the calling thread, so output is in a's order.engine/start-outer: one group per pool task, first a's groups with their matches in b, then b's unmatched groups.start-onyx-join/start-onyx-outer.JoinedDataFrame.computepicks the engine once; local runs read the dataframe's 300-row batches instead of Onyx's 10 segments per batch.join/join-funcreturns the lookup function instead ofdefing it. The globalsourceand the rolling join'smemo/resare locals now;defn-joinremains as a one-line wrapper for the Onyx sink.Tests
engine-testgainsjoin,join-selectandjoin-failure-is-staged: inner, swapped inner, left, right, outer, swapped outer, rolling forward, rolling backward and rolling backward with:limit, each on disk and in memory, compared against Onyx with Onyx made unreachable during the local run. The tests were seen failing before the implementation. The failure test checks for a stagedExecutionExceptionand that no.clojask/joinfiles remain.Suite: 60 tests / 304 assertions, green locally on both engines (JDK 25). CI runs the default engine.
Not changed: an operation that throws inside a join is swallowed on both engines unless
:exception true.Benchmark
Synthetic 1.8M-row CSV joined to a 2.3k-row table (rolling: 22k rows), 4 workers, one run after warm-up, seconds. Sorted outputs are byte-identical between engines.