Repository navigation
tp: run steps which keep their rows as transforms - #7798
Merged
Merged
Conversation
Adds benchmarks running each tree and type step through a pipeline over a scan, over one row and over 2^20 rows. Over many rows the step's own work dominates; over one row, the pipeline's work around it does.
AssertType converts one column of each batch and keeps every row, so it changes the batch it is given in place rather than copying it into a batch of its own. | Benchmark (per run) | Operator | Transform | |---|--:|--:| | BM_StepsAssertTypeAsIs/1 | 88.9 ns | 61.4 ns | | BM_StepsAssertTypeWiden/1 | 102.2 ns | 66.7 ns | | BM_StepsAssertTypeWiden/1048576 | 4.00 ms | 3.97 ms |
TreeNumberNodes, TreeAccumulateUp and TreeAccumulateDown each append columns to the batch they are given and keep every row, so they change it in place rather than copying it into a batch of their own. Tests run them on a fresh copy of their input, as a pipeline hands each batch over. | Benchmark (per run) | Operator | Transform | |---|--:|--:| | BM_StepsTreeNumberNodes/1 | 114.8 ns | 87.0 ns | | BM_StepsTreeAccumulateDown/1 | 182.8 ns | 119.8 ns | | BM_StepsTreeAccumulateUp/1 | 242.6 ns | 132.4 ns | | BM_StepsTreeAccumulateUp/1048576 | 52.1 ms | 51.7 ms |
🎨 Perfetto UI Builds & Tests
|
sashwinbalaji
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs AssertType, TreeNumberNodes, TreeAccumulateUp and TreeAccumulateDown
as transforms, changing the batch they are given in place.
Why
Each of these converts or appends columns and keeps every row: one batch
in, the same rows out. As operators they copied every batch into one of
their own and paid the operator's dispatch around it, which dominates on
small inputs.
Changes
scan, over one row and over 2^20 rows, as a plan runs it.
input, as a pipeline hands each batch over.
IntervalFlatten, TreeParentFirst, TreeChildFirst, Sort and GroupBy stay
operators: they emit rows other than their input's, hold rows back or
emit after the input ends.
Benchmarks
Per run, operator vs transform:
Over 2^20 rows every step is within 1% of before, as is every
TreeNumberNodes and tree order benchmark.
Testing
Existing unit tests, now run through Process.