Repository navigation
perf(mysql): move values out of rows instead of clonning when converting to Arrow - #94
Merged
Merged
Conversation
rows_to_arrow cloned every cell with Row::get_opt and cloned the column name for every cell. Take ownership of the rows and move each value out with Row::take_opt, borrowing the column name instead.
sgrebnov
marked this pull request as ready for review
October 18, 2026 18:50
There was a problem hiding this comment.
🟢 Approved
The ownership and iteration changes are straightforward, with no blocking correctness issues identified.
No open issues 🎉
What changed in this PR
Reduces allocations in MySQL-to-Arrow conversion by moving cell values rather than cloning them.
Changes:
- Consumes owned rows and converts values one column at a time.
- Borrows column names, allocating error context only when needed.
- Updates the query caller to transfer row ownership.
| File | Description |
|---|---|
| core/src/sql/db_connection_pool/dbconnection/mysqlconn.rs | Passes owned rows to the converter. |
| core/src/sql/arrow_sql_gen/mysql.rs | Moves cell values and avoids repeated column-name clones. |
Review effort: Balanced
sgrebnov
added a commit
to spiceai/spiceai
that referenced
this pull request
Oct 18, 2026
Re-audited: the only commit since 54d9010 (spiceai/datafusion-table-providers#94) touches MySQL row conversion, which no ledgered patch covers.
mikeyasaservice
pushed a commit
to mikeyasaservice/spiceai
that referenced
this pull request
Oct 19, 2026
…version (spiceai#15271) * chore(deps): bump datafusion-table-providers for faster MySQL row conversion Pick up spiceai/datafusion-table-providers#94, which moves values out of MySQL rows instead of cloning them when converting to Arrow. * docs(fork-patches): record datafusion-table-providers pin c5bcad1 Re-audited: the only commit since 54d9010 (spiceai/datafusion-table-providers#94) touches MySQL row conversion, which no ledgered patch covers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
rows_to_arrowfor MySQL cloned every cell (Row::get_optclones theValueout of a borrowed row) and cloned the column name for every cell to use as error context. This change:Vec<Row>and moves each value out withRow::take_opt;The only caller,
MySQLConnection::query_arrow, already owns the rows. Output is unchanged.Measurements
rows_to_arrowon 800,000 synthetic rows (200 chunks of 4,000): 12 columns including fivechar(34), avarchar, twotext/longtextcolumns (one with a 3,300-byte value), atinyintand twotimestamps. Allocations were counted with a counting global allocator, and are identical across runs. CPU time is three alternating runs per build on a loaded machine.The remaining per-cell allocation is
mysql_commondecoding each value intoValue::Bytes, which this change does not touch.Testing
cargo test --features mysql --lib mysql: 5 passed.cargo test --features mysql --test integration mysql: 11 passed, 3 ignored, 1 failed (test_mysql_arrow_oneway). That test fails the same way on the base commit and is unrelated to this change.CONTAINER_REGISTRYwith a trailing slash. The defaultpublic.ecr.aws/docker/libraryresolves to.../librarymysql:latest.