Skip to content

perf(mysql): move values out of rows instead of clonning when converting to Arrow - #94

Merged
sgrebnov merged 1 commit into
spiceai-55from
sgrebnov/mysql-rows-to-arrow-take
Oct 18, 2026
Merged

sgrebnov merged 1 commit into
spiceai-55from
sgrebnov/mysql-rows-to-arrow-take

Conversation

@sgrebnov

Copy link
Copy Markdown

What

rows_to_arrow for MySQL cloned every cell (Row::get_opt clones the Value out of a borrowed row) and cloned the column name for every cell to use as error context. This change:

  • takes Vec<Row> and moves each value out with Row::take_opt;
  • borrows the column name instead of cloning it;
  • fills columns one at a time.

The only caller, MySQLConnection::query_arrow, already owns the rows. Output is unchanged.

Measurements

rows_to_arrow on 800,000 synthetic rows (200 chunks of 4,000): 12 columns including five char(34), a varchar, two text/longtext columns (one with a 3,300-byte value), a tinyint and two timestamps. Allocations were counted with a counting global allocator, and are identical across runs. CPU time is three alternating runs per build on a loaded machine.

before after
heap allocations per row 20.06 0.06
bytes allocated per row 11,234 7,587
CPU per 4,000-row chunk (min) 3.73–3.83 ms 1.89–2.00 ms
CPU per 4,000-row chunk (p99) 4.30–4.52 ms 2.33–3.54 ms

The remaining per-cell allocation is mysql_common decoding each value into Value::Bytes, which this change does not touch.

Testing

  • cargo test --features mysql --lib mysql: 5 passed.
  • cargo test --features mysql --test integration mysql: 11 passed, 3 ignored, 1 failed (test_mysql_arrow_oneway). That test fails the same way on the base commit and is unrelated to this change.
  • To run the integration tests locally, set CONTAINER_REGISTRY with a trailing slash. The default public.ecr.aws/docker/library resolves to .../librarymysql:latest.

rows_to_arrow cloned every cell with Row::get_opt and cloned the column
name for every cell. Take ownership of the rows and move each value out
with Row::take_opt, borrowing the column name instead.
@sgrebnov sgrebnov self-assigned this Oct 18, 2026
@sgrebnov
sgrebnov marked this pull request as ready for review October 18, 2026 18:50
Copilot AI balanced review requested due to automatic review settings October 18, 2026 18:50
@sgrebnov sgrebnov changed the title perf(mysql): move values out of rows when converting to Arrow perf(mysql): move values out of rows instead of clonning when converting to Arrow Oct 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

The ownership and iteration changes are straightforward, with no blocking correctness issues identified.

No open issues 🎉

What changed in this PR

Reduces allocations in MySQL-to-Arrow conversion by moving cell values rather than cloning them.

Changes:

  • Consumes owned rows and converts values one column at a time.
  • Borrows column names, allocating error context only when needed.
  • Updates the query caller to transfer row ownership.
File Description
core/​src/​sql/​db_connection_pool/​dbconnection/​mysqlconn.rs Passes owned rows to the converter.
core/​src/​sql/​arrow_sql_gen/​mysql.rs Moves cell values and avoids repeated column-name clones.

Review effort: Balanced

@sgrebnov
sgrebnov merged commit c5bcad1 into spiceai-55 Oct 18, 2026
24 of 28 checks passed
@sgrebnov
sgrebnov deleted the sgrebnov/mysql-rows-to-arrow-take branch October 18, 2026 18:54
sgrebnov added a commit to spiceai/spiceai that referenced this pull request Oct 18, 2026
Re-audited: the only commit since 54d9010 (spiceai/datafusion-table-providers#94)
touches MySQL row conversion, which no ledgered patch covers.
mikeyasaservice pushed a commit to mikeyasaservice/spiceai that referenced this pull request Oct 19, 2026
…version (spiceai#15271)

* chore(deps): bump datafusion-table-providers for faster MySQL row conversion

Pick up spiceai/datafusion-table-providers#94, which moves values out of
MySQL rows instead of cloning them when converting to Arrow.

* docs(fork-patches): record datafusion-table-providers pin c5bcad1

Re-audited: the only commit since 54d9010 (spiceai/datafusion-table-providers#94)
touches MySQL row conversion, which no ledgered patch covers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants