Repository navigation
CSV.Chunks splits file into uneven chunks #1122
Description
In the following example the "a" variable does not have a consistent size.
using CSV, DataFrames
number_of_lines = 10^6
CSV.write("data.csv", DataFrame(rand(number_of_lines, 10), :auto))
steps = 10
@time for chunk in CSV.Chunks("data.csv"; ntasks=steps)
a = chunk |> DataFrame
display(size((a)))
end
Please also have a look at this discourse post, for context.
Activity
quinnj commentedon Oct 15, 2026
[comment generated via AI] Since CSV 1.1.0 (#1204), each batch is a byte window of
chunkbytes(64 MiB by default) that ends at a row boundary, so every batch except the last has about the same size, whateverntasksis. Checked on main: a 10.8 MB file withchunkbytes=4 MiBgives batches of 155,345, 155,345, and 89,310 rows. For about N equal batches, passchunkbytes=cld(filesize(path), N).