Repository navigation
CSV.Chunks API #1049
Description
From the docstring of CSV.Chunks it seems that ntasks is used to specify number of chunks.
The issue is that this name is strange. ntasks should be rather used to specify number of tasks allocated for reading data. Some other kwarg, e.g. nchunks should be used to specify number of chunks I think.
Additionally documentation could say something about how the size of the chunk is determined. It seems that this is not an equal split, but some other rule (most likely dynamic and approximate).
Activity
quinnj commentedon Oct 15, 2026
[comment generated via AI] Addressed in CSV 1.1.0 (#1204):
ntasksbounds the tasks that parse one batch and does not set the batch count.chunkbytessets the batch size, a byte window that ends at a row boundary (64 MiB by default), and theCSV.Chunksdocstring describes both. Checked on main:ntasks=1,4, and16give the same 3 batches for a 10.8 MB input withchunkbytes=4 MiB.