Repository navigation
CSV.Chunks possible memory leak #1048
Description
Code used (on 4 threads):
julia> using CSV, DataFrames
julia> tables = CSV.Chunks("large.csv", ntasks=100);
julia> for t in tables
df = DataFrame(t)
end
The result is that huge amounts of RAM are allocated by Julia process and never reclaimed, even by GC.gc().
Using clear_thread_states() from #1046 does not resolve the issue.
The expectation is that I use CSV.Chunks to read very large files (larger than RAM), which is effectively made problematic with this problem.
Activity
[-]CSV.Chunks memory leak[/-][+]CSV.Chunks possible memory leak[/+]quinnj commentedon Sep 15, 2026
I tried to reproduce this against both
0.10.16and the new1.0.0, on a synthetic file. Short version: there is no unbounded leak in either version, the 0.10.x symptom is fully explained by four bounded effects, 1.0 fixes the part that mattered — but 1.0 introduces a new cost that still blocks the actual goal in the issue (files larger than RAM). Details below.Setup: macOS arm64, 96 GiB RAM, Julia 1.12.6 and 1.8.5,
JULIA_NUM_THREADS=4. Files:big.csv2.72 GiB / 30M rows x 10 mixed columns,half.csv= its first half,narrow.csv0.149 GiB / 40M rows x 2 small Int columns. The loop is exactly the one reported:for t in CSV.Chunks(f; ntasks=100); df = DataFrame(t); end.1. Not a leak
Running the whole sweep 3x in one process, with a full
GC.gc(true)between rounds (CSV 0.10.16,ntasks=4,big.csv):Base.gc_live_bytes()is flat across rounds, so nothing accumulates. Julia 1.8.5 behaves the same as 1.12.6, so this is not a Julia-version story either.2. What the 0.10.x symptom actually was
(a)
ntaskssets the chunk count, so chunk size scales with the file. In 0.10.xChunks.iterateslices the file into exactlyntaskspieces, so a batch isfilesize/ntasks. Streaming a 1 TB file withntasks=100asks for 10 GB batches — the knob works backwards for the stated use case. Measured onbig.csv:(b) The GC does not run between iterations. At
ntasks=4the live heap reached 2.995 GiB while only one ~1.4 GiBDataFramewas reachable. Dead batches simply had not been collected yet — that is what makes RSS look like it climbs forever.(c) The whole file is mmapped and its resident pages are charged to the process.
Chunksholds the mapping for the whole iteration, so touching every chunk makes every page resident, andGC.gc()cannot drop it while the mapping is referenced. In thentasks=100run RSS went 0.456 -> 3.842 GiB and fell back to 1.118 GiB the momenttables = nothing; GC.gc(true)ran. That drop is 2.724 GiB — exactly the file size.(d) A ~0.7 GiB residue that survives a full GC — and it is DataFrames, not CSV. After the sweep plus three
GC.gc(true)calls, 0.10.16 still showed 0.724 GiB live; running unrelated tasks/allocations afterwards released it (the JuliaLang/julia#40626 signature). Swapping only the consumer localises it:DataFrame(t)(defaultcopycols=true)DataFrame(t; copycols=false)Tables.columntable(t)This also explains why #1046's
clear_thread_states()did not help here: 0.10's per-chunk parse runs withthreaded=false, ntasks=1and spawns no tasks at all, andclear_thread_states()is only called on the multithreadedFilepath.3. What 1.0 fixes
big.csv,ntasks=100,DataFrameper batch:GC.gc(true)Batch memory no longer scales with file size (
chunkbytes = clamp(cld(filesize, ntasks), 1 KiB, 4 MiB)), and there is no stale-root residue. Defaultstringtype=DataStringalso cuts per-cell heap objects: forcing large batches (chunkbytes=731 MB) to compare like for like,DataStringpeaks at 9.63 GiB vs 13.8 GiB forstringtype=String.RSS still does not return to the OS promptly after a GC in either version, but that is a Julia allocator/GC property rather than a CSV one.
4. New problem in 1.0 — the whole-file index is retained
CSV.Chunks(andCSV.Rows) in 1.0 build and keep the structural index for the entire file before the first batch is produced:half.csvbig.csvnarrow.csvThat is
ChunkIndex.tape(4 bytes per field event) plusrowfirstandrowstartrel(4 bytes per row each), roughly4*(ncols + 1) + 8bytes per row. It is independent ofchunkbytes(1 MiB / 16 MiB / 256 MiB all give ~1.48 GiB onbig.csv), and tapes are never released after their batch is consumed. 0.10.16'sChunksconstructor was O(1) by comparison (Δlive = 0.000 GiB).CSV.Rowsis the same and its docstring says so plainly ("The source bytes and complete structural index remain in memory"):narrow.csv,CSV.RowsSo 1.0 is dramatically faster but currently has no constant-memory path for a file larger than RAM, which is the requirement stated in this issue. For wide files the index is ~0.5x the file; for narrow numeric files it can be 6x the file.
5. Doc/behaviour mismatch found on the way
The
CSV.Chunksdocstring says "ntaskssets the target batch count; usechunkbytesfor direct size control". Becausechunkbytesis clamped to 4 MiB,ntasksstops controlling the batch count for any file larger than4 MiB * ntasks:CSV.Chunks("big.csv", ntasks=4)yields 698 batches, not 4.Where that leaves this issue
The original report is explained and the growth-during-iteration behaviour is fixed in 1.0. I'd like to keep this issue open until the retained index is addressed, since that is what the original use case ("larger than RAM") actually needs. A PR is on the way for (4) and (5).
[issue investigated by AI; reviewed by quinnj; please respond on discrepancies/inaccuracies]
quinnj commentedon Oct 15, 2026
[comment generated via AI] Fixed by #1204, released in CSV 1.1.0:
CSV.Chunksindexes one window of the source at a time and releases it with its batch. Checked on main with Julia 1.12: a 46.6 MiB two-column file read withchunkbytes=4 MiBholds 0.2 MiB after construction and at most 14.6 MiB while iterating its 12 batches (1.0.0 retained about 6x the file size for this shape). A path source is memory-mapped; anIOsource is still read fully into memory.