Memory Consumption of CSV.Rows with ZipFile #997

Activity

1. nickrobinson251

   added

   performance

   on Oct 7, 2022
2. quinnj commented on Sep 15, 2026

   

   quinnj

   on Sep 15, 2026

   Member

   More actions

   Thanks for the detailed report — and sorry for the slow reply. I reproduced this and the memory is real, but most of it is not coming from where it looks like it is. Numbers below.

   Repro

   3.13 GiB CSV, 53.6M rows × 8 columns, exactly your schema, zipped to 1.81 GiB (macOS, Julia 1.12.6, CSV v0.10.17):

   z = ZipFile.Reader("t3400.zip")
   a = CSV.Rows(z.files[1]; types=[Int64, String, Float64, Float64, Int8, Float64, Float64, Int64])

   point	RSS
   start	0.30 GiB
   after CSV.Rows(...) returns	8.16 GiB (2.6× uncompressed)
   after GC.gc()	4.96 GiB

   So your "about double the size of the file" observation reproduces exactly.

   Where it actually goes

   It is not buffer_to_tempfile holding a copy. It is ZipFile.jl.

   ZipFile's inflate reader (Zlib.read!) is written as:

   function read!(r::Reader, a::Array{UInt8, N}) where N
       nb = length(a)
       if fillbuf(r, nb) < nb    # fill an internal PipeBuffer with ALL nb bytes
           throw(EOFError())
       end
       read!(r.buf, a)           # then copy it into the destination
       a
   end

   fillbuf grows an internal PipeBuffer until it holds the whole requested read, and only then copies into the destination. CSV asks for the member in one call, so ZipFile inflates 3.13 GiB into its own buffer and then hands you a second 3.13 GiB copy.

   Two measurements that pin it down:

   • Pre-sizing the destination myself (read!(f, Vector{UInt8}(undef, f.uncompressedsize))) — so nothing is growing on CSV's side at all — still peaks at 8.23 GiB. CSV is not in the picture there.
   • Draining the same member with read!(f, blk) in 16 MiB blocks and discarding every block: RSS stays flat at 0.38 GiB for the full 3.13 GiB. Same reader, same data, 20× less memory.

   The request size is the whole story. ZipFile's peak is O(bytes you ask for in one call), not O(stream).

   Workaround today

   Decompress in bounded blocks to a temp file, then hand CSV the path. CSV memory-maps a path source, so the file bytes become evictable page cache instead of heap:

   function unzip_to_tempfile(zipf)
       z = ZipFile.Reader(zipf); f = z.files[1]
       path, io = mktemp()
       blk = Vector{UInt8}(undef, 1 << 24)     # 16 MiB — the important part
       while f._pos < f.uncompressedsize
           n = min(length(blk), Int(f.uncompressedsize) - f._pos)
           n == length(blk) || resize!(blk, n)
           read!(f, blk)
           write(io, blk)
       end
       close(io); close(z)
       return path
   end
   
   path = unzip_to_tempfile("t3400.zip")
   rows = CSV.Rows(path; types=[Int64, String, Float64, Float64, Int8, Float64, Float64, Int64])
   for row in rows
       # ...
   end

   Measured on the same file: decompression stays flat at 0.43 GiB (vs. 8.2 GiB peak), and the parse afterwards is file-backed.

   What CSV holds during iteration, and what changes in v1.0

   Worth being upfront about this, since it is the second half of your question — and v1.0 just went out, so the answer differs by version.

   In 0.10.x CSV.Rows streams: it parses one row at a time and keeps no per-row index. What it retains is the source bytes — for a non-seekable stream like a zip member, spilled to a temp file and memory-mapped, so ~1× uncompressed as page cache.

   In v1.0 the internals are rewritten around a structural index. CSV.Rows builds that index over the whole input up front, then row iteration is pure lookup — cells parse on access. The tradeoff is that the index is retained for as long as you hold the Rows object:

   retained = uncompressed bytes + 4 × nrows × (ncols + 2) bytes
   

   The index is 4 bytes per cell plus 8 bytes per row. Verified exact at 2, 5, 8, 20 and 40 columns. For your file (53.6M rows × 8 cols) that is 40 B/row = 2.00 GiB of index, so 5.13 GiB retained against a 3.13 GiB input — 1.64×. Full v1.0 numbers on the same file:

   	bytes
   source buffer	3.13 GiB
   index (tape / rowfirst / rowstart)	2.00 GiB
   retained while iterating	5.13 GiB
   peak, via ZipFile stream	~10.2 GiB
   peak, via block-spill + path (mmap)	5.65 GiB, of which 3.13 is evictable page cache

   So on v1.0 the guidance is the same but matters more: hand it a path, not a stream. Then the bytes are a mapping the kernel can drop under pressure and only the index (~2 GiB here) is real anonymous memory.

   There is no constant-memory mode for a 3.4 GB single file in v1.0 — the index is whole-file. If you need a hard memory ceiling, splitting the input and looping over parts is still the way.

   Actions

   • Keeping this open as a CSV issue: for a non-seekable IO source, CSV should materialize in bounded blocks rather than one whole-stream read. That caps the peak regardless of how the upstream IO buffers, and it would have avoided this entirely.
   • The Zlib.read! behaviour is worth an issue on ZipFile.jl too — any consumer that reads a large member in one call pays 2×.

   ---

   [issue investigated and resolved by AI; reviewed by quinnj; please respond for discrepancies and inaccuracies]
3. quinnj

   added a commit that references this issue on Sep 15, 2026

   Avoid per-row allocations in typed CSV.Rows access (#1203)

   ...

   ff0c032
4. quinnj commented on Oct 4, 2026

   

   quinnj

   on Oct 4, 2026

   Member

   More actions

   Current main still reads a custom IO source in one whole-stream call. I reproduced the extra ZipFile buffer on Julia 1.12.6 / ZipFile 0.10.1 and opened #1213 to bound those requests.

   For a 256 MiB member, the retained inflate buffer falls from 276.4 MiB to 24.9 MiB. Across two fresh-process trials, median process peak falls from 1549.9 MiB to 1385.2 MiB. The tradeoff is extra allocation/copying during materialization: source-loading time rises from 245 ms to 298 ms in this case.

   CSV still retains the complete materialized input. This addresses the extra upstream buffer; it does not make ZIP-backed Rows constant-memory. The full suites pass on Julia 1.10.12 and 1.12.6, including 28 new source checks, and strict documentation passes. Hosted CI is being followed on #1213.

   AI disclosure: This work was prepared with assistance from OpenAI Codex.