From 9fa236fba7981009f6e41e8739f05ef44413ba30 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 15 Oct 2026 19:11:05 -0600 Subject: [PATCH 1/2] Tighten the manual and docstrings Fix claims that did not match the code, and cut internals and change narration from the reading, writing, index, examples, migration, and release-notes pages and the public docstrings. Corrections, each checked with a probe on main and on CSV 0.10.16: - migration: 0.10 left date-times with more than three fraction digits as text instead of truncating them; 0.10 read `""` as missing; 0.10 inferred T/F as Bool and read 1/0 for types=Bool; 0.10 counted comment rows in footerskip; 0.10 Rows buffered only the current row; CSV.Chunks rejects maxwarnings; 0.10 already promoted Int and Float across sources. - writing: a partition base path "part.csv" writes "part.csv_1", ... - LazyFile: text cells follow stringtype, not always DataString. - 1.0 release notes: Rows typed access allocated until 1.1.0, and 0.10 did not truncate fractions. - The removed `debug` keyword message said "removed in 1.0" twice and both debug messages pointed at a non-public index; point at CSV.problems. Co-Authored-By: Claude Opus 5.5 --- README.md | 11 +-- docs/src/examples.md | 37 +++----- docs/src/index.md | 42 ++++----- docs/src/migration.md | 71 +++++++------- docs/src/reading.md | 193 ++++++++++++++------------------------ docs/src/release-notes.md | 42 +++------ docs/src/writing.md | 81 +++++++--------- src/CSV.jl | 105 ++++++++++++--------- src/api.jl | 4 +- 9 files changed, 254 insertions(+), 332 deletions(-) diff --git a/README.md b/README.md index d95111e5..f1d4b8cc 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,9 @@ CSV.write("output.csv", file) Use `CSV.read("input.csv", DataFrame)` after you load DataFrames.jl. Use `CSV.Rows` to process one row at a time, `CSV.Chunks` to process batches, or -`CSV.lazy` to index a source before you choose which cells to parse. +`CSV.lazy` to index a source before you choose which cells to parse. With +DataDecimals.jl loaded, `types=Dict(:amount => DataDecimals.Decimal64{2})` +reads a column as exact decimals; CSV.jl does not infer decimal types. ## Documentation @@ -40,7 +42,7 @@ Use `CSV.read("input.csv", DataFrame)` after you load DataFrames.jl. Use the latest registered release. - [Development documentation](https://JuliaData.github.io/CSV.jl/latest) describes the `main` branch. -- [1.0 release notes](docs/src/release-notes.md) summarize what changed. +- [Release notes](docs/src/release-notes.md) ## Support @@ -62,8 +64,3 @@ changes and own the final approval. `load` and `save` integration. - [DLMReader.jl](https://github.com/sl-solution/DLMReader.jl) reads delimited data and integrates with InMemoryDatasets.jl. - -CSV.jl uses shared string columns from DataStrings.jl. With -DataDecimals.jl loaded, an explicitly requested decimal type such as -`types=Dict(:amount => DataDecimals.Decimal64{2})` parses exactly; CSV does not -infer decimal types, and fractional numbers infer as `Float64`. diff --git a/docs/src/examples.md b/docs/src/examples.md index 74123111..0c95b70e 100644 --- a/docs/src/examples.md +++ b/docs/src/examples.md @@ -5,17 +5,11 @@ Pages = ["examples.md"] Depth = 2 ``` -Most examples use literal data in an `IOBuffer` so the -documentation build executes them. A `String` source is a file path or URL; -wrap literal text in `IOBuffer`. Load `DataStrings` when you want to name the -default text type. +Most examples read literal text through `IOBuffer`; a `String` source is a +file path or URL. File paths and URLs below are placeholders. ## [Start here: read, inspect, and write](@id first_example) -Install CSV with `import Pkg; Pkg.add("CSV")`. Install any other package named -in an example before using it. File paths and URLs below are placeholders; -replace them with your own sources. - ```@example ex-first using CSV @@ -65,8 +59,8 @@ data = [ # I can just pass a `Vector` of inputs, in this case `IOBuffer(::String)`, but it # could also be a `Vector` of any valid input source, like `AbstractVector{UInt8}`, -# filenames, `IO`, etc. Inputs are parsed on separate tasks and concatenated -# in order. The first input defines the output columns. Later missing columns +# filenames, `IO`, etc. Inputs are concatenated in order. The first input +# defines the output columns. Later missing columns # are filled with `missing`, and later extra columns are ignored. To # send the parsed columns directly to a sink function, we can use `CSV.read`, like # `df = CSV.read(map(IOBuffer, data), DataFrame)`. @@ -302,8 +296,7 @@ totals: 12, 15, 18 grand total: 45 """ -# `footerskip=2` excludes the last two rows from value parsing. -# CSV still scans the input to find the row boundaries. +# `footerskip=2` skips the last two rows file = CSV.File(IOBuffer(data); footerskip=2) ``` @@ -421,9 +414,8 @@ a,b,c 13,14,15 """ -# Only the first three data rows have their values parsed and returned. -# CSV still reads or maps the source and builds its structural index. -# The limit is exact at every thread count. +# `limit=3` parses and returns only the first three data rows; CSV still +# indexes the whole input file = CSV.File(IOBuffer(data); limit=3) ``` @@ -536,8 +528,7 @@ file = CSV.File(IOBuffer(data); quotechar='"', escapechar='"') file = CSV.File(IOBuffer(data); openquotechar='"', closequotechar='"', escapechar='"') ``` -A quote inside an unquoted field (`with "inner quotes"`) is content: the -structural scan notices it and rebuilds its index under the field-start rule. +A quote inside an unquoted field (`with "inner quotes"`) is content. ## [DateFormat](@id dateformat_example) @@ -585,7 +576,7 @@ collect(file.date) ``` Without a `dateformat`, ISO dates, ISO date-times (with `T` or a space), and -times are detected. Date-times prefer `Durations.Timestamp{Dates.Nanosecond}`. +times are detected. Date-times are `Durations.Timestamp{Dates.Nanosecond}`. Wider dates use `Timestamp{Microsecond}` if every value fits exactly; otherwise the column stays text. Load `Dates` and request `types=Dict(:stamp => DateTime)` when a consumer needs `DateTime`. Fractions finer than whole milliseconds then @@ -930,12 +921,10 @@ chunks = CSV.Chunks(IOBuffer("id,value\n1,10\n2,20\n3,30\n"); chunkbytes=8) length(chunks) ``` -`CSV.Rows` defaults to text unless you supply `types`. Both readers retain the -source bytes; `CSV.Rows` also retains the whole structural index, while -`CSV.Chunks` indexes one batch at a time. `CSV.Chunks` checks values across the -input to choose one schema before iteration, so its constructor reads every row -it will yield — `limit` excludes the rest from that check — and knows the batch -count. +`CSV.Rows` returns text unless you supply `types`. Both readers keep the source +in memory; `CSV.Rows` also keeps an index of every row, while `CSV.Chunks` +indexes one batch at a time. `CSV.Chunks` checks every value before the first +batch to choose one schema. ## [Index first and parse later](@id lazy_example) diff --git a/docs/src/index.md b/docs/src/index.md index a613bd8f..24dca5d2 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -2,8 +2,7 @@ CSV.jl reads and writes comma-separated and other delimited text data. Its readers and writer implement the [Tables.jl](https://github.com/JuliaData/Tables.jl) -interfaces, so CSV data can move between Julia table packages without a -CSV-specific adapter. +interfaces, so they work with any Julia table package. ## Installation @@ -38,30 +37,27 @@ String(take!(output)) ## Select a reader -| API | Use it when | Materialization model | -|:----|:------------|:----------------------| -| `CSV.File` | You need a normal in-memory table | Index once, then parse columns | -| `CSV.read` | You want to hand the parsed columns to a sink | Same parse as `CSV.File`; marks columns as safe to take | -| `CSV.lazy` | You need a fast first look or sparse cell access | Index now; parse each accessed cell later | -| `CSV.Rows` | You process rows once and do not need columns | Indexes the source; materializes cells on access | -| `CSV.Chunks` | You process a large table in bounded batches | Indexes and parses one stable-schema batch at a time | - -`CSV.lazy`, `CSV.Rows`, and `CSV.Chunks` do not stream an unbounded input. -They all retain the source bytes. `CSV.lazy` and `CSV.Rows` also retain a -structural index of the whole row window; `CSV.Chunks` holds one batch's. -See [Input and memory -behavior](reading.md#Input-and-memory-behavior) for the exact source rules. +| API | Use it when | What it does | +|:----|:------------|:-------------| +| `CSV.File` | You need a normal in-memory table | Parses every selected column | +| `CSV.read` | You want the columns in another table type | Parses like `CSV.File`, then lets the sink take the columns without copying | +| `CSV.lazy` | You need a fast first look or sparse cell access | Indexes the input; parses each cell when it is accessed | +| `CSV.Rows` | You process rows once and do not need columns | Indexes the input; parses cells on access | +| `CSV.Chunks` | You process a large table in batches | Parses one batch at a time; every batch has the same column types | + +`CSV.lazy`, `CSV.Rows`, and `CSV.Chunks` keep the whole input in memory (or +memory-mapped) while you use them, so none of them streams an unbounded input. +`CSV.lazy` and `CSV.Rows` also keep an index of every row; `CSV.Chunks` keeps +one batch's. See [Input and memory +behavior](reading.md#Input-and-memory-behavior) for details. Results do not +depend on the number of threads. ## Data model -CSV.jl builds one quote-aware structural index. It then parses each selected -column with a type-specialized loop. Parallel execution does not change row -order, `limit` results, or output bytes. - -Date-time columns prefer `Timestamp{Nanosecond}` from Durations.jl, which -uses `Dates.Timestamp` when the standard library provides it. Wider dates use microseconds if all values -fit exactly; otherwise the column stays text. Request `Dates.DateTime` with -`types` when a consumer needs it. +Date-time columns are `Timestamp{Nanosecond}` from Durations.jl (the +`Dates.Timestamp` type where the standard library provides it). Wider dates +use microseconds if all values fit exactly; otherwise the column stays text. +Request `Dates.DateTime` with `types` when a consumer needs it. Text columns use `DataStrings.DataString` by default. Short values are stored in the value; longer values live in buffers the column owns, so an eager table never diff --git a/docs/src/migration.md b/docs/src/migration.md index 92d38885..ff913835 100644 --- a/docs/src/migration.md +++ b/docs/src/migration.md @@ -16,11 +16,12 @@ must use an older Julia release. |:-----|:------------|:-----------|:----------| | Text values | InlineStrings.jl values by default | `DataStrings.DataString` by default | Pass `stringtype=String`, or load InlineStrings.jl and select its type | | Pooling | `(0.2, 500)` default policy | `pool=false` | Pass `pool=(0.2, 500)` to restore the old policy | -| Empty unquoted field | Missing sentinel behavior could be disabled | Always `missing` | Use a quoted empty field for present empty text | +| Empty fields | `,,` and `,"",` read as `missing`; `missingstring=nothing` read both as empty strings | `,,` is always `missing`; `,"",` is an empty string | Quote a field to keep an empty string | | Problems | One warning per problem during recovery | Structured `CSV.problems(file)` plus one summary warning | Inspect problems, set `on_error=:collect` to silence the summary, or `on_error=:error` to throw `CSV.ParseError` | | Row limit | Could be approximate with multiple tasks | Exact at every thread count | Remove `ntasks=1` workarounds used only for exact limits | -| Boolean inference | Accepted the 0.10 parser's broader spellings | `true`, `True`, `TRUE`, `false`, `False`, `FALSE` | Add explicit `truestrings` and `falsestrings` as required | -| Date-time inference | `Dates.DateTime`; extra fraction digits truncated | `Timestamp{Nanosecond}` (Durations.jl; `Dates.Timestamp` when available); wider dates use `Timestamp{Microsecond}` only if every value fits exactly, otherwise text | Pass `types=DateTime` for the old type; finer fractions are problems. With `typemap=Dict(Timestamp{Nanosecond} => DateTime)`, finer fractions promote to text | +| `footerskip` | Counted comment rows | Does not count comment rows | Lower `footerskip` by the number of trailing comment rows | +| Boolean values | Also inferred `T` and `F`, and read `1` and `0` for `types=Bool` | Only `true`, `True`, `TRUE`, `false`, `False`, and `FALSE` | Pass `truestrings` and `falsestrings` lists for other spellings | +| Date-time inference | `Dates.DateTime` for `yyyy-mm-ddTHH:MM:SS` with up to three fraction digits; other date-times stayed text | `Timestamp{Nanosecond}` (Durations.jl; `Dates.Timestamp` when available) for `T` or space separators and up to nine fraction digits; wider dates use `Timestamp{Microsecond}` only if every value fits exactly, otherwise text | Pass `typemap=Dict(Timestamp{Nanosecond} => DateTime)` for `DateTime` columns; as in 0.10, a column with finer fractions stays text. With `types=DateTime`, finer fractions are problems | `DataStrings.DataString` is an `AbstractString`. Convert one value with `String(x)` when a consumer requires `String`. Eager text columns own their bytes, so @@ -36,6 +37,8 @@ throws), and extra trailing zeros are exact. ## Removed, replaced, or preferred reader options +A removed keyword throws an `ArgumentError` that names its replacement. + | 0.10 form | 1.0 form | |:----------|:---------| | `rows_to_check=n` or `lines_to_check=n` | `nsample=n` for type sampling | @@ -43,7 +46,7 @@ throws), and extra trailing zeros are exact. | `tasks=n` | `ntasks=n` | | `lazystrings=true` | `stringtype=DataStrings.DataString`, which is the default | | `silencewarnings=true` | `on_error=:collect` records problems without a summary warning | -| `maxwarnings=n` | Still accepted by problem-retaining readers; `maxproblems=n` is the explicit form | +| `maxwarnings=n` | `maxproblems=n`; `CSV.File` and `CSV.read` still accept `maxwarnings` | | `strict=true` | Still accepted; `on_error=:error` is the explicit form. For `CSV.Rows`, either form throws when an invalid typed cell is accessed | | `type=T` | `types=T` | | `datarow=n` | `skipto=n` | @@ -65,11 +68,11 @@ and `CSV.Chunks` and return a unique file-ordered column set. Use ## Table access -Column and row access on `CSV.File` is unchanged. `CSV.lazy` is new: it builds -the structural index and returns a `CSV.LazyFile` whose cells parse when they -are accessed. It supports `lazyfile[:amount]` and indexed cell access because -it is a column-indexed view, not an iterable row table; `CSV.File(lazyfile)` -performs the full typed parse without repeating the structural scan. +Column and row access on `CSV.File` is unchanged. `CSV.lazy` is new: it +indexes the input and returns a `CSV.LazyFile` whose cells parse when they are +accessed. It supports `lazyfile[:amount]` and `lazyfile[row, :amount]` but is +not an iterable row table; `CSV.File(lazyfile)` parses it fully without +indexing the input again. ## Error and schema behavior @@ -86,13 +89,11 @@ rules are: The default `on_error=:warn` prints one summary warning per read (once per `CSV.Chunks`). Set `on_error=:collect` to record problems silently, or -`on_error=:error` for fail-fast behavior; the latter throws `CSV.ParseError`, -which carries the source-earliest `CSV.Problem`. Use `maxproblems` to cap -retained problem objects. +`on_error=:error` to throw a `CSV.ParseError` that carries the first problem in +source order. Use `maxproblems` to cap retained problem objects. `types=Char` and `types=Symbol` remain supported. A `Char` cell is exactly one -Unicode scalar; other text follows the normal missing-plus-problem policy. A -`Symbol` column is parsed as text and converted once after parsing. +Unicode scalar; other text follows the normal missing-plus-problem policy. ## Source and memory behavior @@ -101,32 +102,32 @@ files, `IO` and `Cmd` sources, URLs, and gzip-decompressed data. This differs from 0.10 paths that could use temporary files for these inputs. Account for the full uncompressed size when a gzip file or non-file stream is large. -`CSV.Rows` and `CSV.lazy` retain the source bytes and a complete structural -index. `CSV.Chunks` retains the source bytes and one batch's index. All three -reduce value or column materialization. None of them provides unbounded -network-stream processing. +`CSV.Rows` no longer streams with minimal memory: 0.10 buffered only the +current row, while 1.0 keeps the source and an index of every row, about 4 +bytes per field plus 8 bytes per row. `CSV.lazy` keeps the same index. +`CSV.Chunks` keeps one batch's index, so use it when an index of the whole +file would not fit in memory. ## Multiple sources -A vector of sources now promotes compatible column types across inputs and -uses deterministic labels for non-path sources. The first source defines the -output columns. Later missing columns are filled with `missing`; later extra -columns are ignored. Text columns concatenate as `DataStrings.DataString` -columns, as in a single-source read; an explicitly requested string type such -as `types=String` or `stringtype=String15` applies to the concatenated column. +The first source defines the output columns, and later sources match them by +name. A column missing from a later source is filled with `missing` (0.10 +threw an error); extra columns are ignored, as before. Column types combine +with `promote_type`. Non-path sources get labels such as `""`. Text +columns concatenate as `DataStrings.DataString` columns, as in a single-source +read; an explicitly requested string type such as `types=String` or +`stringtype=String15` applies to the concatenated column. `source=:origin` also works with a one-element source vector. A source-label name that conflicts with a data column is an error. ## Tables.Scan -`Tables.Scan` is new. A scan is a single request object that describes which -columns to return (optionally renamed or converted), a row filter, and an -offset and limit. `CSV.File(source; scan=request)` and `CSV.read` apply the -whole request inside the parser: unselected columns are never sampled or -parsed, rows the filter rejects never parse their other columns, and the -requested output types convert last, so a filtered-out value cannot raise a -conversion problem. Filters run on the parsed native values. +`Tables.Scan` is new. A scan describes the columns to return (optionally +renamed or converted), a row filter, and an offset and limit. +`CSV.File(source; scan=request)` and `CSV.read` apply it inside the parser, so +a filtered read no longer parses the whole table. See +[Tables.Scan pushdown](@ref). ```julia using CSV, Tables @@ -143,9 +144,7 @@ request = Tables.Scan( page = CSV.File("orders.csv"; scan=request) ``` -A scan owns selection, types, and row bounds, so it is not combined with -`select`, `drop`, `types`, or `limit`. In 0.10 the same work needed a full -read followed by a filter; the scan reads less and returns the same table. +A scan cannot be combined with `select`, `drop`, `types`, or `limit`. ## Writer changes and compatibility @@ -169,8 +168,8 @@ writing ambiguous data. Unknown writer keywords are an `ArgumentError`. `CSV.write(sink, CSV.Chunks(source))` streams every batch under one header. An empty string is quoted. `missing` uses `missingstring` and is unquoted by -default. This guarantees a read/write distinction between present empty text -and a missing value. +default, so empty text and `missing` read back differently; 0.10 wrote both +as an empty field. `floatformat` is new and accepts a Printf-style format. Writer output is deterministic across `ntasks` values. diff --git a/docs/src/reading.md b/docs/src/reading.md index b48451b0..9c17dea1 100644 --- a/docs/src/reading.md +++ b/docs/src/reading.md @@ -39,17 +39,11 @@ from its magic bytes for every source type and is fully decompressed into memory before parsing. Other compression formats must be decompressed before they are passed to CSV.jl. -`CSV.File`, `CSV.read`, and `CSV.Chunks` return columns that own their bytes: -text longer than a `DataStrings.DataString` inline payload is copied into -column-owned buffers during parsing, so a finished table never refers to the -source. Finished worker tasks release their input references, so garbage -collection can unmap the file. Rewriting the file afterwards cannot affect -the table. One value retains at most the parse chunk's text buffer, and a -`Vector{UInt8}` input is never aliased. A `CSV.Chunks` iterator retains its -source for later batches, but only one batch's structural index; the returned -batches own their bytes. `CSV.Rows` and `CSV.lazy` -are the exceptions: their cells are views into the retained source, so keep -the source unchanged while you use them, or convert values with `String`. +Tables from `CSV.File` and `CSV.read`, and `CSV.Chunks` batches, own their +text: they never refer to the source, so rewriting or deleting the file +afterwards does not change them, and a `Vector{UInt8}` source is never +aliased. `CSV.Rows` and `CSV.lazy` cells are views into the source; keep the +source unchanged while you use them, or convert values with `String`. ### Gzipped text in other encodings @@ -79,10 +73,8 @@ rm(dir; recursive=true) # hide String.(file.city) ``` -Use the file's actual encoding; it is not detected automatically. The decoder -supplies UTF-8 bytes to CSV.jl. Closing it releases conversion resources and -checks for an incomplete final character. CSV.jl still reads the converted -input into memory; this is not a streaming table parser. +CSV.jl does not detect encodings; name the file's actual encoding. Closing the +decoder checks for an incomplete final character. ## Headers and row windows @@ -97,9 +89,9 @@ Duplicate names are made unique. `normalizenames=true` converts names to valid Julia identifiers. Use `skipto` for the first data row, `footerskip` to omit rows at the end, and -`limit` to cap the returned data rows. `limit` is exact at every thread count. -Comment rows do not count toward `footerskip`. Empty rows and comment rows keep -their physical positions for header and `skipto` handling. +`limit` to cap the returned data rows. `header` and `skipto` count every +physical row, including empty and comment rows. `footerskip` counts empty rows +but not comment rows. ```@example reading-window using CSV @@ -130,13 +122,10 @@ The main dialect options are: Input must be ASCII or UTF-8. Convert other encodings before parsing. -A quote that does not start its field is content: -`10,Pipe 3" long` and `1,x"y` are ordinary two-field rows. The parallel -structural scan treats every quote as a field boundary; when it meets a quote -that does not start its field, CSV.jl rebuilds the index with a serial pass -that applies the field-start rule. Well-formed input never takes that path. A -quote that does start a field and is never closed is still malformed input and -is reported as an `:unclosed_quote` problem. +A quote that does not start its field is content: `10,Pipe 3" long` and +`1,x"y` are ordinary two-field rows. Such a quote makes CSV.jl index the input +a second time with a slower scanner. A quote that starts a field and is never +closed is reported as an `:unclosed_quote` problem. ## Missing values and value syntax @@ -250,24 +239,19 @@ accepted forms are: - `(ratio, maximum_levels)` such as `pool=(0.2, 500)`; or - a dictionary or vector of per-column policies for `CSV.File`. -The ratio-and-cap policy is available as `pool=(0.2, 500)`. Pooled output -uses PooledArrays.jl. Pool levels own their strings even when -`stringtype=DataStrings.DataString`. +Only text columns are pooled, into PooledArrays.jl vectors. -With `transpose=true`, input rows become output columns. `types`, per-output -`dateformat` dictionaries, `stringtype`, and `pool` use those output column -names. `select` and `drop` are not supported in transpose mode. -Transpose mode indexes the input and parses its output columns in parallel, -like the row-wise readers; `ntasks` and `parallel` apply to it as well. +With `transpose=true`, input rows become output columns, and `types`, +`dateformat`, and `pool` dictionaries use the output column names. `select`, +`drop`, and `scan` are not supported with `transpose=true`. ## Parse problems -Every eager reader keeps rows and records malformed quotes, invalid typed -values, long rows, and other parse problems as `CSV.Problem` values. Inspect -the retained items with `CSV.problems(file)`. Each item contains `row`, `col`, -`pos`, `kind`, and `message` fields. The default `on_error=:warn` also prints -one summary warning per read, so a problem is never silent; `on_error=:collect` -records problems without the warning. +`CSV.File` and `CSV.Chunks` batches record malformed quotes, invalid typed +values, and short or long rows as `CSV.Problem` values instead of failing the +read. `CSV.problems(file)` returns them; each has `row`, `col`, `pos`, `kind`, +and `message` fields. The default `on_error=:warn` also prints one summary +warning per read; `on_error=:collect` records problems without the warning. ```@example reading-problems using CSV @@ -278,33 +262,28 @@ file = CSV.File(IOBuffer("count\n1\ninvalid\n"); types=Int, on_error=:collect) `maxproblems` caps retained diagnostics and defaults to 10,000. `CSV.Chunks` warns once, for its first batch with problems; every batch still -carries its own `CSV.problems(batch)`. `on_error=:error`, or the compatibility -shorthand `strict=true`, throws a `CSV.ParseError` at the first problem in -source order; -the exception carries that `problem`, the total `nproblems`, and the `source` -label. `validate=false` ignores `types`, `dateformat`, and `pool` dictionary -keys that do not match an input column; validation is on by default. +carries its own `CSV.problems(batch)`. `on_error=:error` (or `strict=true`) +throws a `CSV.ParseError` carrying the first problem in source order, the +total `nproblems`, and the `source` label. `validate=false` ignores `types`, +`dateformat`, and `pool` dictionary keys that do not match an input column; +validation is on by default. ## Parallel parsing and sampling -Parsing is parallel by default when Julia has multiple threads. Use -`ntasks=1` or `parallel=false` for one task. `ntasks=N` bounds parsing to at -most `N` worker tasks. A direct `CSV.File` targets about four structural -chunks per task, each between 64 KiB and 1 MiB, unless `chunkbytes` is -explicit. `CSV.File(lazyfile)` -keeps the existing index geometry but applies the same worker bound. -`chunkbytes` directly controls the target structural-index chunk size. Keep -it at the default unless a measurement says otherwise: each parsing pass -walks one chunk at a time, so a chunk that fits in cache keeps the value -loops fast, and a `chunkbytes` of several MiB or more slows wide files with -many columns. - -Type inference samples indexed rows. `nsample` controls its row sample and -`samplebytes` controls the delimiter-detection sample. Parallel and -single-task parses have the same row order and exact row limit. -`fastindex=false` builds the structural index with the byte-at-a-time -reference scanner instead of the 64-byte vector scanner. Use it to check -that the two scanners produce the same result. +Parsing is parallel by default when Julia has multiple threads. `ntasks=N` +uses at most `N` tasks; `ntasks=1` or `parallel=false` uses one. Results do not +depend on the number of tasks. + +The input is split into chunks of about `chunkbytes` bytes: by default about +four per task, each between 64 KiB and 1 MiB. Keep the default unless a +measurement says otherwise: a chunk that fits in cache keeps parsing fast, and +chunks of several MiB slow down files with many columns. `CSV.File(lazyfile)` +reuses the lazy file's chunks. + +Type inference starts from a sample of `nsample` rows and widens a column's +type when a later value needs it. `samplebytes` sets how much input delimiter +detection reads. `fastindex=false` indexes with a slower byte-at-a-time +scanner, to check that it agrees with the default vector scanner. ## Multiple sources @@ -319,10 +298,10 @@ file = CSV.File(sources; source=:origin => ["first", "second"], stringtype=Strin (collect(file.id), collect(file.value), collect(file.origin)) ``` -The first source defines the output column set. Later sources match columns by -name. A missing column is filled with `missing`; an extra column is ignored. -Types promote across sources. Text columns concatenate as `DataString` columns -that own their bytes, as in a single-source read. +The first source defines the output columns. Later sources match columns by +name: a missing column is filled with `missing`, and an extra column is +ignored. Column types combine with `promote_type`, so `Int64` and `Float64` +give `Float64`. `source=:origin` adds a pooled source-label column. Path sources use their paths. Other sources use deterministic labels such as `""`. Use @@ -341,14 +320,10 @@ df = CSV.read("input.csv", DataFrame) ## Lazy indexed access -`CSV.lazy` builds the structural index but does not eagerly type and parse all -columns. Its `CSV.LazyFile` result supports `lazyfile.column`, -`lazyfile[:column]`, `lazyfile[row, column]`, and the Tables.jl column -interface. - -Ordinary lazy text cells are zero-copy views into the retained source. If a -long cell starts beyond the compact view format's Int32 source-offset limit, -CSV copies only that cell into a bounded backing buffer when it is accessed. +`CSV.lazy` indexes the input but parses a cell only when it is accessed. Its +`CSV.LazyFile` result supports `lazyfile.column`, `lazyfile[:column]`, +`lazyfile[row, column]`, and the Tables.jl column interface. It does not infer +types: a cell is text unless `types` names its column's type. ```@example reading-lazy using CSV @@ -377,10 +352,8 @@ rows = CSV.Rows(IOBuffer("id,value\n1,10\n2,20\n"); types=[Int, Int]) [row[:value] for row in rows] ``` -`reusebuffer` is accepted but has no effect. The -row view does not allocate a reusable per-row buffer. Because the structural -index is complete before iteration, `length(rows)` and `names(rows)` are -available, and consumers such as `Tables.columntable` can preallocate. +`length(rows)` and `names(rows)` are known before iteration. `reusebuffer` is +accepted and ignored. The row type carries the column names and types, so a loop over rows that a function receives as an argument reads `row.value` as a typed value with no @@ -396,48 +369,28 @@ then costs a dynamic dispatch. An invalid or malformed cell becomes `missing` when it is accessed. Pass `strict=true` or `on_error=:error` to throw a `CSV.ParseError` at that access -instead. `CSV.Rows` does not retain a problem log and does not accept -`maxproblems` or `maxwarnings`. It rejects `on_error=:warn` because it cannot -produce a read summary. Use `CSV.File` when you need `CSV.problems(file)` -or a summary warning. -List `select` and `drop` forms are supported and use the same stable file-order -semantics as `CSV.File`. +instead. `CSV.Rows` keeps no problem log, so it rejects `on_error=:warn`, +`maxproblems`, and `maxwarnings`; use `CSV.File` for `CSV.problems`. `select` +and `drop` work as in `CSV.File`. ## Batch iteration -`CSV.Chunks` yields Tables.jl-compatible batches and implements -`Tables.partitions`. It infers one schema for the full row window, so each -batch has the same column types. Pooling is evaluated separately in each -batch. - -One batch is one window of the source's data bytes. `chunkbytes` is the target -size of that window and defaults to 64 MiB, or the whole source when it is -smaller. A window ends at a row boundary, so a batch holds every row that starts -within its bytes and one complete row can push it past `chunkbytes`. `ntasks` -bounds the parallel work inside one window; it is not a batch count and does not -affect the batch size. A `CSV.Chunks` pool policy must be one `Bool`, ratio, or -`(ratio, maximum_levels)` value; per-column pool dictionaries and vectors are -only supported by `CSV.File`. -List `select` and `drop` forms project the same file-ordered column set in every -batch. - -Each window is indexed when its batch is produced and released with it, so the -live structural index is one window's, not the whole source's. Only that index -is bounded. The source bytes stay in memory for later batches: an `IO` input, a -byte buffer, and `buffer_in_memory=true` still hold the whole input, and a large -local file stays memory-mapped for the life of the iterator. `CSV.File`, -`CSV.lazy` and `CSV.Rows` index the whole row window at once. Each returned -batch owns its own bytes, so retaining batches retains their columns. - -The constructor validates every row it will yield, before the first batch, to -settle one schema. Value validation excludes the rows after `limit`: the window -that holds the limit is indexed to its end, and the rows past the limit are -dropped before any value is read, so they cannot settle a type. The pass reads -no later window. It starts over when a finding invalidates what it settled: a -quote that did not start its field prepares the source again under the lenient -rule, and a later timestamp that needs microseconds reseeds the column types. -The final pass walks exactly the windows iteration walks, so it also counts -them: `length(chunks)` is the batch count and costs no extra pass. +`CSV.Chunks` yields batches, each a `CSV.File`, and implements +`Tables.partitions`. Every batch has the same column types; pooling is decided +per batch. + +A batch covers about `chunkbytes` bytes of the input (64 MiB by default) and +ends at a row boundary, so one long row can make it larger. `ntasks` sets the +parallelism within a batch, not the number of batches. `pool` takes one policy +for every column, and `select` and `drop` apply to every batch. + +Each batch is indexed when it is produced, and the index is released with it. +The source stays in memory, or memory-mapped, until the iterator is released. +Each batch owns its columns. + +To settle one schema, the constructor checks every value it will return before +the first batch; rows after `limit` are not checked. `length(chunks)` is the +number of batches. ## Tables.Scan pushdown @@ -461,5 +414,5 @@ request = Tables.Scan( file = CSV.File("orders.csv"; scan=request) ``` -A scan owns selection, types, and row bounds. Do not combine `scan` with -`select`, `drop`, `types`, or `limit`. Tables 1.14 is the minimum dependency. +A scan owns selection, types, and row bounds, so `scan` cannot be combined +with `select`, `drop`, `types`, or `limit`. diff --git a/docs/src/release-notes.md b/docs/src/release-notes.md index c83bb5f0..bf584df3 100644 --- a/docs/src/release-notes.md +++ b/docs/src/release-notes.md @@ -16,32 +16,25 @@ that cannot start on a file larger than memory. boundary, so one complete row can push a batch past `chunkbytes`. - `ntasks` bounds the parallel work inside one window. It is not a batch count and no longer affects the batch size. -- The constructor still validates every row it will yield, before the first - batch, to settle one schema: every batch has the same column types, - missingness, and settled text width, as in 1.0. That pass stops where `limit` - stops, and its final run walks the same windows iteration walks, so it also - counts them. `length(chunks)` is the batch count and costs no extra pass. -- Only the structural index is bounded. The source bytes stay in memory for - later batches, and each returned batch owns its own bytes. +- The constructor still checks every row it will return before the first + batch, so every batch has the same column types, as in 1.0. + `length(chunks)` is the batch count. +- Only the index is bounded. The source bytes stay in memory for later + batches, and each returned batch owns its own bytes. Batch boundaries therefore differ from 1.0 for the same options. The rows, column types, values, and diagnostics do not. ## Fixes -- Generic ARM64 package precompilation works on Julia 1.13. The quote scanner - retains its accelerated PMULL path behind a runtime CPU check, with a - shift/XOR fallback. CI checks generic package images and JIT compilation on - ARM64 and x86-64 with Julia 1.10 and current Julia. +- Generic ARM64 package precompilation works on Julia 1.13. - Typed `CSV.Rows` column access avoids per-row allocations. - Floating-point output is quoted when its text conflicts with the delimiter or quote characters, including decimal separators, exponents, and `NaN`/`Inf`. -- A quote that does not start its field makes a reader prepare the source again - under the lenient quote rule. That retry now keeps the whole syntax the first - pass resolved, including a sniffed `ignorerepeated`. 1.0 kept only the - delimiter, so an aligned-column source such as `"a b\n1 2\n3 x\"y\n"` gained - an empty column between `a` and `b`; it now reads as the two columns an - explicit `delim=' ', ignorerepeated=true` always gave. +- A space-aligned file with a quote inside a field, such as + `"a b\n1 2\n3 x\"y\n"`, reads as two columns, as with an explicit + `delim=' ', ignorerepeated=true`. 1.0 added an empty column between `a` and + `b`. # CSV.jl 1.0 release notes @@ -58,8 +51,7 @@ CSV.jl entry points. It requires Julia 1.10 or later. - Inferred text uses `DataStrings.DataString` by default. Short text is stored in the value; longer text lives in column-owned buffers, so eager tables never refer to the source or to a mapped file. -- Typed value parsing uses the low-level parsers of Parsers, now even faster in - version 3. +- Typed values parse with the Parsers.jl 3 kernels. - Parse recovery produces structured `CSV.problems(file)` data. `on_error` selects one summary warning (`:warn`, the default), silent collection (`:collect`), or fail-fast `CSV.ParseError` (`:error`). @@ -71,13 +63,7 @@ CSV.jl entry points. It requires Julia 1.10 or later. detection, parallel gzip compression (one member per row block), and parallel partition output. - String conversion (`stringtype=String`, `types=String`, InlineStrings), - `CSV.Chunks` batch parsing, and gzip output run in parallel. Files with - `comment` rows, `skipto`, and `footerskip` use the vector scanner, and so do - files with a separate `escapechar` or distinct `openquotechar` and - `closequotechar`. `CSV.Rows` - carries its schema in its type, so `row.name` on a typed column returns a - typed value without allocation. Transposed reads parse each row as a typed - column. Several sources concatenate text as `DataString` columns. + `CSV.Chunks` batch parsing, and gzip output run in parallel. ## Compatibility changes @@ -90,8 +76,8 @@ The most important default changes are: - `true`, `True`, `TRUE`, `false`, `False`, and `FALSE` are the default Boolean spellings; - a date-time column is a `Timestamp{Nanosecond}` (Durations.jl) instead of a - `Dates.DateTime`, ISO date-times accept `T` or a space, and no fraction digit - is truncated; and + `Dates.DateTime`, ISO date-times accept `T` or a space, and fractions keep + up to nine digits; and - parse problems are retained as problem objects, with one summary warning per read instead of one warning per problem. diff --git a/docs/src/writing.md b/docs/src/writing.md index a5a863ed..fa077a55 100644 --- a/docs/src/writing.md +++ b/docs/src/writing.md @@ -1,9 +1,7 @@ # Writing data `CSV.write(sink, table; keywords...)` writes any Tables.jl table to a file path -or `IO`. It normally returns the sink. A partitioned string base path returns -the generated string path vector. A FilePathsBase path returns the original -path object. +or `IO` and returns the sink. ```@example writing using CSV @@ -14,26 +12,23 @@ CSV.write(output, table) String(take!(output)) ``` -Column-access tables render rows in parallel when the table is large enough. -Output is ordered and byte-identical for every `ntasks` value. Row-access -sources stream sequentially without being collected. A custom `transform` -also runs sequentially to preserve its row-major callback order. The writer -emits bounded row blocks, so temporary rendering memory does not grow with the -total output size. +Column tables render in parallel on up to `ntasks` tasks (`Threads.nthreads()` +by default); the output is the same for every `ntasks` value. Row tables are +written sequentially without being collected. +The writer works in bounded blocks of rows, so its memory use does not grow +with the output size. ## Headers, appending, and sinks The default writes the Tables.jl column names as the first row. Use -`writeheader=false` to omit them, or pass `header=[...]` to replace them. -The replacement header must contain one name for each source column. -`header=true` and `header=false` remain compatibility forms for controlling -header output. +`writeheader=false` to omit them, or pass `header=[...]` with one name per +column to replace them. `header=true` and `header=false` also turn the header +on and off. `append=true` opens a path for append instead of replacement. It omits the header unless `writeheader=true` is explicit. An `IO` sink is written at its -current position, exactly like `Base.write`: content already in the stream, -such as log lines on `stdout` or a prefix the caller wrote, is never rewound -over or truncated. For an `IO`, `append` only decides the header default. +current position, like `Base.write`; for an `IO`, `append` only decides +whether the header is written. `CSV.write(sink, CSV.Chunks(source))` streams every batch of a chunked read under one header without collecting the batches. @@ -64,10 +59,9 @@ The writer dialect options are: `:minimal` quotes text that contains a delimiter, quote, carriage return, newline, or leading or trailing space. It also quotes an empty string. -`:all` quotes every text field. `:none` rejects text that contains a structural -byte, and it rejects an empty string because that spelling would be ambiguous -with `missing`. It never writes data that the configured dialect would parse -incorrectly. +`:all` quotes every text field. `:none` throws an `ArgumentError` for text that +contains a structural byte and for an empty string, which would read back as +`missing`. Quote-related characters and `decimal` must be single ASCII characters, given as a `Char` or a one-character string. A multi-byte delimiter is written @@ -86,7 +80,7 @@ String(take!(output)) `missingstring` controls the text written for `missing` and defaults to an unquoted empty field. A present empty string is always quoted when quoting is -enabled. This preserves the 1.0 read rule: +enabled. This matches how CSV.jl reads empty fields: - an unquoted empty field is `missing`; and - a quoted empty field is a present empty string. @@ -107,6 +101,10 @@ output, configure the reader to use `decimal=','` as well. `dateformat` accepts a Dates.jl format for all `Date`, `DateTime`, `Timestamp`, and `Time` values. Other values use their ordinary text representation. +DataDecimals values keep their decimal spelling, and `decimal` applies to them +too: `decimal=','` writes `1.20` as `1,20`, quoted when the delimiter is also a +comma. + ```@example writing-values using CSV, Dates @@ -119,14 +117,13 @@ String(take!(output)) ## Gzip output `compress=:auto` writes gzip when a path ends in `.gz` (in any letter case). -Use `compress=:gzip` -to force gzip for any sink, or `compress=:none` to disable it. The compatibility -forms `compress=true` and `compress=false` mean `:gzip` and `:none`. -Compression is streamed to the sink. With more than one task, each row block -compresses in its own task and is written as one gzip member, so the file is a -multi-member gzip stream (RFC 1952). Readers such as `gunzip` and `CSV.File` -concatenate the members. A consumer that reads only one gzip member needs -output written with `ntasks=1`. +Use `compress=:gzip` (or `true`) to force gzip for any sink, or +`compress=:none` (or `false`) to disable it. Compression is streamed to the +sink. With more than one task, a column table's row blocks +compress in parallel, one gzip member each, so the file is a multi-member gzip +stream (RFC 1952). Readers such as `gunzip` and `CSV.File` concatenate the +members. A consumer that reads only one gzip member needs output written with +`ntasks=1`. ```julia CSV.write("output.csv.gz", table) # detected from the suffix @@ -136,24 +133,18 @@ CSV.write(io, table; compress=:gzip) # explicit for an IO ## Partitioned output `partition=true` writes each value from `Tables.partitions(table)` to a -separate sink. A vector of sinks must contain one sink per partition. A base -path generates paths by appending `_1`, `_2`, and so on. Partitions write -concurrently. The partition iterator is consumed once and is not collected. -At most `ntasks` partition writes are active at one time. +separate sink, at most `ntasks` at a time. A vector of sinks must contain one +sink per partition. A base path generates paths by appending `_1`, `_2`, and +so on, and `CSV.write` returns them. ```julia -sinks = ["part-1.csv", "part-2.csv"] -CSV.write(sinks, partitioned_table; partition=true) -generated = CSV.write("part.csv", partitioned_table; partition=true) +CSV.write(["part-1.csv", "part-2.csv"], partitioned_table; partition=true) +CSV.write("part.csv", partitioned_table; partition=true) # ["part.csv_1", "part.csv_2"] ``` -The string-base form returns the generated path vector. The explicit-vector -form returns the supplied vector. - -## Compatibility controls +## Other options -`quotestrings=true` remains an alias for `quotestyle=:all`. New code should use -`quotestyle` because it also expresses `:minimal` and `:none`. +`quotestrings=true` is an alias for `quotestyle=:all`. `transform=(column, value) -> new_value` transforms each cell before writing. The callback uses a 1-based column index and runs in row-major order. A custom @@ -180,8 +171,4 @@ join(rows) ``` Rows render on demand. `join(CSV.RowWriter(table))` is byte-identical to -`CSV.write(io, table)` with the same row-writer options. - -DataDecimals values retain their decimal spelling. The `decimal` option applies -to these values as well as floating-point values. For example, `decimal=','` -writes `1.20` as `1,20` and quotes the field when the delimiter is also a comma. +`CSV.write(io, table)` with the same options. diff --git a/src/CSV.jl b/src/CSV.jl index 0882bbb2..c9762c02 100644 --- a/src/CSV.jl +++ b/src/CSV.jl @@ -7,9 +7,8 @@ Reading — `CSV.File`, `CSV.read`, `CSV.lazy`, `CSV.Rows`, `CSV.Chunks`. Writing — `CSV.write`, `CSV.RowWriter`. Diagnostics — `CSV.problems`, `CSV.Problem`, `CSV.ParseError`. -All public names live under the `CSV` namespace. Reading supports eager, -lazy, row-wise, and chunked workflows. Parsing and writing are deterministic -for any supported thread count. +All public names live under the `CSV` namespace. Results do not depend on the +number of threads. """ module CSV @@ -58,10 +57,10 @@ only that cell into a bounded backing buffer. CSV.LazyFile The indexed table that [`CSV.lazy`](@ref CSV.lazy) returns. It holds the -source bytes and the structural index, and no parsed values. `names(lf)`, -`size(lf)`, and `Tables.columnnames` come from the index. `lf.name` and -`lf[:name]` return a column view whose cells parse when read; `lf.name[i]` -parses one cell. Every cell is `DataString` or `missing` unless `types` +source bytes and the index, and no parsed values. `names(lf)`, `size(lf)`, and +`Tables.columnnames` come from the index. `lf.name` and `lf[:name]` return a +column view whose cells parse when read; `lf.name[i]` parses one cell. A cell +is text (`stringtype`, `DataString` by default) or `missing` unless `types` requested a type for its column. A view keeps the source alive, so a `LazyFile` is a way to look before parsing, not a way to hold a large file cheaply. [`CSV.File`](@ref CSV.File)`(lf)` runs the eager typed parse on the @@ -70,11 +69,10 @@ existing index and keeps the columns `lf` selected. @doc """ CSV.Rows(source; types=nothing, stringtype=DataStrings.DataString, keywords...) -Iterate lightweight Tables.jl row views without allocating eager columns. -Cells materialize on access. The source bytes and complete structural index -remain in memory, so `length(rows)` and `names(rows)` are known before -iteration. `reusebuffer` is accepted but is inert because the row view has -no per-row value buffer. Invalid or malformed cells +Iterate Tables.jl row views; each cell parses when it is accessed. The source +and an index of every row stay in memory, so `length(rows)` and `names(rows)` +are known before iteration. `reusebuffer` is accepted and ignored. Invalid or +malformed cells become `missing` by default; `strict=true` or `on_error=:error` throws a [`CSV.ParseError`](@ref CSV.ParseError) when the cell is accessed. Rows do not retain parse diagnostics, so use [`CSV.File`](@ref CSV.File) when @@ -84,11 +82,9 @@ name, or a `Regex`) project columns in stable file order. @doc """ CSV.Chunks(source; ntasks=Threads.nthreads(), chunkbytes=nothing, keywords...) -Iterate a source as stable-schema [`CSV.File`](@ref CSV.File) batches and -provide the Tables.jl partitions interface. Every batch has the same column -types, including one settled width for an auto-width string request such as -`stringtype=InlineString`. Pooling is evaluated per batch. List `select` -and `drop` forms project every batch in stable file order. With the default +Iterate a source as [`CSV.File`](@ref CSV.File) batches that all have the +same column types, and serve them through `Tables.partitions`. Pooling is +decided per batch; `select` and `drop` apply to every batch. With the default `on_error=:warn`, the first batch with parse problems prints one summary warning; every batch keeps its own [`CSV.problems`](@ref CSV.problems). @@ -114,44 +110,63 @@ batches and `length` is defined. CSV.read(source, sink; keywords...) Parse with the same options as [`CSV.File`](@ref CSV.File), then call the -Tables.jl `sink`. The new columns are passed as `Tables.CopiedColumns`, so a -sink that honors that marker can take ownership without another copy. +Tables.jl `sink`, as in `CSV.read("data.csv", DataFrame)`. The new columns are +passed as `Tables.CopiedColumns`, so a sink that honors that marker can take +ownership without another copy. """ read @doc """ CSV.problems(file) -> Vector{CSV.Problem} -Return the retained parse problems for a `CSV.File`, in source order. The -parser can retain at most `maxproblems` entries; the file display reports any -additional dropped count. The default `on_error=:warn` also prints one summary -warning per read; use `on_error=:collect` to record problems silently, or -`strict=true` / `on_error=:error` to stop at the first parse problem with a -[`CSV.ParseError`](@ref CSV.ParseError). +Return the parse problems a `CSV.File` retained, in source order: at most +`maxproblems` of them (10,000 by default). Showing the file reports how many +were recorded in total. Pass `on_error=:collect` to a reader to record problems +without the summary warning, or `on_error=:error` to throw a +[`CSV.ParseError`](@ref CSV.ParseError) instead. """ problems @doc """ - CSV.write(sink, table; keywords...) - -Write any Tables.jl table as delimited text to a path or `IO`. The writer -supports header control (`writeheader`, `header`), `append` mode, gzip -(`compress`), partitioned sinks (`partition`), quote styles (`quotestyle`, -`quotestrings`), dialect bytes (`delim` of any length, `quotechar`, -`openquotechar`/`closequotechar`, `escapechar`, `newline`, `decimal`, -`missingstring`, `bom`), number and date formats (`floatformat`, -`dateformat`), cell transforms (`transform`), a rendered-row size bound -(`bufsize`), task count (`ntasks`), and deterministic ordered output. -Column-access tables can render row blocks in parallel. Row-access sources -and [`CSV.Chunks`](@ref CSV.Chunks) stream without being collected. An `IO` -sink is written at its current position and never rewound or truncated. A -partitioned string base path returns the generated path vector; other forms -return the supplied sink. `CSV.write(sink; keywords...)` returns a function -of the table, for `table |> CSV.write(path)`. + CSV.write(sink, table; keywords...) -> sink + +Write any Tables.jl `table` as delimited text to a path or `IO` and return +`sink`. An `IO` is written at its current position. Column tables render in +parallel; row tables and [`CSV.Chunks`](@ref CSV.Chunks) stream without being +collected. The output is the same for every `ntasks` value. +`CSV.write(sink; keywords...)` returns a function of the table, for +`table |> CSV.write(path)`. + +# Keywords + +- `delim=','`: a character or a string such as `"::"`. +- `quotechar='"'`, `openquotechar`, `closequotechar`, and `escapechar` (the + closing quote by default). +- `quotestyle=:minimal`: quote text only when it needs quotes; `:all` quotes + all text, and `:none` throws for text that needs quotes. `quotestrings=true` + means `:all`. +- `newline='\\n'`, `missingstring=""`, and `bom=false`. +- `writeheader=true` (`false` when appending) and `header`, a vector of + replacement names. +- `append=false`: append to a path instead of replacing it. +- `compress=:auto`: gzip a path ending in `.gz`; `:gzip` or `:none` force it + on or off. +- `floatformat`: a Printf format such as `"%.2f"`; `decimal='.'`. +- `dateformat`: a Dates format for `Date`, `DateTime`, `Timestamp`, and `Time` + values. +- `transform`: a `(col, val) -> val` function applied to each cell in + row-major order. +- `partition=false`: write each of `Tables.partitions(table)` to its own sink, + from a vector of sinks or a base path that gets `_1`, `_2`, ... appended (the + generated paths are returned). +- `ntasks=Threads.nthreads()`: the maximum number of tasks. +- `bufsize=4 MiB`: the largest allowed row, in bytes. + +See the [Writing data](@ref) guide for details. """ write @doc """ CSV.RowWriter(table; keywords...) -Iterate complete CSV-formatted row strings. The header is first unless it is -disabled. Rows render on demand with the same dialect and value formatting as -[`CSV.write`](@ref CSV.write). With the same formatting options, joining the -iterator gives the same uncompressed bytes as `CSV.write`. +Iterate the rows of `table` as CSV-formatted strings, the header first unless +`writeheader=false`. It takes the dialect and formatting keywords of +[`CSV.write`](@ref CSV.write); joining its rows gives the text `CSV.write` +writes with the same keywords. """ RowWriter # Julia 1.11 added `public`. Build the expression at runtime so Julia 1.10 can # still parse this file. The surface remains deliberately unexported: users diff --git a/src/api.jl b/src/api.jl index b402e170..85b9be94 100644 --- a/src/api.jl +++ b/src/api.jl @@ -52,7 +52,7 @@ const _REMOVED_KW = Dict{Symbol, String}( :silencewarnings => "use on_error=:collect to silence warnings; problems(f) returns retained problems", :inferdecimal => "CSV does not infer decimal types; request one with types, for example " * "types=Dict(:amount => DataDecimals.Decimal64{2})", - :debug => "removed in 1.0; parse problems and the structural index are inspectable directly", + :debug => "inspect parse problems with CSV.problems(file)", :lazystrings => "use stringtype=DataStrings.DataString (the default) or stringtype=String", :tasks => "use ntasks", :threaded => "use ntasks (ntasks=1 disables threading)", @@ -63,7 +63,7 @@ const _REMOVED_KW = Dict{Symbol, String}( :type => "use types (a single Type applies to every column)", :missingstrings => "pass a string or vector to missingstring", :dateformats => "pass per-column formats to dateformat", - :parsingdebug => "parse problems and the structural index are inspectable directly", + :parsingdebug => "inspect parse problems with CSV.problems(file)", ) function _checkkwargs(context::AbstractString, kw, allowed) From 7ec0b5113152e23882317b9fb513d148b99fab83 Mon Sep 17 00:00:00 2001 From: Jacob Quinn Date: Thu, 15 Oct 2026 19:14:41 -0600 Subject: [PATCH 2/2] Drop scanner jargon from the lazy-file paragraph Co-Authored-By: Claude Opus 5.5 --- docs/src/reading.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/reading.md b/docs/src/reading.md index 9c17dea1..bb391b64 100644 --- a/docs/src/reading.md +++ b/docs/src/reading.md @@ -333,9 +333,9 @@ lazyfile = CSV.lazy(IOBuffer("id,price\n1,3.5\n2,4.0\n"); (String(lazyfile.id[2]), lazyfile[1, :price]) ``` -`CSV.File(lazyfile)` performs the full parse without repeating the structural -scan. Row-position, dialect, and input-buffer options were fixed when the lazy -file was created. Selection is sticky: an eager parse starts from the columns +`CSV.File(lazyfile)` performs the full parse without indexing the input again. +Row-position, dialect, and input-buffer options were fixed when the lazy file +was created. Selection is sticky: an eager parse starts from the columns visible on the lazy file and can only select or drop further columns. ## Row iteration