Tighten the docs and fix inaccurate claims

README.md

@@ -32,15 +32,17 @@ CSV.write("output.csv", file)
 
 Use `CSV.read("input.csv", DataFrame)` after you load DataFrames.jl. Use
 `CSV.Rows` to process one row at a time, `CSV.Chunks` to process batches, or
-`CSV.lazy` to index a source before you choose which cells to parse.
+`CSV.lazy` to index a source before you choose which cells to parse. With
+DataDecimals.jl loaded, `types=Dict(:amount => DataDecimals.Decimal64{2})`
+reads a column as exact decimals; CSV.jl does not infer decimal types.
 
 ## Documentation
 
 - [Stable documentation](https://JuliaData.github.io/CSV.jl/stable) describes
   the latest registered release.
 - [Development documentation](https://JuliaData.github.io/CSV.jl/latest)
   describes the `main` branch.
-- [1.0 release notes](docs/src/release-notes.md) summarize what changed.
+- [Release notes](docs/src/release-notes.md)
 
 ## Support
 
@@ -62,8 +64,3 @@ changes and own the final approval.
   `load` and `save` integration.
 - [DLMReader.jl](https://github.com/sl-solution/DLMReader.jl) reads delimited
   data and integrates with InMemoryDatasets.jl.
-
-CSV.jl uses shared string columns from DataStrings.jl. With
-DataDecimals.jl loaded, an explicitly requested decimal type such as
-`types=Dict(:amount => DataDecimals.Decimal64{2})` parses exactly; CSV does not
-infer decimal types, and fractional numbers infer as `Float64`.

docs/src/examples.md

@@ -5,17 +5,11 @@ Pages = ["examples.md"]
 Depth = 2
 ```
 
-Most examples use literal data in an `IOBuffer` so the
-documentation build executes them. A `String` source is a file path or URL;
-wrap literal text in `IOBuffer`. Load `DataStrings` when you want to name the
-default text type.
+Most examples read literal text through `IOBuffer`; a `String` source is a
+file path or URL. File paths and URLs below are placeholders.
 
 ## [Start here: read, inspect, and write](@id first_example)
 
-Install CSV with `import Pkg; Pkg.add("CSV")`. Install any other package named
-in an example before using it. File paths and URLs below are placeholders;
-replace them with your own sources.
-
 ```@example ex-first
 using CSV
 
@@ -65,8 +59,8 @@ data = [
 
 # I can just pass a `Vector` of inputs, in this case `IOBuffer(::String)`, but it
 # could also be a `Vector` of any valid input source, like `AbstractVector{UInt8}`,
-# filenames, `IO`, etc. Inputs are parsed on separate tasks and concatenated
-# in order. The first input defines the output columns. Later missing columns
+# filenames, `IO`, etc. Inputs are concatenated in order. The first input
+# defines the output columns. Later missing columns
 # are filled with `missing`, and later extra columns are ignored. To
 # send the parsed columns directly to a sink function, we can use `CSV.read`, like
 # `df = CSV.read(map(IOBuffer, data), DataFrame)`.
@@ -302,8 +296,7 @@ totals: 12, 15, 18
 grand total: 45
 """
 
-# `footerskip=2` excludes the last two rows from value parsing.
-# CSV still scans the input to find the row boundaries.
+# `footerskip=2` skips the last two rows
 file = CSV.File(IOBuffer(data); footerskip=2)
 ```
 
@@ -421,9 +414,8 @@ a,b,c
 13,14,15
 """
 
-# Only the first three data rows have their values parsed and returned.
-# CSV still reads or maps the source and builds its structural index.
-# The limit is exact at every thread count.
+# `limit=3` parses and returns only the first three data rows; CSV still
+# indexes the whole input
 file = CSV.File(IOBuffer(data); limit=3)
 ```
 
@@ -536,8 +528,7 @@ file = CSV.File(IOBuffer(data); quotechar='"', escapechar='"')
 file = CSV.File(IOBuffer(data); openquotechar='"', closequotechar='"', escapechar='"')
 ```
 
-A quote inside an unquoted field (`with "inner quotes"`) is content: the
-structural scan notices it and rebuilds its index under the field-start rule.
+A quote inside an unquoted field (`with "inner quotes"`) is content.
 
 ## [DateFormat](@id dateformat_example)
 
@@ -585,7 +576,7 @@ collect(file.date)
 ```
 
 Without a `dateformat`, ISO dates, ISO date-times (with `T` or a space), and
-times are detected. Date-times prefer `Durations.Timestamp{Dates.Nanosecond}`.
+times are detected. Date-times are `Durations.Timestamp{Dates.Nanosecond}`.
 Wider dates use `Timestamp{Microsecond}` if every value fits exactly; otherwise
 the column stays text. Load `Dates` and request `types=Dict(:stamp => DateTime)`
 when a consumer needs `DateTime`. Fractions finer than whole milliseconds then
@@ -930,12 +921,10 @@ chunks = CSV.Chunks(IOBuffer("id,value\n1,10\n2,20\n3,30\n"); chunkbytes=8)
 length(chunks)
 ```
 
-`CSV.Rows` defaults to text unless you supply `types`. Both readers retain the
-source bytes; `CSV.Rows` also retains the whole structural index, while
-`CSV.Chunks` indexes one batch at a time. `CSV.Chunks` checks values across the
-input to choose one schema before iteration, so its constructor reads every row
-it will yield — `limit` excludes the rest from that check — and knows the batch
-count.
+`CSV.Rows` returns text unless you supply `types`. Both readers keep the source
+in memory; `CSV.Rows` also keeps an index of every row, while `CSV.Chunks`
+indexes one batch at a time. `CSV.Chunks` checks every value before the first
+batch to choose one schema.
 
 ## [Index first and parse later](@id lazy_example)
 

docs/src/index.md

@@ -2,8 +2,7 @@
 
 CSV.jl reads and writes comma-separated and other delimited text data. Its
 readers and writer implement the [Tables.jl](https://github.com/JuliaData/Tables.jl)
-interfaces, so CSV data can move between Julia table packages without a
-CSV-specific adapter.
+interfaces, so they work with any Julia table package.
 
 ## Installation
 
@@ -38,30 +37,27 @@ String(take!(output))
 
 ## Select a reader
 
-| API | Use it when | Materialization model |
-|:----|:------------|:----------------------|
-| `CSV.File` | You need a normal in-memory table | Index once, then parse columns |
-| `CSV.read` | You want to hand the parsed columns to a sink | Same parse as `CSV.File`; marks columns as safe to take |
-| `CSV.lazy` | You need a fast first look or sparse cell access | Index now; parse each accessed cell later |
-| `CSV.Rows` | You process rows once and do not need columns | Indexes the source; materializes cells on access |
-| `CSV.Chunks` | You process a large table in bounded batches | Indexes and parses one stable-schema batch at a time |
-
-`CSV.lazy`, `CSV.Rows`, and `CSV.Chunks` do not stream an unbounded input.
-They all retain the source bytes. `CSV.lazy` and `CSV.Rows` also retain a
-structural index of the whole row window; `CSV.Chunks` holds one batch's.
-See [Input and memory
-behavior](reading.md#Input-and-memory-behavior) for the exact source rules.
+| API | Use it when | What it does |
+|:----|:------------|:-------------|
+| `CSV.File` | You need a normal in-memory table | Parses every selected column |
+| `CSV.read` | You want the columns in another table type | Parses like `CSV.File`, then lets the sink take the columns without copying |
+| `CSV.lazy` | You need a fast first look or sparse cell access | Indexes the input; parses each cell when it is accessed |
+| `CSV.Rows` | You process rows once and do not need columns | Indexes the input; parses cells on access |
+| `CSV.Chunks` | You process a large table in batches | Parses one batch at a time; every batch has the same column types |
+
+`CSV.lazy`, `CSV.Rows`, and `CSV.Chunks` keep the whole input in memory (or
+memory-mapped) while you use them, so none of them streams an unbounded input.
+`CSV.lazy` and `CSV.Rows` also keep an index of every row; `CSV.Chunks` keeps
+one batch's. See [Input and memory
+behavior](reading.md#Input-and-memory-behavior) for details. Results do not
+depend on the number of threads.
 
 ## Data model
 
-CSV.jl builds one quote-aware structural index. It then parses each selected
-column with a type-specialized loop. Parallel execution does not change row
-order, `limit` results, or output bytes.
-
-Date-time columns prefer `Timestamp{Nanosecond}` from Durations.jl, which
-uses `Dates.Timestamp` when the standard library provides it. Wider dates use microseconds if all values
-fit exactly; otherwise the column stays text. Request `Dates.DateTime` with
-`types` when a consumer needs it.
+Date-time columns are `Timestamp{Nanosecond}` from Durations.jl (the
+`Dates.Timestamp` type where the standard library provides it). Wider dates
+use microseconds if all values fit exactly; otherwise the column stays text.
+Request `Dates.DateTime` with `types` when a consumer needs it.
 
 Text columns use `DataStrings.DataString` by default. Short values are stored in the
 value; longer values live in buffers the column owns, so an eager table never

docs/src/migration.md

@@ -16,11 +16,12 @@ must use an older Julia release.
 |:-----|:------------|:-----------|:----------|
 | Text values | InlineStrings.jl values by default | `DataStrings.DataString` by default | Pass `stringtype=String`, or load InlineStrings.jl and select its type |
 | Pooling | `(0.2, 500)` default policy | `pool=false` | Pass `pool=(0.2, 500)` to restore the old policy |
-| Empty unquoted field | Missing sentinel behavior could be disabled | Always `missing` | Use a quoted empty field for present empty text |
+| Empty fields | `,,` and `,"",` read as `missing`; `missingstring=nothing` read both as empty strings | `,,` is always `missing`; `,"",` is an empty string | Quote a field to keep an empty string |
 | Problems | One warning per problem during recovery | Structured `CSV.problems(file)` plus one summary warning | Inspect problems, set `on_error=:collect` to silence the summary, or `on_error=:error` to throw `CSV.ParseError` |
 | Row limit | Could be approximate with multiple tasks | Exact at every thread count | Remove `ntasks=1` workarounds used only for exact limits |
-| Boolean inference | Accepted the 0.10 parser's broader spellings | `true`, `True`, `TRUE`, `false`, `False`, `FALSE` | Add explicit `truestrings` and `falsestrings` as required |
-| Date-time inference | `Dates.DateTime`; extra fraction digits truncated | `Timestamp{Nanosecond}` (Durations.jl; `Dates.Timestamp` when available); wider dates use `Timestamp{Microsecond}` only if every value fits exactly, otherwise text | Pass `types=DateTime` for the old type; finer fractions are problems. With `typemap=Dict(Timestamp{Nanosecond} => DateTime)`, finer fractions promote to text |
+| `footerskip` | Counted comment rows | Does not count comment rows | Lower `footerskip` by the number of trailing comment rows |
+| Boolean values | Also inferred `T` and `F`, and read `1` and `0` for `types=Bool` | Only `true`, `True`, `TRUE`, `false`, `False`, and `FALSE` | Pass `truestrings` and `falsestrings` lists for other spellings |
+| Date-time inference | `Dates.DateTime` for `yyyy-mm-ddTHH:MM:SS` with up to three fraction digits; other date-times stayed text | `Timestamp{Nanosecond}` (Durations.jl; `Dates.Timestamp` when available) for `T` or space separators and up to nine fraction digits; wider dates use `Timestamp{Microsecond}` only if every value fits exactly, otherwise text | Pass `typemap=Dict(Timestamp{Nanosecond} => DateTime)` for `DateTime` columns; as in 0.10, a column with finer fractions stays text. With `types=DateTime`, finer fractions are problems |
 
 `DataStrings.DataString` is an `AbstractString`. Convert one value with `String(x)`
 when a consumer requires `String`. Eager text columns own their bytes, so
@@ -36,14 +37,16 @@ throws), and extra trailing zeros are exact.
 
 ## Removed, replaced, or preferred reader options
 
+A removed keyword throws an `ArgumentError` that names its replacement.
+
 | 0.10 form | 1.0 form |
 |:----------|:---------|
 | `rows_to_check=n` or `lines_to_check=n` | `nsample=n` for type sampling |
 | `threaded=false` | `ntasks=1` or `parallel=false` |
 | `tasks=n` | `ntasks=n` |
 | `lazystrings=true` | `stringtype=DataStrings.DataString`, which is the default |
 | `silencewarnings=true` | `on_error=:collect` records problems without a summary warning |
-| `maxwarnings=n` | Still accepted by problem-retaining readers; `maxproblems=n` is the explicit form |
+| `maxwarnings=n` | `maxproblems=n`; `CSV.File` and `CSV.read` still accept `maxwarnings` |
 | `strict=true` | Still accepted; `on_error=:error` is the explicit form. For `CSV.Rows`, either form throws when an invalid typed cell is accessed |
 | `type=T` | `types=T` |
 | `datarow=n` | `skipto=n` |
@@ -65,11 +68,11 @@ and `CSV.Chunks` and return a unique file-ordered column set. Use
 
 ## Table access
 
-Column and row access on `CSV.File` is unchanged. `CSV.lazy` is new: it builds
-the structural index and returns a `CSV.LazyFile` whose cells parse when they
-are accessed. It supports `lazyfile[:amount]` and indexed cell access because
-it is a column-indexed view, not an iterable row table; `CSV.File(lazyfile)`
-performs the full typed parse without repeating the structural scan.
+Column and row access on `CSV.File` is unchanged. `CSV.lazy` is new: it
+indexes the input and returns a `CSV.LazyFile` whose cells parse when they are
+accessed. It supports `lazyfile[:amount]` and `lazyfile[row, :amount]` but is
+not an iterable row table; `CSV.File(lazyfile)` parses it fully without
+indexing the input again.
 
 ## Error and schema behavior
 
@@ -86,13 +89,11 @@ rules are:
 
 The default `on_error=:warn` prints one summary warning per read (once per
 `CSV.Chunks`). Set `on_error=:collect` to record problems silently, or
-`on_error=:error` for fail-fast behavior; the latter throws `CSV.ParseError`,
-which carries the source-earliest `CSV.Problem`. Use `maxproblems` to cap
-retained problem objects.
+`on_error=:error` to throw a `CSV.ParseError` that carries the first problem in
+source order. Use `maxproblems` to cap retained problem objects.
 
 `types=Char` and `types=Symbol` remain supported. A `Char` cell is exactly one
-Unicode scalar; other text follows the normal missing-plus-problem policy. A
-`Symbol` column is parsed as text and converted once after parsing.
+Unicode scalar; other text follows the normal missing-plus-problem policy.
 
 ## Source and memory behavior
 
@@ -101,32 +102,32 @@ files, `IO` and `Cmd` sources, URLs, and gzip-decompressed data. This differs
 from 0.10 paths that could use temporary files for these inputs. Account for
 the full uncompressed size when a gzip file or non-file stream is large.
 
-`CSV.Rows` and `CSV.lazy` retain the source bytes and a complete structural
-index. `CSV.Chunks` retains the source bytes and one batch's index. All three
-reduce value or column materialization. None of them provides unbounded
-network-stream processing.
+`CSV.Rows` no longer streams with minimal memory: 0.10 buffered only the
+current row, while 1.0 keeps the source and an index of every row, about 4
+bytes per field plus 8 bytes per row. `CSV.lazy` keeps the same index.
+`CSV.Chunks` keeps one batch's index, so use it when an index of the whole
+file would not fit in memory.
 
 ## Multiple sources
 
-A vector of sources now promotes compatible column types across inputs and
-uses deterministic labels for non-path sources. The first source defines the
-output columns. Later missing columns are filled with `missing`; later extra
-columns are ignored. Text columns concatenate as `DataStrings.DataString`
-columns, as in a single-source read; an explicitly requested string type such
-as `types=String` or `stringtype=String15` applies to the concatenated column.
+The first source defines the output columns, and later sources match them by
+name. A column missing from a later source is filled with `missing` (0.10
+threw an error); extra columns are ignored, as before. Column types combine
+with `promote_type`. Non-path sources get labels such as `"<source 1>"`. Text
+columns concatenate as `DataStrings.DataString` columns, as in a single-source
+read; an explicitly requested string type such as `types=String` or
+`stringtype=String15` applies to the concatenated column.
 
 `source=:origin` also works with a one-element source vector. A source-label
 name that conflicts with a data column is an error.
 
 ## Tables.Scan
 
-`Tables.Scan` is new. A scan is a single request object that describes which
-columns to return (optionally renamed or converted), a row filter, and an
-offset and limit. `CSV.File(source; scan=request)` and `CSV.read` apply the
-whole request inside the parser: unselected columns are never sampled or
-parsed, rows the filter rejects never parse their other columns, and the
-requested output types convert last, so a filtered-out value cannot raise a
-conversion problem. Filters run on the parsed native values.
+`Tables.Scan` is new. A scan describes the columns to return (optionally
+renamed or converted), a row filter, and an offset and limit.
+`CSV.File(source; scan=request)` and `CSV.read` apply it inside the parser, so
+a filtered read no longer parses the whole table. See
+[Tables.Scan pushdown](@ref).
 
 ```julia
 using CSV, Tables
@@ -143,9 +144,7 @@ request = Tables.Scan(
 page = CSV.File("orders.csv"; scan=request)
 ```
 
-A scan owns selection, types, and row bounds, so it is not combined with
-`select`, `drop`, `types`, or `limit`. In 0.10 the same work needed a full
-read followed by a filter; the scan reads less and returns the same table.
+A scan cannot be combined with `select`, `drop`, `types`, or `limit`.
 
 ## Writer changes and compatibility
 
@@ -169,8 +168,8 @@ writing ambiguous data. Unknown writer keywords are an `ArgumentError`.
 `CSV.write(sink, CSV.Chunks(source))` streams every batch under one header.
 
 An empty string is quoted. `missing` uses `missingstring` and is unquoted by
-default. This guarantees a read/write distinction between present empty text
-and a missing value.
+default, so empty text and `missing` read back differently; 0.10 wrote both
+as an empty field.
 
 `floatformat` is new and accepts a Printf-style format. Writer output is
 deterministic across `ntasks` values.