PERF: read_csv emits pyarrow-backed string arrays directly from the C parser

[jbrockmendel]

Summary

For string columns under the default future.infer_string or dtype_backend="pyarrow", the C parser previously built an ndarray[object] of Python strs and downstream ArrowStringArray._from_sequence ran a per-chunk isna scan plus a second pass to construct the pyarrow buffers. This PR adds a new _string_pyarrow_utf8 cdef in parsers.pyx that fills pyarrow's offsets / data / validity buffers directly from the tokenizer and returns a pyarrow-backed ExtensionArray, skipping both the intermediate object array and the downstream scan.

• Targets pa.large_string() wrapped in ArrowStringArray for the default (StringDtype(na_value=np.nan)) path.
• Targets pa.string() wrapped in ArrowExtensionArray for dtype_backend="pyarrow".
• Gated on encoding_errors="strict" (other modes keep the object path for codec fidelity).
• parse_dates columns are left as object ndarrays so to_datetime can take its fast tslibs path.

Benchmarks

1M rows, GC-controlled, best of 6, pd.read_csv("file.csv") default:

Workload	main	this PR	Speedup
10 string cols, 1000-string pool	810 ms	415 ms	1.95x
5 string cols, all unique	1144 ms	236 ms	4.85x
50k rows x 500 string cols	3270 ms	1074 ms	3.04x
1 string col x 1M rows	81 ms	47 ms	1.72x

Peak RSS (sampled during the read, baseline-subtracted):

Workload	main	this PR	Reduction
1M x 10 dedup	403 MB	198 MB	-51%
1M x 5 all-unique	560 MB	110 MB	-80%
50k x 500	998 MB	441 MB	-56%

The savings scale with non-repeating strings because the old path allocated one PyUnicode per unique value; the new path stores bytes directly in pyarrow's data buffer with no Python-object tax.

Adversarial sweep confirmed no regressions after a fix for parse_dates (the initial version routed noconvert columns through the fast path, which made to_datetime fall back to the Series.map path — a ~13% regression; now noconvert stays on the object path). Bypassed cases (dtype=object, encoding_errors!="strict", explicit ExtensionDtype) correctly no-op and tie main.

Notes / design choices

• pa.string() (int32 offsets) for dtype_backend="pyarrow" matches the prior convention. Chunks exceeding 2 GiB raise OverflowError — unreachable in practice with default chunk sizes, but worth flagging.
• pa.large_string() (int64 offsets) for the default path to match StringDtype's pyarrow storage exactly (avoids a downstream cast).
• Direct TextReader callers now get ArrowStringArray under future.infer_string. Three tests in test_textreader.py were updated to normalize via np.asarray before comparison.

Test plan

• Full pandas/tests/io/parser/ suite passes (4406 passed, 174 skipped, 230 xfailed)
• pre-commit run clean
• Benchmarks above reproduced
• Adversarial sweep (short strings, long strings, heavy dedup, wide/shallow, usecols, na_filter=False, iterator/chunksize, parse_dates, explicit dtype=object, encoding_errors="replace", 100% NA strings)
• CI green

🤖 Generated with Claude Code