Skip to content

Check a quoted field for inner quotes with one vector compare - #1223

Open
quinnj wants to merge 1 commit into
mainfrom
claude/quoted-field-fast-path
Open

quinnj wants to merge 1 commit into
mainfrom
claude/quoted-field-fast-path

Conversation

@quinnj

@quinnj quinnj commented Oct 22, 2026

Copy link
Copy Markdown
Member

Every quoted cell's content is found by findcontent, which scans the field for its close quote; on an all-quoted file that was about 30% of a single-threaded read. The common field ends with its close quote and holds no other one. For up to 63 content bytes, one 64-byte compare (the structural scanner's byte_mask_vec) now proves that; any other field (longer, near the end of the buffer, or with an inner quote) keeps the byte loop, so results are the same by construction. Only the default doubled-quote rule takes the check.

findcontent per field (Apple M3 Max, Julia 1.12.6, interleaved runs):

field main this PR
"alpha, bravo; charlie delta" 11.9-13.3 ns 6.2 ns
"1234 echo St, Apt 56" 11.1 ns 6.2 ns
"1234" 8.2 ns 6.2 ns
"8123.45" 5.3 ns 6.2 ns
"say ""hi""" (escaped) 10.7 ns 11.9 ns

On NEON the mask extraction costs about 6 ns, so very short and escaped fields lose up to 1 ns; x86 extracts the mask in one instruction.

quoted.csv (113 MB, 1.5M rows, 5 quoted columns with embedded delimiters, newlines, and doubled quotes), interleaved process pairs: 1 task CPU 0.40-0.44 s vs 0.46-0.49 s on main (about 10% less); 8 tasks wall 0.073 / 0.079 / 0.094 s vs 0.082 / 0.096 / 0.121 s. An unquoted text file was unchanged within noise.

Verification:

  • New testset quoted field content around the 64-byte check compares findcontent with a plain reference loop for content lengths 0-130 (around the 63/64 limit), 0-65 bytes of buffer after the field, and lone or doubled inner quotes at the first and last content byte (270 cases).
  • 2,000,000 random fields (quotes, delimiters, newlines, NUL and 0xff bytes, any position relative to the buffer end) matched main's loop exactly.
  • 200 seeds each of a ground-truth structural fuzzer and a cross-configuration consistency fuzzer: no differences.
  • Full suite --check-bounds=yes -t4: 332,048 passed.

🤖 Generated with Claude Code

Every quoted cell's content is found by scanning it for the close quote.
The common field ends with its close quote and holds no other one; for
up to 63 content bytes, one 64-byte compare (the scanner's byte mask)
now proves that, and every other field keeps the byte loop. Results are
the same by construction (2M random fields matched main's loop).

findcontent: 20-28-byte clean fields 11-13 -> 6.2 ns; escaped fields
about 1 ns slower. quoted.csv (1.5M rows, all fields quoted): about 10%
less CPU single-threaded, 15-20% less wall at 8 tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (2f84f42) to head (1e6635f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1223   +/-   ##
=======================================
  Coverage   95.79%   95.79%           
=======================================
  Files           9        9           
  Lines        5566     5570    +4     
=======================================
+ Hits         5332     5336    +4     
  Misses        234      234           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant