Skip to content

Re-parse a stale chunk under the column's current type - #1222

Open
quinnj wants to merge 1 commit into
mainfrom
claude/stale-promotion-typemap
Open

quinnj wants to merge 1 commit into
mainfrom
claude/stale-promotion-typemap

Conversation

@quinnj

@quinnj quinnj commented Oct 22, 2026

Copy link
Copy Markdown
Member

A parallel read could give a different column type than a serial read when typemap maps to a type outside the promotion order. Example: a sparse column that the sample sees as all-missing, with typemap=Dict(Int64 => Int32) (or the documented Timestamp{Nanosecond} => DateTime). Serially it reads Int32; in parallel it read as text in most runs:

40,000 rows, 30 int columns + 1 sparse column, nsample=1 main this PR
Int64 => Int32, 2 tasks: runs giving text (of 40) 29 0
Int64 => Int32, 4 tasks 28 0
Timestamp{Nanosecond} => DateTime, 2 tasks 35 0
Timestamp{Nanosecond} => DateTime, 4 tasks 30 0

Root cause: one chunk promotes the column (Missing → Int64, mapped to Int32) while another is still parsing it as Missing. The second chunk finds a value, detects Int64 from the type it parsed under, and joins that with the column's current type: promote_kernel(Int32, Int64) is outside the order, so the column became String. That conflict was found under the old type and says nothing against Int32. _promotemapped, which both the direct and the masked (Scan) driver call, now takes the type the chunk parsed under; when it is not the current type, the chunk re-parses under the current type. Retries stay bounded: each stale retry follows another chunk's promotion, and a column promotes at most a few times (the existing 8-attempt guard).

Found by an independent review of #1221, which makes this window wider.

Verification:

  • New assertions in the gzip, typemap, ... testset: the table above's shape with both maps, one serial read and eight parallel reads that must match it in type and values. At 2 threads (CI's setting) main fails 14 of 18; this branch passes 18 of 18.
  • Full suite --check-bounds=yes -t4: 331,796 passed.

Not changed here: with a typemap, a column mapped to Int32 that later meets a float becomes text, while a sample that saw the float gives Float64 (the inferred type then depends on the sample). That needs promotions to join on the unmapped type and is a separate decision.

🤖 Generated with Claude Code

When one chunk promoted a column while another was still parsing it under
an older type, the second chunk joined its detected type with the
column's current type. With a typemap target outside the promotion order
(Int64 => Int32, or the documented Timestamp{Nanosecond} => DateTime)
that join is String, so a sparse column read Int32 serially and text in
most parallel reads (29 of 40 at 2 threads, 28 of 40 at 4). The conflict
was found under the old type and says nothing against the current one,
so the chunk now re-parses under the current type. Both drivers route
through _promotemapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (2f84f42) to head (049b2b2).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1222   +/-   ##
=======================================
  Coverage   95.79%   95.79%           
=======================================
  Files           9        9           
  Lines        5566     5569    +3     
=======================================
+ Hits         5332     5335    +3     
  Misses        234      234           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant