Skip to content

fix(storage): skip chunks on a cast column only when the cast keeps its stats - #1284

Open
zahariash wants to merge 1 commit into
LadybugDB:mainfrom
zahariash:fix/zone-map-cast-predicates
Open

zahariash wants to merge 1 commit into
LadybugDB:mainfrom
zahariash:fix/zone-map-cast-predicates

Conversation

@zahariash

Copy link
Copy Markdown
Contributor

This work was produced with the help of language models.

Fixes #1262.

With zone maps on (the default), MATCH (n:T) WHERE n.i > 30.5 on an INT64 column returned no rows, in memory and after CHECKPOINT. CAST(n.f AS INT64) = 31 on a DOUBLE column, CAST(n.d AS INT64) = 5 on DECIMAL(18,2) and a DATE or TIMESTAMP_MS column compared with a timestamp(...) literal were empty after CHECKPOINT. The binder casts the column to the literal's type, and the zone-map check read the chunk's min/max as that type: integer stats as doubles, DECIMAL stats with their scale (5.00 as 500), DATE days as microseconds.

Fix. A cast column is used for chunk skipping only when the cast keeps every value and its order in the same stats slot: the same type, integer widening up to 64 bits (unsigned only into a strictly wider signed type) and FLOAT to DOUBLE. 128-bit targets are left out because a negative 64-bit value is stored without sign extension. Other casts stay in the filter and lose chunk skipping, which for them only ever dropped rows. An integer literal that fits the column's type is cast to that type instead, so n.i32 = 40 involves no column cast. The DuckDB and Postgres scans share this conversion and no longer send CAST(f AS INT64) = 31 to the remote database as f = 31; that path has no test here. No storage format change.

Tests. filter/cast.test: ZoneMapCastedColumnNumericLiteral runs every query in memory and after CHECKPOINT. The issue's queries and the DATE, TIMESTAMP_MS and INT128-literal (n.i < 9223372036854775808, negative INT64) cases return no rows on main. CAST(n.i32 AS INT64) with =, <> and a negative value, n.u32 > -1 and n.u32 < 5000000000 on a UINT32 holding 4000000000, and FLOAT against a double literal pass on main and pin the allowed casts. Full suite: no failures; the storage-compatibility suite matches main.

…dicates

A comparison of a column with a literal of another numeric type is bound
as CAST(column) <op> literal, and the zone-map push-down turned it into a
predicate on the column carrying the literal's value. The chunk's min/max
were then read as the literal's type: integer stats as doubles, double
stats as integers, DECIMAL stats with their scale, DATE days as TIMESTAMP
micros. Chunks with matching rows were skipped, so `n.i > 30.5` on an
INT64 column returned nothing.

Push a casted column down only when the cast keeps each value and its
order in the same stats slot: integer widening (including unsigned to a
wider signed type) and FLOAT to DOUBLE. Other casts stay in the filter.
The DuckDB and Postgres scans share the conversion, so they no longer
send a narrowing cast to the remote side as a comparison on the raw
column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: zone maps drop rows for a property compared with a literal of another numeric type

1 participant