Skip to content

Fix writer: exact type mapping, unsigned/64-bit/decimal numbers, time zones - #58

Merged
Nathan-D-R merged 3 commits into
mainfrom
fix/writer-types
Oct 25, 2026
Merged

Nathan-D-R merged 3 commits into
mainfrom
fix/writer-types

Conversation

@Nathan-D-R

@Nathan-D-R Nathan-D-R commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Write-side silent failures. Stacked on #57.

Findings fixed (tests in openqvd-arrow/tests/write_types.rs, run for Arrow 59 and 60; test_write_types.py)

  • S1-3 non-microsecond timestamps, Time32/Time64 and Duration became text typed TIMESTAMP/TIME. The catch-all cast-to-string is gone. Timestamp (s/ms/us/ns), Time32/64, Duration, Date32 and Date64 (whole days) have exact mappings; every other type is UnsupportedArrowType (column and Arrow type named). Dictionary arrays are unpacked to their values; Utf8View is accepted. Tests: nanosecond_timestamps_are_timestamps_not_text, time_and_duration_are_times, second_timestamps_and_overflow, unmapped_types_are_errors_not_text, date64_must_be_whole_days, dictionaries_are_unpacked.
  • S1-5 UInt32 wrapped; Int64 beyond 32 bits and UInt64 became text; Decimal became text. Integers beyond 32 bits are a dual of (exact float, exact decimal text), errors beyond 2^53; Decimal128/256 are a dual with the exact decimal text, errors when the unscaled value exceeds 2^53 or scale exceeds 22. The wrong "saturating" doc comment is replaced by the full mapping table. Tests: unsigned_integers_are_not_wrapped, large_int64_stays_numeric, integers_beyond_2_pow_53_are_errors, decimals_keep_their_digits, decimals_too_wide_for_a_float_are_errors.
  • S1-12 time zones silently dropped. TimezoneNotSupported by default; lenient check timezone_to_utc writes UTC wall-clock time and attaches a Warning (WriteTable::warnings(), Python QvdWarning). Test: time_zone_is_an_error_unless_allowed, test_timezone_is_an_error_unless_allowed.
  • S1-13 unchecked u32 casts for row count, symbol-table size, distinct values: now LayoutOverflow. (A table above 4 Gi rows cannot be built in a test; the checks are u32::try_from at the three sites.)
  • Single-rounding timestamp serials, so write then read is exact for whole seconds (whole_second_timestamps_survive_a_round_trip_exactly, 5,000 values).

Not reproduced / deferred

  • S1-13 has no executable test (needs more than 2^32 rows or 4 GiB of symbols).
  • Nanosecond values are rounded to the nearest microsecond: a Qlik serial is an f64 day count and resolves about 0.6 us, so finer values cannot be stored. Documented in the mapping table.

API breaks

record_batch(es)_to_write_table returns tables carrying warnings(); new error variants and a new LenientCheck::TimezoneToUtc/WarningKind::TimezoneToUtc. Python write now returns None but emits warnings (the private _write returns them).

Tests changed

api.rs::leniency_is_a_set_of_named_checks and test_errors.py::test_leniency_basics no longer assume there is exactly one check.

Corpus

round_trip (read, re-encode with the writer, read, compare every cell) over /mnt/nas/data/QVD: 1,145 files, 1,072 ok, 15 over the size limit, 49 not QVD, 9 fail. The 9 are the three damaged fixtures plus the six NonNumericTemporal files from #57; no writer-related failures.

Lenient mode

Added: timezone_to_utc (write side; no corpus relevance, but tz-aware timestamps are common in Polars/pandas data and strict-only would block them with no workaround except converting first, which the error message explains).

Docs-site edits needed

Arrow-to-QVD type mapping table (from the record_batches_to_write_table doc), timezone handling, Python write warnings.

Review fixes

  • S7: nanosecond remainders are ValueNotRepresentable; new lenient check sub_microsecond_precision rounds and warns once per column (nanosecond_digits_are_an_error_unless_rounding_is_allowed).
  • S8: checked Date32/Date64 arithmetic; dates and timestamps limited to years 0001 to 9999 and checked against the reader's own conversion (openqvd::serial) so accepted values re-read strictly (dates_and_timestamps_the_reader_would_reject_are_not_written).
  • S9: DuplicateColumn in WriteTable::new and when opening a file (duplicate_column_names_are_rejected, duplicate_field_names_are_an_error).
  • N10: round-trip type drift is documented in the mapping table; Duration columns use the INTERVAL format.

@Nathan-D-R
Nathan-D-R force-pushed the fix/temporal-conversion branch from 8bed303 to be678a7 Compare October 25, 2026 13:16
@Nathan-D-R
Nathan-D-R changed the base branch from fix/temporal-conversion to main October 25, 2026 14:09
@Nathan-D-R
Nathan-D-R merged commit b34c056 into main Oct 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant