Repository navigation
Fix writer: exact type mapping, unsigned/64-bit/decimal numbers, time zones - #58
Merged
Merged
Conversation
…hecked date/timestamp range, duplicate columns rejected, INTERVAL for durations
Nathan-D-R
force-pushed
the
fix/temporal-conversion
branch
from
October 25, 2026 13:16
8bed303 to
be678a7
Compare
Nathan-D-R
force-pushed
the
fix/writer-types
branch
from
October 25, 2026 13:16
f7f5582 to
b8e613a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Write-side silent failures. Stacked on #57.
Findings fixed (tests in
openqvd-arrow/tests/write_types.rs, run for Arrow 59 and 60;test_write_types.py)UnsupportedArrowType(column and Arrow type named). Dictionary arrays are unpacked to their values; Utf8View is accepted. Tests:nanosecond_timestamps_are_timestamps_not_text,time_and_duration_are_times,second_timestamps_and_overflow,unmapped_types_are_errors_not_text,date64_must_be_whole_days,dictionaries_are_unpacked.unsigned_integers_are_not_wrapped,large_int64_stays_numeric,integers_beyond_2_pow_53_are_errors,decimals_keep_their_digits,decimals_too_wide_for_a_float_are_errors.TimezoneNotSupportedby default; lenient checktimezone_to_utcwrites UTC wall-clock time and attaches aWarning(WriteTable::warnings(), PythonQvdWarning). Test:time_zone_is_an_error_unless_allowed,test_timezone_is_an_error_unless_allowed.u32casts for row count, symbol-table size, distinct values: nowLayoutOverflow. (A table above 4 Gi rows cannot be built in a test; the checks areu32::try_fromat the three sites.)whole_second_timestamps_survive_a_round_trip_exactly, 5,000 values).Not reproduced / deferred
API breaks
record_batch(es)_to_write_tablereturns tables carryingwarnings(); new error variants and a newLenientCheck::TimezoneToUtc/WarningKind::TimezoneToUtc. Pythonwritenow returns None but emits warnings (the private_writereturns them).Tests changed
api.rs::leniency_is_a_set_of_named_checksandtest_errors.py::test_leniency_basicsno longer assume there is exactly one check.Corpus
round_trip(read, re-encode with the writer, read, compare every cell) over /mnt/nas/data/QVD: 1,145 files, 1,072 ok, 15 over the size limit, 49 not QVD, 9 fail. The 9 are the three damaged fixtures plus the sixNonNumericTemporalfiles from #57; no writer-related failures.Lenient mode
Added:
timezone_to_utc(write side; no corpus relevance, but tz-aware timestamps are common in Polars/pandas data and strict-only would block them with no workaround except converting first, which the error message explains).Docs-site edits needed
Arrow-to-QVD type mapping table (from the
record_batches_to_write_tabledoc), timezone handling, Python write warnings.Review fixes
ValueNotRepresentable; new lenient checksub_microsecond_precisionrounds and warns once per column (nanosecond_digits_are_an_error_unless_rounding_is_allowed).openqvd::serial) so accepted values re-read strictly (dates_and_timestamps_the_reader_would_reject_are_not_written).DuplicateColumninWriteTable::newand when opening a file (duplicate_column_names_are_rejected,duplicate_field_names_are_an_error).INTERVALformat.