Repository navigation
Fix structure checks: table placement, bias, bit width, indices, unused bits, trailing bytes - #56
Merged
Merged
Conversation
This was referenced Oct 25, 2026
…indices, unused bits, trailing bytes
Nathan-D-R
force-pushed
the
fix/header-parsing
branch
from
October 25, 2026 13:16
83749fe to
33da903
Compare
Nathan-D-R
force-pushed
the
fix/structure-validation
branch
from
October 25, 2026 13:16
15a9cb3 to
e3e2377
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structural silent failures, all caught when a file is opened. Stacked on #55.
Findings fixed (tests in
tests/structure.rs)SymbolRegionInRowBlock,SymbolRegionOverlap,SymbolRegionGap. Tests:symbol_table_inside_the_row_block_is_an_error,overlapping_symbol_tables_are_an_error,unclaimed_bytes_between_tables_and_row_block_are_an_error.Bias: only 0 and -2 are accepted, others areUnsupportedBias(kindUnsupported). Test:unobserved_bias_values_are_unsupported.BitWidthnot range-checked: widths above 33 areBitWidthOutOfRange(33 = u32 symbol count plus bias 2). Test:bit_width_beyond_what_an_index_needs_is_an_error(the 72-bit case from the audit).rows(), Arrow conversion,to_write_table, the CLI and Python all see an error. Test:an_index_past_the_symbol_table_is_rejected_when_the_file_opens,undecoded_columns_are_not_validated.bits_outside_every_field_must_be_zero,unused_bits_are_checked_in_wide_records_too.trailing_bytes_after_the_row_block_are_an_error.The format-wide
usedbit vector (sized from the header'sRecordByteSize, up to 4 GB for a crafted header) is replaced by sorted interval checks, so a crafted header can no longer force that allocation.Not reproduced / deferred
None in this group.
API breaks
New
QvdErrorvariants only.Tests changed
edge_cases.rs::checked_rows_errors_on_out_of_range_index, already renamed in #53, now asserts the error at open instead of fromrows()(listed here because its expected failure point moved).tests/header.rs::earliest_terminator...lost a stray trailing byte that the new check rejects.Corpus (the check the audit asked for before choosing error vs. flag)
validate_allover /mnt/nas/data/QVD with every check on: 1,089 files, 1,044 ok, 3 fail, 42 not QVD. All 1,044 readable files satisfy exact tiling, bias in {0,-2}, width <= 33, zero unused bits, no trailing bytes and in-range indices, so these are plain errors with no lenient mode. The three failures are the known damaged fixtures;mynewfile3.qvdnow reports 858 trailing bytes (its real condition) instead of an invalid-UTF-8 error.Lenient mode
Not added: no readable file needs it.
Docs-site edits needed
Reader-behavior page: list the structural checks above.
Review fixes
NoOfRecordsfor zero-width records (a_huge_record_count_with_zero_width_records_opens_quickly). N5:schema()docstring updated.