Skip to content

Fix structure checks: table placement, bias, bit width, indices, unused bits, trailing bytes - #56

Merged
Nathan-D-R merged 3 commits into
mainfrom
fix/structure-validation
Oct 25, 2026
Merged

Nathan-D-R merged 3 commits into
mainfrom
fix/structure-validation

Conversation

@Nathan-D-R

@Nathan-D-R Nathan-D-R commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Structural silent failures, all caught when a file is opened. Stacked on #55.

Findings fixed (tests in tests/structure.rs)

  • S1-9 symbol-table placement: tables must lie in front of the row block and tile that region exactly (no overlap, no gap). Errors: SymbolRegionInRowBlock, SymbolRegionOverlap, SymbolRegionGap. Tests: symbol_table_inside_the_row_block_is_an_error, overlapping_symbol_tables_are_an_error, unclaimed_bytes_between_tables_and_row_block_are_an_error.
  • S1-10 unobserved Bias: only 0 and -2 are accepted, others are UnsupportedBias (kind Unsupported). Test: unobserved_bias_values_are_unsupported.
  • S1-11 BitWidth not range-checked: widths above 33 are BitWidthOutOfRange (33 = u32 symbol count plus bias 2). Test: bit_width_beyond_what_an_index_needs_is_an_error (the 72-bit case from the audit).
  • S2-1 out-of-range symbol index became NULL: one validation pass at open for every decoded column, so rows(), Arrow conversion, to_write_table, the CLI and Python all see an error. Test: an_index_past_the_symbol_table_is_rejected_when_the_file_opens, undecoded_columns_are_not_validated.
  • S3-1 unused record bits must be zero (also for records wider than 16 bytes). Tests: bits_outside_every_field_must_be_zero, unused_bits_are_checked_in_wide_records_too.
  • S3-2 trailing bytes after the row block are an error. Test: trailing_bytes_after_the_row_block_are_an_error.

The format-wide used bit vector (sized from the header's RecordByteSize, up to 4 GB for a crafted header) is replaced by sorted interval checks, so a crafted header can no longer force that allocation.

Not reproduced / deferred

None in this group.

API breaks

New QvdError variants only.

Tests changed

edge_cases.rs::checked_rows_errors_on_out_of_range_index, already renamed in #53, now asserts the error at open instead of from rows() (listed here because its expected failure point moved). tests/header.rs::earliest_terminator... lost a stray trailing byte that the new check rejects.

Corpus (the check the audit asked for before choosing error vs. flag)

validate_all over /mnt/nas/data/QVD with every check on: 1,089 files, 1,044 ok, 3 fail, 42 not QVD. All 1,044 readable files satisfy exact tiling, bias in {0,-2}, width <= 33, zero unused bits, no trailing bytes and in-range indices, so these are plain errors with no lenient mode. The three failures are the known damaged fixtures; mynewfile3.qvd now reports 858 trailing bytes (its real condition) instead of an invalid-UTF-8 error.

Lenient mode

Not added: no readable file needs it.

Docs-site edits needed

Reader-behavior page: list the structural checks above.

Review fixes

  • N4: validation does not scale with NoOfRecords for zero-width records (a_huge_record_count_with_zero_width_records_opens_quickly). N5: schema() docstring updated.

@Nathan-D-R
Nathan-D-R force-pushed the fix/structure-validation branch from 15a9cb3 to e3e2377 Compare October 25, 2026 13:16
@Nathan-D-R
Nathan-D-R changed the base branch from fix/header-parsing to main October 25, 2026 14:01
@Nathan-D-R
Nathan-D-R merged commit 33112df into main Oct 25, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant