String column values misaligned when decoding bit-packed VertiPaq runs (large multi-segment table, >8M rows)

[dcruzat]

Summary

Extracting a large table (~8M rows, multi-segment VertiPaq column store) from a .pbix, one string column comes out misaligned relative to the other columns in the same row — i.e. row N's value for that column actually belongs to a different row, while every other column in row N decodes correctly. Confirmed against an independent ground-truth source (see below), not a data-quality issue in the source file.

I checked #52 (off-by-one bit-pack base for nullable dictionary segments, merged 2026-07-21) first since it looked like the same class of bug — it's very possibly related, but what I'm seeing doesn't fully match a simple off-by-one/global-offset pattern (details below), so filing separately in case it's a distinct residual issue.

Context

Source: a public-sector .pbix report (~8M row table, 30 columns, one table with a string column that's dictionary-encoded and nullable — same shape #52 describes: "nullable dictionary-encoded column split across multiple VertiPaq segments, roughly >1M rows"). I don't have a synthetic minimal repro yet (the file is a government dataset with rows I can't republish, and I haven't isolated a small .pbix that reproduces it), but I do have a solid statistical characterization from cross-checking against an external, independent ground truth.

What I measured

The string column in question (a name/description field) can be cross-checked against an external registry via a separate ID column in the same row that decodes correctly. Out of 64 rows where that ID had a unique match in the external registry (so the "correct" value is unambiguous):

• All non-string columns (numeric, categorical, the ID itself) decoded correctly for all 64 rows.
• ~70% of the misaligned string values matched the value that should be 2 rows away (consistent with a fixed offset, similar in spirit to the Fix off-by-one bit-pack base for nullable dictionary segments #52 off-by-one pattern).
• ~30% showed no consistent offset at all — displacements ranging from 2 rows to over 40,000 rows away, no discernible pattern.

The 30% "no pattern" tail is what makes this look like it might be more than the #52 off-by-one — a fixed-offset bug would be 100% one offset (or a small number of them tied to segment boundaries), not a long tail with no structure.

All other columns in the same table (region/category codes, numeric measures) validated correctly against independent sources — the misalignment appears isolated to this one long string/dictionary column.

What I can't confirm yet

• Exact pbixray version at extraction time (installed via plain pip install pbixray, not pinned — will confirm and follow up if useful).
• Whether the 30% "no pattern" tail is a distinct bug from Fix off-by-one bit-pack base for nullable dictionary segments #52, or Fix off-by-one bit-pack base for nullable dictionary segments #52 doesn't fully cover this table's segment/dictionary configuration.

Happy to help reproduce

I can't share the raw government data, but I can share:

• Row indices + computed offsets (anonymized) for the 64-row calibration set.
• Segment count / dictionary size / run-type breakdown for the affected column, if there's a way to introspect that from pbixray.
• Whatever else would help build a minimal repro — let me know what's useful and I'll pull it together.

[Hugoberry]

Thanks for the very thorough report, @dcruzat. The calibration numbers were genuinely useful for narrowing this down.

I tried to reproduce it with two purpose-built models (a 4-segment one with nullable dictionary columns whose per-segment id ranges are disjoint, and one that stresses the Huffman dictionary path with long strings, mixed string sizes and non-Latin text). Both decode exactly the same as the source data, so I haven't managed to trigger a misalignment yet. Which probably just means I haven't matched the right characteristic of your file.

A few things that would help, none of which need the data itself:

1. pip show pbixray And if it's below 0.15.2, could you re-extract with the current version, please? The fix in Fix off-by-one bit-pack base for nullable dictionary segments #52 landed over a week ago. One thing worth mentioning: that bug shifted ids by a constant amount in dictionary space, which doesn't show up as a constant offset in row space. Row N ends up showing whatever value sits one dictionary entry away, and how far away that row is depends on where that value happens to occur, usually a couple of rows in grouped data, occasionally many thousands. It also only affected some segments, leaving the rest correct. So the mix you measured looks compatible with it, and it's worth ruling out first.
2. Do the affected rows cluster by segment, i.e. is row_index // 1048576 the same (or one of two values) for most of the mismatching rows?
3. If it's easy: does the mismatch also show up when comparing against a DAX export of the same table (DAX Studio EVALUATE)?
4. Is the affected table partitioned (incremental refresh)?

I also have a small script that dumps the storage layout of a single column: segment count, dictionary size and page count, encoding variant, and run breakdown. It prints no actual values, only counts and ids. Happy to share it if that's easier than answering the above.

[Hugoberry]

Hey @dcruzat, I believe I was able to replicate the bug you've experienced in #55
It would be great if you could have another go with version 0.15.3 of the library and see if the fix resolves your issue as well.