Multi-column warning rests on a false premise about pypdf's visitor_text and does not fire on the layouts it targets (CR-A12-023) #482

Wave 2 · theme D2-5: Data audit, masking and ingestion: medium effort · highest severity Medium · 4 finding(s) · effort medium

Summary

A scrambled two-column PDF is ingested into SFT chunks with no warning; the operator trusts the guard.

Primary finding

CR-A12-023 — Multi-column warning rests on a false premise about pypdf's visitor_text and does not fire on the layouts it targets

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. The detector's docstring says visitor_text fires per glyph with the running text matrix. In pypdf it fires per merged text run. A typical two-column page (left and right fragments on the same baseline) is merged into one run per row ('Left column ... Right column ...'), so only the left margin x is observed. The reported tm x also differs across versions: for separate BT/ET blocks, pypdf 4.0.0 and 5.9.0 give x=0 for the second fragment, while 6.19.0 gives the real x=360. With a single BT and Td moves, or a TJ array, the runs are merged in all versions. Result: _maybe_warn_multi_column returned False for every row-interleaved layout (exactly the case where reading order scrambles) on 4.0.0, 5.9.0 and 6.19.0. It only works when the content stream is already in column-by-column order (the unscrambled case), or, on 6.x only, for separate text objects. The documented F-034 'multi-column warning' therefore gives a false sense of coverage.

Where.

• forgelm/ingestion.py:818
• forgelm/ingestion.py:907
• forgelm/ingestion.py:870
• forgelm/ingestion.py:895
• docs/guides/ingestion.md:503

Evidence.

• dbg_vis.py: on fx/twocol.pdf (left at x=72, right at x=360 on the same lines), _collect_multi_column_samples returned ([72.0]*6, 612.0) and _maybe_warn_multi_column returned False on pypdf 4.0.0, 5.9.0 and 6.19.0. dbg_vis2.py shows the x values per variant and version.

Recommended fix. Use pypdf's layout-mode or an x-histogram over per-run positions from a version-stable source, or state clearly that the probe only catches a subset of layouts. Add a test with an interleaved-row fixture.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T07-002 — Ingestion parsers: Multi-column detector via extract_text(visitor_text=...) using tm[4], and its 'visitor_text fires per glyph' rationale (superseded-method)

severity Next release

Item. Multi-column detector via extract_text(visitor_text=...) using tm[4], and its 'visitor_text fires per glyph' rationale · verdict: superseded-method · urgency: Next release

State in ForgeLM. forgelm/ingestion.py:870-881: the visitor appends float(tm[4]) and ignores cm. :895-904 is a min/max two-cluster test with a 0.30 page-width threshold. The threshold is justified at ingestion.py:826-832, docs/guides/ingestion.md:502-508 and docs/guides/ingestion-tr.md:514-517 by the claim that visitor_text 'fires per glyph'.

State of the ecosystem on 2026-10-01. The pypdf docs say the visitor receives 'the current text (as long as possible, can be up to a full line)'. They also say 'It is recommended to use the user_matrix as it takes into account all transformations', with txt2user = mult(tm, cm). 6.18.1 fixed a stale text matrix being passed to visitor_text (#​4062). Arbiter PoC with hand-built PDFs and the snapshot's forgelm.ingestion: on 5.9.0, with two BT blocks at x=72 and x=330, the right fragment arrives with tm x=0.0; ForgeLM collects [72, 0, 0, ...] and emits no warning (false negative). On 6.19.0 the same file gives x=330 and a warning. A column placed with a cm translation (tm[4]=72, cm[4]=258) is missed on both versions. Two columns inside one BT, placed with Td, arrive as one merged line fragment and are missed on both. There were 17 callbacks for 12 Tj strings plus newline fragments, so the callback is per fragment, not per glyph. Newline fragments carry tm x=0.0 and are also collected.

Impact. Quality and observability only, not security. The warning is silent on simple two-column PDFs under the capped pypdf, misses cm-placed columns on every version, and rests on a wrong premise. The documented reason for keeping 0.30 is therefore invalid.

Recommendation. Once the floor is >=6.19.0, compute x from mult(tm, cm)[4], skip whitespace or newline fragments, and re-derive the threshold against real two-column fixtures (not a mocked visitor). Correct the 'per glyph' text in ingestion.py and in the EN and TR guides together.

Sources.

• pypdf docs: Extract Text from a PDF (visitor arguments, user_matrix recommendation) (2026-10-01)
• pypdf docs (readthedocs): Using a visitor (2026-10-01)
• pypdf PR #4062 Text Matrix Fix for visitor_text (2026-09-11)
• pypdf 6.19.0 wheel used for the arbiter PoC (2026-09-16)

DOC-BLEGAL-187 — docs/guides/ingestion-tr.md: false claim — pypdf's visitor receives text per fragment (up to a full line), not per glyph.

severity Low

Where. docs/guides/ingestion-tr.md:514-517 (Turkish mirror: docs/guides/ingestion.md:503-506) · type: false claim

The document says. [IS-T07-03] "pypdf'in visitor_text callback'i glyph başına ateşlendiği için tek-kolonlu bir sayfanın glyph'leri doğal olarak geniş bir x-aralığına yayılır"

What the code does. pypdf's visitor receives text per fragment (up to a full line), not per glyph.

User impact. Same wrong rationale as the EN page.

Proposed correction. pypdf'in visitor_text callback'i glyph başına değil metin parçası başına (bir satıra kadar) ateşlenir; dedektör yalnızca metin matrisini (tm) kullanıp cm dönüşümünü yok saydığından, cm ile konumlanmış kolonları ve pypdf < 6.18.1'de ayrı BT bloklarındaki kolonları kaçırabilir.

DOC-BLEGAL-188 — docs/guides/ingestion.md: false claim — [IS-T07-02] 'compares the min / max x-positions across all extracted Tj glyphs on a page ... Because pypdf's visitor_text callback fires per glyph, a single-…

severity Low

Where. docs/guides/ingestion.md:503-506 (Turkish mirror: docs/guides/ingestion-tr.md:514-517) · type: false claim

The document says. [IS-T07-02] 'compares the min / max x-positions across all extracted Tj glyphs on a page ... Because pypdf's visitor_text callback fires per glyph, a single-column page's glyphs naturally span a wide x-range'

What the code does. pypdf's visitor receives 'the current text (as long as possible, can be up to a full line)', i.e. per text fragment, not per glyph. pypdf recommends the user matrix (tm x cm). Before 6.18.1 a stale text matrix was passed. Arbiter PoC: two-column PDFs are missed on 5.9.0, and a column placed via a cm translation is missed on all versions.

User impact. The stated rationale for the 30% threshold is wrong, which hides the detector's false negatives.

Proposed correction. Replace the rationale: pypdf's visitor_text fires per text fragment (up to a full line), and the detector uses the text matrix only (ignoring cm), so it can miss columns positioned by a cm translation and, on pypdf < 6.18.1, columns in separate BT blocks.

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}