Ingestion parsers: Design positioning: pypdf-only extraction, OCR out of scope, versus docling / marker / unstructured (and MarkItDown, MinerU) (aging) (TECH-T07-006) #492

Wave 2 · theme D2-7: Next-release technology updates · highest severity Next release · 1 finding(s) · effort not estimated

Summary

A lightweight, deterministic, air-gap-friendly pypdf core is defensible. However, the user docs never say when born-digital text extraction is enough or point to the layout-aware or OCR converters. Scanned, multi-column and table-heavy PDFs are therefore a silent quality risk for SFT corpora.

Primary finding

TECH-T07-006 — Ingestion parsers: Design positioning: pypdf-only extraction, OCR out of scope, versus docling / marker / unstructured (and MarkItDown, MinerU) (aging)

severity Next release

Item. Design positioning: pypdf-only extraction, OCR out of scope, versus docling / marker / unstructured (and MarkItDown, MinerU) · verdict: aging · urgency: Next release

State in ForgeLM. docs/guides/ingestion.md:39-41 says only 'OCR is out of scope' and points to Tesseract or AWS Textract. For multi-column layouts, the warning at ingestion.py:931-935 points to camelot-py or pdfplumber. Docling and Marker appear only in roadmap records (docs/roadmap/completed-phases.md:994, :1014). Nothing in docs/guides, docs/reference, docs/usermanuals or README mentions Docling, Marker, unstructured, MarkItDown or MinerU.

State of the ecosystem on 2026-10-01. docling 2.132.0 was released 2026-10-01. It is MIT-licensed, hosted by LF AI & Data, requires Python <4.0, and its README advertises 'Local execution capabilities for sensitive data and air-gapped environments' and 'Extensive OCR support'. marker-pdf 2.0.0 (2026-07-20) has Apache-2.0 code, but its weights are under a modified OpenRAIL-M licence that is free only under $5M funding or revenue. unstructured 0.27.10 (2026-09-27) is Apache-2.0 and requires Python <3.14. pypdf's own docs say 'pypdf is not OCR software'. Whether these tools are 'the baseline' is a judgement; the comparative benchmarks found are secondary.

Impact. A lightweight, deterministic, air-gap-friendly pypdf core is defensible. However, the user docs never say when born-digital text extraction is enough or point to the layout-aware or OCR converters. Scanned, multi-column and table-heavy PDFs are therefore a silent quality risk for SFT corpora.

Recommendation. Add an EN+TR section to docs/guides/ingestion.md, 'When pypdf is not enough / when to pre-convert'. It should state the scope (text-layer, born-digital), give a handoff recipe (Docling, Marker, MinerU or MarkItDown to Markdown, then forgelm ingest --strategy markdown), and list licence and offline caveats (Marker weights; Docling models must be pre-fetched for air-gapped use). Keep OCR and VLM out of the core.

Sources.

• Docling README (MIT, LF AI & Data, air-gapped local execution, OCR) (2026-10-01)
• PyPI JSON docling (2.132.0, 2026-10-01) (2026-10-01)
• PyPI JSON marker-pdf (2.0.0) (2026-07-20)
• Marker README (Apache-2.0 code, OpenRAIL-M weights) (2026-09-13)
• PyPI JSON unstructured (0.27.10, <3.14) (2026-09-27)
• MarkItDown GitHub (2026-10-01)

Acceptance criteria

• Each statement is re-checked against the code at fix time.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}