[Wave 3] Medium — Module sweep: Document ingestion (PDF/DOCX/EPUB/TXT/MD) (part 1/2) #551

Wave 3 · 22 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A12-005 (code, Medium) — --chunk-tokens 0 is accepted, loads the tokenizer, then silently falls back to character chunking while notes record token mode
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A templated CI value of 0 (unset budget) produces character-sized chunks that overflow the model's max_length instead of failing the job.
  Where. forgelm/ingestion.py:2092 (confirmed), forgelm/ingestion.py:2396-2407 (confirmed), forgelm/ingestion.py:2601-2603 (confirmed), forgelm/cli/_parser.py:283-290
  Recommended fix. Validate chunk_tokens as a positive int (reject 0) in ingest_path before the tokenizer load, or use is not None consistently; fix the help text.

• ↳ same root cause: DOC-BHELP-008 (docs, Medium) — forgelm: false claim — '0 is still rejected at ingest_path's own positive-int check'.
  severity Medium
  Where. forgelm/cli/_parser.py:287-294 (--chunk-tokens) · type: false claim
  The document says. '0 is still rejected at ingest_path's own positive-int check'.
  What the code does. --chunk-tokens 0 is not rejected. ingest_path uses a truthiness test, so 0 silently falls back to character chunking. Verified: --chunk-tokens 0 --tokenizer env/tiny-qwen exits 0 with a normal result.
  Code references. forgelm/ingestion.py:2092, forgelm/ingestion.py:2396-2407
  Proposed correction. Reject chunk_tokens <= 0 in ingest_path (is not None check plus positive check), or fix the help.

• ↳ same root cause: DOC-E2E-004 (docs, Low) — forgelm/cli/_parser.py: false claim — 0 is still rejected at ingest_path's own positive-int check
  severity Low
  Where. forgelm/cli/_parser.py:291-292 (--chunk-tokens help) · type: false claim
  The document says. 0 is still rejected at ingest_path's own positive-int check
  What the code does. --chunk-tokens 0 with --tokenizer exits 0 and silently uses character chunking.
  Code references. forgelm/ingestion.py:2092, forgelm/ingestion.py:2396
  Proposed correction. Fix the code (F-INGEST-02) or the help.

• CR-A12-007 (code, Medium) — Ingest where every file fails or yields zero chunks exits 0 with success:true and promotes an empty JSONL over --output
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Nightly ingest -> train pipeline overwrites last good dataset with an empty file; the next stage fails far from the cause, and a hostile document set can drop a victim's documents silently.
  Where. forgelm/ingestion.py:2619-2630 (confirmed), forgelm/ingestion.py:2478-2517, forgelm/cli/subcommands/_ingest.py:238-262, forgelm/ingestion.py:2072-2074
  Recommended fix. Add an opt-in/default non-zero exit (or a distinct code) when chunk_count == 0, or at least do not overwrite an existing --output and document the exit-0 behaviour in the guide and --help.

• CR-A12-008 (code, Medium) — Markdown YAML front-matter regex deletes real content after a leading '---' thematic break (default on, silent)
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Operator ingests a docs tree in which a page opens with '---' as a visual separator. Its intro and the first sections disappear silently from the SFT JSONL.
  Where. forgelm/ingestion.py:180, forgelm/ingestion.py:1430-1433
  Recommended fix. Require the first line after the opening '---' to be non-blank and the block to parse as a YAML mapping (or at least match ^\w[\w-]*:\s). Otherwise keep the content. Log and count each strip.

• CR-A12-009 (code, Medium) — DOCX block-level content controls (w:sdt) and all text inside them are silently dropped
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A legal-contract corpus authored from a Word template puts its clause bodies in content controls. ingest emits only the surrounding boilerplate paragraphs and reports success.
  Where. forgelm/ingestion.py:1079-1098, forgelm/ingestion.py:1213-1218, forgelm/ingestion.py:1094
  Recommended fix. Recurse into w:sdt/w:sdtContent (and w:customXml) when walking the body. Optionally skip the TOC docPartGallery only, and log the number of skipped sdt blocks.

• CR-A12-010 (code, Medium) — DOCX header/footer subtraction still drops short body lines equal to header text, contrary to its docstring; first-page/even-page headers are not collected
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A report whose running header repeats the chapter or document title loses that title heading from the body. Paragraph and markdown chunkers then lose the section's heading context.
  Where. forgelm/ingestion.py:1143-1150, forgelm/ingestion.py:1186-1193, forgelm/ingestion.py:1130-1140
  Recommended fix. Do not subtract whole paragraphs that are headings or the only line of a paragraph, or subtract only when the body line appears in the body as often as there are page breaks. Collect first_page_header/even_page_header and header tables, and correct the docstring.

• CR-A12-011 (code, Medium) — --language-hint is lower-cased for the CLI warning and profile but passed raw to check_script_sanity, so 'TR' silently disables the sanity check
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN TRUST-09 — docs/roadmap/phase-16-trust-surface-hardening.md:586,603 (S10 open)
  Failure scenario. An operator writes 'TR' (BCP-47 region style) in a CI script. Font-corrupt or mojibake corpora pass with a green script_sanity block.
  Where. forgelm/_script_sanity.py:404 (confirmed), forgelm/ingestion.py:554-562, forgelm/cli/subcommands/_ingest.py:89-92, forgelm/cli/subcommands/_ingest.py:135-148
  Recommended fix. Normalise language_hint (strip().lower()) once in ingest_path, and/or make check_script_sanity look up the lower-cased key. Mark untriggered no-op reports separately in report_to_structured.

• CR-A12-012 (code, Medium) — Lossy Turkish pypdf glyph-fallback profile is applied to TXT/MD/DOCX/EPUB and rewrites legitimate characters
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A Turkish primary-school maths corpus in DOCX/TXT gets every '÷' rewritten to 'ğ' and words ending in 'ö' fused with the next word. The training data is corrupted and the only signal is an INFO line.
  Where. forgelm/ingestion.py:535-549 (confirmed), forgelm/_pypdf_normalise.py:61-67, forgelm/_pypdf_normalise.py:141-150, forgelm/ingestion.py:500
  Recommended fix. Apply the glyph profile only to text produced by _extract_pdf (for example, pass the extension into _post_extract_normalise), or document clearly that it applies to all formats.

• CR-A12-013 (code, Medium) — Token-sliding chunker decodes id slices: U+FFFD at byte-BPE cut points and literal special-token text deleted
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A Japanese, Chinese or emoji-heavy corpus ingested with --strategy sliding --chunk-tokens 512 yields rows with U+FFFD at every boundary that falls inside a multi-byte run. A technical document about chat templates loses its examples.
  Where. forgelm/ingestion.py:1901, forgelm/ingestion.py:1897, forgelm/ingestion.py:1900, forgelm/ingestion.py:1892-1902
  Recommended fix. Map token windows back to character offsets (return_offsets_mapping on fast tokenizers) and slice the original string. At minimum, decode with skip_special_tokens=False and move the cut to a character boundary.

• CR-A12-014 (code, Medium) — --strip-pattern ReDoS shape validator accepts catastrophic nested-quantifier shapes ({n,} braces, ')' followed by '{', bounded trailing atom)
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator passes --strip-pattern '(\w+\s?)+$'. Validation passes. On a 10k-file corpus every non-matching line costs the full 5 s budget per file and the pattern silently does nothing. On Windows the ingest hangs.
  Where. forgelm/_strip_pattern.py:268-282, forgelm/_strip_pattern.py:186-198, forgelm/_strip_pattern.py:196, forgelm/_strip_pattern.py:281
  Recommended fix. Treat '{n,}' (no upper bound) as unbounded for both atoms and groups, and inspect the quantifier after ')' including braces. Consider treating a group as unbounded when any atom in it is unbounded, not only the last one.

• CR-A12-015 (code, Medium) — Atomic ingest output uses a fixed '.tmp' opened with symlink-following open('w'), no fsync, no input/output aliasing check
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN TRUST-10 — docs/roadmap/phase-16-trust-surface-hardening.md:463-465,472-474 (S6 open; unit id inferred)
  Failure scenario. A user with write access to a shared output directory plants '.jsonl.tmp' -> another file the operator can write; the operator's next ingest overwrites it. Separately, an operator typo --output notes.txt with input notes.txt destroys the source document.
  Where. forgelm/ingestion.py:2485, forgelm/ingestion.py:2525, forgelm/ingestion.py:2418 (confirmed), forgelm/ingestion.py:2320-2332 (confirmed)
  Recommended fix. Create the temp with tempfile.mkstemp(dir=dst.parent) or os.open(O_CREAT|O_EXCL|O_NOFOLLOW) and chmod it to the umask default. fsync before os.replace. Reject dst when it is in the resolved input file set.

• CR-A12-016 (code, Medium) — A timed-out --strip-pattern is skipped per file with only a WARNING: the skip is not recorded in the result and the pattern is re-run (and times out) on every later file
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A directory of 10,000 scraped pages that each trip an accepted alternation pattern takes about 14 hours instead of failing fast, and every output chunk keeps the boilerplate the operator meant to strip.
  Where. forgelm/_strip_pattern.py:381, forgelm/_strip_pattern.py:434, forgelm/ingestion.py:2608, forgelm/cli/subcommands/_ingest.py:238
  Recommended fix. Count skipped (pattern, file) pairs in the ExtractContext and surface them in notes_structured.strip_pattern_timeouts and notes. Optionally add a flag to make a timeout fatal. Emit a one-time WARNING when a timeout was requested but the alarm cannot be armed (non-POSIX or non-main thread).

• CR-A12-017 (code, Medium) — A lone surrogate in extracted text raises UnicodeEncodeError at write time, aborting the whole run with exit 1 and no file name
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. A crafted PDF, or any real PDF with a broken ToUnicode map, placed in a corpus makes the entire ingest job fail with a misleading config-error exit code. CI retry logic keyed on the exit code then misroutes the failure.
  Where. forgelm/ingestion.py:2143, forgelm/ingestion.py:2198, forgelm/cli/subcommands/_ingest.py:191-212 (confirmed), forgelm/ingestion.py:2181
  Recommended fix. Sanitise extracted text (text.encode('utf-8','replace').decode('utf-8')) after extraction, or catch UnicodeEncodeError per file in _process_one_file and count it as a skip with the file name. Do not let UnicodeError reach the config-error branch.

• CR-A12-018 (code, Medium) — Token-aware chunking tokenizes the whole file in one call: peak memory is 70-200x the file size
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Where. forgelm/ingestion.py:1786-1820 (_count_section_tokens single batch call), forgelm/ingestion.py:1938 (_chunk_paragraph_tokens caller), forgelm/ingestion.py:1840, forgelm/ingestion.py:1892-1900 (_chunk_sliding_tokens encode of the whole text)
  Recommended fix. Count tokens in batches of 1-2k paragraphs, passing return_attention_mask=False and return_token_type_ids=False. For sliding, encode bounded text segments (for example 1 M chars cut on whitespace) and window over each segment.

• CR-A12-019 (code, Medium) — Markdown chunking is quadratic in leading blank lines (_trim_blank_edges uses list.pop(0))
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A Markdown file padded with megabytes of blank lines stalls 'forgelm ingest --strategy markdown'.
  Where. forgelm/ingestion.py:1608-1614, forgelm/ingestion.py:1651-1660, forgelm/ingestion.py:1752, forgelm/ingestion.py:1834
  Recommended fix. Find the first and last non-blank index once and return lines[start:end].

• CR-A12-020 (code, Medium) — The --strip-pattern ReDoS guard's runtime half (SIGALRM timeout and skip-with-warning) is never exercised by any test and is hidden by a false pragma; a skipped pattern is not reported in the result
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A refactor of _apply_one (for example moving signal.alarm after subn, or catching the wrong exception) leaves the guard inert: a crafted document with (a|aa)+$ style input hangs the ingest run indefinitely and no test fails. Separately, a timed-out sensitive-data pattern leaves the data in the JSONL with a success envelope.
  Where. forgelm/_strip_pattern.py:448-450, forgelm/_strip_pattern.py:417-450, forgelm/_strip_pattern.py:381-385, tests/test_strip_pattern.py:135-183
  Recommended fix. Add a test that applies (a|aa)+$ to ~48 'a' with timeout_s=1 and asserts the elapsed time, the WARNING, and that later patterns still apply; remove the false pragma; record skipped patterns in notes_structured (e.g. strip_pattern_timeouts) and in the JSON envelope.

• CR-A12-025 (code, Medium) — Ingest --tokenizer loads an unpinned tokenizer; docs say it is pinned by the Hub revision
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Failure scenario. Re-ingesting the same PDFs a month later produces different chunk boundaries after a tokenizer repo update, contradicting the stated determinism.
  Where. forgelm/ingestion.py:1978-1989, forgelm/cli/_parser.py:308-313, docs/reference/data_ingestion_architecture.md:638-639, forgelm/ingestion.py:2407
  Recommended fix. Add --tokenizer-revision (route through prepare_revision_pin) or correct the doc sentence.

• CR-A12-027 (code, Medium) — DOCX paragraph text inside inline wrapper elements (tracked insertions w:ins/w:moveTo, w:fldSimple, w:smartTag, run-level w:sdt, w:customXml, w:dir/w:bdo) is silently dropped; unaccepted revisions pr…
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator ingests a folder of contracts or policies that still carry unaccepted Track Changes, or form templates filled through content controls. The JSONL drops every inserted clause and every filled value and splices the neighbouring text together ('The fee is EUR'). It reports processed=N and exit 0, and the model is fine-tuned on corrupted sentences. A document drafted entirely with Track Changes on is reported as 'no extractable text', which sends the operator to look for an OCR or empty…
  Where. forgelm/ingestion.py:1175, forgelm/ingestion.py:1060, forgelm/ingestion.py:1094-1098, forgelm/ingestion.py:2184-2186
  Recommended fix. Build paragraph text from descendant runs, excluding deletions and field instructions. Decide on accept-all versus reject-all for revisions, document the choice, and add a per-file note or counter when revisions or content controls are present. Add tests built from raw XML for each wrapper element.

• CR-A12-028 (code, Medium) — DOCX text boxes (DrawingML wps:txbx and VML v:textbox) are silently dropped
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A brochure or report whose key facts sit in text boxes is ingested. The facts never reach the training data, and nothing in the result says so.
  Where. forgelm/ingestion.py:1079-1098 (_iter_docx_blocks yields only top-level w:p / w:tbl children of w:body), forgelm/ingestion.py:1175 (_render_docx_block uses Paragraph.text, which python-docx builds from run-level w:t/w:tab/w:br etc. and which never descends into w:drawin…, forgelm/ingestion.py:1211-1217 (_extract_docx joins only those blocks, with no warning or counter)
  Recommended fix. Extract w:txbxContent paragraphs once each, preferring mc:Choice over mc:Fallback, or emit a per-file note giving the number of text boxes skipped.

• CR-A12-029 (code, Medium) — DOCX table rendering drops nested tables, row-level and cell-level content controls, and duplicates merged-cell text
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A form-style DOCX (an application form with sub-tables per section, or repeating-row content controls) ingests as a table holding only outer labels. A merged header cell appears twice, so tabular Q&A training learns duplicated values.
  Where. forgelm/ingestion.py:1046-1049, forgelm/ingestion.py:1171-1173, docs/usermanuals/en/data/ingestion.md:47
  Recommended fix. Render cell content recursively, including nested w:tbl and w:sdt, and de-duplicate spanned and vMerged cells by underlying w:tc identity. At minimum, warn when a nested table or content control is skipped.

• CR-A12-031 (code, Medium) — Up-front --input-encoding check (codecs.lookup) accepts non-text codecs, so every TXT/MD file is soft-skipped and the run exits 0
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator passes --input-encoding hex or base64, by mistake or from a templated pipeline. Every TXT/MD file is dropped, an empty JSONL is promoted over --output, the exit code is 0, and a later training step runs on nothing.
  Where. forgelm/ingestion.py:2305-2310, forgelm/ingestion.py:1382, forgelm/ingestion.py:2073-2075, docs/guides/ingestion.md:391-396
  Recommended fix. After codecs.lookup(), reject codecs whose CodecInfo is not a text encoding: check getattr(info, '_is_text_encoding', True), or probe with a non-empty decode such as b'a'.decode(name) inside the try. Raise the same ValueError. Add test cases for base64 and rot13 next to the existing bogus-codec test (tests/test_ingestion_reliability.py:1376).

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}