Skip to content

Document how to clean text from PDF parsers before splitting - #42

Merged
Gal-Gilor merged 2 commits into
mainfrom
docs/clean-pdf-text
Oct 18, 2026
Merged

Gal-Gilor merged 2 commits into
mainfrom
docs/clean-pdf-text

Conversation

@Gal-Gilor

@Gal-Gilor Gal-Gilor commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

What changed

Adds a "Cleaning text from PDF parsers" subsection to the README, under Usage between "Breadcrumbs" and "Limitations". It:

  • Explains that PDF parsers produce ligatures, non-breaking spaces and fullwidth characters, that the splitter doesn't clean text, and that cleaning before split_text keeps the spans indexing into the cleaned string.
  • Gives an example: unicodedata.normalize("NFKC", pdf_markdown), then removes the soft hyphen () and zero-width space () with str.replace, then calls MarkdownSplitter().split_text(clean). The two characters are written as escapes so they are visible and survive copy-paste.
  • Has a table of what NFKC fixes and what it leaves alone. Accents are kept (Müller), superscripts are folded (x² becomes x2), and soft hyphens and zero-width spaces need str.replace.

No code changes, no new parameters or dependencies. No CHANGELOG entry: this is a docs-only change.

Verification

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018jNL9rsodv5SMpDXVNMEia
@Gal-Gilor
Gal-Gilor merged commit 0e892e1 into main Oct 18, 2026
5 checks passed
@Gal-Gilor
Gal-Gilor deleted the docs/clean-pdf-text branch October 18, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants