[Gal-Gilor]

What changed

1. markdown_chunkify/splitter.py: _HEADER_PATTERN is now ^ {0,3}(#{1,6})(?:[ \t]+(.*))?$. A new _header_text helper removes trailing spaces and tabs and an optional closing '#' sequence. As before, a closing sequence counts only when a space or tab comes before it, or when it is the whole text. _find_headers adds a header only when its level is at most max_level and its text is not empty. stack in split_text is annotated list[Section] for mypy --strict.
2. tests/test_splitter.py: one test splits "# a" + " \t" * 500_000 + "b" and checks the header. On main it ran for 30 s without finishing, and I stopped it. On this branch it takes about 30 ms.
3. README.md Limitations: a new bullet says sections have no size limit, so long sections need to be split before embedding.
4. CHANGELOG.md 0.3.0 "Fixed": an entry for this fix.

Why

The old pattern had a lazy (.*?) followed by an optional closing sequence and [ \t]*$. Each time the lazy group grew by one character, the regex scanned the trailing whitespace again, so matching time grew with the square of the line length. A header line with a 40,000-character run of spaces and tabs took about 18 seconds. 0.2.0 used the same lazy pattern, so this is also a fix compared with the last release. Text from PDF parsers can contain long runs of whitespace.

Verification

• Regex equivalence: I compared the old regex with the new pattern plus _header_text on 400,017 lines and got identical results (117,461 of them headers). The lines were 400,000 random lines built from #, space, tab, letters, digits, non-ASCII letters, non-breaking space, \x0c, x# and ## with 0-5 space indents, plus 17 fixed edge cases (## Title ##, # # #, ## ##, # a#, # a ## ##, ## # Foo and others).
• Splitter equivalence with main: I loaded both versions in one process and ran 4,003 documents (4,000 random documents plus README.md, CHANGELOG.md and tests/test_splitter.py) at max_level 1-6. That is 24,018 comparisons, with 41,366 sections and 16,128 warnings. model_dump() output and warning messages were identical in every case. The random documents used headers at levels 1-7 with closing sequences and trailing whitespace, text, blank lines, fences, HTML blocks, front matter and CRLF line endings, and about 10% started with a BOM.
• Timing, MarkdownSplitter().split_text(...):

Input	n	main	branch
"# a" + " \t" * n + "b"	5,000	1.13 s	0.08 ms
"# a" + " \t" * n + "b"	20,000	18.04 s	0.05 ms
"# a" + " \t" * n + "b"	1,000,000	not run	3.0 ms
"# a" + " " * n + "#x"	5,000	0.25 s	0.03 ms
"# a" + " " * n + "#x"	20,000	4.09 s	0.03 ms
"# a" + " " * n + "#x"	1,000,000	not run	1.25 ms

• ruff check, ruff format --check and pytest all pass: 91 tests, 100% coverage, and the doctest is collected. The pydantic 2.0.2 job on Python 3.12 also passes: 91 tests.
• mypy --strict markdown_chunkify leaves one error: models.py:30: Decorators on top of @property are not supported [prop-decorator]. This is a known mypy limitation with Pydantic computed_field.
• pyright markdown_chunkify: 0 errors, 0 warnings.