Skip to content

Clean up the splitter internals - #43

Merged
Gal-Gilor merged 2 commits into
mainfrom
refactor/splitter-internals
Oct 18, 2026
Merged

Gal-Gilor merged 2 commits into
mainfrom
refactor/splitter-internals

Conversation

@Gal-Gilor

@Gal-Gilor Gal-Gilor commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

What changed

All changes are in markdown_chunkify/splitter.py.

  • Named header tuple: _find_headers returns list[_Header], a NamedTuple with line_start, content_start, level and text. split_text uses attribute access instead of positional unpacking.
  • Unclosed-block warning: a block_name is set when a fence or HTML block opens. This replaces the chain that rebuilt the name at the end. The warning uses lazy %-formatting and renders the same message as before.
  • One trimming helper: _trimmed_range(text, start, end) replaces _trim. It gives the preamble range (so the text.index(...) lookup is gone), the header content and the header span end. _LEADING_BLANK_LINES_PATTERN drops its ^, because ^ without MULTILINE only matches at position 0, not at the pos passed to match().
  • Docstrings: the module docstring is now a short summary that points to the README's Limitations section. The class docstring lists max_level under Attributes:. The split_text Returns section is now two sentences, and the hanging indent in the text arg is fixed. The split_text example is still the one doctest.

Why

The splitter was harder to read than it needed to be: positional tuple indexes, a warning name rebuilt from the closing tag, and two separate trimming paths for content and spans. Nothing user-facing changes, so there is no CHANGELOG entry.

Verification

  • ruff check and ruff format --check pass.
  • pytest: 94 passed with no test edits. Coverage is 99% overall and 98% for splitter.py. The 2 missed lines are the same front-matter lines that main misses.
  • The doctest passes.
  • uv build works. The wheel imports in a fresh venv and passes the # Café check.
  • Lowest pydantic: Python 3.12 with pydantic==2.0.2, 94 passed.
  • Equivalence with main: both versions loaded in one process, with module paths asserted. The corpus has 5016 documents: 5000 random documents built from line atoms (headers of levels 1-7, fences, HTML blocks, front matter, list and indented fences, non-ASCII text, blank and whitespace-only lines; 1792 with CRLF, 493 with a BOM), plus README.md, CHANGELOG.md and the test file in LF, CRLF and BOM forms, plus a few edge cases. Each document was split at max_level 1-6, which makes 30096 calls and 51735 sections. model_dump() output and the captured WARNING messages matched in every call. 18960 of those calls logged a warning, covering every block kind.
  • splitter.py drops from 351 to 288 lines.

Name the header tuple, track the name of the open block for the
unclosed-block warning, replace _trim with a range helper so spans and
content come from one place, and shorten the docstrings. No behavior
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018jNL9rsodv5SMpDXVNMEia
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018jNL9rsodv5SMpDXVNMEia
@Gal-Gilor
Gal-Gilor merged commit b27f20d into main Oct 18, 2026
5 checks passed
@Gal-Gilor
Gal-Gilor deleted the refactor/splitter-internals branch October 18, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants