Skip to content

Split forgelm/ingestion.py into a sub-package (_readers, _chunkers, _pipeline) — deferred until the next change that adds a reader or a chunker #905

Open

Description

@cemililik

Deferred work · F-PR29-A1-06 / A1-07 / A1-08: ingestion.py · severity Low · cost ~4-8 h · recorded in docs/roadmap/risks-and-decisions.md:80

Summary

forgelm/ingestion.py is a deferred module split: PDF/DOCX/EPUB/TXT/Markdown readers, chunkers and the SFT-JSONL emitter share one file; the seams are already near-independent, which makes this the highest-value split per hour. The module-size guard holds it to a fatal no-growth budget, but nothing pays the debt down until someone takes the split.

Condition or budget

The next change that adds a reader or a chunker

State today (re-checked 2026-10-09)

2110 code lines against a budget of 2110 (1443 at PR #29).

Related

#667 and #668 track how the module-size guard counts lines and accepts budget raises; decide them before relying on a module's budget.

Acceptance criteria

  • The module becomes a sub-package along the recorded seams, behaviour-neutral, with every public name still importable from its current path.
  • Its _DEFERRED_SPLITS entry is removed from tools/check_module_size.py and every resulting module is under the normal ceiling.
  • Test patch targets that named the old module are retargeted to the module that resolves the name at call time (the forgelm/verify.py split found five that silently moved).
  • The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).

Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.

Activity

  1. added
    severity: lowMinor defect, inaccuracy or rough edge
    wave: 4Roadmap wave 4: low / info / watch, when touching the module
    source: roadmapPlanned or deferred work recorded in docs/roadmap/
    deferredDeferred with a recorded condition: check that it holds before starting
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ingestionCode area: ingestiondeferredDeferred with a recorded condition: check that it holds before startingseverity: lowMinor defect, inaccuracy or rough edgesource: roadmapPlanned or deferred work recorded in docs/roadmap/wave: 4Roadmap wave 4: low / info / watch, when touching the module

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions