Skip to content

Choosing-a-trainer table documents KTO rows as {prompt, response, label}, but the KTO loader requires a completion column (F-W1-DOCS-TRAIN-02) #885

Closed
#938
Closed
Choosing-a-trainer table documents KTO rows as {prompt, response, label}, but the KTO loader requires a completion column (F-W1-DOCS-TRAIN-02)#885
#938
Labels
documentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-09Found by the September 2026 independent reviewwave: 3Roadmap wave 3: medium severity, module sweeps
Milestone
Backlog — Wave 3: medium severity (module sweeps)

Description

@cemililik
cemililik
opened on Oct 9, 2026

Wave 3 · group W3 Docs · highest severity Medium · 1 finding(s) · effort docs fix

Summary

The "What to look at in your data" table says {prompt, response, label: bool} rows mean KTO. The KTO loader requires completion and label and stops with a KeyError on response, so data shaped the way the page recommends does not load.

Primary finding

F-W1-DOCS-TRAIN-02 — The choosing-a-trainer table names the wrong KTO answer column

severity Medium · confidence high · checked against the loader

What happens. docs/usermanuals/en/concepts/choosing-trainer.md:51 and the Turkish mirror at the same line. _validate_trainer_columns raises KeyError: KTO trainer requires 'completion' and 'label' (boolean) columns … (forgelm/data.py:600-607). The row above it (:49, {prompt, completion} → SFT) is the SFT-format defect tracked in #403; that issue now lists this page too.

Where.

Evidence (re-verified on 2026-10-09).

  • The KTO branch of _validate_trainer_columns names only completion and label; both table rows are unchanged on development.

Recommended fix. Change the KTO row to {prompt, completion, label: bool} in both languages, matching the KTO page and the loader. Settle the SFT row (:49) with whatever #403 decides for the SFT format.

Related. #403 — the SFT row of the same table; fix both rows together if the PRs overlap.

Acceptance criteria

  • Every data shape in the table loads with the trainer it recommends.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • The finding is resolved or explicitly closed with a reason.

Found by the September 2026 independent review of ForgeLM against commit cab2947 and re-verified against development at f94595f on 2026-10-09. Code links point at cab2947; line numbers may have moved since. Finding IDs (F-W1-…) are stable references for this backlog.

Activity

  1. cemililik
    added this to the Backlog — Wave 3: medium severity (module sweeps) milestone on Oct 9, 2026
  2. cemililik
    added
    documentationImprovements or additions to documentation
    severity: mediumReal defect with limited reach or a workaround
    wave: 3Roadmap wave 3: medium severity, module sweeps
    source: review-2026-09Found by the September 2026 independent review
    on Oct 9, 2026
  3. cemililik
    added this to ForgeLM Project Planon Oct 9, 2026
  4. cemililik
    added a parent issue on Oct 9, 2026
  5. cemililik
    added
    good first issueGood for newcomers
    on Oct 9, 2026

  6. cemililik commented on Oct 11, 2026

    @cemililik
    cemililik
    on Oct 11, 2026
    CollaboratorAuthor

    Where to start

    Thanks for taking a look. Please confirm the issue is unassigned and comment before starting so work does not overlap.

    Start with these files on development:

    Change the KTO table row to prompt/completion/label in both languages, matching the KTO loader. Keep trainer selection explicit; this table does not auto-select a trainer. Check each row touched against its loader; do not introduce a runtime change.

    Adjacent SFT wording is tracked separately; coordinate with #403 if editing that row.

    Run the focused tests:

    pytest tests/test_data.py --no-cov

    Run these guards/checks as applicable to the changed files:

    ruff format --check forgelm/data.py tests/test_data.py
ruff check forgelm/data.py tests/test_data.py
python3 tools/check_bilingual_parity.py --strict
python3 tools/check_bilingual_code_blocks.py --strict
python3 tools/check_anchor_resolution.py --strict
python3 tools/check_tr_links_prefer_mirror.py --strict
python3 tools/check_usermanual_self_contained.py --strict
python3 tools/check_usermanual_schema_drift.py --strict

    If user-facing documentation changes, update the existing Turkish mirror in the same PR; keep commands, code and terminal output in English and structurally aligned. English-only contributor standards need no new mirror.

    Open the PR against development with Fixes #885 once every finding is resolved or explicitly closed with a reason; use Refs #885 for partial work and list the resolved finding IDs.

  7. github-project-automation
    moved this from Todo to In progress in ForgeLM Project Planon Oct 14, 2026
  8. cemililik
    added a commit that references this issue on Oct 17, 2026

    docs(changelog): record the KTO data-shape fix

    e934508
  9. cemililik
    closed this as completedin #938on Oct 17, 2026
  10. github-project-automation
    moved this from In progress to Done in ForgeLM Project Planon Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-09Found by the September 2026 independent reviewwave: 3Roadmap wave 3: medium severity, module sweeps

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions