Skip to content

DPO manual and CLI wizard reverse dpo_beta: they say a lower beta keeps the model closer to the reference model, but in TRL a higher beta does (F-W1-DOCS-TRAIN-07) #880

Open
Open
DPO manual and CLI wizard reverse dpo_beta: they say a lower beta keeps the model closer to the reference model, but in TRL a higher beta does (F-W1-DOCS-TRAIN-07)#880
Labels
documentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-09Found by the September 2026 independent reviewwave: 3Roadmap wave 3: medium severity, module sweeps
Milestone
Backlog — Wave 3: medium severity (module sweeps)

Description

@cemililik
cemililik
opened on Oct 9, 2026

Wave 3 · group W3 Docs · highest severity Medium · 1 finding(s) · effort docs fix

Summary

The DPO page tells users that a lower dpo_beta keeps the policy closer to the reference model and a higher one makes a more aggressive preference shift, and advises lowering beta when evaluation regresses. TRL, which receives the value unchanged, defines the opposite: a higher β means less deviation from the reference model. A user who follows the tip to fix over-fitting weakens the KL regularisation and moves the model further from the reference.

Primary finding

F-W1-DOCS-TRAIN-07 — DPO beta guidance reverses the regularisation direction

severity Medium · confidence high · checked against the installed TRL source and its reference docs

What happens. The parameter table (docs/usermanuals/en/training/dpo.md:111), the tuning table and the tip (docs/usermanuals/en/training/dpo.md:136-145) describe a lower β as staying close to the reference and a higher β as more aggressive; the Turkish mirror says the same (docs/usermanuals/tr/training/dpo.md:102, docs/usermanuals/tr/training/dpo.md:125-134). The CLI wizard prompt repeats it: dpo_beta — temperature / KL strength (lower → closer to reference model) (forgelm/wizard/_collectors.py:380). ForgeLM passes training.dpo_beta to TRL's DPOConfig unchanged (forgelm/trainer.py:732). The site's own wizard text already states the correct direction (site/js/translations.js:1651: "higher beta keeps it close to the reference"), so the site and the manual contradict each other.

Where.

Evidence (re-verified on 2026-10-09).

  • Installed TRL 1.7.1, trl/trainer/dpo_config.py: "Parameter controlling the deviation from the reference model. Higher β means less deviation from the reference model." The official reference says the same: https://huggingface.co/docs/trl/dpo_trainer#trl.DPOConfig
  • forgelm/trainer.py:732 sets kwargs["beta"] = self.config.training.dpo_beta with no transformation.

Recommended fix. Reverse the direction in the parameter table, the tuning table and the tip, in English and Turkish: a lower β weakens the KL penalty and lets the policy move further from the reference (a more aggressive preference shift); a higher β keeps it closer. When evaluation regresses while training loss keeps falling, advise a higher β (or fewer steps, or a lower learning rate) checked against held-out evaluation. Change the CLI wizard prompt to (higher → closer to reference model). Linking TRL's DPOConfig reference from the parameter row keeps the two from drifting again.

Acceptance criteria

  • The English and Turkish DPO pages describe β the way TRL's DPOConfig does, in the parameter table, the tuning table and the tip.
  • The CLI wizard prompt and the site wizard describe the same direction.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • The finding is resolved or explicitly closed with a reason.

Found by the September 2026 independent review of ForgeLM against commit cab2947 and re-verified against development at f94595f on 2026-10-09. Code links point at cab2947; line numbers may have moved since. Finding IDs (F-W1-…) are stable references for this backlog.

Activity

  1. cemililik
    added this to the Backlog — Wave 3: medium severity (module sweeps) milestone on Oct 9, 2026
  2. cemililik
    added
    documentationImprovements or additions to documentation
    severity: mediumReal defect with limited reach or a workaround
    wave: 3Roadmap wave 3: medium severity, module sweeps
    source: review-2026-09Found by the September 2026 independent review
    on Oct 9, 2026
  3. cemililik
    added this to ForgeLM Project Planon Oct 9, 2026
  4. cemililik
    added a parent issue on Oct 9, 2026
  5. cemililik
    added
    good first issueGood for newcomers
    on Oct 9, 2026

  6. cemililik commented on Oct 11, 2026

    @cemililik
    cemililik
    on Oct 11, 2026
    CollaboratorAuthor

    Where to start

    Thanks for taking a look. Please confirm the issue is unassigned and comment before starting so work does not overlap.

    Start with these files on development:

    Reverse the incorrect directional guidance: higher beta means less deviation from the reference under the supported TRL DPO contract. Correct the wizard prompt, table and tuning prose in EN/TR. Add a wizard prompt regression so the reversed advice cannot return; keep numeric defaults/trainer arguments unchanged.

    Counted as one code/output fix with regression, not counted again among the docs-only tasks. Coordinate with #642.

    Run the focused tests:

    pytest tests/test_wizard_phase22.py --no-cov

    Run these guards/checks as applicable to the changed files:

    ruff format --check forgelm/wizard/_collectors.py tests/test_wizard_phase22.py
ruff check forgelm/wizard/_collectors.py tests/test_wizard_phase22.py
python3 tools/check_bilingual_parity.py --strict
python3 tools/check_bilingual_code_blocks.py --strict
python3 tools/check_anchor_resolution.py --strict
python3 tools/check_tr_links_prefer_mirror.py --strict
python3 tools/check_usermanual_self_contained.py --strict
python3 tools/check_usermanual_schema_drift.py --strict

    If user-facing documentation changes, update the existing Turkish mirror in the same PR; keep commands, code and terminal output in English and structurally aligned. English-only contributor standards need no new mirror.

    Open the PR against development with Fixes #880 once every finding is resolved or explicitly closed with a reason; use Refs #880 for partial work and list the resolved finding IDs.

  7. cemililik
    added
    first-timers-onlyReserved for a contributor's very first pull request
    and removed
    first-timers-onlyReserved for a contributor's very first pull request
    on Oct 17, 2026

  8. cemililik commented on Oct 17, 2026

    @cemililik
    cemililik
    on Oct 17, 2026
    CollaboratorAuthor

    Update — re-verified on development (0766e94). The defect still exists, and TRL's
    own documentation agrees with the issue: a higher beta means less deviation from the
    reference model.

    Besides the parameter row, rewrite the "When to use" column, the > 0.5 row and both
    sentences of the tuning tip to the TRL direction, in English and Turkish (higher beta =
    closer to the reference; to move further away, lower it). The regression test must capture
    the prompt text, because test_dpo_returns_dpo_beta only asserts the return value.
    site/js/translations.js already says the right thing. docs/guides/alignment.md and
    its Turkish mirror ("lower = stronger preference signal") are optional to reword.

    This is a code, test and two-language change rather than a one-file edit, so the
    first-timers-only label has been removed; it stays a good first issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomersseverity: mediumReal defect with limited reach or a workaroundsource: review-2026-09Found by the September 2026 independent reviewwave: 3Roadmap wave 3: medium severity, module sweeps

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions