Skip to content

docs/product_strategy.md: false claim — Auto-revert: if safety regression is detected against baseline, training output is automatically rejected before artifacts are written (DOC-D2B09-001) #208

Open

Description

@cemililik

Wave 1 · theme D1-10: Documentation-only fixes · highest severity High · 1 finding(s) · effort docs fix

Summary

Operators expect a baseline-relative regression check and a guarantee that nothing is written. In fact the run is gated on an absolute threshold, the model is written and then deleted, and the whole mechanism is off unless evaluation.auto_revert is set.

Primary finding

DOC-D2B09-001 — docs/product_strategy.md: false claim — Auto-revert: if safety regression is detected against baseline, training output is automatically rejected before artifacts are written

severity High · already tracked: KNOWN-OPEN DOCS-03/DOCS-04 (S15 auto-revert) — docs/roadmap/phase-16-trust-surface-hardening.md:760-765

Where. docs/product_strategy.md:22 (Turkish mirror: docs/product_strategy-tr.md:22) · type: false claim

The document says. Auto-revert: if safety regression is detected against baseline, training output is automatically rejected before artifacts are written

What the code does. There is no baseline comparison. max_safety_regression is an absolute ceiling on the current run's unsafe ratio. Auto-revert is opt-in (default false) and runs after the model has been saved: it deletes the saved final_model directory with shutil.rmtree and emits model.reverted.

Code references. forgelm/safety/_gates.py:255-262, forgelm/config.py:940-945, forgelm/config.py:1191-1194, forgelm/trainer.py:960-1005

User impact. Operators expect a baseline-relative regression check and a guarantee that nothing is written. In fact the run is gated on an absolute threshold, the model is written and then deleted, and the whole mechanism is off unless evaluation.auto_revert is set.

Proposed correction. Rewrite as: "Auto-revert (opt-in, evaluation.auto_revert: true): when the safety gate fails (unsafe-response ratio above the absolute max_safety_regression ceiling, or another configured gate), the saved model directory is deleted and the revert is recorded in the audit log."

Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

  • Each statement is re-checked against the code at fix time.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • Every finding in this issue is resolved or explicitly closed with a reason.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Metadata

Metadata

Assignees

No one assigned

    Labels

    backlog: knownAlready tracked in docs/roadmapblockedWaits for another issue (see Blocked by); set and removed automatically, never by handdocumentationImprovements or additions to documentationseverity: highWrong behaviour or a false claim users rely onsource: review-2026-10Found by the October 2026 code and documentation reviewwave: 1Roadmap wave 1: high severity, small effort

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions