Repository navigation
product_strategy.md safety claims: trend 'trended to catch drift' has no implementation; nonexistent auto_revert_on_safety_failure field; wrong revert semantics (CR-A06-014) #359
Description
Wave 1 · theme D1-11: Other high-severity packages · highest severity High · 1 finding(s) · effort trivial
Summary
A reader copies auto_revert_on_safety_failure: true into YAML and gets an exit-1 config error, or relies on ForgeLM to flag gradual safety drift across runs, which it never does.
Primary finding
CR-A06-014 — product_strategy.md safety claims: trend 'trended to catch drift' has no implementation; nonexistent auto_revert_on_safety_failure field; wrong revert semantics
severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C31-DOC-SCHEMA-PROSE — docs/roadmap/phase-16-trust-surface-hardening.md:773,785-786 (S15 open)
What happens. docs/product_strategy.md:23 says safety scores are 'trended over time to catch gradual drift', but code only appends rows to safety_trend.jsonl and nothing reads them. Line 24 presents auto_revert_on_safety_failure as a first-class config field. It does not exist, and EvaluationConfig is extra='forbid', so writing it is an exit-1 config error. Line 22 says output is 'rejected before artifacts are written' and that the regression is detected 'against baseline'. The safety gate is absolute (max_safety_regression, _gates.py:255-261) and auto-revert deletes artefacts after they are written (trainer.py:960-1000). The TR mirror (product_strategy-tr.md:24) repeats the field claim.
Where.
docs/product_strategy.md:24docs/product_strategy.md:23docs/product_strategy-tr.md:23forgelm/safety/_results.py:137-178forgelm/safety/_gates.py:252-261forgelm/trainer.py:960-998forgelm/config.py:1188-1194forgelm/trainer.py:1619forgelm/trainer.py:1646forgelm/trainer.py:1446-1466
Evidence.
rg safety_trend forgelm toolsfinds only the writer, a re-export and guard docstrings.rg auto_revert_on_safety_failurefinds only product_strategy(.md|-tr.md). trend.py confirms append-only rows.
Recommended fix. Reword to 'appended per run to safety_trend.jsonl for operator-side trending', replace the field name with evaluation.auto_revert, and describe deletion-after-write with an absolute ceiling.
Acceptance criteria
- A regression test reproduces the failure scenario and asserts the corrected behaviour.
- Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
- Every finding in this issue is resolved or explicitly closed with a reason.
Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity