Skip to content

product_strategy.md safety claims: trend 'trended to catch drift' has no implementation; nonexistent auto_revert_on_safety_failure field; wrong revert semantics (CR-A06-014) #359

Open

Description

@cemililik

Wave 1 · theme D1-11: Other high-severity packages · highest severity High · 1 finding(s) · effort trivial

Summary

A reader copies auto_revert_on_safety_failure: true into YAML and gets an exit-1 config error, or relies on ForgeLM to flag gradual safety drift across runs, which it never does.

Primary finding

CR-A06-014 — product_strategy.md safety claims: trend 'trended to catch drift' has no implementation; nonexistent auto_revert_on_safety_failure field; wrong revert semantics

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C31-DOC-SCHEMA-PROSE — docs/roadmap/phase-16-trust-surface-hardening.md:773,785-786 (S15 open)

What happens. docs/product_strategy.md:23 says safety scores are 'trended over time to catch gradual drift', but code only appends rows to safety_trend.jsonl and nothing reads them. Line 24 presents auto_revert_on_safety_failure as a first-class config field. It does not exist, and EvaluationConfig is extra='forbid', so writing it is an exit-1 config error. Line 22 says output is 'rejected before artifacts are written' and that the regression is detected 'against baseline'. The safety gate is absolute (max_safety_regression, _gates.py:255-261) and auto-revert deletes artefacts after they are written (trainer.py:960-1000). The TR mirror (product_strategy-tr.md:24) repeats the field claim.

Where.

Evidence.

  • rg safety_trend forgelm tools finds only the writer, a re-export and guard docstrings. rg auto_revert_on_safety_failure finds only product_strategy(.md|-tr.md). trend.py confirms append-only rows.

Recommended fix. Reword to 'appended per run to safety_trend.jsonl for operator-side trending', replace the field name with evaluation.auto_revert, and describe deletion-after-write with an absolute ceiling.

Acceptance criteria

  • A regression test reproduces the failure scenario and asserts the corrected behaviour.
  • Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
  • Every finding in this issue is resolved or explicitly closed with a reason.

Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: eval-gatesCode area: eval-gatesbacklog: knownAlready tracked in docs/roadmapblockedWaits for another issue (see Blocked by); set and removed automatically, never by handbugSomething isn't workingseverity: highWrong behaviour or a false claim users rely onsource: review-2026-10Found by the October 2026 code and documentation reviewwave: 1Roadmap wave 1: high severity, small effort

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions