Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ All notable changes to ForgeLM are documented here.

### Fixed

- Correct claims in the roadmap and strategy documents: `forgelm chat` does not screen for safety and reads no YAML
config (#91), the live risk rows no longer name a past "August 2026" deadline (#75), and
`docs/product_strategy.md` (English and Turkish) names the real `evaluation.auto_revert` field instead of a
non-existent `auto_revert_on_safety_failure`.
- Fix the KTO bullet on the Alignment overview user-manual page (English and Turkish): it named the
fields `(prompt, response, good?)`, but the KTO loader reads `prompt`, `completion` and a boolean
`label` (#965).
Expand Down
2 changes: 1 addition & 1 deletion docs/product_strategy-tr.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ForgeLM, LLM ince ayarını (fine-tuning) **güvenli, denetlenebilir ve tekrarla
- **3 katmanlı güvenlik kapısı**: ikili geçer/kalır + güven ağırlıklı skor + 14 zarar kategorisinde (S1-S14) önem analizi
- **Otomatik geri alma (auto-revert)**: güvenlik regresyonu tespit edilirse, çıktılar yazılmadan önce eğitim sonucu otomatik olarak reddedilir
- **Çalıştırmalar arası trend izleme**: güvenlik skorları her çalıştırmada loglanır; zamanla oluşan kayan bozulma yakalanır
- **`auto_revert_on_safety_failure`** config alanı doğrudan YAML'a yazılır — CI'da güvenlik kapısı uygulamak için özel script gerekmez
- **`evaluation.auto_revert`** config alanı doğrudan YAML'a yazılır — bir güvenlik, benchmark ya da judge kapısı başarısız olduğunda çalışma eğitilmiş çıktıları silip 3 koduyla çıkabilir; CI'da kapıları uygulamak için özel script gerekmez

### 2. Uyumluluk Yerleşik (Compliance-Native)

Expand Down
2 changes: 1 addition & 1 deletion docs/product_strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Fine-tuning can degrade a model's safety properties — silently. ForgeLM treats
- **3-layer safety gate**: binary pass/fail + confidence-weighted score + per-category severity analysis covering 14 harm categories (S1-S14)
- **Auto-revert**: if safety regression is detected against baseline, training output is automatically rejected before artifacts are written
- **Cross-run trend tracking**: safety scores are logged per-run and trended over time to catch gradual drift
- **`auto_revert_on_safety_failure`** is a first-class config field — teams can enforce safety gates in CI without any custom scripting
- **`evaluation.auto_revert`** is a first-class config field — when a safety, benchmark or judge gate fails, the run can delete the trained artefacts and exit with code 3, so teams can enforce the gates in CI without any custom scripting

### 2. Compliance-Native

Expand Down
16 changes: 9 additions & 7 deletions docs/roadmap/completed-phases.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Completed Phases — Archive

> **Note:** Historical archive of ForgeLM's Phase 1-10 completed work. Kept for reference and onboarding. For active planning, see [../roadmap.md](../roadmap.md).
>
> **Dates.** Where this archive says the EU AI Act is "fully enforced" in "August 2026", that was the planning assumption when the phase was written and is kept as the record. The application dates have since changed and differ by obligation; the current statement is in [../product_strategy.md](../product_strategy.md), and there is no single full-enforcement date.

## Summary table

Expand Down Expand Up @@ -661,18 +663,18 @@ graph TD
```

2. [x] **`forgelm chat` — interactive terminal loop**
Terminal REPL in `forgelm/chat.py`: streaming output (default), `/reset`, `/save [file]`, `/temperature 0.x`, `/system <prompt>`, `/help` commands. `rich` optional rendering. History management with 50-pair cap. The `--safety` per-turn screen flag is planned for v0.6.0+ Pro CLI (today's safety pipeline runs through the YAML `safety:` block during training/eval).
Terminal REPL in `forgelm/chat.py`: streaming output (default), `/reset`, `/save [file]`, `/temperature 0.x`, `/system <prompt>`, `/help` commands. `rich` optional rendering. History management with 50-pair cap. `forgelm chat` performs no safety screening and reads no YAML config; the `--safety` per-turn screen flag is not implemented. Screen a model with `forgelm safety-eval --model <path> --default-probes`, or with the `evaluation.safety` block during training, before exposing it.
```bash
forgelm chat ./outputs/my_run
```

`forgelm chat`'s shipping form takes a positional checkpoint path
plus optional `--adapter`. The per-turn Llama Guard screen via
`--safety` is planned for v0.6.0+ Pro CLI (see Phase 13 roadmap);
today operators get the same screening by enabling
`safety: enabled: true` in the YAML config the chat REPL reads.
The flag-form preview (`forgelm chat ... --safety`) is NOT
runnable today.
plus optional `--adapter`. It performs no safety screening and does
not read a YAML config: a per-turn Llama Guard screen via `--safety`
is not implemented, and the flag-form preview
(`forgelm chat ... --safety`) is NOT runnable. Screen a model with
`forgelm safety-eval --model <path> --default-probes` (or the
`evaluation.safety` block during training) before exposing it.

3. [x] **`forgelm export` — HF → GGUF conversion**
Wraps `llama-cpp-python`'s `convert_hf_to_gguf.py`; no reimplementation. Handles adapter merge (LoRA + base → merged fp16) before conversion. Supports quants: `q2_k`, `q3_k_m`, `q4_k_m`, `q5_k_m`, `q8_0`, `f16`. SHA-256 of output artifact appended to `model_integrity.json`. Optional dependency: `pip install forgelm[export]`.
Expand Down
6 changes: 3 additions & 3 deletions docs/roadmap/risks-and-decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@
| Risk | Impact | Likelihood | Mitigation |
|------|--------|------------|------------|
| **Dependency Breaking Changes** (TRL, PEFT, Unsloth) | Training pipeline breaks without warning | High | Version pinning with upper bounds, CI nightly builds against latest deps, compatibility matrix |
| **EU AI Act Non-Compliance** (August 2026 deadline) | Enterprise customers cannot adopt ForgeLM for high-risk AI | High | Phase 8 deep compliance: Annex IV docs, audit log, risk assessment, human gate, data governance |
| **EU AI Act Non-Compliance** (Chapter III, Sections 1–3, except Article 6(5), apply from 2 December 2027 to high-risk AI systems under Article 6(2) and Annex III, and from 2 August 2028 to systems under Article 6(1) and Annex I; see [product_strategy.md](../product_strategy.md)) | Enterprise customers cannot adopt ForgeLM for high-risk AI | High | Phase 8 deep compliance: Annex IV docs, audit log, risk assessment, human gate, data governance |
| **Safety Degradation from Fine-Tuning** | Fine-tuned models lose alignment, enterprise liability | High | Phase 6 safety evaluation pipeline, auto-revert on safety regression |
| **Alignment Method Lock-In** | ForgeLM supports only ORPO while market demands DPO/GRPO | High | Phase 5 is top priority — critical market expectation |
| **Compliance window closing** (Aug 2, 2026) | Enterprise leads not converted before competitors add compliance features | High | Enterprise outreach Day 1; Turkey market as first pilot; deep compliance features (Audit API, managed cloud) as moat |
| **Compliance window** (Annex III from 2 December 2027, Annex I from 2 August 2028) | Enterprise leads not converted before competitors add compliance features | High | Enterprise outreach Day 1; Turkey market as first pilot; deep compliance features (Audit API, managed cloud) as moat |
| **Community flywheel not started** | 0 stars → 0 enterprise credibility → missed EU AI Act window | High | Phase 10.5 Quickstart prioritized; YouTube Academy; Show HN + Reddit launch |

### Medium Severity
Expand Down Expand Up @@ -39,7 +39,7 @@
2. **Safety-as-a-Feature (Phase 6)** — No competitor integrates safety evaluation into the training pipeline. ForgeLM can own "the safest way to fine-tune an LLM."

### Medium-Term Opportunities
3. **EU AI Act Compliance** — August 2026 deadline creates urgent demand. ForgeLM as the only tool generating compliance artifacts is a powerful enterprise sales argument.
3. **EU AI Act Compliance** — the staggered high-risk application dates (2 December 2027 for Annex III, 2 August 2028 for Annex I) create demand. ForgeLM as the only tool generating compliance artifacts is a powerful enterprise sales argument.
4. **MoE Fine-Tuning** — Qwen3, DeepSeek-V3 dominance means MoE support is becoming table stakes.
5. **Cost Transparency** — GPU cost tracking per run enables enterprise budget planning and optimization. Simple to implement, high perceived value.

Expand Down
Loading