cemililik · Oct 18, 2026
@@ -68,6 +68,13 @@ All notable changes to ForgeLM are documented here.
 
 ### Fixed
 
+- Document the one case where a failed safety gate still exits 0 with `auto_revert: true`: an evaluation that produced no
+  verdict (`safety.evaluation_completed: false` — classifier not loaded, probes file missing or empty, or most verdicts
+  unreadable) keeps the model and the run exits 0, or 4 with the approval gate. The error-handling standard, the exit-codes and
+  JSON-output pages, the audit-event catalogue, the safety guide, the deployer guide and the risk-treatment plan said that exit
+  0 meant every gate passed, that a classifier load failure "halts" the pipeline and leaves no `final_model/`, and that
+  classification errors count as unsafe. The behaviour is unchanged; making it fail closed is a breaking exit-code change
+  left for a major version (#87).
 - Fix the KTO bullet on the Alignment overview user-manual page (English and Turkish): it named the
   fields `(prompt, response, good?)`, but the KTO loader reads `prompt`, `completion` and a boolean
   `label` (#965).

@@ -249,8 +249,11 @@ Referans:
 
 - `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23
   genişletme) güvenlik-incident playbook için.
-- Zincirin kendisi: `audit.classifier_load_failed` event yangın +
-  `pipeline.failed` propagate + koşum `final_model/` dizini ÜRETMEZ.
+- Zincirin kendisi: `audit.classifier_load_failed` event'i tetiklenir
+  ve `safety.evaluation_completed` `evaluation_completed: false` kaydeder.
+  Koşu **durdurulmaz**: model `final_model/` içinde kalır ve koşu 0 ile
+  çıkar (onay kapısıyla 4); bu yüzden exit koduna değil, bu event'lere ya da
+  JSON zarfının `safety` bloğuna göre alarm kurun.
 - F-compliance-110 strict gate: `risk_classification ∈ {high-risk,
   unacceptable}` AND `evaluation.safety.enabled = false` ise config
   validator koşum başlamadan ÖNCE `ConfigError` raise eder. Denetçi

@@ -248,9 +248,11 @@ Reference:
 
 - `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 expansion)
   for the security-incident playbook.
-- The audit chain itself: `audit.classifier_load_failed` event fires
-  + `pipeline.failed` propagates + the run does NOT produce a
-  `final_model/` directory.
+- The audit chain itself: the `audit.classifier_load_failed` event fires
+  and `safety.evaluation_completed` records `evaluation_completed: false`.
+  The run is **not** halted: the model stays in `final_model/` and the
+  run exits 0 (4 with the approval gate), so alert on those events or on
+  the JSON envelope's `safety` block, not on the exit code.
 - F-compliance-110 strict gate: if `risk_classification ∈
   {high-risk, unacceptable}` AND `evaluation.safety.enabled = false`,
   the config validator raises `ConfigError` BEFORE the run starts.

@@ -95,8 +95,7 @@ Kapı 2: Confidence-weighted   → safety_score < min_safety_score?
 Kapı 3: Şiddet               → critical_count > 0? high_count > eşik?
 ```
 
-Üç kapı bağımsızdır. Herhangi bir kapının başarısızlığı (etkinse)
-auto-revert tetikler.
+Üç kapı bağımsızdır. Bir kapının başarısızlığı (etkinse) auto-revert tetikler; güvenlik değerlendirmesi hiç karar üretemediyse bu geçerli değildir (bkz. [Fail-safe davranış](#fail-safe-davranış)).
 
 ### Zarar kategorileri (Llama Guard 3)
 
@@ -204,13 +203,12 @@ iyileştirip bozmadığını izlemesine yardım eder.
 
 ### Fail-safe davranış
 
-- Sınıflandırma hataları **güvensiz** olarak değerlendirilir (fail-safe
-  prensibi)
+- Sınıflandırıcının yanıtlayamadığı bir çift **skorlanmamış** sayılır, güvenli değil
 - Düşük-güvenli sınıflandırmalar manuel inceleme için işaretlenir
   (uyarı log'u)
-- Güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise
-  model otomatik olarak silinir
-- Hat entegrasyonu için `3` exit kodu döner
+- Skorlanmış bir güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise
+  model otomatik olarak silinir ve hat entegrasyonu için `3` exit kodu döner
+- **Değerlendirme karar üretemezse** (`safety.evaluation_completed: false` — sınıflandırıcı yüklenemedi, probe dosyası yok ya da boş, ya da çoğu karar okunamadı), model `auto_revert: true` olsa bile **korunur** ve koşu `0` ile çıkar (onay kapısıyla `4`); çünkü okunmamış bir karar zarar kanıtı değildir. Yalnızca exit koduna değil, JSON çıktısındaki `safety.evaluation_completed` ve `safety.passed`'e bakın
 
 ---
 
@@ -522,6 +520,8 @@ Herhangi bir adım başarısız olursa ve `auto_revert: true` ise:
 Model silinir + Webhook bildirimi (başarısızlık) + Exit kodu 3
 ```
 
+Karar üretemeyen bir güvenlik değerlendirmesi (`evaluation_completed: false`) istisnadır: model korunur ve koşu 0 ile çıkar.
+
 ### İnsan onay kapısı (Madde 14)
 
 Tüm otomatik kontroller geçtikten sonra hattı duraklatmak için

@@ -77,7 +77,7 @@ Gate 2: Confidence-weighted → safety_score < min_safety_score?
 Gate 3: Severity           → critical_count > 0? high_count > threshold?
 ```
 
-All three gates are independent. Any gate failure triggers auto-revert (if enabled).
+All three gates are independent. A gate failure triggers auto-revert (if enabled), unless the safety evaluation produced no verdict at all (see [Fail-Safe Behavior](#fail-safe-behavior)).
 
 ### Harm Categories (Llama Guard 3)
 
@@ -179,10 +179,10 @@ This helps teams track whether data/config changes are improving or degrading sa
 
 ### Fail-Safe Behavior
 
-- Classification errors are treated as **unsafe** (fail-safe principle)
+- A pair the classifier cannot answer is counted as **unscored**, not as safe
 - Low-confidence classifications are flagged for manual review (warning log)
-- If safety evaluation fails and `auto_revert: true`, the model is automatically deleted
-- Exit code `3` is returned for pipeline integration
+- If a scored safety evaluation fails and `auto_revert: true`, the model is automatically deleted and exit code `3` is returned for pipeline integration
+- **If the evaluation produces no verdict** (`safety.evaluation_completed: false` — the classifier failed to load, the probes file is missing or empty, or most verdicts were unreadable), the model is **kept** and the run exits `0` (`4` with the approval gate) even with `auto_revert: true`, because an unread verdict is not evidence of harm. Branch on `safety.evaluation_completed` and `safety.passed` in the JSON output, not on the exit code alone
 
 ---
 
@@ -461,12 +461,14 @@ Model saved + Model card + Compliance artifacts
 Webhook notification (success)
 ```
 
-If any step fails and `auto_revert: true`:
+If a gate fails with a verdict and `auto_revert: true`:
 ```
   ↓ fail
 Model deleted + Webhook notification (failure) + Exit code 3
 ```
 
+A safety evaluation that produced no verdict (`evaluation_completed: false`) is the exception: the model is kept and the run exits 0.
+
 ### Human Approval Gate (Art. 14)
 
 Add `require_human_approval: true` to pause the pipeline after all automated checks pass:

@@ -103,7 +103,7 @@ ekler.
 |---|---|
 | Açıklama | Llama Guard / yapılandırılmış safety classifier yüklenemiyor (HF Hub down, OOM, version mismatch); ForgeLM gate yangın etmeden devam eder |
 | L × I (inherent) | Med × High = HIGH |
-| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` `ConfigError` raise eder; `audit.classifier_load_failed` event arızayı kaydeder; operatör pipeline'ı non-zero exit'te durur |
+| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` `ConfigError` raise eder; `audit.classifier_load_failed` event arızayı kaydeder. Koşu **durdurulmaz**: karar üretemeyen bir güvenlik değerlendirmesi modeli korur ve 0 ile çıkar (onay kapısıyla 4), bu yüzden operatör pipeline'ı yalnızca exit koduna değil, JSON zarfındaki `safety.evaluation_completed` / `safety.passed`'e ya da o audit event'ine bakmalıdır |
 | Residual L × I | Low × Low = LOW |
 | Sahip | ML Lead |
 | Review cadence | Eğitim koşumu başına |

@@ -102,7 +102,7 @@ section §4.
 |---|---|
 | Description | Llama Guard / configured safety classifier fails to load (HF Hub down, OOM, version mismatch); ForgeLM proceeds without the gate fired |
 | L × I (inherent) | Med × High = HIGH |
-| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` raises `ConfigError`; `audit.classifier_load_failed` event records the failure; deployer pipeline halts on non-zero exit |
+| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` raises `ConfigError`; `audit.classifier_load_failed` event records the failure. The run is **not** halted: a safety evaluation with no verdict keeps the model and exits 0 (4 with the approval gate), so the deployer pipeline must branch on the JSON envelope's `safety.evaluation_completed` / `safety.passed` or on that audit event, not on the exit code alone |
 | Residual L × I | Low × Low = LOW |
 | Owner | ML Lead |
 | Review cadence | Per training run |

@@ -134,7 +134,7 @@ Lightweight `TrainResult` dataclass — importable without torch/transformers. C
 Wraps EleutherAI `lm-evaluation-harness`. Runs configurable benchmark tasks, extracts accuracy metrics, applies min_score threshold, and saves results. Optional dependency: `pip install forgelm[eval]`.
 
 ### `safety/`
-Runs a configurable safety classifier (Llama Guard, ShieldGemma) on adversarial test prompts. Generates responses from the fine-tuned model, classifies each as safe/unsafe, and triggers auto-revert if regression exceeds threshold. Errors are treated as unsafe (fail-safe principle).
+Runs a configurable safety classifier (Llama Guard, ShieldGemma) on adversarial test prompts. Generates responses from the fine-tuned model, classifies each as safe/unsafe, and triggers auto-revert if regression exceeds threshold. A pair the classifier cannot answer is counted as unscored, not as safe; when the classifier does not load, the probes file is missing or empty, or most verdicts are unreadable, the evaluation produces no verdict (`evaluation_completed: false`) and the model is kept (the run exits 0).
 
 ### `judge.py`
 LLM-as-Judge evaluation supporting API-based judges (OpenAI-compatible endpoint) and local model judges. Includes robust JSON parsing with markdown code block extraction. Scores on 1-10 scale with configurable minimum threshold.

@@ -161,7 +161,7 @@ damgasıyla kurabilir. Uygulama: `forgelm/webhook.py`.
 | Webhook `event` | Denetim günlüğü karşılığı | Tetikleyici | Kapı (gate) | Zorunlu payload alanları |
 |---|---|---|---|---|
 | `training.start` | `training.started` | `train()` çağrıldı, model yüklenmeden önce. | `webhook.notify_on_start` | `run_name`, `status="started"` |
-| `training.success` | `pipeline.completed` | Koşu revert veya bekleyen onay olmadan tamamlandı. `evaluation.auto_revert: true` ile tüm kapılar geçildi; varsayılan `auto_revert: false` ile bir kapı geçemediği halde yalnızca kaydedildiğinde de tetiklenir (model yine terfi eder). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` |
+| `training.success` | `pipeline.completed` | Koşu revert veya bekleyen onay olmadan tamamlandı. `evaluation.auto_revert: true` ile karara varan tüm kapılar geçildi (karar üretemeyen bir güvenlik değerlendirmesi, `safety.evaluation_completed: false`, modeli korur ve yine buraya varır); varsayılan `auto_revert: false` ile bir kapı geçemediği halde yalnızca kaydedildiğinde de tetiklenir (model yine terfi eder). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` |
 | `training.failure` | `pipeline.failed` | Eğitim sürecinin kendisi hata fırlattı (OOM, veri seti hatası, yakalanmayan istisna). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (maskelenmiş, ≤2048 karakter) |
 | `training.reverted` | `model.reverted` | Eğitim sonrası bir kapı (değerlendirme, güvenlik, hakem, benchmark) çalışmayı reddetti ve `_revert_model` adaptörleri sildi. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (maskelenmiş, ≤2048 karakter) |
 | `approval.required` | `human_approval.required` | Çalışma başarılı oldu, `evaluation.require_human_approval=true`, model insan incelemesi için staging'de (EU AI Act Madde 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` |

@@ -161,7 +161,7 @@ webhook ping → audit entry by `run_name` + timestamp. Implementation:
 | Webhook `event` | Audit-log mirror | Trigger | Gated by | Required payload fields |
 |---|---|---|---|---|
 | `training.start` | `training.started` | `train()` entered, before model load. | `webhook.notify_on_start` | `run_name`, `status="started"` |
-| `training.success` | `pipeline.completed` | Run completed without revert or pending approval. With `evaluation.auto_revert: true` all gates passed; with the default `auto_revert: false` it also fires when a gate failed but was only recorded (model still promoted). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` |
+| `training.success` | `pipeline.completed` | Run completed without revert or pending approval. With `evaluation.auto_revert: true` every gate that produced a verdict passed (a safety evaluation with no verdict, `safety.evaluation_completed: false`, keeps the model and also ends here); with the default `auto_revert: false` it also fires when a gate failed but was only recorded (model still promoted). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` |
 | `training.failure` | `pipeline.failed` | Training itself raised (OOM, dataset error, unhandled exception). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (masked, ≤2048 chars) |
 | `training.reverted` | `model.reverted` | A post-training gate (evaluation, safety, judge, benchmark) rejected the run and `_revert_model` deleted the adapters. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (masked, ≤2048 chars) |
 | `approval.required` | `human_approval.required` | Run succeeded, `evaluation.require_human_approval=true`, model staged for review (EU AI Act Art. 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` |

@@ -19,7 +19,7 @@ EXIT_INTEGRITY_FAILURE = 6
 
 | Code | When | Who reads it |
 |---|---|---|
-| **0** | Training completed. With `auto_revert: true` (the compliance default for high-risk tiers) this also means every gate passed; with the **shipped default** `auto_revert: false` a failed benchmark/safety/judge gate is *recorded* (`benchmark`/`safety`/`judge` block in the JSON, `*_passed: false`) but does **not** change the exit code — parse those blocks, don't trust exit 0 alone | CI/CD success |
+| **0** | Training completed. With `auto_revert: true` (the compliance default for high-risk tiers) this also means every gate that ran to a verdict passed (a safety evaluation that produced no verdict — `safety.evaluation_completed: false` — keeps the model and exits 0 regardless of `auto_revert`); with the **shipped default** `auto_revert: false` a failed benchmark/safety/judge gate is *recorded* (`benchmark`/`safety`/`judge` block in the JSON, `*_passed: false`) but does **not** change the exit code — parse those blocks, don't trust exit 0 alone | CI/CD success |
 | **1** | Config validation failed (YAML schema, Pydantic error). Also used by the `verify-*` subcommands for an operator-actionable input error — the artefact could not be read at all, so nothing was ever compared (plus the one case where everything comparable *was* compared and came out clean; see the 1-vs-6 note below) | CI/CD "fail fast"; user fixes YAML or the verifier's target path |
 | **2** | Training crashed or failed mid-run (OOM, CUDA error, unhandled exception). Also the `verify-*` subcommands' code for a genuine runtime I/O failure on a reachable path (permission denied mid-read, disk error) | CI/CD retry logic |
 | **3** | Training completed but eval/safety/benchmark threshold failed, and auto-revert happened | CI/CD decision: do not deploy |

@@ -11,7 +11,7 @@ ForgeLM's exit codes are a public contract. CI/CD pipelines, schedulers, and das
 
 | Exit | Constant | Meaning | Typical CI action |
 |---|---|---|---|
-| **0** | `EXIT_SUCCESS` | Run completed and the checkpoint was promoted. With `evaluation.auto_revert: true` every gate also passed; with the shipped default `auto_revert: false` a failed benchmark/safety/judge gate is **recorded in the JSON output but does not change the exit code** — see ["What exit 0 actually guarantees"](#what-exit-0-actually-guarantees) below. | Continue pipeline (parse gate blocks if `auto_revert` is off) |
+| **0** | `EXIT_SUCCESS` | Run completed and the checkpoint was promoted. With `evaluation.auto_revert: true` every gate that produced a verdict also passed (a safety evaluation with no verdict keeps the model and exits 0 too); with the shipped default `auto_revert: false` a failed benchmark/safety/judge gate is **recorded in the JSON output but does not change the exit code** — see ["What exit 0 actually guarantees"](#what-exit-0-actually-guarantees) below. | Continue pipeline (parse gate blocks if `auto_revert` is off) |
 | **1** | `EXIT_CONFIG_ERROR` | YAML invalid, file missing, env var unset, or argument malformed. | Fail fast |
 | **2** | `EXIT_TRAINING_ERROR` | Training-time runtime error (any unhandled exception that isn't a config or eval-gate failure: data load, OOM, NaN loss, I/O failure, mid-stream audit-iteration OSError). | Investigate; surface logs |
 | **3** | `EXIT_EVAL_FAILURE` | A benchmark/safety/judge gate failed **and** the model was auto-reverted (requires `evaluation.auto_revert: true`). With `auto_revert: false` a failed gate does not produce exit 3 — the run exits 0 with the failure recorded in the JSON gate blocks. | Investigate; do NOT promote |
@@ -112,7 +112,7 @@ A run that exits 0 has:
 
 **Gates and exit 0.** Whether a *passed* benchmark/safety/judge gate is part of the exit-0 guarantee depends on `evaluation.auto_revert`:
 
-- With `evaluation.auto_revert: true` (the EU AI Act high-risk default), a failed gate auto-reverts the model and exits **3** — so exit 0 *does* mean every configured gate passed.
+- With `evaluation.auto_revert: true` (the EU AI Act high-risk default), a failed gate auto-reverts the model and exits **3** — so exit 0 means every configured gate that produced a verdict passed. **One exception:** a safety evaluation that produced no verdict (`safety.evaluation_completed: false` — the classifier failed to load, the probes file is missing or empty, or most verdicts were unreadable) keeps the model and exits 0 (4 with the approval gate) even with `auto_revert: true`; check `safety.passed` and `safety.evaluation_completed` in the JSON output.
 - With the shipped default `evaluation.auto_revert: false`, a failed gate is **recorded** (the `benchmark` / `safety` / `judge` block in the JSON output carries `*_passed: false`) but the model is still promoted and the run exits **0**. Read those JSON blocks; do not infer gate success from exit 0 alone.
 
 There is no "partial success" exit code by design — turn on `auto_revert` if you want a failing gate to change the exit code.