From 9f5f699772dededdb2e7d5e924865b53fc57c5ae Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 18 Oct 2026 10:20:48 +0300 Subject: [PATCH] docs(compliance): say that a safety evaluation with no verdict keeps the model and exits 0 Exit 0 was documented as "every gate passed", a classifier load failure as halting the pipeline with no final_model/, and classification errors as unsafe. In code an evaluation that produces no verdict (evaluation_completed=False: classifier not loaded, probes file missing or empty, most verdicts unreadable) keeps the model and exits 0 (4 with the approval gate) regardless of auto_revert. The behaviour is unchanged; making it fail closed is a breaking exit-code change left for a major version. Refs #87 Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 7 +++++++ docs/guides/iso_soc2_deployer_guide-tr.md | 7 +++++-- docs/guides/iso_soc2_deployer_guide.md | 8 +++++--- docs/guides/safety_compliance-tr.md | 14 +++++++------- docs/guides/safety_compliance.md | 12 +++++++----- docs/qms/risk_treatment_plan-tr.md | 2 +- docs/qms/risk_treatment_plan.md | 2 +- docs/reference/architecture.md | 2 +- docs/reference/audit_event_catalog-tr.md | 2 +- docs/reference/audit_event_catalog.md | 2 +- docs/standards/error-handling.md | 2 +- docs/usermanuals/en/reference/exit-codes.md | 4 ++-- docs/usermanuals/en/reference/json-output.md | 2 +- docs/usermanuals/tr/reference/exit-codes.md | 4 ++-- docs/usermanuals/tr/reference/json-output.md | 2 +- 15 files changed, 43 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d44dd02..5780dedc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,13 @@ All notable changes to ForgeLM are documented here. ### Fixed +- Document the one case where a failed safety gate still exits 0 with `auto_revert: true`: an evaluation that produced no + verdict (`safety.evaluation_completed: false` — classifier not loaded, probes file missing or empty, or most verdicts + unreadable) keeps the model and the run exits 0, or 4 with the approval gate. The error-handling standard, the exit-codes and + JSON-output pages, the audit-event catalogue, the safety guide, the deployer guide and the risk-treatment plan said that exit + 0 meant every gate passed, that a classifier load failure "halts" the pipeline and leaves no `final_model/`, and that + classification errors count as unsafe. The behaviour is unchanged; making it fail closed is a breaking exit-code change + left for a major version (#87). - Fix the KTO bullet on the Alignment overview user-manual page (English and Turkish): it named the fields `(prompt, response, good?)`, but the KTO loader reads `prompt`, `completion` and a boolean `label` (#965). diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index fd52e63e..afdd7f66 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -249,8 +249,11 @@ Referans: - `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 genişletme) güvenlik-incident playbook için. -- Zincirin kendisi: `audit.classifier_load_failed` event yangın + - `pipeline.failed` propagate + koşum `final_model/` dizini ÜRETMEZ. +- Zincirin kendisi: `audit.classifier_load_failed` event'i tetiklenir + ve `safety.evaluation_completed` `evaluation_completed: false` kaydeder. + Koşu **durdurulmaz**: model `final_model/` içinde kalır ve koşu 0 ile + çıkar (onay kapısıyla 4); bu yüzden exit koduna değil, bu event'lere ya da + JSON zarfının `safety` bloğuna göre alarm kurun. - F-compliance-110 strict gate: `risk_classification ∈ {high-risk, unacceptable}` AND `evaluation.safety.enabled = false` ise config validator koşum başlamadan ÖNCE `ConfigError` raise eder. Denetçi diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md index 3132ef96..c32ae3f4 100644 --- a/docs/guides/iso_soc2_deployer_guide.md +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -248,9 +248,11 @@ Reference: - `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 expansion) for the security-incident playbook. -- The audit chain itself: `audit.classifier_load_failed` event fires - + `pipeline.failed` propagates + the run does NOT produce a - `final_model/` directory. +- The audit chain itself: the `audit.classifier_load_failed` event fires + and `safety.evaluation_completed` records `evaluation_completed: false`. + The run is **not** halted: the model stays in `final_model/` and the + run exits 0 (4 with the approval gate), so alert on those events or on + the JSON envelope's `safety` block, not on the exit code. - F-compliance-110 strict gate: if `risk_classification ∈ {high-risk, unacceptable}` AND `evaluation.safety.enabled = false`, the config validator raises `ConfigError` BEFORE the run starts. diff --git a/docs/guides/safety_compliance-tr.md b/docs/guides/safety_compliance-tr.md index 547edf2f..a4ece7f1 100644 --- a/docs/guides/safety_compliance-tr.md +++ b/docs/guides/safety_compliance-tr.md @@ -95,8 +95,7 @@ Kapı 2: Confidence-weighted → safety_score < min_safety_score? Kapı 3: Şiddet → critical_count > 0? high_count > eşik? ``` -Üç kapı bağımsızdır. Herhangi bir kapının başarısızlığı (etkinse) -auto-revert tetikler. +Üç kapı bağımsızdır. Bir kapının başarısızlığı (etkinse) auto-revert tetikler; güvenlik değerlendirmesi hiç karar üretemediyse bu geçerli değildir (bkz. [Fail-safe davranış](#fail-safe-davranış)). ### Zarar kategorileri (Llama Guard 3) @@ -204,13 +203,12 @@ iyileştirip bozmadığını izlemesine yardım eder. ### Fail-safe davranış -- Sınıflandırma hataları **güvensiz** olarak değerlendirilir (fail-safe - prensibi) +- Sınıflandırıcının yanıtlayamadığı bir çift **skorlanmamış** sayılır, güvenli değil - Düşük-güvenli sınıflandırmalar manuel inceleme için işaretlenir (uyarı log'u) -- Güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise - model otomatik olarak silinir -- Hat entegrasyonu için `3` exit kodu döner +- Skorlanmış bir güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise + model otomatik olarak silinir ve hat entegrasyonu için `3` exit kodu döner +- **Değerlendirme karar üretemezse** (`safety.evaluation_completed: false` — sınıflandırıcı yüklenemedi, probe dosyası yok ya da boş, ya da çoğu karar okunamadı), model `auto_revert: true` olsa bile **korunur** ve koşu `0` ile çıkar (onay kapısıyla `4`); çünkü okunmamış bir karar zarar kanıtı değildir. Yalnızca exit koduna değil, JSON çıktısındaki `safety.evaluation_completed` ve `safety.passed`'e bakın --- @@ -522,6 +520,8 @@ Herhangi bir adım başarısız olursa ve `auto_revert: true` ise: Model silinir + Webhook bildirimi (başarısızlık) + Exit kodu 3 ``` +Karar üretemeyen bir güvenlik değerlendirmesi (`evaluation_completed: false`) istisnadır: model korunur ve koşu 0 ile çıkar. + ### İnsan onay kapısı (Madde 14) Tüm otomatik kontroller geçtikten sonra hattı duraklatmak için diff --git a/docs/guides/safety_compliance.md b/docs/guides/safety_compliance.md index 5569369c..6f4f585a 100644 --- a/docs/guides/safety_compliance.md +++ b/docs/guides/safety_compliance.md @@ -77,7 +77,7 @@ Gate 2: Confidence-weighted → safety_score < min_safety_score? Gate 3: Severity → critical_count > 0? high_count > threshold? ``` -All three gates are independent. Any gate failure triggers auto-revert (if enabled). +All three gates are independent. A gate failure triggers auto-revert (if enabled), unless the safety evaluation produced no verdict at all (see [Fail-Safe Behavior](#fail-safe-behavior)). ### Harm Categories (Llama Guard 3) @@ -179,10 +179,10 @@ This helps teams track whether data/config changes are improving or degrading sa ### Fail-Safe Behavior -- Classification errors are treated as **unsafe** (fail-safe principle) +- A pair the classifier cannot answer is counted as **unscored**, not as safe - Low-confidence classifications are flagged for manual review (warning log) -- If safety evaluation fails and `auto_revert: true`, the model is automatically deleted -- Exit code `3` is returned for pipeline integration +- If a scored safety evaluation fails and `auto_revert: true`, the model is automatically deleted and exit code `3` is returned for pipeline integration +- **If the evaluation produces no verdict** (`safety.evaluation_completed: false` — the classifier failed to load, the probes file is missing or empty, or most verdicts were unreadable), the model is **kept** and the run exits `0` (`4` with the approval gate) even with `auto_revert: true`, because an unread verdict is not evidence of harm. Branch on `safety.evaluation_completed` and `safety.passed` in the JSON output, not on the exit code alone --- @@ -461,12 +461,14 @@ Model saved + Model card + Compliance artifacts Webhook notification (success) ``` -If any step fails and `auto_revert: true`: +If a gate fails with a verdict and `auto_revert: true`: ``` ↓ fail Model deleted + Webhook notification (failure) + Exit code 3 ``` +A safety evaluation that produced no verdict (`evaluation_completed: false`) is the exception: the model is kept and the run exits 0. + ### Human Approval Gate (Art. 14) Add `require_human_approval: true` to pause the pipeline after all automated checks pass: diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md index 1ac0e752..e6a9fb26 100644 --- a/docs/qms/risk_treatment_plan-tr.md +++ b/docs/qms/risk_treatment_plan-tr.md @@ -103,7 +103,7 @@ ekler. |---|---| | Açıklama | Llama Guard / yapılandırılmış safety classifier yüklenemiyor (HF Hub down, OOM, version mismatch); ForgeLM gate yangın etmeden devam eder | | L × I (inherent) | Med × High = HIGH | -| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` `ConfigError` raise eder; `audit.classifier_load_failed` event arızayı kaydeder; operatör pipeline'ı non-zero exit'te durur | +| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` `ConfigError` raise eder; `audit.classifier_load_failed` event arızayı kaydeder. Koşu **durdurulmaz**: karar üretemeyen bir güvenlik değerlendirmesi modeli korur ve 0 ile çıkar (onay kapısıyla 4), bu yüzden operatör pipeline'ı yalnızca exit koduna değil, JSON zarfındaki `safety.evaluation_completed` / `safety.passed`'e ya da o audit event'ine bakmalıdır | | Residual L × I | Low × Low = LOW | | Sahip | ML Lead | | Review cadence | Eğitim koşumu başına | diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md index c19a9d35..a768dd9d 100644 --- a/docs/qms/risk_treatment_plan.md +++ b/docs/qms/risk_treatment_plan.md @@ -102,7 +102,7 @@ section §4. |---|---| | Description | Llama Guard / configured safety classifier fails to load (HF Hub down, OOM, version mismatch); ForgeLM proceeds without the gate fired | | L × I (inherent) | Med × High = HIGH | -| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` raises `ConfigError`; `audit.classifier_load_failed` event records the failure; deployer pipeline halts on non-zero exit | +| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` raises `ConfigError`; `audit.classifier_load_failed` event records the failure. The run is **not** halted: a safety evaluation with no verdict keeps the model and exits 0 (4 with the approval gate), so the deployer pipeline must branch on the JSON envelope's `safety.evaluation_completed` / `safety.passed` or on that audit event, not on the exit code alone | | Residual L × I | Low × Low = LOW | | Owner | ML Lead | | Review cadence | Per training run | diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md index 735b3f2b..e3872c43 100644 --- a/docs/reference/architecture.md +++ b/docs/reference/architecture.md @@ -134,7 +134,7 @@ Lightweight `TrainResult` dataclass — importable without torch/transformers. C Wraps EleutherAI `lm-evaluation-harness`. Runs configurable benchmark tasks, extracts accuracy metrics, applies min_score threshold, and saves results. Optional dependency: `pip install forgelm[eval]`. ### `safety/` -Runs a configurable safety classifier (Llama Guard, ShieldGemma) on adversarial test prompts. Generates responses from the fine-tuned model, classifies each as safe/unsafe, and triggers auto-revert if regression exceeds threshold. Errors are treated as unsafe (fail-safe principle). +Runs a configurable safety classifier (Llama Guard, ShieldGemma) on adversarial test prompts. Generates responses from the fine-tuned model, classifies each as safe/unsafe, and triggers auto-revert if regression exceeds threshold. A pair the classifier cannot answer is counted as unscored, not as safe; when the classifier does not load, the probes file is missing or empty, or most verdicts are unreadable, the evaluation produces no verdict (`evaluation_completed: false`) and the model is kept (the run exits 0). ### `judge.py` LLM-as-Judge evaluation supporting API-based judges (OpenAI-compatible endpoint) and local model judges. Includes robust JSON parsing with markdown code block extraction. Scores on 1-10 scale with configurable minimum threshold. diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md index 3068d80d..65c0a1cb 100644 --- a/docs/reference/audit_event_catalog-tr.md +++ b/docs/reference/audit_event_catalog-tr.md @@ -161,7 +161,7 @@ damgasıyla kurabilir. Uygulama: `forgelm/webhook.py`. | Webhook `event` | Denetim günlüğü karşılığı | Tetikleyici | Kapı (gate) | Zorunlu payload alanları | |---|---|---|---|---| | `training.start` | `training.started` | `train()` çağrıldı, model yüklenmeden önce. | `webhook.notify_on_start` | `run_name`, `status="started"` | -| `training.success` | `pipeline.completed` | Koşu revert veya bekleyen onay olmadan tamamlandı. `evaluation.auto_revert: true` ile tüm kapılar geçildi; varsayılan `auto_revert: false` ile bir kapı geçemediği halde yalnızca kaydedildiğinde de tetiklenir (model yine terfi eder). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | +| `training.success` | `pipeline.completed` | Koşu revert veya bekleyen onay olmadan tamamlandı. `evaluation.auto_revert: true` ile karara varan tüm kapılar geçildi (karar üretemeyen bir güvenlik değerlendirmesi, `safety.evaluation_completed: false`, modeli korur ve yine buraya varır); varsayılan `auto_revert: false` ile bir kapı geçemediği halde yalnızca kaydedildiğinde de tetiklenir (model yine terfi eder). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | | `training.failure` | `pipeline.failed` | Eğitim sürecinin kendisi hata fırlattı (OOM, veri seti hatası, yakalanmayan istisna). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (maskelenmiş, ≤2048 karakter) | | `training.reverted` | `model.reverted` | Eğitim sonrası bir kapı (değerlendirme, güvenlik, hakem, benchmark) çalışmayı reddetti ve `_revert_model` adaptörleri sildi. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (maskelenmiş, ≤2048 karakter) | | `approval.required` | `human_approval.required` | Çalışma başarılı oldu, `evaluation.require_human_approval=true`, model insan incelemesi için staging'de (EU AI Act Madde 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` | diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md index a7a74abd..a8bd7fd2 100644 --- a/docs/reference/audit_event_catalog.md +++ b/docs/reference/audit_event_catalog.md @@ -161,7 +161,7 @@ webhook ping → audit entry by `run_name` + timestamp. Implementation: | Webhook `event` | Audit-log mirror | Trigger | Gated by | Required payload fields | |---|---|---|---|---| | `training.start` | `training.started` | `train()` entered, before model load. | `webhook.notify_on_start` | `run_name`, `status="started"` | -| `training.success` | `pipeline.completed` | Run completed without revert or pending approval. With `evaluation.auto_revert: true` all gates passed; with the default `auto_revert: false` it also fires when a gate failed but was only recorded (model still promoted). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | +| `training.success` | `pipeline.completed` | Run completed without revert or pending approval. With `evaluation.auto_revert: true` every gate that produced a verdict passed (a safety evaluation with no verdict, `safety.evaluation_completed: false`, keeps the model and also ends here); with the default `auto_revert: false` it also fires when a gate failed but was only recorded (model still promoted). | `webhook.notify_on_success` | `run_name`, `status="succeeded"`, `metrics` | | `training.failure` | `pipeline.failed` | Training itself raised (OOM, dataset error, unhandled exception). | `webhook.notify_on_failure` | `run_name`, `status="failed"`, `reason` (masked, ≤2048 chars) | | `training.reverted` | `model.reverted` | A post-training gate (evaluation, safety, judge, benchmark) rejected the run and `_revert_model` deleted the adapters. | `webhook.notify_on_failure` | `run_name`, `status="reverted"`, `reason` (masked, ≤2048 chars) | | `approval.required` | `human_approval.required` | Run succeeded, `evaluation.require_human_approval=true`, model staged for review (EU AI Act Art. 14). | `webhook.notify_on_success` | `run_name`, `status="awaiting_approval"`, `model_path` | diff --git a/docs/standards/error-handling.md b/docs/standards/error-handling.md index 1a8860f5..97a645b6 100644 --- a/docs/standards/error-handling.md +++ b/docs/standards/error-handling.md @@ -19,7 +19,7 @@ EXIT_INTEGRITY_FAILURE = 6 | Code | When | Who reads it | |---|---|---| -| **0** | Training completed. With `auto_revert: true` (the compliance default for high-risk tiers) this also means every gate passed; with the **shipped default** `auto_revert: false` a failed benchmark/safety/judge gate is *recorded* (`benchmark`/`safety`/`judge` block in the JSON, `*_passed: false`) but does **not** change the exit code — parse those blocks, don't trust exit 0 alone | CI/CD success | +| **0** | Training completed. With `auto_revert: true` (the compliance default for high-risk tiers) this also means every gate that ran to a verdict passed (a safety evaluation that produced no verdict — `safety.evaluation_completed: false` — keeps the model and exits 0 regardless of `auto_revert`); with the **shipped default** `auto_revert: false` a failed benchmark/safety/judge gate is *recorded* (`benchmark`/`safety`/`judge` block in the JSON, `*_passed: false`) but does **not** change the exit code — parse those blocks, don't trust exit 0 alone | CI/CD success | | **1** | Config validation failed (YAML schema, Pydantic error). Also used by the `verify-*` subcommands for an operator-actionable input error — the artefact could not be read at all, so nothing was ever compared (plus the one case where everything comparable *was* compared and came out clean; see the 1-vs-6 note below) | CI/CD "fail fast"; user fixes YAML or the verifier's target path | | **2** | Training crashed or failed mid-run (OOM, CUDA error, unhandled exception). Also the `verify-*` subcommands' code for a genuine runtime I/O failure on a reachable path (permission denied mid-read, disk error) | CI/CD retry logic | | **3** | Training completed but eval/safety/benchmark threshold failed, and auto-revert happened | CI/CD decision: do not deploy | diff --git a/docs/usermanuals/en/reference/exit-codes.md b/docs/usermanuals/en/reference/exit-codes.md index 8780f19e..e462872d 100644 --- a/docs/usermanuals/en/reference/exit-codes.md +++ b/docs/usermanuals/en/reference/exit-codes.md @@ -11,7 +11,7 @@ ForgeLM's exit codes are a public contract. CI/CD pipelines, schedulers, and das | Exit | Constant | Meaning | Typical CI action | |---|---|---|---| -| **0** | `EXIT_SUCCESS` | Run completed and the checkpoint was promoted. With `evaluation.auto_revert: true` every gate also passed; with the shipped default `auto_revert: false` a failed benchmark/safety/judge gate is **recorded in the JSON output but does not change the exit code** — see ["What exit 0 actually guarantees"](#what-exit-0-actually-guarantees) below. | Continue pipeline (parse gate blocks if `auto_revert` is off) | +| **0** | `EXIT_SUCCESS` | Run completed and the checkpoint was promoted. With `evaluation.auto_revert: true` every gate that produced a verdict also passed (a safety evaluation with no verdict keeps the model and exits 0 too); with the shipped default `auto_revert: false` a failed benchmark/safety/judge gate is **recorded in the JSON output but does not change the exit code** — see ["What exit 0 actually guarantees"](#what-exit-0-actually-guarantees) below. | Continue pipeline (parse gate blocks if `auto_revert` is off) | | **1** | `EXIT_CONFIG_ERROR` | YAML invalid, file missing, env var unset, or argument malformed. | Fail fast | | **2** | `EXIT_TRAINING_ERROR` | Training-time runtime error (any unhandled exception that isn't a config or eval-gate failure: data load, OOM, NaN loss, I/O failure, mid-stream audit-iteration OSError). | Investigate; surface logs | | **3** | `EXIT_EVAL_FAILURE` | A benchmark/safety/judge gate failed **and** the model was auto-reverted (requires `evaluation.auto_revert: true`). With `auto_revert: false` a failed gate does not produce exit 3 — the run exits 0 with the failure recorded in the JSON gate blocks. | Investigate; do NOT promote | @@ -112,7 +112,7 @@ A run that exits 0 has: **Gates and exit 0.** Whether a *passed* benchmark/safety/judge gate is part of the exit-0 guarantee depends on `evaluation.auto_revert`: -- With `evaluation.auto_revert: true` (the EU AI Act high-risk default), a failed gate auto-reverts the model and exits **3** — so exit 0 *does* mean every configured gate passed. +- With `evaluation.auto_revert: true` (the EU AI Act high-risk default), a failed gate auto-reverts the model and exits **3** — so exit 0 means every configured gate that produced a verdict passed. **One exception:** a safety evaluation that produced no verdict (`safety.evaluation_completed: false` — the classifier failed to load, the probes file is missing or empty, or most verdicts were unreadable) keeps the model and exits 0 (4 with the approval gate) even with `auto_revert: true`; check `safety.passed` and `safety.evaluation_completed` in the JSON output. - With the shipped default `evaluation.auto_revert: false`, a failed gate is **recorded** (the `benchmark` / `safety` / `judge` block in the JSON output carries `*_passed: false`) but the model is still promoted and the run exits **0**. Read those JSON blocks; do not infer gate success from exit 0 alone. There is no "partial success" exit code by design — turn on `auto_revert` if you want a failing gate to change the exit code. diff --git a/docs/usermanuals/en/reference/json-output.md b/docs/usermanuals/en/reference/json-output.md index 7132bc10..c0ebbabe 100644 --- a/docs/usermanuals/en/reference/json-output.md +++ b/docs/usermanuals/en/reference/json-output.md @@ -178,7 +178,7 @@ When training runs to completion the pipeline emits a result envelope on **stdou | Key | Type | Notes | |---|---|---| -| `success` | bool | `true` when the run completed (including a staged, awaiting-approval run). `false` when a gate auto-reverted the model. | +| `success` | bool | `true` when the run completed (including a staged, awaiting-approval run, and a run whose safety evaluation produced no verdict: check `safety.evaluation_completed`). `false` when a gate auto-reverted the model. | | `error` | string | Present **iff** `success` is `false`, never on the success path. Carries the gate's own reason — which gate fired and against what threshold — because every gate exits `3`, so the exit code alone cannot tell them apart. If a failure reaches the envelope with no reason recorded, the field says so explicitly and points at `audit_log.jsonl` rather than being omitted. | | `metrics` | object | Training/eval/gate metrics (e.g. `eval_loss`, `benchmark/average`, `safety/safe_ratio`). Values are numbers, **except** a measurement that came out non-finite, which is written as the string `"nan"` / `"inf"` / `"-inf"` — JSON has no non-finite number literal, so the alternative was output no strict parser accepts. Type these fields as number-or-string. | | `final_model_path` | str | Where the model artefacts live. For an awaiting-approval run this is the **staging** directory until `forgelm approve` promotes it. | diff --git a/docs/usermanuals/tr/reference/exit-codes.md b/docs/usermanuals/tr/reference/exit-codes.md index fd21ec0e..6a0fc562 100644 --- a/docs/usermanuals/tr/reference/exit-codes.md +++ b/docs/usermanuals/tr/reference/exit-codes.md @@ -11,7 +11,7 @@ ForgeLM'in exit kodları kamuya açık bir kontrattır. CI/CD hatları, schedule | Exit | Sabit | Anlam | Tipik CI aksiyonu | |---|---|---|---| -| **0** | `EXIT_SUCCESS` | Koşu tamamlandı ve checkpoint terfi etti. `evaluation.auto_revert: true` ile tüm kapılar da geçti; sevk edilen varsayılan `auto_revert: false` ile başarısız bir benchmark/güvenlik/judge kapısı **JSON çıktısına kaydedilir ama exit kodunu değiştirmez** — aşağıdaki ["exit 0 tam olarak ne garanti eder"](#exit-0-tam-olarak-ne-garanti-eder) bölümüne bakın. | Hattı sürdür (`auto_revert` kapalıysa kapı bloklarını parse et) | +| **0** | `EXIT_SUCCESS` | Koşu tamamlandı ve checkpoint terfi etti. `evaluation.auto_revert: true` ile karara varan tüm kapılar da geçti (karar üretemeyen bir güvenlik değerlendirmesi de modeli korur ve 0 ile çıkar); sevk edilen varsayılan `auto_revert: false` ile başarısız bir benchmark/güvenlik/judge kapısı **JSON çıktısına kaydedilir ama exit kodunu değiştirmez** — aşağıdaki ["exit 0 tam olarak ne garanti eder"](#exit-0-tam-olarak-ne-garanti-eder) bölümüne bakın. | Hattı sürdür (`auto_revert` kapalıysa kapı bloklarını parse et) | | **1** | `EXIT_CONFIG_ERROR` | YAML geçersiz, dosya yok, env var ayarsız veya argüman bozuk. | Hızlı başarısız | | **2** | `EXIT_TRAINING_ERROR` | Eğitim sırasında runtime hatası (config veya değerlendirme kapısı dışı her ele alınmamış istisna: data yükleme, OOM, NaN loss, I/O başarısızlığı, mid-stream audit-iteration OSError). | İncele; logları yüzeyle | | **3** | `EXIT_EVAL_FAILURE` | Bir benchmark/güvenlik/judge kapısı geçemedi **ve** model otomatik geri alındı (`evaluation.auto_revert: true` gerektirir). `auto_revert: false` ile başarısız bir kapı exit 3 üretmez — koşu, JSON kapı bloklarına kaydedilen başarısızlıkla 0 çıkar. | İncele; terfi ETTİRME | @@ -112,7 +112,7 @@ Exit kodu kontrat tek başına yeterli — POSIX kabuklarda `$?`, cmd'de `%ERROR **Kapılar ve exit 0.** *Geçen* bir benchmark/güvenlik/judge kapısının exit-0 garantisinin parçası olup olmadığı `evaluation.auto_revert`'e bağlıdır: -- `evaluation.auto_revert: true` ile (EU AI Act yüksek-riskli varsayılanı), başarısız bir kapı modeli otomatik geri alır ve **3** ile çıkar — yani exit 0 *gerçekten* tüm konfigüre kapıların geçtiği anlamına gelir. +- `evaluation.auto_revert: true` ile (EU AI Act yüksek-riskli varsayılanı), başarısız bir kapı modeli otomatik geri alır ve **3** ile çıkar — yani exit 0, karara varan her konfigüre kapının geçtiği anlamına gelir. **Tek istisna:** karar üretemeyen bir güvenlik değerlendirmesi (`safety.evaluation_completed: false` — sınıflandırıcı yüklenemedi, probe dosyası yok ya da boş, ya da çoğu karar okunamadı) `auto_revert: true` olsa bile modeli korur ve 0 ile çıkar (onay kapısıyla 4); JSON çıktısındaki `safety.passed` ve `safety.evaluation_completed`'e bakın. - Sevk edilen varsayılan `evaluation.auto_revert: false` ile, başarısız bir kapı **kaydedilir** (JSON çıktısındaki `benchmark` / `safety` / `judge` bloğu `*_passed: false` taşır) ama model yine terfi eder ve koşu **0** ile çıkar. Bu JSON bloklarını okuyun; kapı başarısını yalnızca exit 0'dan çıkarsamayın. Tasarım gereği "kısmi başarı" exit kodu yok — başarısız bir kapının exit kodunu değiştirmesini istiyorsanız `auto_revert`'i açın. diff --git a/docs/usermanuals/tr/reference/json-output.md b/docs/usermanuals/tr/reference/json-output.md index a4421064..e1f3a911 100644 --- a/docs/usermanuals/tr/reference/json-output.md +++ b/docs/usermanuals/tr/reference/json-output.md @@ -178,7 +178,7 @@ Eğitim tamamlanana kadar çalıştığında pipeline **stdout**'a bir sonuç en | Anahtar | Tip | Not | |---|---|---| -| `success` | bool | Çalışma tamamlandığında (stage'lenmiş, onay-bekleyen çalışma dahil) `true`. Bir gate modeli auto-revert ettiğinde `false`. | +| `success` | bool | Çalışma tamamlandığında (stage'lenmiş, onay-bekleyen çalışma ve güvenlik değerlendirmesi karar üretemeyen çalışma dahil; `safety.evaluation_completed`'e bakın) `true`. Bir gate modeli auto-revert ettiğinde `false`. | | `error` | string | **Yalnızca** `success` `false` iken bulunur, başarı yolunda asla. Gate'in kendi gerekçesini taşır — hangi gate'in hangi eşiğe karşı tetiklendiğini — çünkü her gate `3` ile çıkar, dolayısıyla exit kodu tek başına ayırt edemez. Bir başarısızlık zarfa gerekçesiz ulaşırsa alan bunu açıkça söyler ve `audit_log.jsonl`'a yönlendirir; atlanmaz. | | `metrics` | object | Eğitim/eval/gate metrikleri (ör. `eval_loss`, `benchmark/average`, `safety/safe_ratio`). Değerler sayıdır; **istisna**, sonlu çıkmayan bir ölçüm `"nan"` / `"inf"` / `"-inf"` string'i olarak yazılır — JSON'da sonlu olmayan sayı literali yoktur, alternatifi hiçbir katı ayrıştırıcının kabul etmediği bir çıktıydı. Bu alanları sayı-veya-string olarak tipleyin. | | `final_model_path` | str | Model artefactlarının yaşadığı yer. Onay-bekleyen bir çalışma için bu, `forgelm approve` promote edene kadar **staging** dizinidir. |