diff --git a/CHANGELOG.md b/CHANGELOG.md index c224ea95..84e7b970 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,37 @@ All notable changes to ForgeLM are documented here. ### Fixed +- Stop saying that Microsoft Teams and Discord accept the webhook payload directly. The user manual (English and Turkish) and the + `webhook` field descriptions said all three ingest it; the body is a Slack-style `attachments` block with no top-level `text`, + `content` or `embeds`. Discord needs `content` or `embeds` and answers HTTP 400; Teams retired Office 365 Connectors in May 2026 and + its Workflows webhooks expect an Adaptive Card. Both need a relay. The page now shows the real payload, and the access-control SOP + no longer claims that a payload carries `FORGELM_OPERATOR` (it does not) or lists five events (there are eight). The `url_env` + description said it overrides `url`; it is read only when `url` is empty, so `url` wins. A formatter for Teams and Discord is a + separate, larger change and stays open on the issue (#112). +- Say that the DeepSpeed preset names (`zero2`, `zero3`, `zero3_offload`) work only from a source checkout. They resolve to + `configs/deepspeed/.json` beside the package or in the current directory, and neither the wheel nor the Docker image ships + that folder, so on a `pip install` the names fail with a configuration error (and the unset-`deepspeed_config` default is + `zero2`). The configuration and distributed-training references, the troubleshooting entry (its sample error is now the real + message), the enterprise guide, the user manuals, `config_template.yaml` and the field description now say so and point at a + JSON path; the template's `torchrun … forgelm` launch line is `-m forgelm`, the only form that runs the checkout. The + unverified "`zero3_offload` fits 70B on 8×24 GB" sentence is gone, and the `zero2` preset's own start-up failure is noted + (#410). Shipping the presets in the package, and a test that resolves one from an installed wheel, are code changes and stay + open on the issues (#110, #111, #115). +- Show the real output directory on the first-run, project-layout and human-oversight pages (English and Turkish). They described a + `checkpoints//artifacts/` bundle with a `manifest.json` that hashes "every file inside … for tamper-evidence", a + `config_snapshot.yaml`, a `safety_report.json` and a `data_audit_report.json`. None of those exist: a run writes `audit_log.jsonl` with its + `.manifest.json` genesis pin, a `compliance/` directory (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, + `data_governance_report.json`, and `annex_iv_metadata.json` only when the YAML names a provider, system or purpose), `safety/` and + `benchmark/` only when enabled, and `final_model/` with the weights, model card, `deployer_instructions.md` and `model_integrity.json`. + Only the audit log and `final_model/` are tamper-checked; nothing hashes `compliance/`, and the pages now say so. The invented training log + (a DPO pass in an SFT-only template, `Llama Guard S1-S14: clean`) is replaced by a description of the real stages, the approval-gate + sample shows the two log lines the trainer prints, and the `forgelm doctor` sample uses the real renderer's format (ASCII header and + `[+ pass]` markers, "discoverable" wording, an abridged row list) (#96, #534). +- Describe `--offline` / `model.offline` as what it does. It sets `HF_HUB_OFFLINE`, `TRANSFORMERS_OFFLINE` and `HF_DATASETS_OFFLINE`; it does not + stop webhooks, the LLM-judge and synthetic-data API clients, or W&B / MLflow. The `--offline` help text, the `model.offline` field + description, the configuration and CLI references (English and Turkish) and the enterprise deployment guide said "Air-gapped mode" and + "prevents any network calls"; the air-gap page no longer lists Comet, which `training.report_to` does not accept. Gating those clients + behind the flag is a code change and stays open (#74, #100, #893). - `training.rope_scaling` works on transformers 5 again. The setting was passed to `from_pretrained(rope_scaling=...)`, which no longer works: the documented `{type: linear, factor: 4}` raised `KeyError: 'rope_type'` while the model was built, and `{rope_type: ..., factor: ...}` replaced the whole parameter dict so `rope_theta` was lost (`TypeError`). diff --git a/config_template.yaml b/config_template.yaml index acf197cb..415cc2c9 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -162,10 +162,10 @@ data: # dare_seed: 42 # DARE: RNG seed for the random drop mask # Distributed training (multi-GPU). Requires: pip install forgelm[distributed] -# Launch with: torchrun --nproc_per_node=N forgelm --config my_config.yaml +# Launch with: torchrun --nproc_per_node=N -m forgelm --config my_config.yaml # distributed: # strategy: "deepspeed" # "deepspeed" or "fsdp" -# deepspeed_config: "zero2" # preset: "zero2", "zero3", "zero3_offload" or path to JSON +# deepspeed_config: "ds.json" # path to a DeepSpeed JSON (the presets "zero2", "zero3", "zero3_offload" need a source checkout) # # --- FSDP options (when strategy: "fsdp") --- # # fsdp_strategy: "full_shard" # "full_shard" or "shard_grad_op" (FSDP2 supports no other) # # fsdp_auto_wrap: true diff --git a/docs/guides/enterprise_deployment.md b/docs/guides/enterprise_deployment.md index f9c70e60..67231902 100644 --- a/docs/guides/enterprise_deployment.md +++ b/docs/guides/enterprise_deployment.md @@ -121,7 +121,7 @@ ForgeLM sets `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, `HF_DATASETS_OFFLINE= ```yaml distributed: strategy: "deepspeed" - deepspeed_config: "zero2" # or "zero3", "zero3_offload" + deepspeed_config: "/etc/forgelm/ds_zero2.json" # your own DeepSpeed JSON; the zero2/zero3/zero3_offload preset names need a source checkout ``` ```bash @@ -152,7 +152,7 @@ See the [Distributed Training Guide](../reference/distributed_training.md) for d | `trust_remote_code` | `false` | Prevents arbitrary code execution from model repos | | `auth.hf_token` | Use `HUGGINGFACE_TOKEN` env var | Don't hardcode tokens in YAML | | `webhook.url` | Always use `url_env` with environment variable | Webhook tokens excluded from model cards (v0.3.1rc1+), but avoid direct URLs for credential hygiene | -| `offline` | `true` for air-gapped | Prevents any network calls | +| `offline` | `true` for air-gapped | Blocks Hugging Face Hub, transformers and datasets network calls (it sets `HF_HUB_OFFLINE`, `TRANSFORMERS_OFFLINE`, `HF_DATASETS_OFFLINE`). It does **not** stop webhooks, judge or synthetic-data API clients, or W&B / MLflow: disable those (omit `webhook:`, use local judge and teacher models, set `report_to: none`) for a true air gap | --- diff --git a/docs/guides/troubleshooting-tr.md b/docs/guides/troubleshooting-tr.md index c4cb2408..29e6331c 100644 --- a/docs/guides/troubleshooting-tr.md +++ b/docs/guides/troubleshooting-tr.md @@ -363,11 +363,12 @@ nvidia-smi ### DeepSpeed Config bulunamadı ```text -FileNotFoundError: DeepSpeed preset 'zero2' not found +DeepSpeed preset 'zero2' not found at /configs/deepspeed/zero2.json ``` -ForgeLM proje kökünden çalıştırdığınızdan emin olun ya da mutlak yol -kullanın: +Ön ayar adları yalnızca bir kaynak checkout'unda çözülür (paketin yanındaki ya da geçerli dizindeki bir +`configs/deepspeed/` dizini); `pip install` veya Docker imajı bunları içermez. ForgeLM proje kökünden çalıştırın ya da +bir DeepSpeed JSON dosyasının yolunu kullanın: ```yaml distributed: diff --git a/docs/guides/troubleshooting.md b/docs/guides/troubleshooting.md index 7533eedb..9549875b 100644 --- a/docs/guides/troubleshooting.md +++ b/docs/guides/troubleshooting.md @@ -344,10 +344,12 @@ nvidia-smi ### DeepSpeed Config Not Found ```text -FileNotFoundError: DeepSpeed preset 'zero2' not found +DeepSpeed preset 'zero2' not found at /configs/deepspeed/zero2.json ``` -Ensure you're running from the ForgeLM project root, or use an absolute path: +The preset names resolve only from a source checkout (a `configs/deepspeed/` directory beside the package or in the +current directory); a `pip install` or the Docker image does not contain them. Run from the ForgeLM project root, or +use the path of a DeepSpeed JSON file: ```yaml distributed: diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md index 13341c3a..67f3ea60 100644 --- a/docs/qms/access_control-tr.md +++ b/docs/qms/access_control-tr.md @@ -230,20 +230,26 @@ webhook: ForgeLM webhook gövdelerini HMAC ile **imzalamaz** — `WebhookConfig` üzerinde `webhook.secret_env` alanı yoktur (bkz. -`forgelm/config.py::WebhookConfig`). Hedef-tarafı atıf (a) HTTPS + -`webhook.url_env` üzerinden URL-gizliliğine, (b) curated payload -içinde taşınan `FORGELM_OPERATOR` kimliğine, ve (c) alıcı sistemin -kendi bearer-token / imzalı-istek kontrollerine (Slack signing -secret, Teams connector token) düşer. - -Webhook wire-format olay sözlüğü (5 olay): `training.start`, -`training.success`, `training.failure`, `training.reverted`, -`approval.required` — Phase 8. In-process notifier metot adları -`notify_*` öneki kullanır (`notify_start`, `notify_success`, -`notify_failure`, `notify_reverted`, `notify_awaiting_approval`) -ve bu beş wire olayına dispatch eder. Her payload `FORGELM_OPERATOR` -kimliğini taşır böylece alıcı sistem (Slack, Teams, custom -incident-management) bildirimi attribute edebilir. +`forgelm/config.py::WebhookConfig`). Hedef-tarafı özgünlük (a) HTTPS + +`webhook.url_env` üzerinden URL-gizliliğine ve (b) alıcı tarafın kendi +uyguladığı kimlik doğrulamaya düşer. Bir Slack incoming webhook'unun gizli +URL'sinden başka bir doğrulaması yoktur (Slack'in signing secret'ı Slack'ten +*gelen* istekleri doğrular, bir webhook URL'sine gelenleri değil); bu yüzden +özel bir relay kendi kimlik doğrulamasını istemelidir, çünkü ForgeLM imza +göndermez. Webhook payload'u `FORGELM_OPERATOR` +kimliğini **taşımaz**; bir modeli kimin çalıştırdığı veya onayladığı +bildirimden değil, audit log'dan okunur. + +Webhook wire-format olay sözlüğü sekiz olaydır: beş tek-aşamalı +yaşam-döngüsü olayı `training.start`, `training.success`, +`training.failure`, `training.reverted` ve `approval.required`, ayrıca +çok-aşamalı bir koşunun bunlarla birlikte yaydığı üç `pipeline.*` +olayı (`pipeline.started`, `pipeline.completed`, +`pipeline.stage_reverted`). In-process notifier metotları `notify_*` +öneki kullanır (`notify_start`, `notify_success`, `notify_failure`, +`notify_reverted`, `notify_awaiting_approval`, +`notify_pipeline_started`, `notify_pipeline_completed`, +`notify_pipeline_reverted`) ve bu wire olaylarına dispatch eder. ## 8. Doğrulama checklist'i diff --git a/docs/qms/access_control.md b/docs/qms/access_control.md index e3cb1aed..9e5f344a 100644 --- a/docs/qms/access_control.md +++ b/docs/qms/access_control.md @@ -228,21 +228,26 @@ webhook: ForgeLM does **not** HMAC-sign webhook bodies — there is no `webhook.secret_env` field on `WebhookConfig` (see -`forgelm/config.py::WebhookConfig`). Destination-side attribution falls to (a) -HTTPS + URL-secrecy via `webhook.url_env`, (b) the -`FORGELM_OPERATOR` identity carried inside the curated payload, and -(c) the receiving system's own bearer-token / signed-request -controls (Slack signing secret, Teams connector token). - -The webhook wire-format event vocabulary (5 events) is -`training.start`, `training.success`, `training.failure`, -`training.reverted`, and `approval.required` — Phase 8. The -in-process notifier method names use the `notify_*` prefix -(`notify_start`, `notify_success`, `notify_failure`, -`notify_reverted`, `notify_awaiting_approval`) and dispatch to those -five wire events. Every payload carries the `FORGELM_OPERATOR` -identity so the receiving system (Slack, Teams, custom -incident-management) can attribute the notification. +`forgelm/config.py::WebhookConfig`). Destination-side authenticity falls to +(a) HTTPS + URL-secrecy via `webhook.url_env` and (b) authentication +the receiving side applies itself. A Slack incoming webhook has none +beyond its secret URL (Slack's signing secret verifies requests *from* +Slack, not requests to a webhook URL), so a custom relay should require +its own authentication, because ForgeLM sends no signature. A webhook +payload does **not** carry the +`FORGELM_OPERATOR` identity; attribution of who ran or approved a model +comes from the audit log, not from the notification. + +The webhook wire-format event vocabulary has eight events: the five +single-stage lifecycle events `training.start`, `training.success`, +`training.failure`, `training.reverted` and `approval.required`, plus the +three `pipeline.*` events (`pipeline.started`, `pipeline.completed`, +`pipeline.stage_reverted`) that a multi-stage run emits alongside them. The +in-process notifier methods use the `notify_*` prefix (`notify_start`, +`notify_success`, `notify_failure`, `notify_reverted`, +`notify_awaiting_approval`, `notify_pipeline_started`, +`notify_pipeline_completed`, `notify_pipeline_reverted`) and dispatch to +those wire events. ## 8. Verification checklist diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index f9b88b43..0bf47d57 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -15,7 +15,7 @@ Tam açıklamalı örnek için `config_template.yaml` dosyasına bakın. | `load_in_4bit` | bool | `true` | QLoRA 4-bit NF4 kuantizasyon | | `backend` | string | `"transformers"` | `"transformers"` veya `"unsloth"` (desteklenen mimarilerde daha hızlı, Linux) | | `trust_remote_code` | bool | `false` | Model depolarından özel kod çalıştırma. **Güvenlik riski** | -| `offline` | bool | `false` | İzole mod: HF Hub çağrısı yok. Modeller/veri setleri yerel olmalı | +| `offline` | bool | `false` | HF offline değişkenlerini ayarlar; Hub, transformers ve datasets ağ çağrısı yapmaz. Modeller/veri setleri yerel olmalı. Webhook'ları, judge veya sentetik-veri API istemcilerini ve W&B / MLflow'u durdurmaz | | `revision` | string | `null` | Temel modeli + tokenizer'ı bir HF Hub commit SHA'sına (40-hex) veya branch/tag'e sabitle. **Bugün uygulanıyor.** Bkz. [Hub revision pinleme](#hub-revision-pinleme) | | `bnb_4bit_use_double_quant` | bool | `true` | Ekstra VRAM tasarrufu için çift kuantizasyon | | `bnb_4bit_quant_type` | string | `"nf4"` | Kuantizasyon tipi (`"nf4"` veya `"fp4"`) | @@ -397,7 +397,7 @@ kaynağı kaydedilmemiştir. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| | `strategy` | string | `null` | `"deepspeed"` veya `"fsdp"` (null = tek GPU) | -| `deepspeed_config` | string | `null` | Ön ayar (`"zero2"`, `"zero3"`, `"zero3_offload"`) veya JSON yolu | +| `deepspeed_config` | string | `null` | JSON yolu ya da yalnızca kaynak checkout'unda çözülen (pip veya Docker kurulumunda çözülmeyen) bir ön ayar adı (`"zero2"`, `"zero3"`, `"zero3_offload"`) | | `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` veya `"shard_grad_op"` (`"hybrid_shard"` ve `"no_shard"` reddedilir: FSDP2 bunları desteklemez) | | `fsdp_auto_wrap` | bool | `true` | Transformer katmanlarını otomatik sar (transformers 5.10+'ta yok sayılır, katmanlar hep sarılır) | | `fsdp_offload` | bool | `false` | Parametreleri CPU'ya taşı | @@ -433,11 +433,11 @@ kaynağı kaydedilmemiştir. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `url` | string | `null` | Webhook hedef URL | -| `url_env` | string | `null` | URL'yi içeren ortam değişkeni adı | +| `url` | string | `null` | Webhook hedef URL'si: bir Slack incoming webhook'u ya da bir relay (Teams ve Discord payload'u doğrudan reddeder) | +| `url_env` | string | `null` | URL'yi içeren ortam değişkeni adı; yalnızca `url` boşken okunur | | `notify_on_start` | bool | `true` | Eğitim başlangıcında bildir | | `notify_on_success` | bool | `true` | Başarıda bildir | -| `notify_on_failure` | bool | `true` | Hata durumunda bildir | +| `notify_on_failure` | bool | `true` | Eğitim hata verdiğinde ya da eğitim sonrası bir kapı modeli geri aldığında bildir | | `timeout` | int | `10` | HTTP istek zaman aşımı (saniye). Notifier ≥ 1s'ye clamp'ler. v0.5.5'te varsayılan 10s'ye çıkarıldı (önceden 5s'di) — Slack/Teams gateway gecikme atışları production'da düzenli olarak 5s'yi aşıyor ve bir webhook zaman aşımı audit chain'i sessizce zayıflatıyor (webhook arızası best-effort). | | `allow_private_destinations` | bool | `false` | RFC1918 / loopback / link-local hedeflere webhook gönderimine izin verir (cluster içi Slack proxy, on-prem Teams gateway gibi). Varsayılan yalnızca genel internet — SSRF koruması | | `require_https` | bool | `false` | TLS-only zorlama. `true`, plaintext bir `http://` URL'ini reddeder (SSRF chokepoint raise eder; POST atlanır), warn-and-send yerine. Varsayılan `false`, warn-then-send davranışını korur | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 5d830e0a..87e8639a 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -15,7 +15,7 @@ See `config_template.yaml` for a complete annotated example. | `load_in_4bit` | bool | `true` | Enable QLoRA 4-bit NF4 quantization | | `backend` | string | `"transformers"` | `"transformers"` or `"unsloth"` (faster on supported architectures, Linux only) | | `trust_remote_code` | bool | `false` | Allow custom code from model repos. **Security risk** — only enable for models that require it | -| `offline` | bool | `false` | Air-gapped mode: no HF Hub calls. Models/datasets must be local | +| `offline` | bool | `false` | Sets the HF offline variables, so Hub, transformers and datasets make no network calls. Models/datasets must be local. Does not stop webhooks, judge or synthetic-data API clients, or W&B / MLflow | | `revision` | string | `null` | Pin the base model + tokenizer to an HF Hub commit SHA (40-hex) or a branch/tag. **Honoured today.** See [Hub revision pinning](#hub-revision-pinning) | | `bnb_4bit_use_double_quant` | bool | `true` | Double quantization for extra VRAM savings | | `bnb_4bit_quant_type` | string | `"nf4"` | Quantization type (`"nf4"` or `"fp4"`) | @@ -293,11 +293,11 @@ exits 0, and nothing is deleted, blocked or scheduled automatically. The | Field | Type | Default | Description | |-------|------|---------|-------------| -| `url` | string | `null` | Webhook destination URL | -| `url_env` | string | `null` | Env var name containing URL | +| `url` | string | `null` | Webhook destination URL: a Slack incoming webhook, or a relay (Teams and Discord reject the payload directly) | +| `url_env` | string | `null` | Env var name containing the URL; read only when `url` is empty | | `notify_on_start` | bool | `true` | Notify on training start | | `notify_on_success` | bool | `true` | Notify on success | -| `notify_on_failure` | bool | `true` | Notify on failure | +| `notify_on_failure` | bool | `true` | Notify when training raises or a post-training gate reverts the model | | `timeout` | int | `10` | HTTP request timeout (seconds). Clamped to ≥ 1s by the notifier. Default raised to 10s in v0.5.5 (was 5s) — Slack/Teams gateway latency spikes regularly cross 5s in production, and a webhook timeout silently degrades the audit chain (webhook failure is best-effort). | | `allow_private_destinations` | bool | `false` | Opt in to webhooks pointing at RFC1918 / loopback / link-local hosts (in-cluster Slack proxy, on-prem Teams gateway). Defaults to public-internet only — SSRF guard | | `require_https` | bool | `false` | TLS-only enforcement. `true` refuses a plaintext `http://` URL (the SSRF chokepoint raises; the POST is skipped) instead of warn-and-send. Default `false` preserves warn-then-send | @@ -310,7 +310,7 @@ exits 0, and nothing is deleted, blocked or scheduled automatically. The | Field | Type | Default | Description | |-------|------|---------|-------------| | `strategy` | string | `null` | `"deepspeed"` or `"fsdp"` (null = single GPU) | -| `deepspeed_config` | string | `null` | Preset (`"zero2"`, `"zero3"`, `"zero3_offload"`) or JSON path | +| `deepspeed_config` | string | `null` | JSON path, or a preset name (`"zero2"`, `"zero3"`, `"zero3_offload"`) that resolves only from a source checkout, not from a pip or Docker install | | `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` or `"shard_grad_op"` (`"hybrid_shard"` and `"no_shard"` are rejected: FSDP2 does not support them) | | `fsdp_auto_wrap` | bool | `true` | Auto-wrap transformer layers (ignored on transformers 5.10+, which always wraps them) | | `fsdp_offload` | bool | `false` | Offload parameters to CPU | diff --git a/docs/reference/distributed_training-tr.md b/docs/reference/distributed_training-tr.md index 0df37e60..6054462b 100644 --- a/docs/reference/distributed_training-tr.md +++ b/docs/reference/distributed_training-tr.md @@ -49,7 +49,17 @@ torchrun --nproc_per_node=4 -m forgelm.cli --config my_config.yaml ## DeepSpeed Yapılandırması -ForgeLM, üç yerleşik DeepSpeed ön ayarı sunar. `deepspeed_config` parametresine bu isimlerden birini veya kendi JSON dosyanızın yolunu verin. +Depo, `configs/deepspeed/` altında üç DeepSpeed ön ayarı barındırır. `deepspeed_config` parametresine bunların adlarından birini veya kendi JSON dosyanızın yolunu verin. + +**Bir ön ayar adı yalnızca `configs/deepspeed/.json` kurulu paketin yanında ya da geçerli dizinde mevcutsa çalışır.** +Kaynak checkout'u bu dosyaları sağlar; `pip install forgelm` wheel'i ve Docker imajı sağlamaz, orada bir ön ayar adı +yapılandırma hatasıyla (exit 1) başarısız olur. Deponun `configs/deepspeed/` klasöründeki JSON dosyalarını kopyalayın +(çalışma dizinindeki bir `configs/deepspeed/` dizinine ya da `deepspeed_config`'i yoluna yönelttiğiniz herhangi bir yere) +veya kendinizinkini yazın. + +**Bilinen sorun ([#410](https://github.com/HodeTech/ForgeLM/issues/410)):** `deepspeed_config` ayarlanmadığında da +kullanılan `zero2` ön ayarı `allgather_bucket_size` değerini `"auto"` yapar; transformers'ın DeepSpeed entegrasyonu bu +değeri doldurmaz, bu yüzden onunla bir ZeRO-2 koşusu durur. Düzelene kadar kendi ZeRO-2 JSON'unuzu kullanın. ### Ön Ayarlar (Presets) @@ -64,8 +74,8 @@ ForgeLM, üç yerleşik DeepSpeed ön ayarı sunar. `deepspeed_config` parametre ```yaml distributed: strategy: "deepspeed" - deepspeed_config: "zero2" # Ön ayar adı - # deepspeed_config: "./my_ds_config.json" # Veya özel dosya yolu + deepspeed_config: "./my_ds_config.json" # Kendi JSON'unuzun yolu (her kurulumda çalışır) + # deepspeed_config: "zero2" # Ön ayar adı: yalnızca kaynak checkout'u ``` ### ZeRO Aşama Karşılaştırması diff --git a/docs/reference/distributed_training.md b/docs/reference/distributed_training.md index 013018a0..95becdf9 100644 --- a/docs/reference/distributed_training.md +++ b/docs/reference/distributed_training.md @@ -49,7 +49,17 @@ torchrun --nproc_per_node=4 -m forgelm.cli --config my_config.yaml ## DeepSpeed Configuration -ForgeLM provides three built-in DeepSpeed presets. Set `deepspeed_config` to one of these names, or provide a path to your own JSON file. +The repository ships three DeepSpeed presets under `configs/deepspeed/`. Set `deepspeed_config` to one of their names, or provide a path to your own JSON file. + +**A preset name works only when `configs/deepspeed/.json` is available beside the installed package or in the +current directory.** A source checkout provides those files; a `pip install forgelm` wheel and the Docker image do not, +so there a preset name fails with a configuration error (exit 1). Copy the JSON files from the repository's +`configs/deepspeed/` folder (into a `configs/deepspeed/` directory in the working directory, or anywhere if you point +`deepspeed_config` at the path), or write your own. + +**Known issue ([#410](https://github.com/HodeTech/ForgeLM/issues/410)):** the `zero2` preset, which is also used when +`deepspeed_config` is unset, sets `allgather_bucket_size` to `"auto"`, a value transformers' DeepSpeed integration does +not fill in, so a ZeRO-2 run with it aborts. Use your own ZeRO-2 JSON until that is fixed. ### Presets @@ -64,8 +74,8 @@ ForgeLM provides three built-in DeepSpeed presets. Set `deepspeed_config` to one ```yaml distributed: strategy: "deepspeed" - deepspeed_config: "zero2" # Preset name - # deepspeed_config: "./my_ds_config.json" # Or custom path + deepspeed_config: "./my_ds_config.json" # Path to your own JSON (works on every install) + # deepspeed_config: "zero2" # Preset name: source checkout only ``` ### ZeRO Stage Comparison diff --git a/docs/usermanuals/en/compliance/human-oversight.md b/docs/usermanuals/en/compliance/human-oversight.md index 373f7db2..61e05a72 100644 --- a/docs/usermanuals/en/compliance/human-oversight.md +++ b/docs/usermanuals/en/compliance/human-oversight.md @@ -48,12 +48,8 @@ There is **no** `compliance.human_approval` field, no `approval.*` block, no `si The trainer halts after eval and prints: ```text -[2026-04-29 14:33:10] Human approval required. - Run ID: abc123 - Bundle: checkpoints/run/artifacts/ - - To approve: forgelm approve abc123 --output-dir checkpoints/run --comment "..." - To reject: forgelm reject abc123 --output-dir checkpoints/run --comment "..." +2026-04-29 14:33:10 [INFO] forgelm.trainer: Human approval required. Model staged at: checkpoints/run/final_model.staging.abc123 +2026-04-29 14:33:10 [INFO] forgelm.trainer: Review results in checkpoints/run/compliance/ and run `forgelm approve abc123 --output-dir checkpoints/run` to promote, or `forgelm reject abc123 --output-dir checkpoints/run` to preserve for forensic review. ``` The reviewer runs the approval command from any machine with access to the audit-log + staging directory. ForgeLM resolves their identity from `FORGELM_OPERATOR`, HMAC-chains the decision event, and (on `approve`) renames the staging directory to the canonical `final_model/` path. diff --git a/docs/usermanuals/en/getting-started/first-run.md b/docs/usermanuals/en/getting-started/first-run.md index 99a2b6ee..5b09a35c 100644 --- a/docs/usermanuals/en/getting-started/first-run.md +++ b/docs/usermanuals/en/getting-started/first-run.md @@ -29,22 +29,23 @@ flowchart LR Before anything else, run `forgelm doctor` to check that Python, PyTorch, CUDA, and any optional dependencies are wired up correctly: ```shell -$ forgelm doctor -forgelm doctor — environment check - - [✓ pass] python.version Python 3.11.4 (CPython). - [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. - [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB). - [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training (Linux GPUs only)). - [✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)). - [! warn] extras.distributed Optional extra missing — install with: pip install 'forgelm[distributed]' (purpose: DeepSpeed ZeRO + offload distributed training (Linux + macOS only)). - [✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200). - [✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. - [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs. - -Summary: 7 pass, 2 warn, 0 fail. +$ forgelm doctor --offline +forgelm doctor - environment check + + [+ pass] python.version Python 3.11.4 (CPython). + [! warn] torch.cuda torch 2.4.0 installed but CUDA is unavailable. CPU-only runs are supported but training will be very slow; expect to use this only for tiny experiments or smoke tests. + [+ pass] numpy.torch_abi torch 2.4.0 + numpy 2.1.0 are ABI-compatible. + [! warn] gpu.inventory No CUDA devices visible; CPU-only mode. + [! warn] extras.distributed Optional extra missing — install with: pip install 'forgelm[distributed]' (purpose: DeepSpeed ZeRO + offload distributed training (Linux + macOS only)). + [+ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs so the audit log identifies a stable identity. + ... + +Summary: 7 pass, 12 warn, 0 fail. ``` +This is an excerpt: the real output has around 19 rows (it also checks the ForgeLM install, every optional extra, and the HF cache or Hub), and the rows, values and counts depend on your machine. A `pass` on an extra means the module is discoverable, not that it was import-tested; `[+ pass]`, `[! warn]` and `[x fail]` are the status markers. + `--output-format json` returns a structured envelope (`{"success": bool, "checks": [...], "summary": {...}}`) so CI can filter on individual probe results without parsing the table. Pass `--offline` to skip the HF Hub network probe and inspect the local cache instead — useful for air-gapped deployments. :::tip @@ -137,17 +138,10 @@ Possible verdicts: ```shell $ forgelm --config configs/quickstart-customer-support.yaml -[2026-04-28 14:01:32] config validated -[2026-04-28 14:01:33] auditing data/customer-support.jsonl (12,400 rows, 3 splits) -[2026-04-28 14:01:35] PII flags: 0 critical, 5 medium · cross-split overlap: 0 -[2026-04-28 14:01:37] SFT epoch 1/3 · loss=2.31 → 1.42 -[2026-04-28 14:18:55] DPO preference pass · β=0.1 · KL=4.2 -[2026-04-28 14:32:11] benchmark hellaswag=0.62 truthfulqa=0.48 -[2026-04-28 14:33:02] Llama Guard S1-S14: clean -[2026-04-28 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv_metadata.json -[2026-04-28 14:33:04] ✔ finished, exit 0 ``` +The output is a stream of timestamped `[INFO]` lines from the `forgelm.*` loggers, with the progress bar and the per-step loss lines that transformers and TRL print. The `customer-support` template is an SFT run with no `evaluation:` block, so the stages you get are: config validation, training, then the files below being written. Stages such as the benchmark, the safety evaluation, the judge and the human-approval gate run only when the YAML enables them. The command ends with exit code `0` on success (see [Exit codes](#/reference/exit-codes)). + ## 6. Try the model ```shell @@ -160,20 +154,32 @@ continues until the end of the current billing period… ## What you got on disk ```text -checkpoints/customer-support/ -├── artifacts/ -│ ├── annex_iv_metadata.json ← Article 11 technical documentation -│ ├── audit_log.jsonl ← Article 12 append-only event log -│ ├── data_audit_report.json ← Article 10 data governance evidence -│ ├── safety_report.json ← Llama Guard verdict -│ ├── benchmark_results.json ← Per-task accuracy -│ └── manifest.json ← SHA-256 over every artifact -├── README.md ← Article 13 model card -├── config_snapshot.yaml ← The exact YAML this run used -└── adapter_model.safetensors ← LoRA weights (or merged checkpoint) +checkpoints/customer-support/ ← training.output_dir +├── audit_log.jsonl ← Article 12 append-only event log, hash-chained +├── audit_log.jsonl.manifest.json ← genesis pin: lets verify-audit detect truncation +├── compliance/ +│ ├── compliance_report.json ← Article 11 manifest (the canonical machine-readable bundle) +│ ├── training_manifest.yaml ← operator-readable summary of the run +│ ├── data_provenance.json ← Article 10 dataset fingerprint +│ └── data_governance_report.json ← Article 10 data-governance evidence +├── checkpoint-/ ← intermediate Trainer checkpoints +└── final_model/ + ├── adapter_model.safetensors ← LoRA weights (or merged checkpoint) + ├── README.md ← Article 13 model card + ├── deployer_instructions.md ← notes for whoever deploys the model + └── model_integrity.json ← SHA-256 checksums of the files in final_model/ ``` -That `artifacts/` directory is the deliverable for compliance reviews. Every file inside is hashed in `manifest.json` for tamper-evidence. +Some files appear only when the YAML asks for them: + +| File | Appears when | +|---|---| +| `compliance/annex_iv_metadata.json` | `compliance.provider_name`, `system_name` or `intended_purpose` is set | +| `safety/safety_results.json` | the safety evaluation is enabled | +| `benchmark/benchmark_results.json` | the benchmark gate is enabled | +| `final_model.staging./` instead of `final_model/` | `evaluation.require_human_approval` is on, until `forgelm approve` promotes it | + +There is no `artifacts/` directory and no single manifest over the whole bundle. Two things are tamper-checked: the audit log by its hash chain (`forgelm verify-audit`) and `final_model/` by `model_integrity.json` (`forgelm verify-integrity`). Nothing hashes `compliance/`, so keep those files where an edit would show, for example in version control or write-once storage. ## Next steps diff --git a/docs/usermanuals/en/getting-started/project-layout.md b/docs/usermanuals/en/getting-started/project-layout.md index 05b26f00..6883382c 100644 --- a/docs/usermanuals/en/getting-started/project-layout.md +++ b/docs/usermanuals/en/getting-started/project-layout.md @@ -22,10 +22,9 @@ my-finetune/ │ └── data_audit_report.json ├── checkpoints/ — model outputs (gitignored) │ └── customer-support/ -│ ├── adapter_model.safetensors -│ ├── README.md — model card (Article 13) -│ ├── config_snapshot.yaml — frozen copy of the config used -│ └── artifacts/ — compliance evidence bundle +│ ├── audit_log.jsonl — append-only, hash-chained event log +│ ├── compliance/ — Article 10/11 evidence (JSON + YAML) +│ └── final_model/ — weights, model card (README.md), model_integrity.json ├── ingested/ — raw documents → JSONL via `forgelm ingest` └── .forgelm/ — local cache (also gitignored) ``` @@ -50,11 +49,11 @@ ForgeLM's defaults play well with version control if you commit the right things **Commit:** - `configs/*.yaml` — your runs are configurations; the YAML is the source of truth. - `audit/*.json` — small, machine-readable, useful in PR reviews ("did the audit numbers regress?"). -- `checkpoints/*/README.md` and `checkpoints/*/artifacts/` (model card and audit bundle, both small). +- `checkpoints/*/final_model/README.md` and `checkpoints/*/compliance/` (model card and compliance evidence, both small). **Gitignore:** - `data/` — usually too large; track via DVC or your preferred dataset registry instead. -- `checkpoints/*/adapter_model.safetensors` and other weight files — too large for git, push to HuggingFace Hub or model registry. +- `checkpoints/*/final_model/*.safetensors`, the `checkpoint-*` directories and other weight files — too large for git, push to HuggingFace Hub or model registry. - `ingested/` — re-buildable from raw documents. - `.forgelm/` — local cache. @@ -125,10 +124,10 @@ Run the dev variant during development, the full one for release. CI runs the fu ## Where compliance artifacts go -Every successful (or failed) training run creates a `checkpoints//artifacts/` directory containing the evidence bundle described in [Compliance Overview](#/compliance/overview). This is the directory you hand to a regulator or pin in your CI artifacts. +A training run writes its evidence into `training.output_dir`: `audit_log.jsonl` (with its `.manifest.json` genesis pin) at the top level and a `compliance/` directory next to `final_model/`; [Your First Run](#/getting-started/first-run) shows the full tree and which files depend on the YAML. There is no `artifacts/` directory. Hand `compliance/` and `audit_log.jsonl` to a reviewer, or pin them in your CI artifacts. :::warn -**Don't merge the artifacts/ directory across runs.** Each run is a separate evidence bundle; mixing them breaks the SHA-256 manifest and undermines tamper-evidence. +**Keep each run's output directory separate.** Each run has its own audit chain and run id; copying two runs' files into one directory mixes them. Only the audit log (`forgelm verify-audit`) and `final_model/` (`forgelm verify-integrity`) are tamper-checked; nothing hashes `compliance/`. ::: ## See also diff --git a/docs/usermanuals/en/operations/air-gap.md b/docs/usermanuals/en/operations/air-gap.md index 40a2807c..c22cff50 100644 --- a/docs/usermanuals/en/operations/air-gap.md +++ b/docs/usermanuals/en/operations/air-gap.md @@ -59,7 +59,7 @@ ERROR: model "meta-llama/Llama-Guard-3-8B" not in local cache. | Subsystem | Flag effect or required operator preparation | |---|---| | HuggingFace Hub downloads | `--offline` sets the HF offline flags; pre-cache the required artifacts. | -| W&B / MLflow / Comet | Disable reporting or configure local-only MLflow with a file URI; `--offline` does not disable these integrations. | +| W&B / MLflow | Set `training.report_to` to `none` (or `tensorboard`), or configure local-only MLflow with a file URI; `--offline` does not disable these integrations. | | OpenAI / Anthropic judges | Disable API-backed evaluation or use a local judge; `--offline` does not disable API clients. | | Webhooks | `--offline` does not disable configured delivery. Omit the `webhook:` block, or leave `url` / `url_env` unset, to disable it; `allow_private_destinations` permits private destinations in addition to public ones. | | pip install (during runtime) | Install required dependencies before disconnecting; `--offline` does not control pip. | diff --git a/docs/usermanuals/en/operations/webhooks.md b/docs/usermanuals/en/operations/webhooks.md index d7cf7107..ffccf7bf 100644 --- a/docs/usermanuals/en/operations/webhooks.md +++ b/docs/usermanuals/en/operations/webhooks.md @@ -1,11 +1,11 @@ --- title: Webhooks -description: Slack and Teams notifications on training events — start, success, failure, auto-revert. +description: Slack-format webhook notifications on training events — start, success, failure, auto-revert. Teams and Discord need a relay. --- # Webhooks -ForgeLM fires structured webhooks on training milestones. Wire them into Slack, Teams, or any incident tool that accepts JSON payloads — get the right context to the right humans without anyone watching a log. +ForgeLM fires structured webhooks on training milestones. Wire them into Slack, or into any incident tool or relay that accepts this JSON payload (Microsoft Teams and Discord need a relay; see below) — get the right context to the right humans without anyone watching a log. ## Quick example @@ -17,8 +17,9 @@ webhook: notify_on_failure: true # default true (covers training.failure + training.reverted) ``` -The notifier emits a generic JSON payload — Slack and Teams ingest it directly via -their incoming-webhook endpoints. Per-event subscription is not currently +The notifier emits one generic JSON payload with a Slack-style `attachments` block. +Slack's incoming webhooks accept it; Microsoft Teams and Discord do not (see +"Slack / Teams / Discord ingestion" below). Per-event subscription is not currently configurable; toggle the three `notify_on_*` flags to coarse-control which lifecycle events fire. @@ -50,40 +51,56 @@ surface mirrored in the ## Payload shape -Single generic JSON shape — Slack / Teams / Discord all accept it -directly via incoming-webhook endpoints; ForgeLM does **not** wrap it in -provider-specific templates: +One generic JSON shape. Every payload carries `event`, `run_name`, `status`, `metrics`, `reason`, +`model_path` and an `attachments` array with one Slack-style block (`title`, `text`, `color`). +ForgeLM does **not** wrap it in provider-specific templates, and it has no top-level `text`, +`content` or `embeds` field: ```json { "event": "training.reverted", - "run_name": "customer-support-v1.2.0", + "run_name": "llama3-support-sft", "status": "reverted", - "reason": "safety regression: S5 hate-speech +0.18 over baseline" + "metrics": {}, + "reason": "Safety gate failed: unsafe_rate 0.08 exceeds max_unsafe_rate 0.05", + "model_path": null, + "attachments": [ + { + "title": "Training Reverted: llama3-support-sft", + "text": "Auto-revert fired. Generated artifacts were deleted because a post-training gate (evaluation, safety, judge, or benchmark) rejected the run.\n\nReason: Safety gate failed: unsafe_rate 0.08 exceeds max_unsafe_rate 0.05", + "color": "#ff9900" + } + ] } ``` -Payload keys vary by event; the full per-event field list is in the -[Audit Event Catalog on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/audit_event_catalog.md) -under the *Webhook lifecycle events* table. +The `pipeline.*` events add a few fields. The full per-event field list is in the +[Webhook Payload Schema on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/webhook_schema.md). ## Slack / Teams / Discord ingestion -The single generic JSON payload shown above is what Slack, Teams, and Discord -all expect on their incoming-webhook endpoints. There are **no per-provider +The payload above is a Slack-style body. There are **no per-provider templates** in `WebhookConfig` (no `template:`, no `events:` allow-list, no `channel:` / `mention_on_failure:` formatting knobs, no per-destination fan-out array). Routing and formatting happen on the receiving side: -- **Slack** — paste the payload into a Slack workflow or incoming-webhook - app; Slack renders the JSON's top-level fields. To get a richer - formatted card, point the webhook at a relay (Slack workflow / AWS - Lambda / your own gateway) that translates the ForgeLM payload into - Slack Block Kit. -- **Microsoft Teams** — similar pattern. Teams renders incoming JSON - natively but the visual is plain; for MessageCard / Adaptive Card - formatting, run a relay. -- **Discord** — accepts the JSON directly via the bot/webhook URL. +- **Slack** — an incoming webhook accepts the body and shows the + `attachments` block (title, text, colour). That is the legacy attachment + format, which Slack treats as secondary; for a richer card point the + webhook at a relay (Slack workflow / AWS Lambda / your own gateway) that + translates the ForgeLM payload into Slack Block Kit. +- **Microsoft Teams** — not accepted directly. Office 365 Connectors were + retired in May 2026, and a Teams Workflows (Power Automate) webhook expects + an Adaptive Card message, which this payload is not. Run a relay that wraps + the payload in an Adaptive Card. +- **Discord** — not accepted directly. A Discord webhook needs `content` or + `embeds` (or a file or poll) in the body, and ForgeLM sends none of them, so + Discord answers with HTTP 400. Run a relay that maps `attachments[0]` to + `content` or an embed. + +A receiver that rejects a delivery does not fail the run, and nothing is paged: +the rejection is a `WARNING` in the run's log (see *Common pitfalls*). Send a +test event to a new destination before relying on it. For multiple destinations, run multiple separate ForgeLM training configs (each pinned to its own `webhook.url_env`) or fan out from a @@ -97,7 +114,7 @@ Real `WebhookConfig` (see `forgelm/config.py::WebhookConfig`): | Field | Default | Notes | |---|---|---| | `url` | `null` | Inline URL — prefer `url_env` for secret hygiene. | -| `url_env` | `null` | Env-var name carrying the URL. Overrides `url` when set. | +| `url_env` | `null` | Env-var name carrying the URL. Read only when `url` is empty; when both are set, `url` wins. | | `notify_on_start` | `true` | Gates `training.start` AND `pipeline.started`. | | `notify_on_success` | `true` | Gates `training.success`, `approval.required`, AND a successful `pipeline.completed`. | | `notify_on_failure` | `true` | Gates `training.failure`, `training.reverted`, `pipeline.stage_reverted`, AND a failing `pipeline.completed`. | @@ -117,7 +134,7 @@ live outside ForgeLM. - **TLS strongly recommended.** ForgeLM permits both HTTPS and HTTP webhook URLs — HTTP destinations log a `Webhook URL uses HTTP (not HTTPS). Data will be sent unencrypted.` warning but are not rejected by default (see `forgelm/webhook.py` `_send`). On a regulated estate set `webhook.require_https: true` to make a plaintext `http://` URL a hard failure (the delivery is refused, not sent). Pin `https://` URLs in production. - **Curated payload.** ForgeLM never includes raw training data, full configs, or unredacted PII in webhook payloads. The notifier wraps a fixed-shape JSON; there is no `webhook.redact` toggle because there's nothing user-controllable to redact. - **Server-Side Request Forgery (SSRF) guard.** ForgeLM blocks webhook URLs pointing at internal IPs — RFC 1918, loopback, link-local (incl. cloud IMDS at `169.254.169.254`), RFC 6598 Shared Address Space / Carrier-Grade-NAT (`100.64.0.0/10`, which also covers Alibaba Cloud ECS's IMDS at `100.100.100.200` — the stdlib `ipaddress` predicates don't flag that block as private or reserved on their own), plus reserved and multicast destinations — unless you explicitly opt-in with `webhook.allow_private_destinations: true`. This prevents misconfigured runs from probing your internal network or leaking the payload to a cloud metadata service. -- **No HMAC body signing.** ForgeLM does not sign webhook bodies — destination-side authenticity falls to TLS + URL secrecy via `url_env` plus the receiving system's bearer-token / signed-request controls (Slack signing secret, Teams connector token). +- **No HMAC body signing.** ForgeLM does not sign webhook bodies — destination-side authenticity falls to TLS + URL secrecy via `url_env` plus whatever authentication the receiver applies itself. A Slack incoming webhook has none beyond its secret URL (Slack's signing secret verifies requests *from* Slack, not requests to a webhook URL), so put your own authentication in front of a custom relay: ForgeLM sends no signature. ## Common pitfalls diff --git a/docs/usermanuals/en/reference/cli.md b/docs/usermanuals/en/reference/cli.md index e286fbb8..5ee639a8 100644 --- a/docs/usermanuals/en/reference/cli.md +++ b/docs/usermanuals/en/reference/cli.md @@ -44,7 +44,7 @@ Run `forgelm --help` for any of these. | `--dry-run` | Validate configuration and check model/dataset access; no training. | | `--fit-check` | Estimate peak training VRAM; no model load. Requires `--config`. | | `--resume [PATH]` | Resume training. Bare `--resume` auto-detects last checkpoint; `--resume PATH` resumes from a specific one. | -| `--offline` | Air-gapped mode: disable all HF Hub network calls. Models and datasets must be available locally. | +| `--offline` | Offline mode: set the Hugging Face offline variables so Hub, transformers and datasets make no network calls. Models and datasets must be available locally. Does not stop webhooks, judge or synthetic-data API clients, or experiment trackers. | | `--benchmark-only MODEL_PATH` | Run benchmark evaluation on an existing model without training. Requires `evaluation.benchmark` config. | | `--merge` | Run model merging from the `merge:` config block. No training. | | `--stage NAME` | Multi-stage pipelines only: run a single named stage in isolation. Non-first stages need the previous stage's on-disk output, or `--input-model`. | diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index ea38e082..ae2a455b 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -228,7 +228,7 @@ The merge field is `method` (not `algorithm`), and there is no `dare_ties` choic ```yaml distributed: strategy: null # null (single-GPU, no distributed wrapping) | deepspeed | fsdp - deepspeed_config: null # path to a DeepSpeed JSON, or preset name: zero2 | zero3 | zero3_offload + deepspeed_config: null # path to a DeepSpeed JSON (preset names zero2 | zero3 | zero3_offload need a source checkout) fsdp_strategy: "full_shard" # full_shard | shard_grad_op fsdp_auto_wrap: true # ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # offload FSDP parameters to CPU between forward/backward @@ -262,8 +262,8 @@ compliance: ```yaml webhook: - url: null # Slack / Teams / Discord / custom; prefer url_env - url_env: null # env var carrying the webhook URL + url: null # Slack incoming webhook, or a relay for Teams / Discord; prefer url_env + url_env: null # env var carrying the webhook URL; read only when url is empty notify_on_start: true notify_on_success: true notify_on_failure: true diff --git a/docs/usermanuals/en/training/distributed.md b/docs/usermanuals/en/training/distributed.md index 8679d546..72321962 100644 --- a/docs/usermanuals/en/training/distributed.md +++ b/docs/usermanuals/en/training/distributed.md @@ -66,7 +66,7 @@ ZeRO-2 shards the optimiser state (the heaviest VRAM component for adaptive opti ```yaml distributed: strategy: "deepspeed" - deepspeed_config: "zero2" # preset name (or a filesystem path to a DeepSpeed JSON) + deepspeed_config: "./ds_zero2.json" # a filesystem path to a DeepSpeed JSON (preset names such as "zero2" need a source checkout) training: gradient_accumulation_steps: 4 @@ -95,7 +95,7 @@ training: gradient_accumulation_steps: 8 ``` -The `zero3_offload` preset fits 70B on 8×24 GB by offloading optimiser state and parameters to CPU. ZeRO-Infinity's NVMe offload isn't a built-in preset — point `deepspeed_config` at a custom DeepSpeed JSON with `offload_param`/`offload_optimizer` set to `device: nvme` for that case. +The `zero3_offload` preset offloads optimiser state and parameters to CPU, trading speed for GPU memory; whether a given model fits depends on your GPU memory and CPU RAM. The preset names `zero2`, `zero3` and `zero3_offload` resolve only from a source checkout (the JSON files live in the repository's `configs/deepspeed/` folder, not in the pip wheel or the Docker image), and the `zero2` preset is currently broken ([#410](https://github.com/HodeTech/ForgeLM/issues/410)), so give `deepspeed_config` the path of a JSON file you control. ZeRO-Infinity's NVMe offload isn't a preset — point `deepspeed_config` at a custom DeepSpeed JSON with `offload_param`/`offload_optimizer` set to `device: nvme` for that case. | Model | GPUs | ZeRO-3 + offload? | |---|---|---| diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index 65a1f336..4761258d 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -48,12 +48,8 @@ webhook: Trainer eval'den sonra durur ve yazdırır: ```text -[2026-04-29 14:33:10] Human approval required. - Run ID: abc123 - Bundle: checkpoints/run/artifacts/ - - To approve: forgelm approve abc123 --output-dir checkpoints/run --comment "..." - To reject: forgelm reject abc123 --output-dir checkpoints/run --comment "..." +2026-04-29 14:33:10 [INFO] forgelm.trainer: Human approval required. Model staged at: checkpoints/run/final_model.staging.abc123 +2026-04-29 14:33:10 [INFO] forgelm.trainer: Review results in checkpoints/run/compliance/ and run `forgelm approve abc123 --output-dir checkpoints/run` to promote, or `forgelm reject abc123 --output-dir checkpoints/run` to preserve for forensic review. ``` Reviewer audit-log + staging dizinine erişimi olan herhangi bir makineden onay komutunu çalıştırır. ForgeLM kimliği `FORGELM_OPERATOR`'dan resolve eder, kararı HMAC ile zincirler ve (`approve`'da) staging dizinini kanonik `final_model/` yoluna rename eder. diff --git a/docs/usermanuals/tr/getting-started/first-run.md b/docs/usermanuals/tr/getting-started/first-run.md index 1f65a3eb..46b1cd20 100644 --- a/docs/usermanuals/tr/getting-started/first-run.md +++ b/docs/usermanuals/tr/getting-started/first-run.md @@ -29,22 +29,23 @@ flowchart LR Her şeyden önce Python, PyTorch, CUDA ve opsiyonel bağımlılıkların doğru bağlandığını kontrol etmek için `forgelm doctor` çalıştırın: ```shell -$ forgelm doctor -forgelm doctor — environment check - - [✓ pass] python.version Python 3.11.4 (CPython). - [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. - [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB). - [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training (Linux GPUs only)). - [✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)). - [! warn] extras.distributed Optional extra missing — install with: pip install 'forgelm[distributed]' (purpose: DeepSpeed ZeRO + offload distributed training (Linux + macOS only)). - [✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200). - [✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. - [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs. - -Summary: 7 pass, 2 warn, 0 fail. +$ forgelm doctor --offline +forgelm doctor - environment check + + [+ pass] python.version Python 3.11.4 (CPython). + [! warn] torch.cuda torch 2.4.0 installed but CUDA is unavailable. CPU-only runs are supported but training will be very slow; expect to use this only for tiny experiments or smoke tests. + [+ pass] numpy.torch_abi torch 2.4.0 + numpy 2.1.0 are ABI-compatible. + [! warn] gpu.inventory No CUDA devices visible; CPU-only mode. + [! warn] extras.distributed Optional extra missing — install with: pip install 'forgelm[distributed]' (purpose: DeepSpeed ZeRO + offload distributed training (Linux + macOS only)). + [+ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs so the audit log identifies a stable identity. + ... + +Summary: 7 pass, 12 warn, 0 fail. ``` +Bu bir alıntıdır: gerçek çıktı yaklaşık 19 satırdır (ForgeLM kurulumunu, her opsiyonel extra'yı ve HF cache'i ya da Hub'ı da denetler); satırlar, değerler ve sayılar makinenize göre değişir. Bir extra için `pass`, modülün bulunabildiği anlamına gelir, import testinden geçtiği anlamına gelmez; durum işaretleri `[+ pass]`, `[! warn]` ve `[x fail]`'dir. + `--output-format json` yapısal bir zarf döner (`{"success": bool, "checks": [...], "summary": {...}}`); CI ayrı ayrı probe sonuçlarını tabloyu parse etmeden filtreleyebilir. `--offline` HF Hub ağ probe'unu atlar ve onun yerine yerel cache'i inceler — air-gap dağıtımları için faydalı. :::tip @@ -137,17 +138,10 @@ Olası verdict'ler: ```shell $ forgelm --config configs/quickstart-customer-support.yaml -[2026-04-29 14:01:32] config validated -[2026-04-29 14:01:33] auditing data/customer-support.jsonl (12,400 rows, 3 splits) -[2026-04-29 14:01:35] PII flags: 0 critical, 5 medium · cross-split overlap: 0 -[2026-04-29 14:01:37] SFT epoch 1/3 · loss=2.31 → 1.42 -[2026-04-29 14:18:55] DPO preference pass · β=0.1 · KL=4.2 -[2026-04-29 14:32:11] benchmark hellaswag=0.62 truthfulqa=0.48 -[2026-04-29 14:33:02] Llama Guard S1-S14: clean -[2026-04-29 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv_metadata.json -[2026-04-29 14:33:04] ✔ finished, exit 0 ``` +Çıktı, `forgelm.*` logger'larından gelen zaman damgalı `[INFO]` satırlarının ve transformers ile TRL'in bastığı ilerleme çubuğu ile adım başı loss satırlarının akışıdır. `customer-support` şablonu `evaluation:` bloğu olmayan bir SFT koşusudur; bu yüzden göreceğiniz aşamalar şunlardır: config doğrulama, eğitim, ardından aşağıdaki dosyaların yazılması. Benchmark, güvenlik değerlendirmesi, judge ve insan-onay kapısı gibi aşamalar yalnızca YAML bunları açtığında çalışır. Komut başarıda `0` çıkış koduyla biter ([Çıkış kodları](#/reference/exit-codes)). + ## 6. Modeli dene ```shell @@ -160,20 +154,32 @@ Erişiminiz mevcut faturalama döneminin sonuna kadar devam eder… ## Diskte ne var ```text -checkpoints/customer-support/ -├── artifacts/ -│ ├── annex_iv_metadata.json ← Article 11 teknik dokümantasyon -│ ├── audit_log.jsonl ← Article 12 append-only event log -│ ├── data_audit_report.json ← Article 10 veri yönetimi kanıtı -│ ├── safety_report.json ← Llama Guard verdict -│ ├── benchmark_results.json ← Görev başı doğruluk -│ └── manifest.json ← Her artifact üzerinde SHA-256 -├── README.md ← Article 13 model card -├── config_snapshot.yaml ← Bu koşunun kullandığı tam YAML -└── adapter_model.safetensors ← LoRA ağırlıkları (veya birleştirilmiş checkpoint) +checkpoints/customer-support/ ← training.output_dir +├── audit_log.jsonl ← Article 12 append-only event log, hash zincirli +├── audit_log.jsonl.manifest.json ← genesis pin: verify-audit'in kesilmeyi tespit etmesini sağlar +├── compliance/ +│ ├── compliance_report.json ← Article 11 manifesti (kanonik makine-okunur paket) +│ ├── training_manifest.yaml ← koşunun operatör-okunur özeti +│ ├── data_provenance.json ← Article 10 veri seti parmak izi +│ └── data_governance_report.json ← Article 10 veri yönetimi kanıtı +├── checkpoint-/ ← ara Trainer checkpoint'leri +└── final_model/ + ├── adapter_model.safetensors ← LoRA ağırlıkları (veya birleştirilmiş checkpoint) + ├── README.md ← Article 13 model card + ├── deployer_instructions.md ← modeli dağıtacak kişi için notlar + └── model_integrity.json ← final_model/ içindeki dosyaların SHA-256 özetleri ``` -O `artifacts/` dizini compliance incelemeleri için teslim edilebilirdir. İçindeki her dosya `manifest.json`'da tamper-evidence için hashlenmiştir. +Bazı dosyalar yalnızca YAML istediğinde oluşur: + +| Dosya | Ne zaman oluşur | +|---|---| +| `compliance/annex_iv_metadata.json` | `compliance.provider_name`, `system_name` veya `intended_purpose` ayarlıysa | +| `safety/safety_results.json` | güvenlik değerlendirmesi açıksa | +| `benchmark/benchmark_results.json` | benchmark kapısı açıksa | +| `final_model/` yerine `final_model.staging./` | `evaluation.require_human_approval` açıksa, `forgelm approve` onu terfi ettirene kadar | + +`artifacts/` dizini ve tüm paketi kapsayan tek bir manifest yoktur. İki şey tamper-check'lidir: audit log kendi hash zinciriyle (`forgelm verify-audit`) ve `final_model/` `model_integrity.json` ile (`forgelm verify-integrity`). `compliance/` dosyalarını hiçbir şey hash'lemez; bu yüzden onları bir değişikliğin görüneceği bir yerde tutun, örneğin sürüm kontrolünde ya da bir kez yazılabilir depolamada. ## Sonraki adımlar diff --git a/docs/usermanuals/tr/getting-started/project-layout.md b/docs/usermanuals/tr/getting-started/project-layout.md index e24bf6ef..568043f9 100644 --- a/docs/usermanuals/tr/getting-started/project-layout.md +++ b/docs/usermanuals/tr/getting-started/project-layout.md @@ -22,10 +22,9 @@ my-finetune/ │ └── data_audit_report.json ├── checkpoints/ — model çıktıları (gitignored) │ └── customer-support/ -│ ├── adapter_model.safetensors -│ ├── README.md — model card (Article 13) -│ ├── config_snapshot.yaml — kullanılan config'in dondurulmuş kopyası -│ └── artifacts/ — compliance kanıt paketi +│ ├── audit_log.jsonl — append-only, hash zincirli olay günlüğü +│ ├── compliance/ — Article 10/11 kanıtı (JSON + YAML) +│ └── final_model/ — ağırlıklar, model card (README.md), model_integrity.json ├── ingested/ — `forgelm ingest` üzerinden ham döküman → JSONL └── .forgelm/ — yerel cache (gitignored) ``` @@ -50,11 +49,11 @@ ForgeLM varsayılanları doğru şeyleri commit'leyip gerisini ignore'larsanız **Commit:** - `configs/*.yaml` — koşularınız konfigürasyon; YAML kaynak doğrudur. - `audit/*.json` — küçük, makine-okunur, PR review'larında faydalı ("audit numaraları geriledi mi?"). -- `checkpoints/*/README.md` ve `checkpoints/*/artifacts/` (model card ve audit paketi, ikisi de küçük). +- `checkpoints/*/final_model/README.md` ve `checkpoints/*/compliance/` (model card ve compliance kanıtı, ikisi de küçük). **Gitignore:** - `data/` — genelde fazla büyük; DVC veya tercih ettiğiniz dataset registry'sine alın. -- `checkpoints/*/adapter_model.safetensors` ve diğer ağırlık dosyaları — git için fazla büyük; HuggingFace Hub veya model registry'ye gönderin. +- `checkpoints/*/final_model/*.safetensors`, `checkpoint-*` dizinleri ve diğer ağırlık dosyaları — git için fazla büyük; HuggingFace Hub veya model registry'ye gönderin. - `ingested/` — ham dokümanlardan yeniden inşa edilebilir. - `.forgelm/` — yerel cache. @@ -123,10 +122,10 @@ Geliştirme sırasında dev varyantı, release için tam olanını koşturun. CI ## Compliance artifact'ları nereye gider -Her başarılı (veya başarısız) eğitim koşusu, [Compliance Genel Bakış](#/compliance/overview) bölümünde anlatılan kanıt paketini içeren bir `checkpoints//artifacts/` dizini oluşturur. Regülatöre veya CI artifact'larında pinleyeceğiniz dizin budur. +Bir eğitim koşusu kanıtını `training.output_dir` içine yazar: en üstte `audit_log.jsonl` (`.manifest.json` genesis pin'iyle) ve `final_model/` yanında bir `compliance/` dizini; tam ağacı ve hangi dosyaların YAML'a bağlı olduğunu [İlk Koşunuz](#/getting-started/first-run) gösterir. `artifacts/` dizini yoktur. `compliance/` ve `audit_log.jsonl`'i bir incelemeciye teslim edin ya da CI artifact'larında pinleyin. :::warn -**`artifacts/` dizinini koşular arasında birleştirmeyin.** Her koşu ayrı bir kanıt paketidir; karıştırmak SHA-256 manifesti bozar ve tamper-evidence'ı baltalar. +**Her koşunun çıktı dizinini ayrı tutun.** Her koşunun kendi audit zinciri ve run id'si vardır; iki koşunun dosyalarını tek dizine kopyalamak onları karıştırır. Yalnızca audit log (`forgelm verify-audit`) ve `final_model/` (`forgelm verify-integrity`) tamper-check'lidir; `compliance/`'ı hiçbir şey hash'lemez. ::: ## Bkz. diff --git a/docs/usermanuals/tr/operations/air-gap.md b/docs/usermanuals/tr/operations/air-gap.md index c5744481..39ede22f 100644 --- a/docs/usermanuals/tr/operations/air-gap.md +++ b/docs/usermanuals/tr/operations/air-gap.md @@ -59,7 +59,7 @@ ERROR: model "meta-llama/Llama-Guard-3-8B" not in local cache. | Alt sistem | Bayrağın etkisi veya operatörün yapması gereken hazırlık | |---|---| | HuggingFace Hub indirmeleri | `--offline`, HF çevrimdışı ayarlarını yapar; gerekli artefaktları önceden önbelleğe alın. | -| W&B / MLflow / Comet | Raporlamayı kapatın veya MLflow'u file URI ile yalnızca yerel çalışacak şekilde yapılandırın; `--offline` bu entegrasyonları kapatmaz. | +| W&B / MLflow | `training.report_to`'yu `none` (veya `tensorboard`) yapın ya da MLflow'u file URI ile yalnızca yerel çalışacak şekilde yapılandırın; `--offline` bu entegrasyonları kapatmaz. | | OpenAI / Anthropic judge'lar | API tabanlı değerlendirmeyi kapatın veya yerel judge kullanın; `--offline` API istemcilerini kapatmaz. | | Webhook'lar | `--offline`, yapılandırılmış teslimatları kapatmaz. Kapatmak için `webhook:` bloğunu çıkarın veya `url` / `url_env` değerlerini boş bırakın; `allow_private_destinations`, genel hedeflerin yanında özel hedeflere de izin verir. | | Runtime'da pip install | Gerekli bağımlılıkları bağlantıyı kesmeden kurun; `--offline`, pip'i yönetmez. | diff --git a/docs/usermanuals/tr/operations/webhooks.md b/docs/usermanuals/tr/operations/webhooks.md index 6f80a377..722e15d5 100644 --- a/docs/usermanuals/tr/operations/webhooks.md +++ b/docs/usermanuals/tr/operations/webhooks.md @@ -1,11 +1,11 @@ --- title: Webhook'lar -description: Eğitim olaylarında Slack ve Teams bildirimleri — başlangıç, başarı, hata, otomatik geri alma. +description: Eğitim olaylarında Slack biçiminde webhook bildirimleri — başlangıç, başarı, hata, otomatik geri alma. Teams ve Discord için relay gerekir. --- # Webhook'lar -ForgeLM eğitim dönüm noktalarında yapılandırılmış webhook fırlatır. Bunları Slack, Teams veya JSON payload kabul eden herhangi bir incident aracına bağlayın — hiç kimsenin log izlemesine gerek kalmadan doğru bağlamı doğru insanlara getirin. +ForgeLM eğitim dönüm noktalarında yapılandırılmış webhook fırlatır. Bunları Slack'e ya da bu JSON payload'u kabul eden herhangi bir incident aracına veya relay'e bağlayın (Microsoft Teams ve Discord için relay gerekir; aşağıya bakın) — hiç kimsenin log izlemesine gerek kalmadan doğru bağlamı doğru insanlara getirin. ## Hızlı örnek @@ -17,8 +17,9 @@ webhook: notify_on_failure: true # default true (training.failure + training.reverted'i kapsar) ``` -Notifier generic JSON payload yayar — Slack ve Teams bunu incoming-webhook -uçlarından doğrudan kabul eder. Per-event abonelik şu an konfigüre +Notifier, Slack tarzı bir `attachments` bloğu taşıyan tek bir generic JSON payload yayar. +Slack'in incoming webhook'ları bunu kabul eder; Microsoft Teams ve Discord etmez +(aşağıdaki "Slack / Teams / Discord ingest" bölümüne bakın). Per-event abonelik şu an konfigüre edilemiyor; üç `notify_on_*` flag'iyle hangi yaşam-döngüsü olaylarının fırlatılacağı kaba ayarlanır. @@ -50,41 +51,58 @@ ile aynalanan kanonik yüzeydir: ## Payload yapısı -Tek generic JSON şekil — Slack / Teams / Discord hepsi bunu incoming-webhook -uçlarından doğrudan kabul eder; ForgeLM provider'a-özel template'lerle -sarmalama **yapmaz**: +Tek generic JSON şekil. Her payload `event`, `run_name`, `status`, `metrics`, `reason`, +`model_path` ve tek bir Slack tarzı blok (`title`, `text`, `color`) içeren bir `attachments` +dizisi taşır. ForgeLM provider'a-özel template'lerle sarmalama **yapmaz** ve üst düzeyde +`text`, `content` veya `embeds` alanı yoktur: ```json { "event": "training.reverted", - "run_name": "customer-support-v1.2.0", + "run_name": "llama3-support-sft", "status": "reverted", - "reason": "safety regression: S5 hate-speech +0.18 over baseline" + "metrics": {}, + "reason": "Safety gate failed: unsafe_rate 0.08 exceeds max_unsafe_rate 0.05", + "model_path": null, + "attachments": [ + { + "title": "Training Reverted: llama3-support-sft", + "text": "Auto-revert fired. Generated artifacts were deleted because a post-training gate (evaluation, safety, judge, or benchmark) rejected the run.\n\nReason: Safety gate failed: unsafe_rate 0.08 exceeds max_unsafe_rate 0.05", + "color": "#ff9900" + } + ] } ``` -Payload anahtarları event'e göre değişir; tam per-event alan listesi -[GitHub'daki Audit Event Kataloğu](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/audit_event_catalog.md) -*Webhook lifecycle events* tablosundadır. +`pipeline.*` event'leri birkaç alan ekler. Tam per-event alan listesi +[GitHub'daki Webhook Payload Şeması](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/webhook_schema.md) +sayfasındadır. ## Slack / Teams / Discord ingest -Yukarıdaki tek generic JSON payload, Slack, Teams ve Discord'un -incoming-webhook endpoint'lerinde beklediği şeydir. `WebhookConfig`'te +Yukarıdaki payload Slack tarzı bir gövdedir. `WebhookConfig`'te **provider başına template yoktur** (no `template:`, no `events:` allow-list, no `channel:` / `mention_on_failure:` formatlama knob'ları, no per-destination fan-out array). Routing ve formatlama alıcı tarafta yapılır: -- **Slack** — payload'u Slack workflow veya incoming-webhook - uygulamasına yapıştırın; Slack JSON'un top-level alanlarını render - eder. Daha zengin formatlanmış bir kart için, webhook'u ForgeLM - payload'unu Slack Block Kit'e çeviren bir relay'e (Slack +- **Slack** — incoming webhook gövdeyi kabul eder ve `attachments` + bloğunu (başlık, metin, renk) gösterir. Bu eski attachment biçimidir + ve Slack onu ikincil sayar; daha zengin bir kart için webhook'u + ForgeLM payload'unu Slack Block Kit'e çeviren bir relay'e (Slack workflow / AWS Lambda / kendi gateway'iniz) yönlendirin. -- **Microsoft Teams** — benzer pattern. Teams gelen JSON'u natively - render eder ama görsel düz; MessageCard / Adaptive Card - formatlama için bir relay çalıştırın. -- **Discord** — JSON'u doğrudan bot/webhook URL üzerinden kabul eder. +- **Microsoft Teams** — doğrudan kabul edilmez. Office 365 Connector'lar + Mayıs 2026'da kapatıldı ve bir Teams Workflows (Power Automate) + webhook'u Adaptive Card mesajı bekler; bu payload öyle değildir. + Payload'u Adaptive Card'a saran bir relay çalıştırın. +- **Discord** — doğrudan kabul edilmez. Discord webhook'u gövdede + `content` veya `embeds` (ya da dosya ya da anket) ister; ForgeLM bunların + hiçbirini göndermez, bu yüzden Discord HTTP 400 döner. `attachments[0]`'ı + `content`'e veya bir embed'e eşleyen bir relay çalıştırın. + +Teslimatı reddeden bir alıcı run'ı başarısız etmez ve kimseye alarm gitmez: +ret, run log'unda bir `WARNING`'dir (bkz. *Sık hatalar*). Yeni bir hedefe güvenmeden +önce ona bir test event'i gönderin. Birden çok hedef için, her biri kendi `webhook.url_env`'ine pinlenmiş birden çok ayrı ForgeLM training config'i çalıştırın @@ -99,7 +117,7 @@ Gerçek `WebhookConfig` (bkz. `forgelm/config.py::WebhookConfig`): | Alan | Vars. | Notlar | |---|---|---| | `url` | `null` | Inline URL — secret hijyeni için `url_env`'i tercih edin. | -| `url_env` | `null` | URL'i taşıyan env var adı. Set edildiğinde `url`'i override eder. | +| `url_env` | `null` | URL'i taşıyan env var adı. Yalnızca `url` boşken okunur; ikisi de set ise `url` kazanır. | | `notify_on_start` | `true` | `training.start` VE `pipeline.started`'ı gate'ler. | | `notify_on_success` | `true` | `training.success`, `approval.required` VE başarılı bir `pipeline.completed`'ı gate'ler. | | `notify_on_failure` | `true` | `training.failure`, `training.reverted`, `pipeline.stage_reverted` VE başarısız bir `pipeline.completed`'ı gate'ler. | @@ -118,7 +136,7 @@ payload zaten curated), ve routing hepsi ForgeLM'in dışında yaşar. - **TLS şiddetle önerilir.** ForgeLM hem HTTPS hem HTTP webhook URL'lerine izin verir — HTTP hedefleri `Webhook URL uses HTTP (not HTTPS). Data will be sent unencrypted.` uyarısı loglar ama varsayılan olarak reddedilmez (bkz. `forgelm/webhook.py` `_send`). Regüle bir ortamda `webhook.require_https: true` ile plaintext bir `http://` URL'ini hard failure yapın (teslimat reddedilir, gönderilmez). Üretimde `https://` URL'leri pinleyin. - **Curated payload.** ForgeLM webhook payload'larına asla raw eğitim verisi, tam config'ler veya unredacted PII koymaz. Notifier sabit-şekilli bir JSON sarar; `webhook.redact` toggle'ı yoktur çünkü kullanıcı-kontrollü redakte edilecek bir şey yok. - **SSRF guard.** ForgeLM iç IP'lere işaret eden webhook URL'lerini engeller — RFC 1918, loopback, link-local (bulut IMDS `169.254.169.254` dahil), RFC 6598 Shared Address Space / Carrier-Grade-NAT (`100.64.0.0/10`; bu blok, Alibaba Cloud ECS'in IMDS'ini de kapsar — `100.100.100.200` — stdlib `ipaddress` predicate'leri bu bloğu kendiliğinden private veya reserved olarak işaretlemez), ayrıca reserved ve multicast hedefler — `webhook.allow_private_destinations: true` ile açıkça opt-in olmadıkça. Yanlış konfigüre koşuların iç ağınızı sondalamasını veya payload'ı bir bulut metadata servisine sızdırmasını önler. -- **HMAC body imzalama yok.** ForgeLM webhook gövdelerini imzalamaz — hedef-tarafı authenticity TLS + `url_env` üzerinden URL gizliliği artı alıcı sistemin bearer-token / signed-request kontrollerine (Slack signing secret, Teams connector token) düşer. +- **HMAC body imzalama yok.** ForgeLM webhook gövdelerini imzalamaz — hedef-tarafı authenticity TLS + `url_env` üzerinden URL gizliliği artı alıcının kendi uyguladığı kimlik doğrulamaya düşer. Bir Slack incoming webhook'unun gizli URL'sinden başka doğrulaması yoktur (Slack'in signing secret'ı Slack'ten *gelen* istekleri doğrular, bir webhook URL'sine gelenleri değil); bu yüzden özel bir relay'in önüne kendi kimlik doğrulamanızı koyun: ForgeLM imza göndermez. ## Sık hatalar diff --git a/docs/usermanuals/tr/reference/cli.md b/docs/usermanuals/tr/reference/cli.md index eae80b15..c552e54b 100644 --- a/docs/usermanuals/tr/reference/cli.md +++ b/docs/usermanuals/tr/reference/cli.md @@ -44,7 +44,7 @@ Bunlardan herhangi biri için `forgelm --help`. | `--dry-run` | Config'i ve model/dataset erişimini doğrula; eğitim yok. | | `--fit-check` | Eğitim VRAM tahmini; model yüklenmez. `--config` gerektirir. | | `--resume [PATH]` | Eğitime kaldığı yerden devam. Çıplak `--resume` son checkpoint'i otomatik bulur; `--resume PATH` belirli bir yerden. | -| `--offline` | Air-gap modu: tüm HF Hub ağ çağrılarını kapat. Modeller ve dataset'ler yerel olarak mevcut olmalı. | +| `--offline` | Offline mod: Hugging Face offline değişkenlerini ayarlar; Hub, transformers ve datasets ağ çağrısı yapmaz. Modeller ve dataset'ler yerel olarak mevcut olmalı. Webhook'ları, judge veya sentetik-veri API istemcilerini ve deney izleyicilerini durdurmaz. | | `--benchmark-only MODEL_PATH` | Mevcut bir model üzerinde benchmark koştur, eğitim yok. `evaluation.benchmark` config'i gerektirir. | | `--merge` | Config'in `merge:` bloğundan model birleştirmeyi koştur. Eğitim yok. | | `--stage NAME` | Yalnızca çok aşamalı pipeline'lar: adı verilen tek bir stage'i izole koştur. İlk olmayan stage'ler önceki stage'in on-disk çıktısına ya da `--input-model` override'ına ihtiyaç duyar. | diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 690f806b..753f164c 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -230,7 +230,7 @@ Merge alanı `algorithm` değil `method`'dur, `dare_ties` seçeneği, `base_mode ```yaml distributed: strategy: null # null (tek GPU, dağıtık sarmalama yok) | deepspeed | fsdp - deepspeed_config: null # bir DeepSpeed JSON'a yol, veya preset adı: zero2 | zero3 | zero3_offload + deepspeed_config: null # bir DeepSpeed JSON'a yol (preset adları zero2 | zero3 | zero3_offload kaynak checkout'u ister) fsdp_strategy: "full_shard" # full_shard | shard_grad_op fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # forward/backward arasında FSDP parametrelerini CPU'ya offload et @@ -264,8 +264,8 @@ compliance: ```yaml webhook: - url: null # Slack / Teams / Discord / özel; url_env tercihli - url_env: null # webhook URL'sini taşıyan env değişkeni + url: null # Slack incoming webhook'u ya da Teams / Discord için relay; url_env tercihli + url_env: null # webhook URL'sini taşıyan env değişkeni; yalnızca url boşken okunur notify_on_start: true notify_on_success: true notify_on_failure: true diff --git a/docs/usermanuals/tr/training/distributed.md b/docs/usermanuals/tr/training/distributed.md index 7e73a6d7..7c1dc991 100644 --- a/docs/usermanuals/tr/training/distributed.md +++ b/docs/usermanuals/tr/training/distributed.md @@ -66,7 +66,7 @@ ZeRO-2 optimizer state'i sharder (Adam gibi adaptif optimizer'larda en ağır VR ```yaml distributed: strategy: "deepspeed" - deepspeed_config: "zero2" # preset adı (veya bir DeepSpeed JSON dosya yolu) + deepspeed_config: "./ds_zero2.json" # bir DeepSpeed JSON dosya yolu ("zero2" gibi preset adları kaynak checkout'u ister) training: gradient_accumulation_steps: 4 @@ -95,7 +95,7 @@ training: gradient_accumulation_steps: 8 ``` -`zero3_offload` preset'i, optimizer state ve parametreleri CPU'ya boşaltarak 70B'i 8×24 GB'a sığdırır. ZeRO-Infinity'nin NVMe offload'u hazır bir preset değildir — bunun için `deepspeed_config`'i, `offload_param`/`offload_optimizer` alanları `device: nvme` olarak ayarlanmış özel bir DeepSpeed JSON dosyasına yönlendirin. +`zero3_offload` preset'i optimizer state ve parametreleri CPU'ya boşaltır, hızdan GPU belleği kazandırır; belirli bir modelin sığıp sığmayacağı GPU belleğinize ve CPU RAM'inize bağlıdır. `zero2`, `zero3` ve `zero3_offload` preset adları yalnızca kaynak checkout'unda çözülür (JSON dosyaları pip wheel'inde ya da Docker imajında değil, deponun `configs/deepspeed/` klasöründedir) ve `zero2` preset'i şu an bozuktur ([#410](https://github.com/HodeTech/ForgeLM/issues/410)); bu yüzden `deepspeed_config`'e kontrolünüzdeki bir JSON dosyasının yolunu verin. ZeRO-Infinity'nin NVMe offload'u bir preset değildir — bunun için `deepspeed_config`'i, `offload_param`/`offload_optimizer` alanları `device: nvme` olarak ayarlanmış özel bir DeepSpeed JSON dosyasına yönlendirin. | Model | GPU | ZeRO-3 + offload? | |---|---|---| diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 562c6343..b76d985b 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1368,7 +1368,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument( "--offline", action="store_true", - help="Air-gapped mode: disable all HF Hub network calls. Models and datasets must be available locally.", + help="Offline mode: set the Hugging Face offline variables so Hub, transformers and datasets make no network calls. Models and datasets must be available locally. Does not stop webhooks, judge or synthetic-data API clients, or experiment trackers.", ) mode_group.add_argument( "--benchmark-only", diff --git a/forgelm/config.py b/forgelm/config.py index 33e90e25..27f644ca 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -269,7 +269,7 @@ class ModelConfig(BaseModel): ) offline: bool = Field( default=False, - description="Air-gapped mode: refuse HF Hub network calls. Models/datasets/extras must be available locally.", + description="Offline mode: set the HF offline variables (Hub, transformers, datasets). Models/datasets/extras must be available locally. Webhooks, judge/synthetic API clients and trackers are not stopped.", ) revision: Optional[str] = Field( default=None, @@ -768,7 +768,7 @@ class DistributedConfig(BaseModel): ) deepspeed_config: Optional[str] = Field( default=None, - description="DeepSpeed config: filesystem path to a DS JSON OR preset name (`zero2`, `zero3`, `zero3_offload`).", + description="DeepSpeed config: filesystem path to a DS JSON. The preset names (`zero2`, `zero3`, `zero3_offload`) resolve only from a source checkout, not a pip or Docker install.", ) fsdp_strategy: Literal["full_shard", "shard_grad_op"] = Field( default="full_shard", @@ -1523,17 +1523,17 @@ class WebhookConfig(BaseModel): url: Optional[str] = Field( default=None, - description="Webhook URL (Slack / Teams / Discord / custom). Use `url_env` to read from env to avoid committing secrets.", + description="Webhook URL (a Slack incoming webhook, or a relay that forwards to Teams / Discord / custom). Use `url_env` to avoid committing secrets.", ) url_env: Optional[str] = Field( - default=None, description="Env var name carrying the webhook URL (overrides `url` when set)." + default=None, description="Env var name carrying the webhook URL; read only when `url` is empty." ) notify_on_start: bool = Field(default=True, description="POST a `notify_start` event when training begins.") notify_on_success: bool = Field( default=True, description="POST a `notify_success` event when training completes successfully." ) notify_on_failure: bool = Field( - default=True, description="POST a `notify_failure` event when training fails (any non-zero exit)." + default=True, description="POST a failure event when training raises or a post-training gate reverts the model." ) timeout: int = Field( default=10, @@ -1677,7 +1677,7 @@ class ForgeConfig(BaseModel): description="Post-training evaluation block (loss / benchmark / safety / judge / human-approval gate).", ) webhook: Optional[WebhookConfig] = Field( - default=None, description="Webhook notification block (Slack / Teams / Discord / custom)." + default=None, description="Webhook notification block (Slack-format JSON; Teams and Discord need a relay)." ) distributed: Optional[DistributedConfig] = Field( default=None, description="DeepSpeed / FSDP multi-GPU configuration block."