diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index ae7d4821..32b9e699 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -237,14 +237,17 @@ jobs: # EXPLICITLY so this job tests the true documented minimum — trl puts # no upper bound on transformers, so an unpinned resolve would silently # pull the newest 5.x and stop exercising the floor. transformers - # 5.3.0 is the CVE-2026-4372 fix floor and requires torch>=2.4 / - # huggingface_hub>=1.3 / tokenizers>=0.22; trl 1.0.0 pulls the - # accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the pyproject - # [project].dependencies minimums. + # 5.10.1 is the first installable release past CVE-2026-4372, + # CVE-2026-5241 and CVE-2026-9856 (5.10.0, which carries the last of + # them, was yanked); it requires huggingface_hub>=1.5 / + # tokenizers>=0.22, and ForgeLM itself needs torch>=2.4; trl 1.0.0 + # pulls the accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the + # pyproject [project].dependencies minimums (tests/test_packaging.py + # compares them). pip install \ "torch==2.4.0" \ - "transformers==5.3.0" \ - "huggingface_hub==1.3.0" \ + "transformers==5.10.1" \ + "huggingface_hub==1.5.0" \ "peft==0.19.0" \ "accelerate==1.4.0" \ "datasets==4.7.0" \ diff --git a/CHANGELOG.md b/CHANGELOG.md index b3ab9531..c224ea95 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,14 @@ All notable changes to ForgeLM are documented here. ### Changed +- Raise the `transformers` floor from `>=5.3.0` to `>=5.10.1` (and `huggingface_hub` from `>=1.3.0` to `>=1.5.0`, which + transformers 5.10 requires). `5.3.0` was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue + `config.json` overrides `trust_remote_code: false`, fixed in 5.5.0) and CVE-2026-9856 (a `chat_template` name in + `tokenizer_config.json` is used unsanitised as a file name by `save_pretrained`, so `final_model/` saves, GGUF export + and merges could write outside the output directory; fixed in 5.10.0, which PyPI yanked, hence 5.10.1). The nightly + minimum-versions job pins the new floor, and `tests/test_packaging.py` keeps it from drifting from + `pyproject.toml` and checks that a traversal `chat_template` name cannot escape on save. Environments pinned to + transformers 5.3 to 5.9 must upgrade (#109). - Record monitoring default provenance in `defaulted_fields` and mark literal URL projection with `endpoint_redacted` (also true when the URL already contains only scheme/host/port). Literal endpoints retain scheme/host/port in JSON evidence, including IPv6 brackets; userinfo, path, @@ -79,6 +87,36 @@ All notable changes to ForgeLM are documented here. ### Fixed +- `training.rope_scaling` works on transformers 5 again. The setting was passed to `from_pretrained(rope_scaling=...)`, + which no longer works: the documented `{type: linear, factor: 4}` raised `KeyError: 'rope_type'` while the model was + built, and `{rope_type: ..., factor: ...}` replaced the whole parameter dict so `rope_theta` was lost (`TypeError`). + The requested keys are now merged over the model's own `rope_parameters` (the legacy key `type` is read as + `rope_type`), applied to the text config of a multimodal checkpoint, and the finished config is loaded. The + `sliding_window_attention` override travels in the same config object. Verified on transformers 5.13.0 for + `linear`, `dynamic` and `yarn` on a tiny local Llama; not exercised on the 5.3-5.9 range the floor still admits (#103). +- The `forgelm` CLI no longer sends Hugging Face's anonymous usage ping. TRL's trainers call `send_telemetry` when they are + built, and `huggingface_hub` sends it unless `HF_HUB_DISABLE_TELEMETRY` is true or offline mode is on, which contradicted the + privacy page ("does not send usage data anywhere"). `main()` now sets `HF_HUB_DISABLE_TELEMETRY=1` before anything imports + `huggingface_hub`, and an operator's own value wins (`HF_HUB_DISABLE_TELEMETRY=0` opts back in). Using ForgeLM as a library + does not set it (#95). +- `retention.enforce` is documented for what it is: reserved. No command reads it, so `warn_on_excess` and + `block_on_excess` never warned or blocked anything and nothing was deleted automatically; retention is only reported, + read-only, by `forgelm purge --check-policy` (always exit 0). The config description, the wizard prompt, the + configuration reference, the GDPR erasure guide and `config_template.yaml` now say so (English and Turkish), and + the CLI logs a warning when a config sets a non-default `enforce` instead of staying silent (#92, #98). +- `distributed.fsdp_strategy: no_shard` and `hybrid_shard` are rejected when the config loads (exit 1). They were documented, passed `--dry-run`, + and then failed with exit 2 at trainer construction (`fsdp=hybrid_shard is only available with FSDP1`), because transformers 5.10+ runs + FSDP2 by default. `fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted but change nothing under FSDP2; + ForgeLM now logs a warning when either is set, and the field descriptions, reference pages, user manuals and `config_template.yaml` + say so. Moving to FSDP2 settings (`fsdp=True` plus an explicit `fsdp_config`) is not part of this change; the string form is + deprecated in transformers and slated for removal in v5.20 (#104). +- `forgelm purge --row-id` without `--output-dir` now logs a warning that names the directory it resolved (the corpus's) + and the `final_model/` the memorisation check will look for. The documentation promised that warning but the code never + logged it, and a default run looks for a model next to the corpus, where a training run rarely writes one, so it finished + with a completed-erasure record and no memorisation event. The audit-event catalogue, the purge reference, the GDPR + erasure guide, the user manual and the risk-treatment plan now state the real condition: a `final_model/` directly under + `--output-dir`, not "any run that consumed the corpus" (English and Turkish). Reading `training.output_dir` from + `--config` is still open (#102). - Correct the GGUF pages (English and Turkish): the export example shows the real output, K-quants are described as the two-step operation they are, the invented "manifest" and `verify-gguf` output (vocabulary and tokenizer checks that do not exist) are replaced by the real ones, a clean `verify-gguf` exit is said to prove only what ran, and the sidecar diff --git a/config_template.yaml b/config_template.yaml index 8aec1398..acf197cb 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -167,7 +167,7 @@ data: # strategy: "deepspeed" # "deepspeed" or "fsdp" # deepspeed_config: "zero2" # preset: "zero2", "zero3", "zero3_offload" or path to JSON # # --- FSDP options (when strategy: "fsdp") --- -# # fsdp_strategy: "full_shard" # "full_shard", "shard_grad_op", "no_shard", "hybrid_shard" +# # fsdp_strategy: "full_shard" # "full_shard" or "shard_grad_op" (FSDP2 supports no other) # # fsdp_auto_wrap: true # # fsdp_offload: false @@ -276,7 +276,7 @@ data: # staging_ttl_days: 7 # final_model.staging./ TTL after `forgelm reject` # ephemeral_artefact_retention_days: 90 # compliance bundles + audit reports # raw_documents_retention_days: 90 # ingested raw docs (PDF/DOCX/EPUB/TXT/MD) -# enforce: "log_only" # "log_only" | "warn_on_excess" | "block_on_excess" +# enforce: "log_only" # reserved: no command acts on it yet ("log_only" | "warn_on_excess" | "block_on_excess") # Phase 14 — Multi-stage training pipeline chains (SFT → DPO → GRPO). # Uncomment the ``pipeline:`` block below to run multiple training stages diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 64a1bb16..7fa432c1 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -19,7 +19,7 @@ **Yapmaz:** -- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, corpus'u tüketen herhangi bir koşum için `final_model/` mevcut olduğunda `data.erasure_warning_memorisation` yayar; böylece operatör boşluğu görür. +- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. - Audit log'un kendisinden entry silmez. Madde 17(3)(b) audit / muhasebe kayıtlarını bir hukuki yükümlülük savunması olarak korur; tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. @@ -78,10 +78,10 @@ retention: staging_ttl_days: 7 # `forgelm reject` üzerine aksiyon almak için bir iş haftası ephemeral_artefact_retention_days: 90 # üç-aylık review cadence'i raw_documents_retention_days: 90 # data audit yeniden çalıştırmadan önce ingestion penceresi - enforce: log_only # log_only / warn_on_excess / block_on_excess + enforce: log_only # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess) ``` -Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi devre dışı bırakır (sınırsız tut). `enforce`, trainer pre-flight gate'inin ihlallere nasıl tepki vereceğini kontrol eder. +Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi devre dışı bırakır (sınırsız tut). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur, her zaman 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. `enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz, bu yüzden üç değer de `log_only` gibi davranır. ## Kaldırıldı: `evaluation.staging_ttl_days` @@ -96,7 +96,7 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden önce (`--check-policy` salt-okunur; audit event yaymaz) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme bittiğinde | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Yalnız row modunda.** Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. (Run modu bu uyarıyı yüzeylemez — synthetic veri corpus-scoped'dur, run-scoped değil.) | Tüm `completed` field'ları + `synthetic_files` | | `data.erasure_warning_external_copies` | Yüklü config webhook block'u içeriyor | Tüm `completed` field'ları + `webhook_targets` | diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 24941879..fd4f7e21 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -19,7 +19,7 @@ **Does not:** -- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` exists for any run that consumed the corpus, so the operator sees the gap. +- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. - Delete entries from the audit log itself. Article 17(3)(b) preserves audit / accounting records as a legal-obligation defence; the tool records the erasure as a new event rather than rewriting history. - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. @@ -78,10 +78,10 @@ retention: staging_ttl_days: 7 # one work-week to act on a `forgelm reject` ephemeral_artefact_retention_days: 90 # quarterly review cadence raw_documents_retention_days: 90 # ingestion-window before re-running data audit - enforce: log_only # log_only / warn_on_excess / block_on_excess + enforce: log_only # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess) ``` -Setting any horizon to `0` disables the policy for that artefact kind (retain indefinitely). `enforce` controls how the trainer pre-flight gate reacts to violations. +Setting any horizon to `0` disables the policy for that artefact kind (retain indefinitely). The horizons are reported by `forgelm purge --check-policy` (read-only, always exits 0); nothing is deleted or blocked automatically. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. ## Removed: `evaluation.staging_ttl_days` @@ -96,7 +96,7 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, before any deletion (`--check-policy` is read-only and emits no audit events) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finishes | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × `final_model/` exists for any run that consumed the corpus. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a `final_model/` directory exists directly under `--output-dir`. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Row mode only.** Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. (Run mode does not surface this warning — synthetic data is corpus-scoped, not run-scoped.) | All `completed` fields + `synthetic_files` | | `data.erasure_warning_external_copies` | Loaded config has a webhook block (both modes). | All `completed` fields + `webhook_targets` | diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md index 1ac0e752..9570b600 100644 --- a/docs/qms/risk_treatment_plan-tr.md +++ b/docs/qms/risk_treatment_plan-tr.md @@ -90,7 +90,7 @@ ekler. |---|---| | Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM, silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` event'ı yayar — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | +| Treatment | ForgeLM, purge'ün `--output-dir`'i altında doğrudan bir `final_model/` varsa `data.erasure_warning_memorisation` event'ı yayar (satırı hangi koşumun tükettiğini bilmez ve varsayılan dizin eğitim çıktısı değil corpus'unkidir) — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | | Residual L × I | High × Med = HIGH (risk **kabul edildi**, bkz. §5) | | Sahip | Data Protection Officer (DPO) | | Review cadence | Madde 17 talebi başına | diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md index c19a9d35..43258b00 100644 --- a/docs/qms/risk_treatment_plan.md +++ b/docs/qms/risk_treatment_plan.md @@ -89,7 +89,7 @@ section §4. |---|---| | Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when the run that consumed the deleted row has a `final_model/` artefact — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | +| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when a `final_model/` exists directly under the purge's `--output-dir` (it does not know which run consumed the row, and the default directory is the corpus's, not the training output) — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | | Residual L × I | High × Med = HIGH (risk **accepted**, see §5) | | Owner | Data Protection Officer (DPO) | | Review cadence | Per Article 17 request | diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md index 3068d80d..051d863e 100644 --- a/docs/reference/audit_event_catalog-tr.md +++ b/docs/reference/audit_event_catalog-tr.md @@ -78,7 +78,7 @@ Hash zinciri, satır diske düştükten (`flush` + `fsync`) sonra ilerler; kirli | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunur; event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. | Tüm `completed` field'ları + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | 17 | diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md index a7a74abd..e5d05c32 100644 --- a/docs/reference/audit_event_catalog.md +++ b/docs/reference/audit_event_catalog.md @@ -78,7 +78,7 @@ The hash chain advances after the line lands on disk (`flush` + `fsync`), so an | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × `final_model/` exists for any run that consumed this corpus. | All `completed` fields + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | 17 | diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index b78fa07f..f9b88b43 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -270,10 +270,10 @@ training: Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarını belirler. Ufuklar GDPR Madde 5(1)(e) "saklama sınırlaması" ve Madde 17 -"silme hakkı" tarihlerini onurlandırır. `enforce` anahtarı yalnız-loglama, -uyarı ve sert-engelleme modları arasında geçiş yaparak regüle edilen bir CI -kapısının saklama ufkunu eski bir çalışma alanını yeniden kullanarak sessizce -uzatmasını engeller. +"silme hakkı" tarihlerini onurlandırır. Ufuklar **raporlanır**, zorlanmaz: +`forgelm purge --check-policy` ufku aşanları listeler ve her zaman 0 ile çıkar; +hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı +ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| @@ -281,7 +281,7 @@ uzatmasını engeller. | `staging_ttl_days` | int | `7` | `forgelm reject` kararından sonra `final_model.staging./` dizininin planlı temizlenmeden önce saklanacağı gün sayısı. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | | `ephemeral_artefact_retention_days` | int | `90` | Uyumluluk paketleri, veri denetim raporları ve diğer çalışma kapsamlı türetilmiş artefaktların saklanma süresi (gün). `0` süresiz saklama. | | `raw_documents_retention_days` | int | `90` | İngest edilmiş ham belgelerin (PDF / DOCX / EPUB / TXT / Markdown) operatörün ingestion-output dizininde saklanma süresi (gün). `0` süresiz saklama. | -| `enforce` | string | `"log_only"` | Politika uygulama modu: `"log_only"` (yalnızca audit log), `"warn_on_excess"` (stderr'e yapılandırılmış uyarı), `"block_on_excess"` (`EXIT_EVAL_FAILURE` = 3 ile trainer ön-kontrolünü iptal eder). | +| `enforce` | string | `"log_only"` | Ayrılmış politika modu. Henüz hiçbir komut onu okumaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa CLI uyarır. | > **Kaldırıldı:** `evaluation.staging_ttl_days` (v0.5.5 itibarıyla kullanımdan > kaldırılmıştı) v0.8.0'da kaldırılmıştır. Artık geçerli tek form @@ -398,10 +398,10 @@ kaynağı kaydedilmemiştir. |------|-----|-----------|----------| | `strategy` | string | `null` | `"deepspeed"` veya `"fsdp"` (null = tek GPU) | | `deepspeed_config` | string | `null` | Ön ayar (`"zero2"`, `"zero3"`, `"zero3_offload"`) veya JSON yolu | -| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"`, `"shard_grad_op"`, `"hybrid_shard"`, `"no_shard"` | -| `fsdp_auto_wrap` | bool | `true` | Transformer katmanlarını otomatik sar | +| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` veya `"shard_grad_op"` (`"hybrid_shard"` ve `"no_shard"` reddedilir: FSDP2 bunları desteklemez) | +| `fsdp_auto_wrap` | bool | `true` | Transformer katmanlarını otomatik sar (transformers 5.10+'ta yok sayılır, katmanlar hep sarılır) | | `fsdp_offload` | bool | `false` | Parametreleri CPU'ya taşı | -| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` veya `"backward_post"` | +| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` veya `"backward_post"` (transformers 5.10+'ta yok sayılır; FSDP2 prefetch politikası almaz) | | `fsdp_state_dict_type` | string | `"FULL_STATE_DICT"` | `"FULL_STATE_DICT"` veya `"SHARDED_STATE_DICT"` | ## `synthetic` (İsteğe bağlı — Sentetik Veri Üretimi) diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 0f83d16e..5d830e0a 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -270,9 +270,10 @@ across retries. Each retry attempt is logged to the audit trail. Defines maximum retention horizons for compliance, training, and evaluation artefacts. Horizons honour GDPR Article 5(1)(e) "storage limitation" and -Article 17 "right to erasure" deadlines. The `enforce` knob switches between -log-only, warning, and hard-block modes so a regulated CI gate cannot -silently extend the retention horizon by re-using a stale workspace. +Article 17 "right to erasure" deadlines. The horizons are **reported**, not +enforced: `forgelm purge --check-policy` lists what is past them and always +exits 0, and nothing is deleted, blocked or scheduled automatically. The +`enforce` knob is reserved: no command reads it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| @@ -280,7 +281,7 @@ silently extend the retention horizon by re-using a stale workspace. | `staging_ttl_days` | int | `7` | Days to retain `final_model.staging./` after a `forgelm reject` decision before scheduled cleanup. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | | `ephemeral_artefact_retention_days` | int | `90` | Days to retain compliance bundles, data audit reports, and other run-scoped derived artefacts. Set to `0` to retain indefinitely. | | `raw_documents_retention_days` | int | `90` | Days to retain ingested raw documents (PDF / DOCX / EPUB / TXT / Markdown) under the operator's ingestion-output directory. Set to `0` to retain indefinitely. | -| `enforce` | string | `"log_only"` | Policy enforcement mode: `"log_only"` (audit-log only), `"warn_on_excess"` (structured stderr warning), `"block_on_excess"` (abort trainer pre-flight with `EXIT_EVAL_FAILURE` = 3). | +| `enforce` | string | `"log_only"` | Reserved policy mode. No command reads it yet: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and the CLI warns when a config sets either of the last two. | > **Removed:** `evaluation.staging_ttl_days` (deprecated as of v0.5.5) was > removed in v0.8.0. `retention.staging_ttl_days` is now the only accepted form. @@ -310,10 +311,10 @@ silently extend the retention horizon by re-using a stale workspace. |-------|------|---------|-------------| | `strategy` | string | `null` | `"deepspeed"` or `"fsdp"` (null = single GPU) | | `deepspeed_config` | string | `null` | Preset (`"zero2"`, `"zero3"`, `"zero3_offload"`) or JSON path | -| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"`, `"shard_grad_op"`, `"hybrid_shard"`, `"no_shard"` | -| `fsdp_auto_wrap` | bool | `true` | Auto-wrap transformer layers | +| `fsdp_strategy` | string | `"full_shard"` | `"full_shard"` or `"shard_grad_op"` (`"hybrid_shard"` and `"no_shard"` are rejected: FSDP2 does not support them) | +| `fsdp_auto_wrap` | bool | `true` | Auto-wrap transformer layers (ignored on transformers 5.10+, which always wraps them) | | `fsdp_offload` | bool | `false` | Offload parameters to CPU | -| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` or `"backward_post"` | +| `fsdp_backward_prefetch` | string | `"backward_pre"` | `"backward_pre"` or `"backward_post"` (ignored on transformers 5.10+, whose FSDP2 takes no prefetch policy) | | `fsdp_state_dict_type` | string | `"FULL_STATE_DICT"` | `"FULL_STATE_DICT"` or `"SHARDED_STATE_DICT"` | --- diff --git a/docs/reference/distributed_training-tr.md b/docs/reference/distributed_training-tr.md index 32e6378f..0df37e60 100644 --- a/docs/reference/distributed_training-tr.md +++ b/docs/reference/distributed_training-tr.md @@ -120,8 +120,8 @@ PyTorch Fully Sharded Data Parallel (FSDP), DeepSpeed'e alternatif olarak PyTorc |----------|----------|-----------------| | `full_shard` | Parametreleri, gradyanları ve optimizer durumlarını böl | En Yüksek | | `shard_grad_op` | Yalnızca gradyanları ve optimizer durumlarını böl | Orta | -| `hybrid_shard` | Düğüm içi tam bölme, düğümler arası çoğaltma | Çoklu Düğüm | -| `no_shard` | Standart DDP (bölme yok) | Yok | + +`hybrid_shard` ve `no_shard` **kabul edilmez**: `fsdp_strategy` `full_shard` ya da `shard_grad_op` olmalıdır. ForgeLM'in gerektirdiği transformers 5.10 ve sonrası varsayılan olarak FSDP2 çalıştırır ve FSDP2 bu ikisini de desteklemez: eskiden `--dry-run`'ı geçip model yüklendikten sonra exit 2 ile (`fsdp=hybrid_shard is only available with FSDP1`) patlıyorlardı, şimdi doğrulamada exit 1 ile reddediliyorlar. ### Kullanım @@ -129,12 +129,14 @@ PyTorch Fully Sharded Data Parallel (FSDP), DeepSpeed'e alternatif olarak PyTorc distributed: strategy: "fsdp" fsdp_strategy: "full_shard" - fsdp_auto_wrap: true # Auto-wrap transformer layers + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # Offload parameters AND gradients (between forward/backward) to CPU - fsdp_backward_prefetch: "backward_pre" # Prefetch strategy + fsdp_backward_prefetch: "backward_pre" # transformers 5.10+'ta yok sayılır (FSDP2 prefetch politikası almaz) fsdp_state_dict_type: "FULL_STATE_DICT" # State dict handling ``` +`fsdp_auto_wrap: false` ve `fsdp_backward_prefetch: backward_post` hâlâ kabul edilir, ancak FSDP2'de hiçbir şeyi değiştirmezler; bu yüzden ikisinden biri varsayılanından farklı ayarlandığında ForgeLM bir uyarı loglar. + ### FSDP'yi DeepSpeed yerine ne zaman seçmeli - **FSDP**: Native PyTorch, ekstra bağımlılık yok, daha basit kurulum, diff --git a/docs/reference/distributed_training.md b/docs/reference/distributed_training.md index b83b29f9..013018a0 100644 --- a/docs/reference/distributed_training.md +++ b/docs/reference/distributed_training.md @@ -118,8 +118,8 @@ PyTorch Fully Sharded Data Parallel (FSDP) is an alternative to DeepSpeed that's |----------|-------------|----------------| | `full_shard` | Shard parameters, gradients, and optimizer states | Highest | | `shard_grad_op` | Shard gradients and optimizer states only | Medium | -| `hybrid_shard` | Full shard within nodes, replicate across nodes | Multi-node | -| `no_shard` | Standard DDP (no sharding) | None | + +`hybrid_shard` and `no_shard` are **not accepted**: `fsdp_strategy` must be `full_shard` or `shard_grad_op`. transformers 5.10 and later, which ForgeLM requires, run FSDP2 by default, and FSDP2 supports neither: they used to pass `--dry-run` and then fail with exit 2 (`fsdp=hybrid_shard is only available with FSDP1`) after the model was loaded, and now fail validation with exit 1. ### Usage @@ -127,12 +127,14 @@ PyTorch Fully Sharded Data Parallel (FSDP) is an alternative to DeepSpeed that's distributed: strategy: "fsdp" fsdp_strategy: "full_shard" - fsdp_auto_wrap: true # Auto-wrap transformer layers + fsdp_auto_wrap: true # Ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # Offload parameters AND gradients (between forward/backward) to CPU - fsdp_backward_prefetch: "backward_pre" # Prefetch strategy + fsdp_backward_prefetch: "backward_pre" # Ignored on transformers 5.10+ (FSDP2 takes no prefetch policy) fsdp_state_dict_type: "FULL_STATE_DICT" # State dict handling ``` +`fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted, but under FSDP2 they change nothing, so ForgeLM logs a warning when either is set away from its default. + ### When to Choose FSDP over DeepSpeed - **FSDP**: Native PyTorch, no extra dependency, simpler setup, good for most use cases diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index de03e4d2..cf111861 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -35,7 +35,7 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | `--row-id ID` | evet | Silinecek identifier. Audit emisyonundan önce `forgelm.cli.subcommands._purge._hash_target_id` üzerinden hash'lenir. | | `--corpus PATH` | evet | Tek bir JSONL dosyası. Dizin modu reddedilir — operatörler kendi script'lerinde döngü kurar. | | `--row-matches {one,all}` | hayır (varsayılan `one`) | `one` >=2 eşleşmede reddeder; `all` her eşleşmeyi siler (operatör niyeti onaylar). | -| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, `forgelm reverse-pii` ile cross-tool korelasyon için operatörün `--output-dir`'i sabitleyebilmesi adına çözülen dizini adlandıran bir WARNING yayar. | +| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, operatörün `--output-dir`'i eğitim çıktı dizinine ve `forgelm reverse-pii` ile cross-tool korelasyon için sabitleyebilmesi adına çözülen dizini ve memorisation kontrolünün arayacağı `final_model/`'i adlandıran bir WARNING yayar. | | `--justification TEXT` | hayır | Operatör-sağlanan, her erasure event'ına yazılan sebep. Dahili ticket id'nizi referans verin; subject identifier yapıştırmayın. | | `--dry-run` | hayır | Silmeyi önizler + audit zincirini (`dry_run=true` ile) yayar; disk'e dokunmaz. | @@ -80,7 +80,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | `data.erasure_requested` | Herhangi bir `--row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunurdur ve event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'li), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum yok VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | | `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index ebbffe38..2c44b85f 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -35,7 +35,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | `--row-id ID` | yes | Identifier to erase. Hashed before audit emission via `forgelm.cli.subcommands._purge._hash_target_id`. | | `--corpus PATH` | yes | Single JSONL file. Directory mode is rejected — operators loop in their own script. | | `--row-matches {one,all}` | no (default `one`) | `one` refuses on >=2 matches; `all` deletes every match (operator confirms intent). | -| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir so the operator can pin `--output-dir` for cross-tool correlation with `forgelm reverse-pii`. | +| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir and the `final_model/` the memorisation check will look for, so the operator can pin `--output-dir` to the training output directory and for cross-tool correlation with `forgelm reverse-pii`. | | `--justification TEXT` | no | Operator-supplied reason recorded on every erasure event. Reference your internal ticket id; do not paste subject identifiers. | | `--dry-run` | no | Preview the deletion + emit the audit chain (with `dry_run=true`) without modifying disk. | @@ -80,7 +80,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | `data.erasure_requested` | First step of any `--row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × `final_model/` exists for any run that consumed this corpus. | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | | `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md index 0b74c5a2..a81303ae 100644 --- a/docs/reference/supply_chain_security-tr.md +++ b/docs/reference/supply_chain_security-tr.md @@ -151,7 +151,7 @@ bir [`tools/pip_audit_ignores.yaml`](../../tools/pip_audit_ignores.yaml) taşır (şu an dokuz no-fix torch advisory'si ve bir OSV-yanlış-sınıflandırılmış markdown kaydı; daha önce taşıdığı iki transformers advisory'si `transformers>=5.3.0` floor'u onları pip-audit -tarafından raporlanamaz hâle getirince kaldırıldı). Bu dosya yalnız projenin kendi workflow'u tarafından +tarafından raporlanamaz hâle getirince kaldırıldı; floor artık `>=5.10.1`). Bu dosya yalnız projenin kendi workflow'u tarafından `--ignores` ile tüketilir; deployer'lar miras almaz. Her release cycle'da gözden geçirilir. diff --git a/docs/reference/supply_chain_security.md b/docs/reference/supply_chain_security.md index 13cac6c0..e070e770 100644 --- a/docs/reference/supply_chain_security.md +++ b/docs/reference/supply_chain_security.md @@ -149,7 +149,7 @@ ForgeLM's own nightly does carry a checked-in project-internal triage (currently nine no-fix torch advisories and one OSV-misclassified markdown record; the two transformers advisories it once carried were dropped once the `transformers>=5.3.0` floor made them -unreportable by pip-audit). That file is consumed only by the +unreportable by pip-audit; the floor is now `>=5.10.1`). That file is consumed only by the project's own workflow via `--ignores`; deployers do not inherit it. Review at every release cycle. diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 6cf43cb9..359a8561 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -42,7 +42,7 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - `data.erasure_requested` — emitted FIRST, before any deletion. - `data.erasure_completed` — emitted LAST after the disk operation succeeded. - `data.erasure_failed` — emitted instead when the disk operation raised, or no matching row/run was found. -- `data.erasure_warning_memorisation` — corpus-row erasure × `final_model/` exists for any run that consumed this corpus (the row is gone from disk but may still be memorised in the trained weights). +- `data.erasure_warning_memorisation` — corpus-row erasure × a `final_model/` exists directly under `--output-dir` (the row is gone from disk but may still be memorised in the trained weights). - `data.erasure_warning_synthetic_data_present` — corpus-row erasure × `synthetic_data*.jsonl` exists in `output_dir`. - `data.erasure_warning_external_copies` — loaded config has a non-empty `webhook` block; downstream consumers may have received notices. @@ -110,7 +110,7 @@ Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of e ::: :::warn -**Running `forgelm purge` without checking for memorisation.** If a `final_model/` exists for any run that consumed the corpus, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap; act on it (re-train without the erased row, or document the residual risk in your DPIA). +**Running `forgelm purge` without checking for memorisation.** If a trained model exists, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap only when a `final_model/` sits directly under `--output-dir` — which defaults to the corpus's directory, so pass your training output directory; act on it (re-train without the erased row, or document the residual risk in your DPIA). ::: :::tip diff --git a/docs/usermanuals/en/getting-started/installation.md b/docs/usermanuals/en/getting-started/installation.md index a943d0f0..d5aad6d2 100644 --- a/docs/usermanuals/en/getting-started/installation.md +++ b/docs/usermanuals/en/getting-started/installation.md @@ -22,7 +22,7 @@ ForgeLM ships as a single PyPI package with optional dependency groups (called * ::: :::note -**Intel Macs (x86_64) are not supported.** ForgeLM requires `torch>=2.4.0` (pulled in by `transformers>=5.3.0`), and PyPI publishes no `torch>=2.4` wheel for x86_64 macOS. Use Apple Silicon, Linux, or Windows. +**Intel Macs (x86_64) are not supported.** ForgeLM requires `torch>=2.4.0` (the minimum that `transformers` 5 supports), and PyPI publishes no `torch>=2.4` wheel for x86_64 macOS. Use Apple Silicon, Linux, or Windows. ::: ## Base install diff --git a/docs/usermanuals/en/operations/air-gap.md b/docs/usermanuals/en/operations/air-gap.md index aacf64e5..40a2807c 100644 --- a/docs/usermanuals/en/operations/air-gap.md +++ b/docs/usermanuals/en/operations/air-gap.md @@ -63,7 +63,7 @@ ERROR: model "meta-llama/Llama-Guard-3-8B" not in local cache. | OpenAI / Anthropic judges | Disable API-backed evaluation or use a local judge; `--offline` does not disable API clients. | | Webhooks | `--offline` does not disable configured delivery. Omit the `webhook:` block, or leave `url` / `url_env` unset, to disable it; `allow_private_destinations` permits private destinations in addition to public ones. | | pip install (during runtime) | Install required dependencies before disconnecting; `--offline` does not control pip. | -| Telemetry | ForgeLM has no own telemetry service; configure external integrations separately. | +| Telemetry | ForgeLM has no own telemetry service. The `forgelm` CLI sets `HF_HUB_DISABLE_TELEMETRY=1` (unless you set it yourself) so TRL and `huggingface_hub` send no anonymous usage ping; using ForgeLM as a library does not set it. Configure external integrations separately. | ## Local synthetic data diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index c4ba73c2..ea38e082 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -229,10 +229,10 @@ The merge field is `method` (not `algorithm`), and there is no `dare_ties` choic distributed: strategy: null # null (single-GPU, no distributed wrapping) | deepspeed | fsdp deepspeed_config: null # path to a DeepSpeed JSON, or preset name: zero2 | zero3 | zero3_offload - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # auto-wrap transformer layers (recommended) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # offload FSDP parameters to CPU between forward/backward - fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post + fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post; ignored on transformers 5.10+ fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` @@ -363,7 +363,7 @@ retention: staging_ttl_days: 7 # days to retain staging model after forgelm reject ephemeral_artefact_retention_days: 90 # compliance bundles, audit reports raw_documents_retention_days: 90 # ingested PDF/DOCX/EPUB/TXT/Markdown - enforce: "log_only" # log_only | warn_on_excess | block_on_excess + enforce: "log_only" # reserved: no command acts on it yet (log_only | warn_on_excess | block_on_excess) ``` ## `pipeline:` diff --git a/docs/usermanuals/en/training/distributed.md b/docs/usermanuals/en/training/distributed.md index 7aa622ff..8679d546 100644 --- a/docs/usermanuals/en/training/distributed.md +++ b/docs/usermanuals/en/training/distributed.md @@ -111,12 +111,14 @@ FSDP shards similarly to ZeRO-3 but uses PyTorch's native FullyShardedDataParall ```yaml distributed: strategy: "fsdp" - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # auto-wrap transformer layers (recommended) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # ignored on transformers 5.10+ (FSDP2 always wraps transformer layers) fsdp_offload: false # offload parameters to CPU between forward/backward fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` +`no_shard` and `hybrid_shard` are rejected when the config loads: transformers 5.10+, which ForgeLM requires, runs FSDP2, and FSDP2 supports neither. `fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted but change nothing under FSDP2, so ForgeLM logs a warning when either is set. + `DistributedConfig` has no `fsdp_auto_wrap_policy` or `fsdp_offload_params` field — auto-wrap is the plain boolean `fsdp_auto_wrap`, and CPU offload is `fsdp_offload` (no `_params` suffix). ## Gradient accumulation diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index f430e5c0..77e6b09a 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -42,7 +42,7 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - `data.erasure_requested` — herhangi bir silmeden ÖNCE, ilk yayılır. - `data.erasure_completed` — disk operasyonu başarılı sonra son yayılır. - `data.erasure_failed` — disk operasyonu raise ettiğinde veya eşleşen satır/koşum bulunmadığında bunun yerine yayılır. -- `data.erasure_warning_memorisation` — corpus-satır silme × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). +- `data.erasure_warning_memorisation` — corpus-satır silme × `--output-dir` altında doğrudan bir `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). - `data.erasure_warning_synthetic_data_present` — corpus-satır silme × `output_dir`'de `synthetic_data*.jsonl` mevcut. - `data.erasure_warning_external_copies` — yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. @@ -110,7 +110,7 @@ Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın d ::: :::warn -**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Corpus'u tüketen herhangi bir koşum için `final_model/` mevcutsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yüzeyselleştirir; harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). +**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Eğitilmiş bir model varsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yalnızca `--output-dir` altında doğrudan bir `final_model/` varsa yüzeyselleştirir — `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin; harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). ::: :::tip diff --git a/docs/usermanuals/tr/getting-started/installation.md b/docs/usermanuals/tr/getting-started/installation.md index bdf196c9..5d5ecd01 100644 --- a/docs/usermanuals/tr/getting-started/installation.md +++ b/docs/usermanuals/tr/getting-started/installation.md @@ -22,7 +22,7 @@ ForgeLM, PyPI'da tek bir paket olarak gelir; ağır özellikler için opsiyonel ::: :::note -**Intel Mac'ler (x86_64) desteklenmiyor.** ForgeLM `torch>=2.4.0` gerektirir (`transformers>=5.3.0` üzerinden gelir) ve PyPI'da x86_64 macOS için `torch>=2.4` wheel'i yok. Apple Silicon, Linux veya Windows kullanın. +**Intel Mac'ler (x86_64) desteklenmiyor.** ForgeLM `torch>=2.4.0` gerektirir (`transformers` 5'in desteklediği asgari sürüm) ve PyPI'da x86_64 macOS için `torch>=2.4` wheel'i yok. Apple Silicon, Linux veya Windows kullanın. ::: ## Temel kurulum diff --git a/docs/usermanuals/tr/operations/air-gap.md b/docs/usermanuals/tr/operations/air-gap.md index 58dc233e..c5744481 100644 --- a/docs/usermanuals/tr/operations/air-gap.md +++ b/docs/usermanuals/tr/operations/air-gap.md @@ -63,7 +63,7 @@ ERROR: model "meta-llama/Llama-Guard-3-8B" not in local cache. | OpenAI / Anthropic judge'lar | API tabanlı değerlendirmeyi kapatın veya yerel judge kullanın; `--offline` API istemcilerini kapatmaz. | | Webhook'lar | `--offline`, yapılandırılmış teslimatları kapatmaz. Kapatmak için `webhook:` bloğunu çıkarın veya `url` / `url_env` değerlerini boş bırakın; `allow_private_destinations`, genel hedeflerin yanında özel hedeflere de izin verir. | | Runtime'da pip install | Gerekli bağımlılıkları bağlantıyı kesmeden kurun; `--offline`, pip'i yönetmez. | -| Telemetry | ForgeLM'in kendi telemetri servisi yoktur; harici entegrasyonları ayrıca yapılandırın. | +| Telemetry | ForgeLM'in kendi telemetri servisi yoktur. `forgelm` CLI'ı `HF_HUB_DISABLE_TELEMETRY=1`'i (siz ayarlamadıysanız) ayarlar; böylece TRL ve `huggingface_hub` anonim kullanım ping'i göndermez. ForgeLM'i kütüphane olarak kullanmak bunu ayarlamaz. Harici entegrasyonları ayrıca yapılandırın. | ## Yerel sentetik veri diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 39f8e44b..690f806b 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -231,10 +231,10 @@ Merge alanı `algorithm` değil `method`'dur, `dare_ties` seçeneği, `base_mode distributed: strategy: null # null (tek GPU, dağıtık sarmalama yok) | deepspeed | fsdp deepspeed_config: null # bir DeepSpeed JSON'a yol, veya preset adı: zero2 | zero3 | zero3_offload - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # transformer katmanlarını otomatik sarmala (önerilir) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # forward/backward arasında FSDP parametrelerini CPU'ya offload et - fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post + fsdp_backward_prefetch: "backward_pre" # backward_pre | backward_post; transformers 5.10+'ta yok sayılır fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` @@ -371,7 +371,7 @@ retention: staging_ttl_days: 7 # forgelm reject sonrası staging modelini saklama süresi ephemeral_artefact_retention_days: 90 # uyumluluk paketleri, denetim raporları raw_documents_retention_days: 90 # ingest edilen PDF/DOCX/EPUB/TXT/Markdown - enforce: "log_only" # log_only | warn_on_excess | block_on_excess + enforce: "log_only" # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only | warn_on_excess | block_on_excess) ``` ## `pipeline:` diff --git a/docs/usermanuals/tr/training/distributed.md b/docs/usermanuals/tr/training/distributed.md index 4dcfd239..7e73a6d7 100644 --- a/docs/usermanuals/tr/training/distributed.md +++ b/docs/usermanuals/tr/training/distributed.md @@ -111,12 +111,14 @@ FSDP, ZeRO-3 gibi sharder ama PyTorch'un yerli FullyShardedDataParallel'ini kull ```yaml distributed: strategy: "fsdp" - fsdp_strategy: "full_shard" # full_shard | shard_grad_op | no_shard | hybrid_shard - fsdp_auto_wrap: true # transformer katmanlarını otomatik sar (önerilir) + fsdp_strategy: "full_shard" # full_shard | shard_grad_op + fsdp_auto_wrap: true # transformers 5.10+'ta yok sayılır (FSDP2 transformer katmanlarını hep sarar) fsdp_offload: false # forward/backward arasında parametreleri CPU'ya boşalt fsdp_state_dict_type: "FULL_STATE_DICT" # FULL_STATE_DICT | SHARDED_STATE_DICT ``` +`no_shard` ve `hybrid_shard` config yüklenirken reddedilir: ForgeLM'in gerektirdiği transformers 5.10+ FSDP2 çalıştırır ve FSDP2 bu ikisini desteklemez. `fsdp_auto_wrap: false` ve `fsdp_backward_prefetch: backward_post` hâlâ kabul edilir ancak FSDP2'de hiçbir şeyi değiştirmez; bu yüzden ikisinden biri ayarlandığında ForgeLM bir uyarı loglar. + `DistributedConfig`'te `fsdp_auto_wrap_policy` veya `fsdp_offload_params` alanı yoktur — auto-wrap düz bir `fsdp_auto_wrap` boolean'ıdır, CPU offload ise `fsdp_offload`'dur (`_params` eki yoktur). ## Gradient accumulation diff --git a/forgelm/cli/_config_load.py b/forgelm/cli/_config_load.py index 4e5ac477..8357d535 100644 --- a/forgelm/cli/_config_load.py +++ b/forgelm/cli/_config_load.py @@ -14,6 +14,22 @@ from ._logging import logger +def _warn_reserved_retention_enforce(config) -> None: + """``retention.enforce`` is validated and offered by both wizards, but no command reads it. + + A config that sets ``warn_on_excess`` or ``block_on_excess`` is asking for a gate that does not exist; say so, + rather than accept it silently. (This lives here, not in the schema, to keep ``config.py`` within its + deferred-split line budget.) + """ + mode = getattr(getattr(config, "retention", None), "enforce", "log_only") + if mode != "log_only": + logger.warning( + "retention.enforce=%r is accepted but no command acts on it yet: retention horizons are only reported " + "by `forgelm purge --check-policy`, and the trainer is not warned or blocked.", + mode, + ) + + def _load_config_or_exit(config_path: str, json_output: bool): """Load config and translate exceptions into the right exit code. @@ -25,7 +41,9 @@ def _load_config_or_exit(config_path: str, json_output: bool): """ try: logger.info("Loading configuration from %s...", config_path) - return load_config(config_path) + config = load_config(config_path) + _warn_reserved_retention_enforce(config) + return config except FileNotFoundError as e: msg = f"Config file not found: {e}" except ConfigError as e: @@ -47,6 +65,20 @@ def _load_config_or_exit(config_path: str, json_output: bool): sys.exit(EXIT_CONFIG_ERROR) +def _disable_hub_telemetry() -> None: + """Keep the ``forgelm`` CLI from sending Hugging Face's anonymous usage ping. + + A training run makes ``huggingface_hub`` send a small usage ping to huggingface.co: TRL's trainers call + ``send_telemetry`` when they are built, and ``huggingface_hub`` skips it only when ``HF_HUB_DISABLE_TELEMETRY`` is + true or offline mode is on. ForgeLM's privacy page says the tool sends no usage data anywhere, so the CLI sets + the variable by default. ``setdefault`` leaves an operator's own value alone (``HF_HUB_DISABLE_TELEMETRY=0`` + opts back in). It must run before ``huggingface_hub`` is imported, which reads it once at import; importing + ``forgelm.cli`` does not import it, and ``main()`` calls this first. Using ForgeLM as a library does not go + through ``main()``, so a library user sets the variable themselves. + """ + os.environ.setdefault("HF_HUB_DISABLE_TELEMETRY", "1") + + def _apply_offline_flag(config, offline_arg: bool) -> None: if offline_arg: config.model.offline = True @@ -55,3 +87,27 @@ def _apply_offline_flag(config, offline_arg: bool) -> None: os.environ["TRANSFORMERS_OFFLINE"] = "1" os.environ["HF_DATASETS_OFFLINE"] = "1" logger.info("Offline mode enabled. All HF Hub network calls are disabled.") + + +def _warn_inert_fsdp_options(config) -> None: + """Say so when an FSDP knob that transformers' FSDP2 default ignores is set away from its default. + + ``fsdp_auto_wrap: false`` and ``fsdp_backward_prefetch: backward_post`` are still accepted, but under + transformers 5.10+ (FSDP2) the transformer-layer wrap policy is always applied and no backward prefetch is + passed on, so the run behaves as if the defaults had been kept. (This lives here, not in the schema, to keep + ``config.py`` within its deferred-split line budget.) + """ + dist = getattr(config, "distributed", None) + if getattr(dist, "strategy", None) != "fsdp": + return + if not dist.fsdp_auto_wrap: + logger.warning( + "distributed.fsdp_auto_wrap=false has no effect: transformers 5.10+ runs FSDP2, which always wraps " + "transformer layers." + ) + if dist.fsdp_backward_prefetch != "backward_pre": + logger.warning( + "distributed.fsdp_backward_prefetch=%r has no effect: transformers 5.10+ runs FSDP2, which does not take " + "a backward-prefetch policy.", + dist.fsdp_backward_prefetch, + ) diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index e4b64387..972ee75d 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -10,7 +10,7 @@ import sys from .._strict_json import dumps_strict -from ._config_load import _apply_offline_flag, _load_config_or_exit +from ._config_load import _apply_offline_flag, _disable_hub_telemetry, _load_config_or_exit, _warn_inert_fsdp_options from ._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS, EXIT_TRAINING_ERROR, _clamp_exit_code from ._logging import _setup_logging, logger from ._no_train_modes import _maybe_run_no_train_mode @@ -163,6 +163,7 @@ def main(): # (= 128+SIGINT) — outside the documented public 0/1/2/3/4 surface # and surprising to CI/CD scripts that branch on exit code 2. _force_utf8_console_streams() + _disable_hub_telemetry() try: return _main_inner() except KeyboardInterrupt: @@ -241,6 +242,7 @@ def _main_inner() -> None: config = _load_config_or_exit(args.config, json_output) _apply_offline_flag(config, args.offline) + _warn_inert_fsdp_options(config) if config.pipeline is not None: _dispatch_pipeline_mode(config, args, json_output) diff --git a/forgelm/cli/subcommands/_audit.py b/forgelm/cli/subcommands/_audit.py index d9b4f4a1..b4e17ff7 100644 --- a/forgelm/cli/subcommands/_audit.py +++ b/forgelm/cli/subcommands/_audit.py @@ -55,8 +55,7 @@ def _exit_on_critical_secrets(verdict: dict, *, allow_secrets: bool, defer_exit: chain, GGUF sidecar); training data was never signed. - ``3`` fits exactly: an evaluation gate examined an artefact and said no. This is the same semantics as ``forgelm safety-eval``'s - non-passing branch (``subcommands/_safety_eval.py``) and the - retention ``block_on_excess`` gate. A new public exit code is a + non-passing branch (``subcommands/_safety_eval.py``). A new public exit code is a permanent contract addition and this incident does not need one. ``allow_secrets`` is an explicit per-invocation escape hatch for the diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 781d9208..97f139aa 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -714,6 +714,17 @@ def _run_purge_row_id(args, output_format: str) -> None: """ _validate_row_id_args(args, output_format) output_dir = args.output_dir or os.path.dirname(os.path.abspath(args.corpus)) or "." + if not args.output_dir: + # The salt, the audit chain and the memorisation check (`/final_model/`) all live under the + # output directory. Defaulting to the corpus directory is almost never where a training run wrote its model, + # so say what was resolved instead of letting the check pass quietly over the wrong directory. + logger.warning( + "--output-dir not given: using the corpus directory %s for the audit log, the salt and the memorisation " + "check (which looks for %s). Pass --output-dir to check your trained model and to " + "correlate with `forgelm reverse-pii`.", + output_dir, + os.path.join(output_dir, "final_model"), + ) os.makedirs(output_dir, exist_ok=True) try: diff --git a/forgelm/config.py b/forgelm/config.py index 4114f2df..33e90e25 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -770,17 +770,17 @@ class DistributedConfig(BaseModel): default=None, description="DeepSpeed config: filesystem path to a DS JSON OR preset name (`zero2`, `zero3`, `zero3_offload`).", ) - fsdp_strategy: Literal["full_shard", "shard_grad_op", "no_shard", "hybrid_shard"] = Field( + fsdp_strategy: Literal["full_shard", "shard_grad_op"] = Field( default="full_shard", - description="FSDP sharding strategy. `full_shard` is the production default; `hybrid_shard` for multi-node intra-node sharding.", + description="FSDP sharding strategy. `full_shard` is the production default; `shard_grad_op` shards gradients and optimizer states only. `no_shard` and `hybrid_shard` are not accepted: transformers 5.10+ runs FSDP2, which supports neither.", ) - fsdp_auto_wrap: bool = Field(default=True, description="FSDP: auto-wrap transformer layers (recommended).") + fsdp_auto_wrap: bool = Field(default=True, description="FSDP auto-wrap; ignored by FSDP2 (transformers 5.10+).") fsdp_offload: bool = Field( default=False, description="FSDP: offload parameters to CPU between forward and backward (slower, less VRAM)." ) fsdp_backward_prefetch: Literal["backward_pre", "backward_post"] = Field( default="backward_pre", - description="FSDP backward-prefetch policy. `backward_pre` overlaps comm + compute; `backward_post` is more memory-conservative.", + description="FSDP backward-prefetch policy. Not honoured by transformers 5.10+ (FSDP2 prefetches on its own); `backward_post` logs a warning.", ) fsdp_state_dict_type: Literal["FULL_STATE_DICT", "SHARDED_STATE_DICT"] = Field( default="FULL_STATE_DICT", @@ -1598,12 +1598,12 @@ class RetentionConfig(BaseModel): days (typical ingestion-window before re-running data audit). Setting any horizon to ``0`` disables the policy for that artefact - kind (retain indefinitely). ``enforce`` controls how the trainer - pre-flight gate reacts to violations: ``log_only`` records a - notice, ``warn_on_excess`` emits a structured warning, and - ``block_on_excess`` aborts training with EXIT_EVAL_FAILURE so a - regulated CI cannot accidentally extend the retention horizon by - re-using a stale workspace. + kind (retain indefinitely). The horizons are *reported*, not enforced: + ``forgelm purge --check-policy`` lists artefacts past them and always + exits 0; nothing is deleted, blocked or scheduled automatically. + ``enforce`` is reserved: no command acts on it yet, so + ``warn_on_excess`` and ``block_on_excess`` behave like ``log_only`` + (the CLI logs a warning when a config sets either). """ model_config = ConfigDict(extra="forbid") @@ -1645,9 +1645,9 @@ class RetentionConfig(BaseModel): enforce: Literal["log_only", "warn_on_excess", "block_on_excess"] = Field( default="log_only", description=( - "Policy enforcement mode. `log_only` records violations in the audit log without operator-visible output; " - "`warn_on_excess` adds a structured warning to stderr; `block_on_excess` aborts the trainer pre-flight with " - "EXIT_EVAL_FAILURE (3) so a regulated CI gate does not silently extend the retention horizon." + "Reserved: no command reads it yet. Retention horizons are only reported, read-only, by `forgelm purge " + "--check-policy` (always exit 0); nothing is deleted, warned about or blocked automatically. " + "`warn_on_excess` and `block_on_excess` are accepted but behave like `log_only`; the CLI warns when set." ), ) diff --git a/forgelm/model.py b/forgelm/model.py index de6c7577..4d96ea14 100644 --- a/forgelm/model.py +++ b/forgelm/model.py @@ -420,6 +420,35 @@ def _device_map_for(config: Any, is_distributed: bool): return None # CPU-only +def _config_with_rope_scaling( + name_or_path: str, rope_scaling: dict, *, trust_remote_code: bool, revision: Optional[str] = None +) -> Any: + """Load the model's config and apply ``training.rope_scaling`` to it. + + transformers 5 keeps the RoPE settings in ``config.rope_parameters`` (which also carries ``rope_theta``) and + expects the key ``rope_type``. Passing ``rope_scaling=`` to ``from_pretrained`` no longer works: the documented + ``{type: linear, factor: 4}`` raises ``KeyError: 'rope_type'`` while the model is built, and + ``{rope_type: ..., factor: ...}`` replaces the whole dict so ``rope_theta`` is lost (``TypeError`` in the + frequency computation). So the requested keys are merged *over* the model's own ``rope_parameters``, the legacy + key ``type`` is read as ``rope_type``, and the finished config object is what gets loaded. + + For a multimodal checkpoint the RoPE settings live in the text sub-config, which is where they are applied. + A transformers without ``rope_parameters`` gets the legacy ``rope_scaling`` attribute instead. + """ + from transformers import AutoConfig + + model_config = AutoConfig.from_pretrained(name_or_path, trust_remote_code=trust_remote_code, revision=revision) + target = model_config.get_text_config() if hasattr(model_config, "get_text_config") else model_config + requested = dict(rope_scaling) + if "type" in requested and "rope_type" not in requested: + requested["rope_type"] = requested.pop("type") + if hasattr(target, "rope_parameters"): + target.rope_parameters = {**(getattr(target, "rope_parameters", None) or {}), **requested} + else: + target.rope_scaling = requested + return model_config + + def _build_model_kwargs(config: Any, trust_remote_code: bool, revision: Optional[str] = None) -> dict: """Assemble from_pretrained kwargs (device_map, BnB, RoPE, sliding window, revision). @@ -461,12 +490,19 @@ def _build_model_kwargs(config: Any, trust_remote_code: bool, revision: Optional ) rope_scaling = config.training.rope_scaling + sliding_window = config.training.sliding_window_attention if rope_scaling: logger.info("RoPE scaling enabled: %s", rope_scaling) - kwargs["rope_scaling"] = rope_scaling - - sliding_window = config.training.sliding_window_attention - if sliding_window: + model_config = _config_with_rope_scaling( + config.model.name_or_path, rope_scaling, trust_remote_code=trust_remote_code, revision=revision + ) + if sliding_window: + # Once ``config=`` is a finished object, from_pretrained hands unrecognised keyword arguments to the + # model constructor instead of the config, so the override is applied to the config itself. + logger.info("Sliding window attention override: %d tokens", sliding_window) + model_config.sliding_window = sliding_window + kwargs["config"] = model_config + elif sliding_window: logger.info("Sliding window attention override: %d tokens", sliding_window) kwargs["sliding_window"] = sliding_window diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py index 83c0039a..d7574192 100644 --- a/forgelm/wizard/_collectors.py +++ b/forgelm/wizard/_collectors.py @@ -855,11 +855,11 @@ def _collect_retention() -> Optional[Dict[str, Any]]: ) raw_docs = _prompt_int("raw_documents_retention_days (0 = retain indefinitely)", 90, min_val=0, max_val=3650) enforce = _prompt_choice( - "enforcement mode:", + "enforcement mode (reserved: no command acts on it yet; `forgelm purge --check-policy` only reports):", [ - "log_only (record violations only)", - "warn_on_excess (stderr warning, run continues)", - "block_on_excess (abort trainer with EXIT_EVAL_FAILURE)", + "log_only (default; behaves the same as the other two for now)", + "warn_on_excess (accepted, behaves like log_only)", + "block_on_excess (accepted, behaves like log_only)", ], default=1, ) diff --git a/pyproject.toml b/pyproject.toml index adb6053f..f1e2ce1e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -50,7 +50,11 @@ dependencies = [ # the transformers-5 migration (v0.9.0). Apple Silicon, Linux and Windows # are unaffected. The previous `numpy<2; darwin x86_64` ABI-guard marker is # gone with the platform it protected (torch 2.4+ ships against NumPy 2.x). - "transformers>=5.3.0,<6.0.0", + # 5.10.1 is the first installable release past three advisories that reach ForgeLM's own load and save path from + # a Hub repo: CVE-2026-4372 (fixed 5.3.0), CVE-2026-5241 (a LightGlue config.json overrides trust_remote_code; + # fixed 5.5.0) and CVE-2026-9856 (chat_template names used unsanitised as file names by save_pretrained; fixed + # 5.10.0, which PyPI yanked, hence 5.10.1). It also needs huggingface_hub>=1.5.0. + "transformers>=5.10.1,<6.0.0", "peft>=0.19.0,<1.0.0", "datasets>=4.7.0,<6.0.0", "accelerate>=1.4.0,<2.0.0", @@ -58,7 +62,7 @@ dependencies = [ "pydantic>=2.0.0,<3.0.0", "trl>=1.0.0,<2.0.0", "tensorboard>=2.15.0,<3.0.0", - "huggingface_hub>=1.3.0,<2.0.0", + "huggingface_hub>=1.5.0,<2.0.0", "requests>=2.32.2,<3.0.0", # requests-toolbelt is required by forgelm._http for the HostHeaderSSLAdapter # used to close the DNS-rebinding TOCTOU window in safe_post / safe_get diff --git a/tests/test_cli.py b/tests/test_cli.py index a9765895..1fa412e6 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -319,3 +319,118 @@ def test_facade_reexports_every_subcommand_registrar(self): ] for name in registrars: assert callable(getattr(cli, name, None)), f"{name} is not re-exported from forgelm.cli" + + +class TestHubTelemetryDefault: + """The CLI keeps TRL / huggingface_hub from sending the anonymous usage ping (#95).""" + + @staticmethod + def _unset(monkeypatch): + """Make the variable absent *and* make monkeypatch restore that, so a later set does not leak.""" + monkeypatch.setenv("HF_HUB_DISABLE_TELEMETRY", "placeholder") + monkeypatch.delenv("HF_HUB_DISABLE_TELEMETRY") + + @staticmethod + def _clean_env(): + env = {k: v for k, v in os.environ.items() if k != "HF_HUB_DISABLE_TELEMETRY"} + env["PYTHONPATH"] = os.getcwd() + os.pathsep + env.get("PYTHONPATH", "") + return env + + def test_it_is_set_when_the_operator_did_not_choose(self, monkeypatch): + from forgelm.cli._config_load import _disable_hub_telemetry + + self._unset(monkeypatch) + _disable_hub_telemetry() + assert os.environ["HF_HUB_DISABLE_TELEMETRY"] == "1" + + @pytest.mark.parametrize("chosen", ["0", "1", "false"]) + def test_an_operators_own_value_wins(self, monkeypatch, chosen): + from forgelm.cli._config_load import _disable_hub_telemetry + + monkeypatch.setenv("HF_HUB_DISABLE_TELEMETRY", chosen) + _disable_hub_telemetry() + assert os.environ["HF_HUB_DISABLE_TELEMETRY"] == chosen + + def test_main_sets_it_before_anything_else_runs(self, monkeypatch): + from forgelm.cli import _dispatch + + self._unset(monkeypatch) + seen = {} + + def fake_inner(): + seen["at_dispatch"] = os.environ.get("HF_HUB_DISABLE_TELEMETRY") + return 0 + + with patch.object(_dispatch, "_main_inner", fake_inner): + _dispatch.main() + assert seen["at_dispatch"] == "1" + + def test_importing_the_cli_does_not_import_huggingface_hub(self): + # huggingface_hub reads HF_HUB_DISABLE_TELEMETRY once, at import. main() sets the variable, so nothing the CLI + # module imports at load time may import it first (a top-level `import transformers` in forgelm.cli would). + import subprocess + import sys + + code = "import sys, forgelm.cli; print(any(m == 'huggingface_hub' or m.startswith('huggingface_hub.') for m in sys.modules))" + out = subprocess.run( + [sys.executable, "-c", code], env=self._clean_env(), capture_output=True, text=True, timeout=120 + ) + assert out.returncode == 0, out.stderr + assert out.stdout.strip() == "False" + + def test_a_fresh_process_that_calls_it_reads_the_variable_as_true(self): + pytest.importorskip("huggingface_hub") + import subprocess + import sys + + code = ( + "from forgelm.cli._config_load import _disable_hub_telemetry; _disable_hub_telemetry(); " + "import huggingface_hub.constants as c; print(c.HF_HUB_DISABLE_TELEMETRY)" + ) + out = subprocess.run( + [sys.executable, "-c", code], env=self._clean_env(), capture_output=True, text=True, timeout=120 + ) + assert out.returncode == 0, out.stderr + assert out.stdout.strip() == "True" + + +class TestReservedRetentionEnforceWarning: + """`retention.enforce` is accepted but no command reads it; the CLI says so (#92).""" + + @staticmethod + def _config(enforce): + from forgelm.config import ForgeConfig + from tests._helpers.factories import minimal_config + + return ForgeConfig(**minimal_config(retention={"enforce": enforce})) + + def test_the_default_is_silent(self, caplog): + import logging + + from forgelm.cli._config_load import _warn_reserved_retention_enforce + + with caplog.at_level(logging.WARNING): + _warn_reserved_retention_enforce(self._config("log_only")) + assert not [r for r in caplog.records if "retention.enforce" in r.getMessage()] + + @pytest.mark.parametrize("mode", ["warn_on_excess", "block_on_excess"]) + def test_a_gate_that_does_not_exist_is_called_out(self, caplog, mode): + import logging + + from forgelm.cli._config_load import _warn_reserved_retention_enforce + + with caplog.at_level(logging.WARNING): + _warn_reserved_retention_enforce(self._config(mode)) + warned = [r.getMessage() for r in caplog.records if "retention.enforce" in r.getMessage()] + assert len(warned) == 1 and mode in warned[0] and "no command acts on it" in warned[0] + + def test_loading_a_config_file_triggers_it(self, tmp_path, caplog, minimal_config): + import logging + + from forgelm.cli._config_load import _load_config_or_exit + + path = tmp_path / "c.yaml" + path.write_text(yaml.dump(minimal_config(retention={"enforce": "block_on_excess"}))) + with caplog.at_level(logging.WARNING): + _load_config_or_exit(str(path), json_output=False) + assert any("retention.enforce" in r.getMessage() for r in caplog.records) diff --git a/tests/test_config.py b/tests/test_config.py index 10b81fd1..70296912 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -935,6 +935,17 @@ def test_legacy_evaluation_staging_ttl_days_is_rejected(self): with pytest.raises(ValidationError, match="staging_ttl_days"): ForgeConfig(**minimal_config(evaluation={"staging_ttl_days": 14})) + +class TestRetentionConfig: + """`retention` horizons are reported, not enforced; `enforce` is reserved (#92).""" + + def test_the_description_does_not_promise_a_gate(self): + from forgelm.config import RetentionConfig + + description = RetentionConfig.model_fields["enforce"].description or "" + assert "no command reads it" in description + assert "EXIT_EVAL_FAILURE" not in description and "aborts" not in description + def test_canonical_retention_staging_ttl_days_still_works(self): cfg = ForgeConfig(**minimal_config(retention={"staging_ttl_days": 30})) assert cfg.retention.staging_ttl_days == 30 diff --git a/tests/test_distributed.py b/tests/test_distributed.py index d7127735..b51d7819 100644 --- a/tests/test_distributed.py +++ b/tests/test_distributed.py @@ -66,9 +66,9 @@ def test_deepspeed_config(self, minimal_config): assert cfg.distributed.deepspeed_config == "zero3" def test_fsdp_config(self, minimal_config): - cfg = ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "hybrid_shard"})) + cfg = ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "shard_grad_op"})) assert cfg.distributed.strategy == "fsdp" - assert cfg.distributed.fsdp_strategy == "hybrid_shard" + assert cfg.distributed.fsdp_strategy == "shard_grad_op" def test_unsloth_distributed_raises(self, minimal_config): """Unsloth + distributed should raise ValueError.""" @@ -306,3 +306,113 @@ def test_fsdp_strategy_does_not_require_deepspeed(self, monkeypatch, minimal_con kwargs: dict = {} ForgeTrainer._apply_distributed_config(object.__new__(ForgeTrainer), kwargs, cfg.distributed) assert "deepspeed" not in kwargs + + +# --- FSDP: strategies transformers' FSDP2 default cannot express (#104) --- + + +class TestFsdp1OnlyStrategiesAreRejectedAtConfigTime: + """``no_shard`` and ``hybrid_shard`` used to pass ``--dry-run`` and exit 2 at trainer construction. + + transformers 5.10+ runs FSDP2 by default and raises ``fsdp=hybrid_shard is only available with FSDP1`` when + ``TrainingArguments`` is built, so the config is refused up front (exit 1) instead of after the model is loaded. + """ + + @pytest.mark.parametrize("strategy", ["no_shard", "hybrid_shard"]) + def test_the_schema_refuses_them(self, strategy): + with pytest.raises(ValidationError, match="fsdp_strategy"): + DistributedConfig(strategy="fsdp", fsdp_strategy=strategy) + + @pytest.mark.parametrize("strategy", ["no_shard", "hybrid_shard"]) + def test_dry_run_exits_with_a_config_error(self, tmp_path, minimal_config, strategy): + from unittest.mock import patch + + from forgelm.cli import EXIT_CONFIG_ERROR, main + + cfg_path = tmp_path / "config.yaml" + cfg_path.write_text(yaml.dump(minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": strategy}))) + with patch("sys.argv", ["forgelm", "--config", str(cfg_path), "--dry-run"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == EXIT_CONFIG_ERROR + + @pytest.mark.parametrize("strategy", ["full_shard", "shard_grad_op"]) + @pytest.mark.parametrize("auto_wrap", [True, False]) + @pytest.mark.parametrize("offload", [True, False]) + @pytest.mark.parametrize("prefetch", ["backward_pre", "backward_post"]) + @pytest.mark.parametrize("state_dict", ["FULL_STATE_DICT", "SHARDED_STATE_DICT"]) + def test_every_accepted_variant_builds_training_arguments( + self, tmp_path, strategy, auto_wrap, offload, prefetch, state_dict + ): + """What the schema accepts must be what ``TrainingArguments`` accepts (no config that dies later).""" + from types import SimpleNamespace + + from transformers import TrainingArguments + + from forgelm.trainer import ForgeTrainer + + dist_cfg = DistributedConfig( + strategy="fsdp", + fsdp_strategy=strategy, + fsdp_auto_wrap=auto_wrap, + fsdp_offload=offload, + fsdp_backward_prefetch=prefetch, + fsdp_state_dict_type=state_dict, + ) + kwargs: dict = {} + ForgeTrainer._apply_distributed_config(SimpleNamespace(), kwargs, dist_cfg) + TrainingArguments(output_dir=str(tmp_path), use_cpu=True, report_to=[], **kwargs) + + +class TestInertFsdpOptionsAreCalledOut: + """``fsdp_auto_wrap: false`` and ``fsdp_backward_prefetch: backward_post`` are ignored by FSDP2; say so (#104).""" + + @staticmethod + def _config(**distributed): + from tests._helpers.factories import minimal_config + + return ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", **distributed})) + + @staticmethod + def _warnings(config, caplog): + import logging + + from forgelm.cli._config_load import _warn_inert_fsdp_options + + with caplog.at_level(logging.WARNING): + _warn_inert_fsdp_options(config) + return [r.getMessage() for r in caplog.records if "distributed.fsdp_" in r.getMessage()] + + def test_the_defaults_are_silent(self, caplog): + assert self._warnings(self._config(), caplog) == [] + + def test_turning_auto_wrap_off_is_called_out(self, caplog): + warned = self._warnings(self._config(fsdp_auto_wrap=False), caplog) + assert len(warned) == 1 and "fsdp_auto_wrap=false has no effect" in warned[0] + + def test_backward_post_is_called_out(self, caplog): + warned = self._warnings(self._config(fsdp_backward_prefetch="backward_post"), caplog) + assert len(warned) == 1 and "backward_post" in warned[0] + + def test_other_strategies_and_no_distributed_block_are_silent(self, caplog): + from tests._helpers.factories import minimal_config + + deepspeed = ForgeConfig(**minimal_config(distributed={"strategy": "deepspeed", "fsdp_auto_wrap": False})) + plain = ForgeConfig(**minimal_config()) + assert self._warnings(deepspeed, caplog) == [] + assert self._warnings(plain, caplog) == [] + + def test_a_dry_run_reaches_the_warning(self, tmp_path, caplog, minimal_config): + import logging + from unittest.mock import patch + + from forgelm.cli import EXIT_SUCCESS, main + + cfg_path = tmp_path / "config.yaml" + cfg_path.write_text(yaml.dump(minimal_config(distributed={"strategy": "fsdp", "fsdp_auto_wrap": False}))) + with caplog.at_level(logging.WARNING): + with patch("sys.argv", ["forgelm", "--config", str(cfg_path), "--dry-run"]): + with pytest.raises(SystemExit) as exc_info: + main() + assert exc_info.value.code == EXIT_SUCCESS + assert any("fsdp_auto_wrap=false has no effect" in r.getMessage() for r in caplog.records) diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index bcd3c6e9..06c27775 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -152,6 +152,56 @@ def test_row_erasure_target_id_is_hashed_not_cleartext(self, tmp_path: Path) -> # --------------------------------------------------------------------------- +class TestImplicitOutputDirWarning: + """`--output-dir` defaults to the corpus directory; the user is told, because the memorisation check looks for + `/final_model/` and a training run rarely writes its model next to the corpus (#102).""" + + def test_the_implicit_directory_is_named_in_a_warning(self, tmp_path: Path, caplog) -> None: + import logging + + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "data" / "train.jsonl" + corpus.parent.mkdir() + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=None) + with caplog.at_level(logging.WARNING, logger="forgelm"): + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + assert ei.value.code == 0 + warned = [r.getMessage() for r in caplog.records if "--output-dir not given" in r.getMessage()] + assert len(warned) == 1 + assert str(corpus.parent) in warned[0] and os.path.join(str(corpus.parent), "final_model") in warned[0] + + def test_an_explicit_output_dir_is_silent(self, tmp_path: Path, caplog) -> None: + import logging + + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "train.jsonl" + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=str(tmp_path)) + with caplog.at_level(logging.WARNING, logger="forgelm"): + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + assert not [r for r in caplog.records if "--output-dir not given" in r.getMessage()] + + def test_no_memorisation_event_without_a_final_model_in_the_resolved_directory(self, tmp_path: Path) -> None: + # The default directory is the corpus's; a model written elsewhere is invisible to the check. The warning + # above is what tells the operator. + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "data" / "train.jsonl" + corpus.parent.mkdir() + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + (tmp_path / "training_out" / "final_model").mkdir(parents=True) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=None) + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + events = _read_audit_events(corpus.parent / "audit_log.jsonl") + assert "data.erasure_warning_memorisation" not in [e["event"] for e in events] + + class TestSaltPersistence: def test_salt_file_created_on_first_use_with_mode_0600(self, tmp_path: Path) -> None: from forgelm.cli.subcommands._purge import _resolve_salt diff --git a/tests/test_model.py b/tests/test_model.py index 771a671e..cac537e0 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -115,15 +115,105 @@ def test_quantization_config_added_with_cuda(self): assert "quantization_config" in kwargs fake_bnb.assert_called_once() - def test_rope_and_sliding_window_passed_through(self): + def test_sliding_window_alone_is_a_keyword_argument(self): cfg = _cfg( model={"name_or_path": "org/model", "load_in_4bit": False}, - training={"rope_scaling": {"type": "linear", "factor": 2.0}, "sliding_window_attention": 4096}, + training={"sliding_window_attention": 4096}, ) with patch("torch.cuda.is_available", return_value=False): kwargs = model_mod._build_model_kwargs(cfg, trust_remote_code=False) - assert kwargs["rope_scaling"] == {"type": "linear", "factor": 2.0} - assert kwargs["sliding_window"] == 4096 + assert kwargs["sliding_window"] == 4096 and "config" not in kwargs + + def test_rope_scaling_travels_in_a_config_object_not_a_keyword(self): + # transformers 5 no longer accepts `rope_scaling=` on from_pretrained (#103): the settings go into the + # config, together with the sliding-window override, which would otherwise reach the model constructor. + model_config = SimpleNamespace(rope_parameters={"rope_theta": 10000.0, "rope_type": "default"}) + fake_transformers = MagicMock() + fake_transformers.AutoConfig.from_pretrained.return_value = model_config + cfg = _cfg( + model={"name_or_path": "org/model", "load_in_4bit": False}, + training={"rope_scaling": {"type": "linear", "factor": 2.0}, "sliding_window_attention": 4096}, + ) + with ( + patch("torch.cuda.is_available", return_value=False), + patch.dict("sys.modules", {"transformers": fake_transformers}), + ): + kwargs = model_mod._build_model_kwargs(cfg, trust_remote_code=False, revision="abc123") + assert "rope_scaling" not in kwargs and "sliding_window" not in kwargs + assert kwargs["config"] is model_config + assert model_config.rope_parameters == {"rope_theta": 10000.0, "rope_type": "linear", "factor": 2.0} + assert model_config.sliding_window == 4096 + fake_transformers.AutoConfig.from_pretrained.assert_called_once_with( + "org/model", trust_remote_code=False, revision="abc123" + ) + + +class TestConfigWithRopeScaling: + """Against a real (tiny, local, CPU) Llama config: the merged settings must build and run a model (#103).""" + + @staticmethod + def _tiny_model_dir(tmp_path): + from transformers import LlamaConfig + + LlamaConfig( + vocab_size=64, + hidden_size=32, + intermediate_size=64, + num_hidden_layers=1, + num_attention_heads=2, + num_key_value_heads=2, + max_position_embeddings=128, + ).save_pretrained(tmp_path) + return str(tmp_path) + + @pytest.mark.parametrize( + "requested, expected_type", + [ + ({"type": "linear", "factor": 4.0}, "linear"), # the key every documented example and the wizard use + ({"rope_type": "linear", "factor": 4.0}, "linear"), + ({"type": "dynamic", "factor": 2.0}, "dynamic"), + ({"rope_type": "yarn", "factor": 2.0, "original_max_position_embeddings": 128}, "yarn"), + ], + ) + def test_the_merged_config_builds_a_model_that_runs(self, tmp_path, requested, expected_type): + import torch + from transformers import AutoModelForCausalLM + + path = self._tiny_model_dir(tmp_path) + model_config = model_mod._config_with_rope_scaling(path, requested, trust_remote_code=False) + assert model_config.rope_parameters["rope_type"] == expected_type + assert model_config.rope_parameters["rope_theta"] == 10000.0 # kept from the model, not lost + assert "type" not in model_config.rope_parameters + logits = AutoModelForCausalLM.from_config(model_config)(input_ids=torch.tensor([[1, 2, 3, 4]])).logits + assert tuple(logits.shape) == (1, 4, 64) + + def test_requested_rope_theta_overrides_the_models(self, tmp_path): + path = self._tiny_model_dir(tmp_path) + model_config = model_mod._config_with_rope_scaling( + path, {"type": "linear", "factor": 2.0, "rope_theta": 500000.0}, trust_remote_code=False + ) + assert model_config.rope_parameters["rope_theta"] == 500000.0 + + def test_a_transformers_without_rope_parameters_gets_the_legacy_attribute(self): + legacy = SimpleNamespace() + fake_transformers = MagicMock() + fake_transformers.AutoConfig.from_pretrained.return_value = legacy + with patch.dict("sys.modules", {"transformers": fake_transformers}): + out = model_mod._config_with_rope_scaling( + "org/model", {"type": "linear", "factor": 2.0}, trust_remote_code=False + ) + assert out is legacy and legacy.rope_scaling == {"rope_type": "linear", "factor": 2.0} + + def test_multimodal_settings_land_in_the_text_config(self): + text = SimpleNamespace(rope_parameters={"rope_theta": 1.0, "rope_type": "default"}) + outer = SimpleNamespace(get_text_config=lambda: text) + fake_transformers = MagicMock() + fake_transformers.AutoConfig.from_pretrained.return_value = outer + with patch.dict("sys.modules", {"transformers": fake_transformers}): + out = model_mod._config_with_rope_scaling( + "org/vlm", {"type": "linear", "factor": 2.0}, trust_remote_code=False + ) + assert out is outer and text.rope_parameters["rope_type"] == "linear" class TestLoadTokenizer: @@ -152,6 +242,47 @@ def test_existing_pad_token_preserved(self): assert fake_tok.pad_token == "[PAD]" +class TestChatTemplateNameCannotEscapeTheSaveDirectory: + """CVE-2026-9856: a ``chat_template`` name from a Hub repo was used as a file name by ``save_pretrained``. + + ForgeLM loads a tokenizer from a Hub repo and saves it again to ``final_model/`` (trainer, GGUF export, merge), so + a crafted ``tokenizer_config.json`` could write a file outside the output directory. Fixed in transformers + 5.10.0 (yanked) and 5.10.1, which is why the declared floor is ``>=5.10.1``. This uses the installed transformers + and a tiny local tokenizer, so it also fails on a floor install that predates the fix. + """ + + def _crafted_tokenizer_dir(self, root): + import json + + from tokenizers import Tokenizer, models, pre_tokenizers + from transformers import PreTrainedTokenizerFast + + tok = Tokenizer(models.WordLevel({"[UNK]": 0, "hello": 1, "": 2}, unk_token="[UNK]")) + tok.pre_tokenizer = pre_tokenizers.Whitespace() + source = root / "hub_repo" + PreTrainedTokenizerFast(tokenizer_object=tok, unk_token="[UNK]", eos_token="").save_pretrained(source) + config_file = source / "tokenizer_config.json" + config = json.loads(config_file.read_text(encoding="utf-8")) + config["chat_template"] = [ + {"name": "../../../ESCAPED_BY_CHAT_TEMPLATE", "template": "{{ messages }}"}, + {"name": "default", "template": "{{ messages }}"}, + ] + config_file.write_text(json.dumps(config), encoding="utf-8") + return source + + def test_a_traversal_name_is_refused_and_nothing_is_written_outside(self, tmp_path): + source = self._crafted_tokenizer_dir(tmp_path) + out = tmp_path / "run" / "final_model" + cfg = _cfg(model={"name_or_path": str(source)}) + + with pytest.raises(ValueError, match="Invalid chat template name"): + tokenizer = model_mod._load_tokenizer(cfg, trust_remote_code=False) + tokenizer.save_pretrained(out) + + escaped = [p for p in tmp_path.rglob("*") if p.is_file() and "ESCAPED" in p.name] + assert escaped == [], f"a chat_template name wrote outside the save directory: {escaped}" + + class TestBuildLoraConfig: """``_build_lora_config`` resolves the PEFT method and target modules.""" diff --git a/tests/test_packaging.py b/tests/test_packaging.py index ab8c562e..3018791c 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -244,6 +244,85 @@ def test_setuptools_security_floor_meets_pysec_2026_3447() -> None: ) +def _load_runtime_dependencies() -> list: + try: + import tomllib # type: ignore[import-not-found] + except ModuleNotFoundError: # pragma: no cover — Python 3.10 fallback + try: + import tomli as tomllib # type: ignore[import-not-found, no-redef] + except ModuleNotFoundError: + pytest.skip("Neither tomllib (3.11+) nor tomli is available; runtime dependency assertion skipped.") + pyproject_path = Path(__file__).resolve().parent.parent / "pyproject.toml" + with pyproject_path.open("rb") as fh: + pyproject = tomllib.load(fh) + return pyproject.get("project", {}).get("dependencies", []) + + +# transformers 5.10.1 is the first installable release past three advisories that reach ForgeLM's own load and +# save path from a Hub repo: CVE-2026-4372 (5.3.0), CVE-2026-5241 (a LightGlue config.json overrides +# trust_remote_code; 5.5.0) and CVE-2026-9856 (a chat_template name used as a file name by save_pretrained; +# 5.10.0, which PyPI yanked). A floor of ">=5.3.0" was documented as "the CVE fix floor" and was not. +_TRANSFORMERS_SECURITY_FLOOR = Version("5.10.1") + + +def test_transformers_floor_is_past_cve_2026_5241_and_cve_2026_9856() -> None: + """``transformers`` must not admit a release that predates the chat_template / trust_remote_code fixes.""" + found = [ + Requirement(spec) + for spec in _load_runtime_dependencies() + if canonicalize_name(Requirement(spec).name) == "transformers" + ] + assert len(found) == 1, f"expected exactly one transformers requirement in [project].dependencies, found {found!r}" + specifier_set = found[0].specifier + lower_bounds = [ + Version(spec.version.rstrip(".*")) for spec in specifier_set if spec.operator in _LOWER_BOUND_OPERATORS + ] + assert lower_bounds, f"transformers entry {str(found[0])!r} has no lower bound" + assert max(lower_bounds) >= _TRANSFORMERS_SECURITY_FLOOR, ( + f"transformers entry {str(found[0])!r} floors at {max(lower_bounds)}, below {_TRANSFORMERS_SECURITY_FLOOR}, " + "the first installable release past CVE-2026-5241 and CVE-2026-9856. Do not lower it without re-verifying " + "both advisories against the release you lower it to." + ) + for vulnerable in ("5.3.0", "5.4.0", "5.9.0", "5.10.0"): + assert not specifier_set.contains(Version(vulnerable), prereleases=True), ( + f"transformers entry {str(found[0])!r} still admits {vulnerable}" + ) + + +def test_nightly_floor_job_pins_the_runtime_dependency_minimums() -> None: + """Every ``name==X`` the nightly minimum-versions job installs is the ``>=X`` floor in pyproject. + + The job installs with ``--no-deps`` afterwards, so it can only test the documented minimum if its list is the + same list. It is a second hand-kept copy of every floor, and it kept saying ``transformers==5.3.0`` after the + floor was meant to be a security floor. + """ + import re + + nightly = (Path(__file__).resolve().parents[1] / ".github" / "workflows" / "nightly.yml").read_text( + encoding="utf-8" + ) + step = nightly.split("Install minimum dependency versions", 1)[1].split("- name:", 1)[0] + pins = { + canonicalize_name(name): version for name, version in re.findall(r'"([A-Za-z0-9_.-]+)==([0-9][^"]*)"', step) + } + assert pins, "the nightly minimum-versions step no longer installs `name==version` pins — update this test" + + floors = {} + for spec in _load_runtime_dependencies(): + req = Requirement(spec) + lowers = [s.version for s in req.specifier if s.operator == ">="] + if lowers: + floors[canonicalize_name(req.name)] = lowers[0] + + for name, version in pins.items(): + assert name in floors, f"the nightly floor job pins {name}=={version}, which is not a runtime dependency" + assert version == floors[name], ( + f"the nightly floor job pins {name}=={version} but pyproject.toml declares {name}>={floors[name]}; " + "the job would test a minimum nobody documents" + ) + assert {"transformers", "huggingface-hub"} <= set(pins), "the floor job must pin transformers and huggingface_hub" + + def _lint_job_pins() -> dict[str, Requirement]: """Requirements the ci.yml lint job installs directly. diff --git a/tools/check_module_size.py b/tools/check_module_size.py index 35bb0b0d..5fc6af13 100644 --- a/tools/check_module_size.py +++ b/tools/check_module_size.py @@ -488,7 +488,7 @@ class _DeferredSplit: ), ), "forgelm/cli/subcommands/_purge.py": _DeferredSplit( - budget=1216, + budget=1224, deferred_at_loc=1215, budget_history=( "2026-10-09 (Phase 16 S1-S3 review, 1215 -> 1216, +1): the `dumps_strict` import. " @@ -496,6 +496,10 @@ class _DeferredSplit: "strict-JSON guard's hand-picked module list; the guard now scans all of " "`forgelm/`, so they route through the chokepoint like every other writer. " "One line, taken as a raise rather than a split per decision C-18.", + "2026-10-18 (wave-0 triage, #102, 1216 -> 1224, +8): the warning `purge --row-id` logs when " + "`--output-dir` is omitted. The default (the corpus directory) silently pointed the memorisation " + "check at a directory that never holds a trained model, so the message has to sit where the " + "default is resolved. Taken as a raise rather than a split per decision C-18.", ), reason=( "GDPR purge: row-id resolution + run-id resolution + retention-policy " diff --git a/tools/pip_audit_ignores.yaml b/tools/pip_audit_ignores.yaml index cd16a831..e70e7619 100644 --- a/tools/pip_audit_ignores.yaml +++ b/tools/pip_audit_ignores.yaml @@ -40,6 +40,9 @@ ignores: # so a suppression would be dead weight: # - CVE-2026-1839 — fixed in transformers 5.0.0rc3. # - PYSEC-2025-217 / CVE-2025-14929 (X-CLIP RCE) — OSV last_affected 5.0.0rc0. + # The floor has since moved to transformers>=5.10.1, the first installable + # release past CVE-2026-5241 (fixed 5.5.0) and CVE-2026-9856 (fixed 5.10.0, + # yanked), so neither needs a suppression either. # ----- torch (added 2026-05-21, triage in issue #58) ----- #