fix: rope_scaling on transformers 5, hub telemetry, reserved retention.enforce, transformers CVE floor, FSDP strategies, purge output dir

.github/workflows/nightly.yml

@@ -237,14 +237,17 @@ jobs:
           # EXPLICITLY so this job tests the true documented minimum — trl puts
           # no upper bound on transformers, so an unpinned resolve would silently
           # pull the newest 5.x and stop exercising the floor. transformers
-          # 5.3.0 is the CVE-2026-4372 fix floor and requires torch>=2.4 /
-          # huggingface_hub>=1.3 / tokenizers>=0.22; trl 1.0.0 pulls the
-          # accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the pyproject
-          # [project].dependencies minimums.
+          # 5.10.1 is the first installable release past CVE-2026-4372,
+          # CVE-2026-5241 and CVE-2026-9856 (5.10.0, which carries the last of
+          # them, was yanked); it requires huggingface_hub>=1.5 /
+          # tokenizers>=0.22, and ForgeLM itself needs torch>=2.4; trl 1.0.0
+          # pulls the accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the
+          # pyproject [project].dependencies minimums (tests/test_packaging.py
+          # compares them).
           pip install \
             "torch==2.4.0" \
-            "transformers==5.3.0" \
-            "huggingface_hub==1.3.0" \
+            "transformers==5.10.1" \
+            "huggingface_hub==1.5.0" \
             "peft==0.19.0" \
             "accelerate==1.4.0" \
             "datasets==4.7.0" \

CHANGELOG.md

@@ -37,6 +37,14 @@ All notable changes to ForgeLM are documented here.
 
 ### Changed
 
+- Raise the `transformers` floor from `>=5.3.0` to `>=5.10.1` (and `huggingface_hub` from `>=1.3.0` to `>=1.5.0`, which
+  transformers 5.10 requires). `5.3.0` was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue
+  `config.json` overrides `trust_remote_code: false`, fixed in 5.5.0) and CVE-2026-9856 (a `chat_template` name in
+  `tokenizer_config.json` is used unsanitised as a file name by `save_pretrained`, so `final_model/` saves, GGUF export
+  and merges could write outside the output directory; fixed in 5.10.0, which PyPI yanked, hence 5.10.1). The nightly
+  minimum-versions job pins the new floor, and `tests/test_packaging.py` keeps it from drifting from
+  `pyproject.toml` and checks that a traversal `chat_template` name cannot escape on save. Environments pinned to
+  transformers 5.3 to 5.9 must upgrade (#109).
 - Record monitoring default provenance in `defaulted_fields` and mark literal URL projection with
   `endpoint_redacted` (also true when the URL already contains only scheme/host/port). Literal
   endpoints retain scheme/host/port in JSON evidence, including IPv6 brackets; userinfo, path,
@@ -79,6 +87,36 @@ All notable changes to ForgeLM are documented here.
 
 ### Fixed
 
+- `training.rope_scaling` works on transformers 5 again. The setting was passed to `from_pretrained(rope_scaling=...)`,
+  which no longer works: the documented `{type: linear, factor: 4}` raised `KeyError: 'rope_type'` while the model was
+  built, and `{rope_type: ..., factor: ...}` replaced the whole parameter dict so `rope_theta` was lost (`TypeError`).
+  The requested keys are now merged over the model's own `rope_parameters` (the legacy key `type` is read as
+  `rope_type`), applied to the text config of a multimodal checkpoint, and the finished config is loaded. The
+  `sliding_window_attention` override travels in the same config object. Verified on transformers 5.13.0 for
+  `linear`, `dynamic` and `yarn` on a tiny local Llama; not exercised on the 5.3-5.9 range the floor still admits (#103).
+- The `forgelm` CLI no longer sends Hugging Face's anonymous usage ping. TRL's trainers call `send_telemetry` when they are
+  built, and `huggingface_hub` sends it unless `HF_HUB_DISABLE_TELEMETRY` is true or offline mode is on, which contradicted the
+  privacy page ("does not send usage data anywhere"). `main()` now sets `HF_HUB_DISABLE_TELEMETRY=1` before anything imports
+  `huggingface_hub`, and an operator's own value wins (`HF_HUB_DISABLE_TELEMETRY=0` opts back in). Using ForgeLM as a library
+  does not set it (#95).
+- `retention.enforce` is documented for what it is: reserved. No command reads it, so `warn_on_excess` and
+  `block_on_excess` never warned or blocked anything and nothing was deleted automatically; retention is only reported,
+  read-only, by `forgelm purge --check-policy` (always exit 0). The config description, the wizard prompt, the
+  configuration reference, the GDPR erasure guide and `config_template.yaml` now say so (English and Turkish), and
+  the CLI logs a warning when a config sets a non-default `enforce` instead of staying silent (#92, #98).
+- `distributed.fsdp_strategy: no_shard` and `hybrid_shard` are rejected when the config loads (exit 1). They were documented, passed `--dry-run`,
+  and then failed with exit 2 at trainer construction (`fsdp=hybrid_shard is only available with FSDP1`), because transformers 5.10+ runs
+  FSDP2 by default. `fsdp_auto_wrap: false` and `fsdp_backward_prefetch: backward_post` are still accepted but change nothing under FSDP2;
+  ForgeLM now logs a warning when either is set, and the field descriptions, reference pages, user manuals and `config_template.yaml`
+  say so. Moving to FSDP2 settings (`fsdp=True` plus an explicit `fsdp_config`) is not part of this change; the string form is
+  deprecated in transformers and slated for removal in v5.20 (#104).
+- `forgelm purge --row-id` without `--output-dir` now logs a warning that names the directory it resolved (the corpus's)
+  and the `final_model/` the memorisation check will look for. The documentation promised that warning but the code never
+  logged it, and a default run looks for a model next to the corpus, where a training run rarely writes one, so it finished
+  with a completed-erasure record and no memorisation event. The audit-event catalogue, the purge reference, the GDPR
+  erasure guide, the user manual and the risk-treatment plan now state the real condition: a `final_model/` directly under
+  `--output-dir`, not "any run that consumed the corpus" (English and Turkish). Reading `training.output_dir` from
+  `--config` is still open (#102).
 - Correct the GGUF pages (English and Turkish): the export example shows the real output, K-quants are described as
   the two-step operation they are, the invented "manifest" and `verify-gguf` output (vocabulary and tokenizer checks that
   do not exist) are replaced by the real ones, a clean `verify-gguf` exit is said to prove only what ran, and the sidecar

config_template.yaml

@@ -167,7 +167,7 @@ data:
 #   strategy: "deepspeed"           # "deepspeed" or "fsdp"
 #   deepspeed_config: "zero2"       # preset: "zero2", "zero3", "zero3_offload" or path to JSON
 #   # --- FSDP options (when strategy: "fsdp") ---
-#   # fsdp_strategy: "full_shard"   # "full_shard", "shard_grad_op", "no_shard", "hybrid_shard"
+#   # fsdp_strategy: "full_shard"   # "full_shard" or "shard_grad_op" (FSDP2 supports no other)
 #   # fsdp_auto_wrap: true
 #   # fsdp_offload: false
 
@@ -276,7 +276,7 @@ data:
 #   staging_ttl_days: 7               # final_model.staging.<run_id>/ TTL after `forgelm reject`
 #   ephemeral_artefact_retention_days: 90   # compliance bundles + audit reports
 #   raw_documents_retention_days: 90        # ingested raw docs (PDF/DOCX/EPUB/TXT/MD)
-#   enforce: "log_only"               # "log_only" | "warn_on_excess" | "block_on_excess"
+#   enforce: "log_only"               # reserved: no command acts on it yet ("log_only" | "warn_on_excess" | "block_on_excess")
 
 # Phase 14 — Multi-stage training pipeline chains (SFT → DPO → GRPO).
 # Uncomment the ``pipeline:`` block below to run multiple training stages

docs/guides/gdpr_erasure-tr.md

@@ -19,7 +19,7 @@
 
 **Yapmaz:**
 
-- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, corpus'u tüketen herhangi bir koşum için `final_model/` mevcut olduğunda `data.erasure_warning_memorisation` yayar; böylece operatör boşluğu görür.
+- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez.
 - Audit log'un kendisinden entry silmez. Madde 17(3)(b) audit / muhasebe kayıtlarını bir hukuki yükümlülük savunması olarak korur; tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder.
 - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür.
 - Yedekleri silmez. Operatörün `<output_dir>` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir.
@@ -78,10 +78,10 @@ retention:
   staging_ttl_days: 7                     # `forgelm reject` üzerine aksiyon almak için bir iş haftası
   ephemeral_artefact_retention_days: 90   # üç-aylık review cadence'i
   raw_documents_retention_days: 90        # data audit yeniden çalıştırmadan önce ingestion penceresi
-  enforce: log_only                       # log_only / warn_on_excess / block_on_excess
+  enforce: log_only                       # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess)
 ```
 
-Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi devre dışı bırakır (sınırsız tut). `enforce`, trainer pre-flight gate'inin ihlallere nasıl tepki vereceğini kontrol eder.
+Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi devre dışı bırakır (sınırsız tut). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur, her zaman 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. `enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz, bu yüzden üç değer de `log_only` gibi davranır.
 
 ## Kaldırıldı: `evaluation.staging_ttl_days`
 
@@ -96,7 +96,7 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d
 | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden önce (`--check-policy` salt-okunur; audit event yaymaz) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` |
 | `data.erasure_completed` | Başarılı silme bittiğinde | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) |
 | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti | Tüm `requested` field'ları + `error_class`, `error_message` |
-| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` |
+| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` |
 | `data.erasure_warning_synthetic_data_present` | **Yalnız row modunda.** Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. (Run modu bu uyarıyı yüzeylemez — synthetic veri corpus-scoped'dur, run-scoped değil.) | Tüm `completed` field'ları + `synthetic_files` |
 | `data.erasure_warning_external_copies` | Yüklü config webhook block'u içeriyor | Tüm `completed` field'ları + `webhook_targets` |
 

docs/guides/gdpr_erasure.md

@@ -19,7 +19,7 @@
 
 **Does not:**
 
-- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` exists for any run that consumed the corpus, so the operator sees the gap.
+- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus.
 - Delete entries from the audit log itself. Article 17(3)(b) preserves audit / accounting records as a legal-obligation defence; the tool records the erasure as a new event rather than rewriting history.
 - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder.
 - Erase backups. Replicas, snapshots, and storage backups outside the operator's `<output_dir>` are infrastructure-side concerns.
@@ -78,10 +78,10 @@ retention:
   staging_ttl_days: 7                     # one work-week to act on a `forgelm reject`
   ephemeral_artefact_retention_days: 90   # quarterly review cadence
   raw_documents_retention_days: 90        # ingestion-window before re-running data audit
-  enforce: log_only                       # log_only / warn_on_excess / block_on_excess
+  enforce: log_only                       # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess)
 ```
 
-Setting any horizon to `0` disables the policy for that artefact kind (retain indefinitely). `enforce` controls how the trainer pre-flight gate reacts to violations.
+Setting any horizon to `0` disables the policy for that artefact kind (retain indefinitely). The horizons are reported by `forgelm purge --check-policy` (read-only, always exits 0); nothing is deleted or blocked automatically. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`.
 
 ## Removed: `evaluation.staging_ttl_days`
 
@@ -96,7 +96,7 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev
 | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, before any deletion (`--check-policy` is read-only and emits no audit events) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` |
 | `data.erasure_completed` | Successful deletion finishes | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) |
 | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity | All `requested` fields + `error_class`, `error_message` |
-| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × `final_model/` exists for any run that consumed the corpus. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` |
+| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a `final_model/` directory exists directly under `--output-dir`. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` |
 | `data.erasure_warning_synthetic_data_present` | **Row mode only.** Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. (Run mode does not surface this warning — synthetic data is corpus-scoped, not run-scoped.) | All `completed` fields + `synthetic_files` |
 | `data.erasure_warning_external_copies` | Loaded config has a webhook block (both modes). | All `completed` fields + `webhook_targets` |
 

docs/qms/risk_treatment_plan-tr.md

@@ -90,7 +90,7 @@ ekler.
 |---|---|
 | Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir |
 | L × I (inherent) | High × Med = HIGH |
-| Treatment | ForgeLM, silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` event'ı yayar — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). |
+| Treatment | ForgeLM, purge'ün `--output-dir`'i altında doğrudan bir `final_model/` varsa `data.erasure_warning_memorisation` event'ı yayar (satırı hangi koşumun tükettiğini bilmez ve varsayılan dizin eğitim çıktısı değil corpus'unkidir) — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). |
 | Residual L × I | High × Med = HIGH (risk **kabul edildi**, bkz. §5) |
 | Sahip | Data Protection Officer (DPO) |
 | Review cadence | Madde 17 talebi başına |

docs/qms/risk_treatment_plan.md

@@ -89,7 +89,7 @@ section §4.
 |---|---|
 | Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights |
 | L × I (inherent) | High × Med = HIGH |
-| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when the run that consumed the deleted row has a `final_model/` artefact — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). |
+| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when a `final_model/` exists directly under the purge's `--output-dir` (it does not know which run consumed the row, and the default directory is the corpus's, not the training output) — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). |
 | Residual L × I | High × Med = HIGH (risk **accepted**, see §5) |
 | Owner | Data Protection Officer (DPO) |
 | Review cadence | Per Article 17 request |