diff --git a/CHANGELOG.md b/CHANGELOG.md index 41d1450c..3a5c8a47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,29 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Fixed + +- **`forgelm purge --row-id` warns about the weights a run actually leaves behind.** This restores the documented + contract that `data.erasure_warning_memorisation` fires when a trained model exists for the erased row. The check + used to look only for a literal `<--output-dir>/final_model/`, so a default invocation (`--output-dir` omitted, + which uses the corpus directory), a non-default `training.final_model_dir`, a run awaiting or refused human + approval (its model exists only as `.staging./`) and a run whose auto-revert deleted the + model but left `checkpoint-*` all exited `0` with no warning. It now reports the model directory, its + `.staging[.]` siblings and `checkpoint-*` directories under `--output-dir` and, with + `--config`, under `training.output_dir` / `training.final_model_dir`, every `pipeline` stage's `training` block + and `merge.output_dir`. The event gains `memorisation_artefact_paths` (the absolute path of every artefact + found); `affected_run_ids` also reads the run's own audit log under the configured training directory; a + directory that cannot be listed is logged at WARNING instead of reading as "nothing found". Models you export or + copy outside those directories stay invisible to the check (#102, CR-A10-022, CR-A10-042). +- `forgelm purge --run-id --kind staging` reads `--config` for `training.final_model_dir`, and + `--check-policy` takes it from a full config, so a run with a non-default model directory has its + `.staging./` found, erased and aged like the default `final_model.staging./`. + A `retention:`-only file still means the default `final_model`. No retention horizon covers `checkpoint-*` + directories; the purge, retention and configuration docs now say so (#102, CR-A10-022, CR-A10-042). +- The user manual states that `forgelm purge --row-id` matches the row's `id` / `row_id` field only, never the row's + text or a line number, and points to `forgelm reverse-pii` for finding every row that mentions the subject + (#102, DOC-D3B02-005). + ## [0.11.1] — 2026-10-23 _(**Numbered as a patch by a maintainer decision, as a one-time exception.** diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 89085def..59cd9693 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -479,10 +479,10 @@ The design above shipped in v0.5.5, with the deviations listed after this list: - Retention horizons are reported by `forgelm purge --check-policy` and never enforced; `retention.enforce` is reserved and no command acts on it (the training CLI logs a warning when it is set). There is no trainer pre-flight gate: nothing exits 3 or refuses a run on age. - The legacy `final_model.staging/` is purged only when it holds a `staging_run_id` marker equal to the run id. Nothing in ForgeLM writes that marker. - `--check-policy` ages the raw-documents directories (`raw_documents/`, `ingestion_output/`) from the audit log's first event, not per run; it reads `--config` only when given (no `./forgelm.yaml` discovery) and, without a `retention:` block, reports that nothing was checked. `--dry-run` combined with `--check-policy` is not rejected. -- The `data.erasure_warning_external_copies` event is emitted in row mode only, and only when `--config` is given; run mode never loads the config. +- The `data.erasure_warning_external_copies` event is emitted in row mode only, and only when `--config` is given; run mode reads the config only for `training.final_model_dir`, to name the staging directory. - Mode flags that conflict (for example `--row-id` with `--run-id`, or `--kind logs`) are rejected by the argument parser, which exits 2 without a JSON envelope; other configuration errors exit 1. A run id is not looked up in the audit log: it is "unknown" only when no matching artefact exists. - With `FORGELM_AUDIT_SECRET` set, the per-directory salt is XOR-combined with the first 16 bytes of the secret (`salt_source=env_var`), so a row's `target_id` changes when the secret rotates. `forgelm reverse-pii` derives its `query_hash` the same way in its default plaintext mode; `--salt-source` switches it to a hash-mask scan that looks for digests inside the corpus. -- `data.erasure_warning_memorisation` fires whenever `/final_model/` exists; the check does not tie that model to the erased row. +- `data.erasure_warning_memorisation` fires whenever a model directory, a `.staging[.]` directory or a `checkpoint-*` directory exists under `--output-dir` or, with `--config`, under the training directories the config names (and `merge.output_dir`); the check does not tie any of them to the erased row. - The recovery paths of §4.4 assume the temp file survives a failed rename; the shipped code removes it (best effort) instead. - Neither the trainer pre-flight gate nor the `forgelm audit` retention notes of §3.4 exist. - The `recommendation` field listed for the new events in §5.1 is not emitted, and the test table in §7 predates the shipped tests (for example, a dry-run appends audit events, and erasing the same row twice exits 1 with `NoMatchingRow`); `tests/test_gdpr_erasure.py` is the source of truth. diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 1e4aef7b..0a8c6b42 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -22,7 +22,7 @@ **Yapmaz:** -- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. +- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, bir model dizini, bir `.staging.` staging dizini (insan onayını bekleyen veya reddedilen bir koşum) ya da bir `checkpoint-*` dizini bulduğunda `data.erasure_warning_memorisation` yayar — `--output-dir` altında (varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin) ve `--config` ile config'in adlandırdığı eğitim dizinlerinde (pipeline stage'leri ve `merge.output_dir` dahil) — böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez ve başka yere kopyaladığınız ya da export ettiğiniz modelleri görmez. - Audit log'un kendisinden entry silmez. Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Madde 17(1)'deki silme nedenlerinden biri geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki taleplerin ileri sürülmesini, kullanılmasını veya savunulmasını kapsar. Madde 17(1)'deki nedenlerin hiçbiri geçerli değilse (girdilerin toplandıkları amaç için hâlâ gerekli olması yalnızca 17(1)(a)'yı dışlar; rızanın geri çekilmesini, Madde 21 itirazını ve hukuka aykırı işlemeyi de kontrol edin) silme gerekmez; ancak talebe yine de gecikmeksizin ve en geç talebin alınmasından itibaren bir ay içinde yanıt verilir; yanıtta gerekçe, bir denetim otoritesine şikâyet etme ve yargı yoluna başvurma imkânı bildirilir (Madde 12(4)). - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve veri sahibi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). Tool, row modunda `--config` ile verdiğiniz config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. @@ -60,7 +60,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` -- `--kind staging` — `/final_model.staging./`'i siler; legacy `final_model.staging/`'i yalnızca içeriği ``'ye eşit bir `staging_run_id` işaretçisi taşıyorsa siler (ForgeLM bu işaretçiyi yazmaz; aksi halde legacy dizini kendiniz silin). +- `--kind staging` — `/.staging./`'i siler (`--config` başka bir `training.final_model_dir` adlandırmadıkça `final_model`); legacy `.staging/`'i yalnızca içeriği ``'ye eşit bir `staging_run_id` işaretçisi taşıyorsa siler (ForgeLM bu işaretçiyi yazmaz; aksi halde legacy dizini kendiniz silin). - `--kind artefacts` — `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosyayı siler (yalnızca `.json` değil). **Standart paketi bulamaz:** trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...), bu yüzden normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Bu, gerçek bir run id için (`fg-…`) geçerlidir: sabit bir adda geçen genel bir token (örneğin `json` veya `report`) o dosyalarla eşleşir ve onları siler; bu yüzden her zaman tam run id'yi verin. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. - `--kind logs` **bilinçli olarak yok**: audit log append-only'dir ve tool onu asla silmez (girdilerin tutulup tutulamayacağına kimin karar verdiği için yukarıdaki "Yapmaz" listesine bakın). @@ -70,7 +70,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` -Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. `--config`, eğitim config'i ya da yalnızca `retention:` bloğunu içeren bir dosya olabilir; başka bir üst düzey anahtar içeren dosya tam config olarak doğrulanır. +Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: `.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. `--config`, eğitim config'i ya da yalnızca `retention:` bloğunu içeren bir dosya olabilir; başka bir üst düzey anahtar içeren dosya tam config olarak doğrulanır. `final_model_dir`, tam bir config'in `training` bloğundan gelir; yalnızca `retention:` içeren dosya varsayılan `final_model`'i bırakır. **Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve raporu kendileri sınar, örneğin `set -o pipefail` altında `jq -e '.success == true and has("count") and .count == 0'`: `count`, hiçbir `retention:` bloğu yüklenmediğinde yoktur; böylece gate hiç kontrol edilmemiş bir dizinde geçmek yerine başarısız olur, ve `--output-dir` gerçek eğitim çıktı dizini olmalıdır (olmayan bir dizin `count: 0` raporlar). Bu, public exit-code kontratını her ForgeLM subcommand'ında tutarlı tutar. @@ -85,7 +85,7 @@ retention: enforce: log_only # ayrılmış: henüz hiçbir komut onu dikkate almaz (log_only / warn_on_excess / block_on_excess) ``` -Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapatır (her iki durumda da hiçbir şey silinmez). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur; kaç tane bulursa bulsun 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. ForgeLM `raw_documents/` veya `ingestion_output/` dizinlerini oluşturmaz; `raw_documents_retention_days` yalnızca kaynak dokümanları orada tutuyorsanız etkili olur. `enforce` ayrılmıştır: henüz hiçbir komut onu dikkate almaz, bu yüzden üç değer de `log_only` gibi davranır. +Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapatır (her iki durumda da hiçbir şey silinmez). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur; kaç tane bulursa bulsun 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. ForgeLM `raw_documents/` veya `ingestion_output/` dizinlerini oluşturmaz; `raw_documents_retention_days` yalnızca kaynak dokümanları orada tutuyorsanız etkili olur. Hiçbir horizon `checkpoint-*` dizinlerini kapsamaz: bir koşum bunları `training.output_dir` altına kaydeder (`save_total_limit`'in tuttuğu kadarını) ve rapor bunları hiç listelemez; bu yüzden depolama katmanınız üzerinden kendiniz silin. `enforce` ayrılmıştır: henüz hiçbir komut onu dikkate almaz, bu yüzden üç değer de `log_only` gibi davranır. ## Kaldırıldı: `evaluation.staging_ttl_days` @@ -100,9 +100,9 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapa | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden önce (`--check-policy` salt-okunur; audit event yaymaz) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme bittiğinde | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında ya da `--config` ile config'in adlandırdığı eğitim dizinlerinde bir model dizini, bir staging dizini veya bir `checkpoint-*` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids`, `memorisation_artefact_paths` | | `data.erasure_warning_synthetic_data_present` | **Yalnız row modunda.** Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. (Run modu bu uyarıyı yüzeylemez — synthetic veri corpus-scoped'dur, run-scoped değil.) | Tüm `completed` field'ları + `synthetic_files` | -| `data.erasure_warning_external_copies` | **Yalnız row modunda, `--config` verildiğinde.** Yüklü config webhook block'u içeriyor. (Run modu `--config` yüklemez.) | Tüm `completed` field'ları + `webhook_targets` | +| `data.erasure_warning_external_copies` | **Yalnız row modunda, `--config` verildiğinde.** Yüklü config webhook block'u içeriyor. (Run modu `--config`'i yalnızca staging dizinini adlandırmak için `training.final_model_dir` değerini okumak üzere yükler.) | Tüm `completed` field'ları + `webhook_targets` | ## Silme sonrası zincir doğrulama diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 74df41f7..91d66b04 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -21,7 +21,7 @@ **Does not:** -- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. +- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when it finds a model directory, a `.staging.` staging directory (a run awaiting or refused human approval) or a `checkpoint-*` directory — under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory) and, with `--config`, under the training directories the config names (pipeline stages and `merge.output_dir` included) — so the operator sees the gap. It cannot tell which runs consumed the corpus, and it does not see models you copied or exported elsewhere. - Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history, and never deletes an audit-log entry. Whether entries that name the data subject may be kept is the controller's decision, and it has to be documented. If an Article 17(1) ground applies, the entries stay only where an Article 17(3) exception covers them: 17(3)(b) applies only to the extent processing is necessary for compliance with a legal obligation under Union or Member State law to which the controller is subject (or for a task in the public interest or in the exercise of official authority), and 17(3)(e) covers the establishment, exercise or defence of legal claims. If no Article 17(1) ground applies (the entries still being necessary for the purpose they were collected for rules out only 17(1)(a); also check consent withdrawal, an Article 21 objection and unlawful processing), erasure is not owed, but the request still gets an answer without delay and at the latest within one month of receipt, giving the reasons and the possibility of complaining to a supervisory authority and seeking a judicial remedy (Article 12(4)). - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (GDPR Article 19 requires telling each recipient the data was disclosed to about the erasure, unless that proves impossible or involves disproportionate effort, and telling the data subject about those recipients on request; Article 17(2) adds reasonable steps where the controller made the data public). The tool emits `data.erasure_warning_external_copies` in row mode when the config you pass with `--config` has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. @@ -59,7 +59,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` -- `--kind staging` removes `/final_model.staging./`, and the legacy `final_model.staging/` only if it carries a `staging_run_id` marker whose content equals the `` (ForgeLM does not write that marker; otherwise remove the legacy directory yourself). +- `--kind staging` removes `/.staging./` (`final_model` unless `--config` names another `training.final_model_dir`), and the legacy `.staging/` only if it carries a `staging_run_id` marker whose content equals the `` (ForgeLM does not write that marker; otherwise remove the legacy directory yourself). - `--kind artefacts` removes any file directly under `/compliance/` whose name contains the `` bounded by `_`, `-`, `.` or the start/end of the name (not only `.json`). **It does not find the standard bundle:** the trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, so on a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. That holds for a real run id (`fg-…`): a generic token that appears in a fixed name, such as `json` or `report`, does match those files and deletes them, so always pass the full run id. Remove the bundle through your storage layer until the resolver is fixed. - `--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it (see the "Does not" list above for who decides whether its entries may be kept). @@ -69,7 +69,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` -Read-only scan. Walks the output directory, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. `--config` can be the training config or a file that holds only the `retention:` block; a file with any other top-level key is validated as a full config. +Read-only scan. Walks the output directory, compares each artefact's age (canonical: the run's first audit event for a `.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. `--config` can be the training config or a file that holds only the `retention:` block; a file with any other top-level key is validated as a full config. `final_model_dir` comes from a full config's `training` block; a `retention:`-only file leaves the default `final_model`. **Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and test the report themselves, for example `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`: `count` is absent when no `retention:` block was loaded, so the gate fails instead of passing on a directory that was never checked, and `--output-dir` must be the real training output directory (a missing one reports `count: 0`). This keeps the public exit-code contract consistent across every ForgeLM subcommand. @@ -84,7 +84,7 @@ retention: enforce: log_only # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess) ``` -Setting any horizon to `0` turns the report off for that artefact kind (nothing is deleted either way). The horizons are reported by `forgelm purge --check-policy` (read-only; exits 0 however many it finds); nothing is deleted or blocked automatically. ForgeLM does not create `raw_documents/` or `ingestion_output/`; `raw_documents_retention_days` has an effect only if you keep source documents there. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. +Setting any horizon to `0` turns the report off for that artefact kind (nothing is deleted either way). The horizons are reported by `forgelm purge --check-policy` (read-only; exits 0 however many it finds); nothing is deleted or blocked automatically. ForgeLM does not create `raw_documents/` or `ingestion_output/`; `raw_documents_retention_days` has an effect only if you keep source documents there. No horizon covers `checkpoint-*` directories: a run saves them under `training.output_dir` (as many as `save_total_limit` keeps), and the report never lists them, so remove them through your storage layer. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. ## Removed: `evaluation.staging_ttl_days` @@ -99,9 +99,9 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, before any deletion (`--check-policy` is read-only and emits no audit events) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finishes | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a `final_model/` directory exists directly under `--output-dir`. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a model directory, a staging directory or a `checkpoint-*` directory exists under `--output-dir` or, with `--config`, under the training directories the config names. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids`, `memorisation_artefact_paths` | | `data.erasure_warning_synthetic_data_present` | **Row mode only.** Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. (Run mode does not surface this warning — synthetic data is corpus-scoped, not run-scoped.) | All `completed` fields + `synthetic_files` | -| `data.erasure_warning_external_copies` | **Row mode only, when `--config` is given.** Loaded config has a webhook block. (Run mode never loads `--config`.) | All `completed` fields + `webhook_targets` | +| `data.erasure_warning_external_copies` | **Row mode only, when `--config` is given.** Loaded config has a webhook block. (Run mode reads `--config` only for `training.final_model_dir`, to name the staging directory.) | All `completed` fields + `webhook_targets` | ## Verifying a chain post-erasure diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index 9ee66a10..75a7fe80 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -293,8 +293,8 @@ forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ --output-dir ./outputs # Audit chain `target_id` HASH'LENMİŞ ŞEKİLDE # `data.erasure_requested` → `data.erasure_completed` (veya -# `data.erasure_failed`) damgalar. Eğer `--output-dir` altında doğrudan bir -# `final_model/` dizini varsa `data.erasure_warning_memorisation` DA +# `data.erasure_failed`) damgalar. Eğer `--output-dir` altında bir model, staging +# veya `checkpoint-*` dizini varsa `data.erasure_warning_memorisation` DA # tetiklenir; satırı hangi modelin gördüğünü söylemez, bu yüzden # corpus ile eğitilmiş her modeli kapsam içinde sayın ve memorisation # kalan ezberleme riskini veri sahibine bildirin. diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md index 74e23dcd..b3fa07a4 100644 --- a/docs/guides/iso_soc2_deployer_guide.md +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -294,8 +294,9 @@ forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ --output-dir ./outputs # The audit chain stamps `data.erasure_requested` → # `data.erasure_completed` (or `data.erasure_failed`) with -# `target_id` HASHED. If a `final_model/` directory exists directly -# under `--output-dir`, `data.erasure_warning_memorisation` ALSO fires; +# `target_id` HASHED. If a model, staging or `checkpoint-*` directory +# exists under `--output-dir`, `data.erasure_warning_memorisation` ALSO +# fires; # it does not say which model saw the row, so treat every model # trained on the corpus as in scope and communicate the memorisation # residual risk to the subject. diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md index ee0f437d..64bbe2c3 100644 --- a/docs/qms/risk_treatment_plan-tr.md +++ b/docs/qms/risk_treatment_plan-tr.md @@ -90,7 +90,7 @@ ekler. |---|---| | Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM, purge'ün `--output-dir`'i altında doğrudan bir `final_model/` varsa `data.erasure_warning_memorisation` event'ı yayar (satırı hangi koşumun tükettiğini bilmez ve varsayılan dizin eğitim çıktısı değil corpus'unkidir) — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | +| Treatment | ForgeLM, purge'ün `--output-dir`'i altında ya da `--config` ile config'in adlandırdığı eğitim dizinlerinde bir model, staging veya `checkpoint-*` dizini varsa `data.erasure_warning_memorisation` event'ı yayar (satırı hangi koşumun tükettiğini bilmez, varsayılan dizin eğitim çıktısı değil corpus'unkidir ve başka yere export edilen ya da kopyalanan modeller ona görünmez) — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | | Residual L × I | High × Med = HIGH (risk **kabul edildi**, bkz. §5) | | Sahip | Data Protection Officer (DPO) | | Review cadence | Madde 17 talebi başına | diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md index cf6d5efe..ecc0c34f 100644 --- a/docs/qms/risk_treatment_plan.md +++ b/docs/qms/risk_treatment_plan.md @@ -89,7 +89,7 @@ section §4. |---|---| | Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when a `final_model/` exists directly under the purge's `--output-dir` (it does not know which run consumed the row, and the default directory is the corpus's, not the training output) — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | +| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when a model, staging or `checkpoint-*` directory exists under the purge's `--output-dir` or, with `--config`, under the training directories the config names (it does not know which run consumed the row, the default directory is the corpus's, not the training output, and models exported or copied elsewhere are invisible to it) — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | | Residual L × I | High × Med = HIGH (risk **accepted**, see §5) | | Owner | Data Protection Officer (DPO) | | Review cadence | Per Article 17 request | diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md index d1ba101c..e34ad5c3 100644 --- a/docs/qms/sop_incident_response-tr.md +++ b/docs/qms/sop_incident_response-tr.md @@ -277,10 +277,11 @@ Madde 13(2)'de 30 gün). 2. [ ] **`forgelm purge --row-id --corpus data/.jsonl --output-dir ` çalıştır**. 3. [ ] **`data.erasure_warning_memorisation` olayını kontrol et** — - yalnızca `--output-dir` altında doğrudan bir `final_model/` - dizini varsa tetiklenir ve satırı hangi modelin gördüğünü - söylemez; uyarı tetiklenmese bile corpus ile eğitilmiş her - modeli kapsam içinde say. + yalnızca `--output-dir` altında (veya `--config` ile config'in + adlandırdığı eğitim dizinlerinde) bir model, staging ya da + `checkpoint-*` dizini bulursa tetiklenir ve satırı hangi modelin + gördüğünü söylemez; uyarı tetiklenmese bile corpus ile eğitilmiş + her modeli kapsam içinde say. 4. [ ] **Memorisation caveat'ı sahibe bildir** yanıt mektubunda (şablon: "satırı sildik ama önceden eğitilmiş bir model bunu memorise etmiş olabilir; uygun şekilde yeniden eğiteceğiz veya diff --git a/docs/qms/sop_incident_response.md b/docs/qms/sop_incident_response.md index 1265760f..8ce2bc0b 100644 --- a/docs/qms/sop_incident_response.md +++ b/docs/qms/sop_incident_response.md @@ -279,10 +279,11 @@ month; 30 days under KVKK Art. 13(2)). 2. [ ] **Run** `forgelm purge --row-id --corpus data/.jsonl --output-dir `. 3. [ ] **Check** for `data.erasure_warning_memorisation` event — - it fires only when a `final_model/` directory exists directly - under `--output-dir` and does not say which model saw the row, so - treat every model trained on the corpus as in scope even when no - warning fires. + it fires only when it finds a model, staging or `checkpoint-*` + directory under `--output-dir` (or, with `--config`, under the + training directories the config names) and does not say which + model saw the row, so treat every model trained on the corpus as + in scope even when no warning fires. 4. [ ] **Communicate the memorisation caveat** to the subject in the response letter (template: "we deleted the row but a previously-trained model may have memorised it; we will diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md index d2a8b755..9718be6c 100644 --- a/docs/reference/audit_event_catalog-tr.md +++ b/docs/reference/audit_event_catalog-tr.md @@ -78,7 +78,7 @@ Hash zinciri, satır diske düştükten (`flush` + `fsync`) sonra ilerler; kirli | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunur; event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × memorisation-risk artefact'lar mevcut. Her zaman taranır: birebir `<--output-dir>/final_model` yerleşimi (`--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). `--config` verilip yüklendiğinde ayrıca taranır: `--output-dir` altında yapılandırılan her `final_model_dir`, yapılandırılan her `training.output_dir` / `training.final_model_dir` yerleşimi (üst düzeyde ve her `pipeline` stage'inde) ve `merge.output_dir`. Raporlanır: model dizini (``), onun `.staging[.]` kardeşleri (insan onayını bekleyen veya reddedilen bir koşum) ve `checkpoint-*` dizinleri. ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` (koşum kimliği bulunamazsa: `unknown — no matching staging directory or human_approval.granted event`), `memorisation_artefact_paths` (bulunan her artefact'ın mutlak path'i) | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Yalnızca `--config` ile row-modu purge: config'in `webhook` bloğu boş değil (literal `url` ya da export edilmiş `url_env`); downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | 17 | diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md index 2eed9532..130fa447 100644 --- a/docs/reference/audit_event_catalog.md +++ b/docs/reference/audit_event_catalog.md @@ -78,7 +78,7 @@ The hash chain advances after the line lands on disk (`flush` + `fsync`), so an | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × memorisation-risk artefacts exist. Always scanned: the literal `<--output-dir>/final_model` layout (`--output-dir` defaults to the corpus's directory, so pass your training output directory). Also scanned when `--config` is given and loads: `--output-dir` paired with each configured `final_model_dir`, each configured `training.output_dir` / `training.final_model_dir` layout (top-level and per-`pipeline` stage), and `merge.output_dir`. Reported: the model directory (``), its `.staging[.]` siblings (a run awaiting or refused human approval) and `checkpoint-*` directories. ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` (when no run ID is available: `unknown — no matching staging directory or human_approval.granted event`), `memorisation_artefact_paths` (absolute path of every artefact found) | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Row-mode purge with `--config` only: the config's `webhook` block is non-empty (literal `url`, or `url_env` exported); downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | 17 | diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index bc4d65e7..9718a177 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -279,12 +279,17 @@ Ufuklar **raporlanır, zorlanmaz**: `forgelm purge --check-policy` ufku aşanlar listeler ve kaç tane bulursa bulsun 0 ile çıkar (yüklenemeyen `--config` 1 ile çıkar); hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `retention:` bloğu yoksa hiçbir şey kontrol edilmez; `retention: {}` varsayılanları devreye alır. +Staging dizinleri `training.final_model_dir`'i izler (yalnızca `retention:` içeren dosya +bir tane adlandırmaz; rapor varsayılan `final_model`'i arar). **Hiçbir ufuk +`checkpoint-*` dizinlerini kapsamaz**: bir koşum bunları `training.output_dir` +altına kaydeder (`save_total_limit`'in tuttuğu kadarını) ve rapor bunları hiç +listelemez; bu yüzden depolama katmanınız üzerinden kendiniz silin. `enforce` anahtarı ayrılmıştır: henüz hiçbir komut onu dikkate almaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| | `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "süresi geçmiş" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir. `0` bu raporu kapatır (hiçbir şey silinmez); log'u yalnızca girdiler gerekli kaldığı sürece ya da adlandırabileceğiniz belirli bir yasal saklama yükümlülüğü gerektirdiği sürece tutun. | -| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve eski `final_model.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` bu raporu kapatır. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | +| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `.staging./` dizinini (ve eski `.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` bu raporu kapatır. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | | `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` bu raporu kapatır. | | `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. ForgeLM bu dizinleri oluşturmaz; ufuk yalnızca orada tuttuğunuz kaynak dokümanlara uygulanır. `0` bu raporu kapatır. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Hiçbir komut onu kullanmaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa eğitim ve `--dry-run` uyarır. | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 00abe768..b0c8bd82 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -278,13 +278,18 @@ The horizons are **reported, not enforced**: `forgelm purge --check-policy` lists what is past them and exits 0 however many it finds (an unloadable `--config` exits 1), and nothing is deleted, blocked or scheduled automatically. Without a `retention:` block nothing is checked at all; -`retention: {}` switches the defaults on. The `enforce` knob is reserved: no -command acts on it yet. +`retention: {}` switches the defaults on. Staging directories follow +`training.final_model_dir` (a `retention:`-only file names none, so the report +looks for the default `final_model`). **No horizon covers `checkpoint-*` +directories**: a run saves them under `training.output_dir` (as many as +`save_total_limit` keeps) and the report never lists them, so remove them through +your storage layer. The +`enforce` knob is reserved: no command acts on it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| | `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period. Set to `0` to stop reporting it (nothing is ever deleted); keep it only for as long as the entries stay necessary or a specific legal retention duty you can name requires it. | -| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to stop reporting it. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | +| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `.staging./` directory (and the legacy `.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to stop reporting it. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | | `ephemeral_artefact_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/compliance/` and `/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to stop reporting it. | | `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. ForgeLM does not create these directories, so the horizon applies only to source documents you keep there. Set to `0` to stop reporting it. | | `enforce` | string | `"log_only"` | Reserved policy mode. No command acts on it: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and training and `--dry-run` warn when a config sets either of the last two. | diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index e003c9fa..8467e651 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -35,7 +35,8 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | `--row-id ID` | evet | Silinecek identifier. Audit emisyonundan önce `forgelm.cli.subcommands._purge._hash_target_id` üzerinden hash'lenir. | | `--corpus PATH` | evet | Tek bir JSONL dosyası. Dizin modu reddedilir — operatörler kendi script'lerinde döngü kurar. | | `--row-matches {one,all}` | hayır (varsayılan `one`) | `one` >=2 eşleşmede reddeder; `all` her eşleşmeyi siler (operatör niyeti onaylar). | -| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, operatörün `--output-dir`'i eğitim çıktı dizinine ve `forgelm reverse-pii` ile cross-tool korelasyon için sabitleyebilmesi adına çözülen dizini ve memorisation kontrolünün arayacağı `final_model/`'i adlandıran bir WARNING yayar. | +| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, operatörün `--output-dir`'i eğitim çıktı dizinine ve `forgelm reverse-pii` ile cross-tool korelasyon için sabitleyebilmesi adına çözülen dizini ve memorisation kontrolünün orada arayacağı `final_model/`, staging ve `checkpoint-*` dizinlerini adlandıran bir WARNING yayar. | +| `--config PATH` | hayır | memorisation kontrolünün taradığı eğitim yerleşimlerini adlandırır: `training.output_dir` / `training.final_model_dir`, her `pipeline` stage'inin kendi `training` bloğu ve `merge.output_dir`; `webhook` bloğu external-copies uyarısını besler. Yüklenemeyen config WARNING seviyesinde loglanır ve purge onsuz devam eder. | | `--justification TEXT` | hayır | Operatör-sağlanan, her erasure event'ına yazılan sebep. Dahili ticket id'nizi referans verin; subject identifier yapıştırmayın. | | `--dry-run` | hayır | Silmeyi önizler + audit zincirini (`dry_run=true` ile) yayar; disk'e dokunmaz. | @@ -47,7 +48,7 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | Kind | Hedef | |---|---| -| `staging` | `/final_model.staging./`; legacy `final_model.staging/` yalnızca içerdiği `staging_run_id` dosyasının içeriği ``'ye eşitse. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | +| `staging` | `/.staging./` (`--config` başka bir `training.final_model_dir` adlandırmadıkça `final_model`; pipeline stage'lerinin değerleri dahil); legacy `.staging/` yalnızca içerdiği `staging_run_id` dosyasının içeriği ``'ye eşitse. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | | `artefacts` | `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosya (yalnızca `.json` değil). | > **`--kind artefacts` standart compliance paketini bulamaz.** Trainer ve exporter sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`; yapılandırıldığında `risk_assessment.json` / `annex_iv_metadata.json`) ve hiçbiri run id içermez. Normal bir çıktı dizininde komut bu yüzden hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar ve 1 ile çıkar; hiçbir şey silmez. Bu, gerçek bir run id için (`fg-…`) geçerlidir: sabit bir adda geçen genel bir token (örneğin `json` veya `report`) o dosyalarla eşleşir ve onları siler; bu yüzden her zaman tam run id'yi verin. Yalnızca adına run id'yi kendiniz yazdığınız dosyalara etki eder. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. @@ -56,7 +57,7 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir ### Retention-policy raporu (`--check-policy`) -Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. Config, tam bir eğitim config'i ya da tek üst düzey anahtarı `retention:` olan bir YAML dosyası olabilir; başka bir üst düzey anahtar içeren dosya tam config olarak doğrulanır. +Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. Config, tam bir eğitim config'i ya da tek üst düzey anahtarı `retention:` olan bir YAML dosyası olabilir; başka bir üst düzey anahtar içeren dosya tam config olarak doğrulanır. Staging dizinleri `.staging[.]` olarak aranır: tam bir config `final_model_dir`'i `training` bloğundan (ve pipeline stage'lerinden) sağlar, yalnızca `retention:` içeren dosya varsayılan `final_model`'i bırakır. **Hiçbir horizon `checkpoint-*` dizinlerini kapsamaz**: bir koşum bunları `training.output_dir` altına kaydeder (`save_total_limit`'in tuttuğu kadarını) ve rapor bunları hiç listelemez; bu yüzden depolama katmanınız üzerinden kendiniz silin. **Paylaşılan çıktı dizinleri.** `compliance/`, `data_audit_report.json` ve ham-belge dizinlerinin yaşı, dosyanın en son yazıldığı andan değil, audit log'un *ilk* event'inden sayılır. Birçok koşum için yeniden kullanılan bir dizinde bugün yazılmış bir paket de ufku aşmış olarak raporlanabilir: raporun listelediği bir şeyi silmeden önce dosyanın kendi tarihine bakın. @@ -84,7 +85,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | `data.erasure_requested` | Herhangi bir `--row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunurdur ve event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'li), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum yok VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × memorisation-risk artefact'lar mevcut. Her zaman taranır: birebir `<--output-dir>/final_model` yerleşimi (`--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). `--config` verilip yüklendiğinde ayrıca taranır: `--output-dir` altında yapılandırılan her `final_model_dir`, yapılandırılan her `training.output_dir` / `training.final_model_dir` yerleşimi (üst düzeyde ve her `pipeline` stage'inde) ve `merge.output_dir`. Raporlanır: model dizini (``), onun `.staging[.]` kardeşleri (insan onayını bekleyen veya reddedilen bir koşum) ve `checkpoint-*` dizinleri. ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` (koşum kimliği bulunamazsa: `unknown — no matching staging directory or human_approval.granted event`), `memorisation_artefact_paths` (bulunan her artefact'ın mutlak path'i) | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | | `data.erasure_warning_external_copies` | **Yalnız row modunda, `--config` verildiğinde.** Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index 35a4f363..9e6ef1a5 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -35,7 +35,8 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | `--row-id ID` | yes | Identifier to erase. Hashed before audit emission via `forgelm.cli.subcommands._purge._hash_target_id`. | | `--corpus PATH` | yes | Single JSONL file. Directory mode is rejected — operators loop in their own script. | | `--row-matches {one,all}` | no (default `one`) | `one` refuses on >=2 matches; `all` deletes every match (operator confirms intent). | -| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir and the `final_model/` the memorisation check will look for, so the operator can pin `--output-dir` to the training output directory and for cross-tool correlation with `forgelm reverse-pii`. | +| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir and the `final_model/` (and the staging and `checkpoint-*` directories) the memorisation check will look for there, so the operator can pin `--output-dir` to the training output directory and for cross-tool correlation with `forgelm reverse-pii`. | +| `--config PATH` | no | Names the training layouts the memorisation check scans: `training.output_dir` / `training.final_model_dir`, every `pipeline` stage's own `training` block, and `merge.output_dir`; its `webhook` block feeds the external-copies warning. A config that cannot be loaded is logged at WARNING and the purge continues without it. | | `--justification TEXT` | no | Operator-supplied reason recorded on every erasure event. Reference your internal ticket id; do not paste subject identifiers. | | `--dry-run` | no | Preview the deletion + emit the audit chain (with `dry_run=true`) without modifying disk. | @@ -47,7 +48,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | Kind | Target | |---|---| -| `staging` | `/final_model.staging./`, and the legacy `final_model.staging/` only when it contains a `staging_run_id` file whose content equals ``. ForgeLM does not write that marker, so remove an unmarked legacy staging directory yourself. | +| `staging` | `/.staging./` (`final_model` unless `--config` names another `training.final_model_dir`, a pipeline stage's included), and the legacy `.staging/` only when it contains a `staging_run_id` file whose content equals ``. ForgeLM does not write that marker, so remove an unmarked legacy staging directory yourself. | | `artefacts` | Any file directly under `/compliance/` whose name contains `` bounded by `_`, `-`, `.` or the start/end of the name (not only `.json`). | > **`--kind artefacts` does not find the standard compliance bundle.** The trainer and exporter write fixed basenames (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`, and `risk_assessment.json` / `annex_iv_metadata.json` when configured), none of which contains a run id. On a normal output directory the command therefore finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`) and exits 1; it deletes nothing. That holds for a real run id (`fg-…`): a generic token that appears in a fixed name, such as `json` or `report`, does match those files and deletes them, so always pass the full run id. It acts only on files you have named with the run id yourself. Until the resolver is fixed, remove the bundle through your storage layer. @@ -56,7 +57,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai ### Retention-policy report (`--check-policy`) -Read-only scan. Walks ``, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. The config can be a full training config or a YAML file whose only top-level key is `retention:`; a file with any other top-level key is validated as a full config. +Read-only scan. Walks ``, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. The config can be a full training config or a YAML file whose only top-level key is `retention:`; a file with any other top-level key is validated as a full config. Staging directories are looked up as `.staging[.]`: a full config supplies `final_model_dir` from its `training` block (and its pipeline stages), a `retention:`-only file leaves the default `final_model`. **No horizon covers `checkpoint-*` directories**: a run saves them under `training.output_dir` (as many as `save_total_limit` keeps) and the report never lists them, so remove them through your storage layer. **Shared output directories.** `compliance/`, `data_audit_report.json` and the raw-document directories are aged from the audit log's *first* event, not from when the file was last written. In a directory reused for many runs, a bundle written today can be reported as past its horizon: check the file's own date before deleting anything the report lists. @@ -84,7 +85,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | `data.erasure_requested` | First step of any `--row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × memorisation-risk artefacts exist. Always scanned: the literal `<--output-dir>/final_model` layout (`--output-dir` defaults to the corpus's directory, so pass your training output directory). Also scanned when `--config` is given and loads: `--output-dir` paired with each configured `final_model_dir`, each configured `training.output_dir` / `training.final_model_dir` layout (top-level and per-`pipeline` stage), and `merge.output_dir`. Reported: the model directory (``), its `.staging[.]` siblings (a run awaiting or refused human approval) and `checkpoint-*` directories. ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` (when no run ID is available: `unknown — no matching staging directory or human_approval.granted event`), `memorisation_artefact_paths` (absolute path of every artefact found) | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | | `data.erasure_warning_external_copies` | **Row mode only, when `--config` is given.** Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 27fb4c85..0a5c4897 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -35,6 +35,8 @@ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` +`--row-id` is matched against the row's `id` or `row_id` field only: never against the row's text and never against a line number. A row that merely mentions the subject (an e-mail address inside `text`) is not touched, and a row without an `id` cannot be purged. Run `forgelm reverse-pii --query ` first to find every row that mentions the subject, then purge each one by its id. + ### What gets recorded Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/audit_event_catalog.md)): @@ -42,7 +44,7 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - `data.erasure_requested` — emitted FIRST, before any deletion. - `data.erasure_completed` — emitted LAST after the disk operation succeeded. - `data.erasure_failed` — emitted instead when the disk operation raised, or no matching row/run was found. -- `data.erasure_warning_memorisation` — corpus-row erasure × a `final_model/` exists directly under `--output-dir` (the row is gone from disk but may still be memorised in the trained weights). +- `data.erasure_warning_memorisation` — corpus-row erasure × a model directory, a staging directory (`.staging.`) or a `checkpoint-*` directory exists under `--output-dir` or, with `--config`, under the training directories the config names (the row is gone from disk but may still be memorised in the trained weights). - `data.erasure_warning_synthetic_data_present` — corpus-row erasure × `synthetic_data*.jsonl` exists in `output_dir`. - `data.erasure_warning_external_copies` — corpus-row erasure only, and only when you pass `--config`: the config's `webhook` block is non-empty (a literal `url`, or a `url_env` variable exported in the purge environment); downstream consumers may have received notices. @@ -50,7 +52,7 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - **Re-train models.** Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` so the gap is visible. - **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history, and never deletes an audit-log entry. Whether entries that name the data subject may be kept is the controller's decision, and it has to be documented. If an Article 17(1) ground applies, the entries stay only where an Article 17(3) exception covers them: 17(3)(b) applies only to the extent processing is necessary for compliance with a legal obligation under Union or Member State law to which the controller is subject (or for a task in the public interest or in the exercise of official authority), and 17(3)(e) covers the establishment, exercise or defence of legal claims. If no Article 17(1) ground applies (the entries still being necessary for the purpose they were collected for rules out only 17(1)(a); also check consent withdrawal, an Article 21 objection and unlawful processing), erasure is not owed, but the request still gets an answer without delay and at the latest within one month of receipt, giving the reasons and the possibility of complaining to a supervisory authority and seeking a judicial remedy (Article 12(4)). -- **Remove an unmarked legacy `final_model.staging/`.** `--kind staging` deletes `final_model.staging./`, and the legacy `final_model.staging/` only when it holds a `staging_run_id` file equal to the run id. ForgeLM does not write that marker, so remove an unmarked legacy directory yourself. +- **Remove an unmarked legacy `final_model.staging/`.** `--kind staging` deletes `.staging./` (`final_model` unless `--config` names another `training.final_model_dir`), and the legacy `.staging/` only when it holds a `staging_run_id` file equal to the run id. ForgeLM does not write that marker, so remove an unmarked legacy directory yourself. - **Find the standard compliance bundle with `--kind artefacts`.** The trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, and the resolver matches only filenames that embed it. On a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. That holds for a real run id (`fg-…`): a generic token that appears in a fixed name, such as `json` or `report`, does match those files and deletes them, so always pass the full run id. Remove the bundle through your storage layer. - **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (GDPR Article 19 requires telling each recipient the data was disclosed to about the erasure, unless that proves impossible or involves disproportionate effort, and telling the data subject about those recipients on request; Article 17(2) adds reasonable steps where the controller made the data public). The `data.erasure_warning_external_copies` event is the explicit reminder. - **Make the whole response.** A complete response also needs steps ForgeLM does not perform: verifying the requester's identity with reasonable measures (recital 64; Article 12(6) lets you ask for more information where you have reasonable doubts); answering within one month of receipt (Article 12(3); extendable by two further months for complex or numerous requests, with notice in the first month); for an access request, giving the data subject a copy of the data and the Article 15(1)(a)-(h) information (Article 15(3)) and, where the data went to a third country, the Article 46 safeguards (Article 15(2)), without adversely affecting other people's rights (Article 15(4)); for an erasure request, deciding whether an Article 17(1) ground applies and which Article 17(3) exception, if any, covers what you keep; telling recipients (Article 19); re-running the access search after a purge to confirm the rows are gone; and, if you refuse a request, informing the requester without delay and at the latest within one month of the reasons, the possibility of complaining and the possibility of seeking a judicial remedy (Article 12(4)). Whether the trained model counts as personal data is a separate question (EDPB Opinion 28/2024). @@ -113,7 +115,7 @@ Codes 3 to 6 (`EXIT_EVAL_FAILURE`, `EXIT_AWAITING_APPROVAL`, `EXIT_WIZARD_CANCEL ::: :::warn -**Running `forgelm purge` without checking for memorisation.** If a trained model exists, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap only when a `final_model/` sits directly under `--output-dir` — which defaults to the corpus's directory, so pass your training output directory; act on it (re-train without the erased row, or document the residual risk in your DPIA). +**Running `forgelm purge` without checking for memorisation.** If a trained model exists, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap only for artefacts it can see: a model, staging or `checkpoint-*` directory under `--output-dir` — which defaults to the corpus's directory, so pass your training output directory — or under the training directories a `--config` names; models you exported or copied elsewhere are invisible to it. Act on it (re-train without the erased row, or document the residual risk in your DPIA). ::: :::tip diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index b1838af4..dbdef0dd 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -77,7 +77,7 @@ The scan reports the file and line where the identifier appears (the snippets ar "Delete my data." If a person's data was in your training set: -1. Find the rows with `forgelm reverse-pii`, then remove them with `forgelm purge --row-id --corpus --output-dir ./outputs` (the same directory as `reverse-pii`; use your training output directory so purge can also warn about a trained `final_model/`); every erasure is written to the audit log. `--row-id` is matched against the row's `id` or `row_id` field and never against a line number, and `forgelm ingest` does not write either field, so give every row a stable id **before you train** (see below). A corpus without ids cannot be purged row by row. +1. Find the rows with `forgelm reverse-pii`, then remove them with `forgelm purge --row-id --corpus --output-dir ./outputs` (the same directory as `reverse-pii`; use your training output directory so purge can also warn about a trained model, its staging directory or checkpoints); every erasure is written to the audit log. `--row-id` is matched against the row's `id` or `row_id` field only, never against the row's text or a line number, and `forgelm ingest` does not write either field, so give every row a stable id **before you train** (see below). A corpus without ids cannot be purged row by row. 2. Source documents may still hold it — drop them from your raw store and re-ingest if necessary. 3. The model may have memorised some details — see "model-level erasure" below. diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 2124db61..d2308d0e 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -35,6 +35,8 @@ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` +`--row-id` yalnızca satırın `id` veya `row_id` alanıyla eşleştirilir: satırın metniyle de satır numarasıyla da asla. Veri sahibinden yalnızca söz eden bir satıra (`text` içindeki bir e-posta adresi) dokunulmaz ve `id`'si olmayan bir satır silinemez. Veri sahibinden söz eden her satırı bulmak için önce `forgelm reverse-pii --query ` çalıştırın, sonra her birini kendi id'siyle silin. + ### Neyi kaydeder Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/audit_event_catalog-tr.md)): @@ -42,7 +44,7 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - `data.erasure_requested` — herhangi bir silmeden ÖNCE, ilk yayılır. - `data.erasure_completed` — disk operasyonu başarılı sonra son yayılır. - `data.erasure_failed` — disk operasyonu raise ettiğinde veya eşleşen satır/koşum bulunmadığında bunun yerine yayılır. -- `data.erasure_warning_memorisation` — corpus-satır silme × `--output-dir` altında doğrudan bir `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). +- `data.erasure_warning_memorisation` — corpus-satır silme × `--output-dir` altında ya da `--config` ile config'in adlandırdığı eğitim dizinlerinde bir model dizini, bir staging dizini (`.staging.`) veya bir `checkpoint-*` dizini mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). - `data.erasure_warning_synthetic_data_present` — corpus-satır silme × `output_dir`'de `synthetic_data*.jsonl` mevcut. - `data.erasure_warning_external_copies` — yalnızca corpus-satır silmede ve yalnızca `--config` verdiğinizde: config'in `webhook` bloğu boş değil (literal bir `url` ya da purge ortamında export edilmiş bir `url_env` değişkeni); downstream tüketiciler bildirim almış olabilir. @@ -50,7 +52,7 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - **Modelleri yeniden eğitmez.** Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözümdür. Tool `data.erasure_warning_memorisation` yayar; böylece boşluk görünür olur. - **Audit log'unu silmez.** Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Madde 17(1)'deki silme nedenlerinden biri geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki taleplerin ileri sürülmesini, kullanılmasını veya savunulmasını kapsar. Madde 17(1)'deki nedenlerin hiçbiri geçerli değilse (girdilerin toplandıkları amaç için hâlâ gerekli olması yalnızca 17(1)(a)'yı dışlar; rızanın geri çekilmesini, Madde 21 itirazını ve hukuka aykırı işlemeyi de kontrol edin) silme gerekmez; ancak talebe yine de gecikmeksizin ve en geç talebin alınmasından itibaren bir ay içinde yanıt verilir; yanıtta gerekçe, bir denetim otoritesine şikâyet etme ve yargı yoluna başvurma imkânı bildirilir (Madde 12(4)). -- **İşaretsiz legacy `final_model.staging/` dizinini silmez.** `--kind staging`, `final_model.staging./` dizinini siler; legacy `final_model.staging/` dizinini yalnızca içinde run id'ye eşit bir `staging_run_id` dosyası varsa siler. ForgeLM bu işaretçiyi yazmaz, bu yüzden işaretsiz bir legacy dizini kendiniz silin. +- **İşaretsiz legacy `final_model.staging/` dizinini silmez.** `--kind staging`, `.staging./` dizinini siler (`--config` başka bir `training.final_model_dir` adlandırmadıkça `final_model`); legacy `.staging/` dizinini yalnızca içinde run id'ye eşit bir `staging_run_id` dosyası varsa siler. ForgeLM bu işaretçiyi yazmaz, bu yüzden işaretsiz bir legacy dizini kendiniz silin. - **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca run id'yi embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Bu, gerçek bir run id için (`fg-…`) geçerlidir: sabit bir adda geçen genel bir token (örneğin `json` veya `report`) o dosyalarla eşleşir ve onları siler; bu yüzden her zaman tam run id'yi verin. Paketi depolama katmanınız üzerinden kaldırın. - **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve veri sahibi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. - **Yanıtın tamamını vermez.** Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini makul önlemlerle doğrulamak (Gerekçe 64; Madde 12(6), makul şüphe varsa ek bilgi istemenize izin verir); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); bir erişim talebinde, veri sahibine verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve veri üçüncü bir ülkeye aktarıldıysa Madde 46 güvencelerini (Madde 15(2)) vermek; bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir silme talebinde, Madde 17(1)'deki nedenlerden birinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz gecikmeksizin ve en geç bir ay içinde gerekçeyi, şikâyet etme ve yargı yoluna başvurma imkânını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). @@ -113,7 +115,7 @@ Kod 3 ile 6 arası (`EXIT_EVAL_FAILURE`, `EXIT_AWAITING_APPROVAL`, `EXIT_WIZARD_ ::: :::warn -**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Eğitilmiş bir model varsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yalnızca `--output-dir` altında doğrudan bir `final_model/` varsa yüzeyselleştirir — `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin; harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). +**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Eğitilmiş bir model varsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yalnızca görebildiği artefact'lar için yüzeyselleştirir: `--output-dir` altındaki — varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin — veya bir `--config`'in adlandırdığı eğitim dizinlerindeki bir model, staging ya da `checkpoint-*` dizini; başka yere export ettiğiniz ya da kopyaladığınız modeller ona görünmez. Harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). ::: :::tip diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index e3c862a8..f1c44e86 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -77,7 +77,7 @@ Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler "Verilerimi silin." Bir kişinin verisi eğitim setinizdeydi: -1. Satırları `forgelm reverse-pii` ile bulun, sonra `forgelm purge --row-id --corpus --output-dir ./outputs` ile kaldırın (`reverse-pii` ile aynı dizin; purge'ün eğitilmiş bir `final_model/` için de uyarabilmesi adına eğitim çıktı dizininizi kullanın); her silme audit log'a yazılır. `--row-id` satırın `id` veya `row_id` alanıyla eşleştirilir, satır numarasıyla asla; `forgelm ingest` bu alanlardan hiçbirini yazmaz. Bu yüzden **eğitimden önce** her satıra kalıcı bir id verin (aşağıya bakın). Id'si olmayan bir corpus satır satır silinemez. +1. Satırları `forgelm reverse-pii` ile bulun, sonra `forgelm purge --row-id --corpus --output-dir ./outputs` ile kaldırın (`reverse-pii` ile aynı dizin; purge'ün eğitilmiş bir model, onun staging dizini veya checkpoint'ler için de uyarabilmesi adına eğitim çıktı dizininizi kullanın); her silme audit log'a yazılır. `--row-id` yalnızca satırın `id` veya `row_id` alanıyla eşleştirilir, satırın metniyle ya da satır numarasıyla asla; `forgelm ingest` bu alanlardan hiçbirini yazmaz. Bu yüzden **eğitimden önce** her satıra kalıcı bir id verin (aşağıya bakın). Id'si olmayan bir corpus satır satır silinemez. 2. Ham kaynak dokümanlar hâlâ içerebilir — ham deponuzdan düşürün ve gerekirse yeniden ingest edin. 3. Model bazı detayları ezberlemiş olabilir — aşağıdaki "model-seviyesi silme"ye bakın. diff --git a/forgelm/cli/_config_load.py b/forgelm/cli/_config_load.py index 3d89034b..17bf71bb 100644 --- a/forgelm/cli/_config_load.py +++ b/forgelm/cli/_config_load.py @@ -4,13 +4,13 @@ import os import sys -from typing import Optional +from typing import Optional, Tuple import yaml from pydantic import ConfigDict, TypeAdapter, ValidationError from .._strict_json import dumps_strict -from ..config import ConfigError, RetentionConfig, load_config +from ..config import ConfigError, ForgeConfig, RetentionConfig, load_config from ._exit_codes import EXIT_CONFIG_ERROR from ._logging import logger @@ -74,17 +74,19 @@ def _load_config_or_exit(config_path: str, json_output: bool): sys.exit(EXIT_CONFIG_ERROR) -def _load_retention_policy(config_path: Optional[str]) -> Optional[RetentionConfig]: - """Return the ``retention:`` block that ``forgelm purge --check-policy`` reports against. +def _load_retention_and_config(config_path: Optional[str]) -> Tuple[Optional[RetentionConfig], Optional[ForgeConfig]]: + """Return what ``forgelm purge --check-policy`` needs from ``--config``: the ``retention:`` block and the full config. The file is either a full ForgeLM config or one whose only top-level key is ``retention``: a scheduled retention report needs no training config (#1054). Any other top-level key makes it a full config, validated - as one, so a misspelt ``retension:`` fails instead of being ignored. ``retention: null`` returns ``None``, as - it does in a full config. Raises what ``load_config`` raises (``FileNotFoundError``, ``OSError``, + as one, so a misspelt ``retension:`` fails instead of being ignored. ``retention: null`` yields ``None`` for + the block, as it does in a full config. The second item is the validated config, or ``None`` for a + retention-only file, which names no ``training.final_model_dir`` and so leaves the report on the default + model directory name. Raises what ``load_config`` raises (``FileNotFoundError``, ``OSError``, ``ConfigError``), and like ``ForgeConfig`` leaves raw input values out of validation messages. """ if not config_path: - return None + return None, None if not os.path.exists(config_path): raise FileNotFoundError(f"Configuration file not found: {config_path}") with open(config_path, encoding="utf-8") as fh: @@ -93,10 +95,11 @@ def _load_retention_policy(config_path: Optional[str]) -> Optional[RetentionConf except yaml.YAMLError as exc: raise ConfigError(f"Invalid YAML syntax in {config_path}: {exc}") from exc if not (isinstance(data, dict) and set(data) == {"retention"}): - return load_config(config_path).retention + config = load_config(config_path) + return config.retention, config adapter = TypeAdapter(Optional[RetentionConfig], config=ConfigDict(hide_input_in_errors=True, title="retention")) try: - return adapter.validate_python(data["retention"]) + return adapter.validate_python(data["retention"]), None except ValidationError as exc: raise ConfigError(f"Configuration validation failed:\n{exc}") from exc diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index a8682139..d4aa3d49 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1150,7 +1150,7 @@ def _add_purge_subcommand(subparsers) -> None: type=str, default=argparse.SUPPRESS, metavar="PATH", - help="YAML config. Row mode: detects the webhook block for the external-copies warning. --check-policy: supplies the `retention:` horizons (without it nothing is checked and the report says so). Run mode does not read it.", + help="YAML config. Row mode: names the training directories (`training.output_dir`, `training.final_model_dir`, pipeline stages, `merge.output_dir`) scanned for the memorisation warning, and its webhook block feeds the external-copies warning. Run mode: `training.final_model_dir` names the staging directory to erase. --check-policy: supplies the `retention:` horizons (without it nothing is checked and the report says so); a full config also names the staging directory to report.", ) p.add_argument( "--justification", diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index a283249d..7806f715 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -28,9 +28,11 @@ the chain shows ``request → fail`` instead of ``request → complete``, so an operator scanning the log can spot the unfinished erasure. - ``data.erasure_warning_memorisation`` — corpus row erasure ran - while a ``final_model/`` directory exists directly under the output - dir (which runs consumed the corpus is not determined). The row is - gone from disk but may still be memorised in the trained weights. + while a trained model, a staging directory or a ``checkpoint-*`` + directory exists: under the output dir, and under every training + layout a loaded ``--config`` names (which runs consumed the corpus is + not determined). The row is gone from disk but may still be memorised + in the trained weights. - ``data.erasure_warning_synthetic_data_present`` — corpus row erasure ran with synthetic-data snapshots present; the row may have produced derivative snippets the row→snippet mapping no longer connects. @@ -52,14 +54,13 @@ import hashlib import json import os -import re import secrets import shutil import sys import tempfile import time from pathlib import Path -from typing import Any, Dict, List, NoReturn, Optional, Tuple +from typing import Any, Dict, List, NoReturn, Optional, Sequence, Tuple from ..._strict_json import dumps_strict from .._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS, EXIT_TRAINING_ERROR @@ -93,6 +94,10 @@ # operator handles the residual disposal manually). _VALID_RUN_KINDS: Tuple[str, ...] = ("staging", "artefacts") +# ``training.final_model_dir`` default. A run writes its model to ``/`` and, while +# ``require_human_approval`` holds it back, to the sibling ``.staging.``. +_DEFAULT_FINAL_MODEL_DIR = "final_model" + # Cap for the ``error_message`` field persisted into the append-only audit # chain. ``reverse-pii`` imports :func:`_sanitise_audit_error_message` # below rather than duplicating the mask/bound logic, so both GDPR @@ -386,9 +391,9 @@ def _detect_warning_conditions( Per design §5.1: - - ``data.erasure_warning_memorisation`` when a ``final_model/`` - directory exists directly under the operator's output dir (a - trained model may have memorised the row). + - ``data.erasure_warning_memorisation`` when :func:`_scan_memorisation_artefacts` + finds a trained model, a staging directory or a checkpoint (the + row may be memorised in weights trained on it). - ``data.erasure_warning_synthetic_data_present`` when any ``synthetic_data*.jsonl`` exists (the row may have produced derivative snippets). @@ -399,10 +404,13 @@ def _detect_warning_conditions( warnings: List[str] = [] extras: Dict[str, Dict[str, Any]] = {} - final_model_dir = os.path.join(output_dir, "final_model") - if os.path.isdir(final_model_dir): + memorisation_paths = _scan_memorisation_artefacts(output_dir, config_loaded) + if memorisation_paths: warnings.append(_EVT_WARN_MEMORISATION) - extras[_EVT_WARN_MEMORISATION] = {"affected_run_ids": _scan_run_ids_with_final_model(output_dir)} + extras[_EVT_WARN_MEMORISATION] = { + "affected_run_ids": _affected_run_ids(output_dir, config_loaded, memorisation_paths), + "memorisation_artefact_paths": memorisation_paths, + } synthetic_files = sorted(str(p) for p in Path(output_dir).glob("synthetic_data*.jsonl") if p.is_file()) if synthetic_files: @@ -417,41 +425,113 @@ def _detect_warning_conditions( return warnings, extras -def _scan_run_ids_with_final_model(output_dir: str) -> List[str]: - """Best-effort: enumerate run ids whose ``final_model/`` is present. +def _training_trees(config_loaded: Optional[Any]) -> List[Tuple[str, str]]: + """``(output_dir, final_model_dir)`` of every training run the loaded config describes.""" + stages = getattr(getattr(config_loaded, "pipeline", None), "stages", None) or [] + blocks = [getattr(config_loaded, "training", None), *(getattr(stage, "training", None) for stage in stages)] + return [(block.output_dir, block.final_model_dir) for block in blocks if block is not None] + + +def _final_model_names(config_loaded: Optional[Any]) -> List[str]: + """Model directory names a run may have left a staging sibling for: the config's, then the default.""" + names = [model_dir for _output_dir, model_dir in _training_trees(config_loaded)] + return list(dict.fromkeys([*names, _DEFAULT_FINAL_MODEL_DIR])) - Looks for ``final_model.staging./`` directories left as - forensic artefacts after promotion. - Wave 2b Round-4 review F-W2B-06 fix: once an operator runs - ``forgelm approve ``, the staging directory is renamed to - ``final_model/`` — leaving no staging-suffix directory to match. - Falls back to the audit log's ``human_approval.granted`` events so - the warning event still names a concrete run id. Empty list with - a sentinel placeholder when nothing matches and the audit log is - silent — the warning still fires (the ``final_model/`` is present, - that's what triggered it), but the operator gets a clear hint. +def _memorisation_trees(output_dir: str, config_loaded: Optional[Any]) -> List[Tuple[str, str]]: + """``--output-dir`` under every model directory name in play, plus every layout the loaded config names. + + The operator's ``--output-dir`` need not be the config's ``training.output_dir`` (the config may be + resolved from another working directory), so it is paired with the config's ``final_model_dir`` too. """ - pattern = re.compile(r"final_model\.staging\.([A-Za-z0-9._-]+)$") - run_ids: List[str] = [] + return [*((output_dir, name) for name in _final_model_names(config_loaded)), *_training_trees(config_loaded)] + + +def _scan_memorisation_artefacts(output_dir: str, config_loaded: Optional[Any]) -> List[str]: + """Return sorted paths of memorisation-risk artefacts, or ``[]``. + + Weights trained on an erased row survive in three places, all reported: + + - the model directory (````) and its staging siblings + (``.staging[.]``), which hold the adapters of a + run that is awaiting or was refused human approval; + - ``checkpoint-*`` directories, which every completed run leaves behind + and which survive an auto-revert; + - ``merge.output_dir`` when the loaded config has a ``merge`` block. + + ``output_dir`` is scanned for the default ``final_model`` layout without a + config; a loaded ``--config`` adds its ``training`` block and every + ``pipeline`` stage that carries its own. Symlinked entries are reported where + they are named: the scan stats and lists paths and never reads or deletes them. + """ + found: Dict[str, str] = {} + for root, model_dir in _memorisation_trees(output_dir, config_loaded): + for path in _scan_training_tree(root, model_dir): + found.setdefault(os.path.realpath(path), path) + merge = getattr(config_loaded, "merge", None) + if merge is not None and os.path.isdir(merge.output_dir): + found.setdefault(os.path.realpath(merge.output_dir), os.path.abspath(merge.output_dir)) + return sorted(found.values()) + + +def _is_artefact_name(name: str, model_name: str) -> bool: + """Any ``checkpoint-*`` (a prefix ``final_model_dir`` may not use), or ``.staging[.]``.""" + staging = f"{model_name}.staging" + return name.startswith("checkpoint-") or name == staging or name.startswith(f"{staging}.") + + +def _is_directory(entry: os.DirEntry) -> bool: + """``entry.is_dir()``, where an entry that cannot be examined (a symlink loop) is logged and skipped, not fatal.""" try: - for entry in os.listdir(output_dir): - match = pattern.match(entry) - if match: - run_ids.append(match.group(1)) - except OSError: - return [] - if run_ids: - return sorted(set(run_ids)) - # Staging dir is gone (run was promoted). Walk the audit log for - # granted approvals; their run_id is the one that consumed this - # corpus. - audit_run_ids = _scan_run_ids_from_granted_events(os.path.join(output_dir, "audit_log.jsonl")) - if audit_run_ids: - return audit_run_ids - # Nothing on disk, nothing in audit chain — surface a clear hint - # so the operator knows where to look next. - return ["unknown — no matching staging directory or human_approval.granted event"] + return entry.is_dir() + except OSError as exc: + logger.warning("purge: could not examine %s for the memorisation check (%s); skipped.", entry.path, exc) + return False + + +def _scan_training_tree(output_dir: str, final_model_dir: str) -> List[str]: + """Model, staging and checkpoint directories one training layout holds, as absolute paths. + + Checkpoints sit directly under *output_dir*; staging directories are siblings of the + model directory, which is nested under *output_dir* when ``final_model_dir`` has a ``/``. + """ + root = os.path.abspath(output_dir) + model_path = os.path.normpath(os.path.join(root, final_model_dir)) + model_parent, model_name = os.path.split(model_path) + found = [model_path] if os.path.isdir(model_path) else [] + for parent in dict.fromkeys((model_parent, root)): + try: + with os.scandir(parent) as entries: + found.extend(e.path for e in entries if _is_artefact_name(e.name, model_name) and _is_directory(e)) + except FileNotFoundError: + continue # nothing was trained into this directory + except OSError as exc: + logger.warning( + "purge: could not scan %s for memorisation-risk artefacts (%s); the memorisation check is " + "incomplete for this directory.", + parent, + exc, + ) + return found + + +def _affected_run_ids(output_dir: str, config_loaded: Optional[Any], artefact_paths: List[str]) -> List[str]: + """Run ids behind the artefacts: from staging names, else from granted approvals, else a hint. + + Once ``forgelm approve `` renames a staging directory to the model directory + no suffix names the run, so the audit logs of the scanned layouts are read for + ``human_approval.granted`` events. When neither names a run the warning still + fires, because artefacts exist, and carries a hint instead of an empty list. + """ + marker = ".staging." + from_names = {os.path.basename(p).partition(marker)[2] for p in artefact_paths if marker in os.path.basename(p)} + if from_names - {""}: + return sorted(from_names - {""}) + roots = dict.fromkeys(os.path.abspath(root) for root, _model_dir in _memorisation_trees(output_dir, config_loaded)) + granted = { + rid for root in roots for rid in _scan_run_ids_from_granted_events(os.path.join(root, "audit_log.jsonl")) + } + return sorted(granted) or ["unknown — no matching staging directory or human_approval.granted event"] def _scan_run_ids_from_granted_events(audit_log_path: str) -> List[str]: @@ -478,7 +558,9 @@ def _scan_run_ids_from_granted_events(audit_log_path: str) -> List[str]: run_id = event.get("run_id") if isinstance(run_id, str) and run_id: run_ids.append(run_id) - except OSError: + except (OSError, UnicodeDecodeError): + # The reader runs after the row is gone and on logs outside --output-dir: an unreadable one must not + # abort the erasure chain, only leave the run id to the next source. return [] return sorted(set(run_ids)) @@ -720,10 +802,11 @@ def _run_purge_row_id(args, output_format: str) -> None: # so say what was resolved instead of letting the check pass quietly over the wrong directory. logger.warning( "--output-dir not given: using the corpus directory %s for the audit log, the salt and the memorisation " - "check (which looks for %s). Pass --output-dir to check your trained model and to " + "check (which looks there for %s, staging and checkpoint-* directories, and in the training directories " + "a --config names). Pass --output-dir to check your trained model and to " "correlate with `forgelm reverse-pii`.", output_dir, - os.path.join(output_dir, "final_model"), + os.path.join(output_dir, _DEFAULT_FINAL_MODEL_DIR), ) os.makedirs(output_dir, exist_ok=True) @@ -854,6 +937,9 @@ def _run_purge_run_id(args, output_format: str) -> None: except ConfigError as exc: _output_error_and_exit(output_format, str(exc), EXIT_CONFIG_ERROR) + # Loaded before ``data.erasure_requested`` so a config that cannot be read never strands a request in the chain. + # Only ``--kind staging`` uses it (``final_model_dir`` names the staging directory). + config_loaded = _maybe_load_config(getattr(args, "config", None)) if args.kind == "staging" else None request_fields: Dict[str, Any] = { "target_kind": args.kind, "target_id": args.run_id, @@ -863,7 +949,7 @@ def _run_purge_run_id(args, output_format: str) -> None: } audit.log_event(_EVT_ERASURE_REQUESTED, **request_fields) - target_paths = _resolve_run_kind_targets(output_dir, args.run_id, args.kind) + target_paths = _resolve_run_kind_targets(output_dir, args.run_id, args.kind, _final_model_names(config_loaded)) if not target_paths: # The standard compliance bundle has fixed filenames with no run id, so a real run id never matches it. hint = " The standard compliance bundle has fixed filenames without a run id; remove it through your storage layer." @@ -957,11 +1043,17 @@ def _run_purge_run_id(args, output_format: str) -> None: ) -def _staging_targets_for_run(base: Path, run_id: str) -> List[Path]: - """Return existing ``final_model.staging`` paths for ``run_id``. +def _staging_targets_for_run( + base: Path, run_id: str, model_names: Sequence[str] = (_DEFAULT_FINAL_MODEL_DIR,) +) -> List[Path]: + """Return existing ``.staging`` paths for ``run_id``, one candidate per *model_names* entry. + + *model_names* are the ``training.final_model_dir`` values the loaded config names (see + :func:`_final_model_names`); the trainer stages a run next to its model directory, so a + non-default ``final_model_dir`` has a non-default staging name. Wave 2b final-review F-21-STAGING: only the Phase 9 v2 explicit form - ``final_model.staging./`` is considered. Earlier revisions + ``.staging./`` is considered. Earlier revisions also unconditionally included the legacy unscoped ``final_model.staging/`` candidate as a "pre-Phase-9-v2 backward compatibility" path, but that path could belong to any pre-v2 run @@ -975,12 +1067,13 @@ def _staging_targets_for_run(base: Path, run_id: str) -> List[Path]: via the Phase 9 v2 path or remove the directory manually. """ targets: List[Path] = [] - scoped = base / f"final_model.staging.{run_id}" - if scoped.exists(): - targets.append(scoped) - legacy = base / "final_model.staging" - if legacy.exists() and _legacy_staging_owned_by(legacy, run_id): - targets.append(legacy) + for name in model_names: + scoped = base / f"{name}.staging.{run_id}" + if scoped.exists(): + targets.append(scoped) + legacy = base / f"{name}.staging" + if legacy.exists() and _legacy_staging_owned_by(legacy, run_id): + targets.append(legacy) return targets @@ -1003,9 +1096,14 @@ def _legacy_staging_owned_by(legacy_dir: Path, run_id: str) -> bool: return recorded == run_id -def _artefact_targets_for_run(base: Path, run_id: str) -> List[Path]: +def _artefact_targets_for_run( + base: Path, run_id: str, model_names: Sequence[str] = (_DEFAULT_FINAL_MODEL_DIR,) +) -> List[Path]: """Return ``compliance/`` files whose name embeds ``run_id`` as a token. + *model_names* is unused: every resolver in :data:`_RUN_KIND_RESOLVERS` takes the same arguments, and + compliance files are named by run id alone. + Any regular file directly under ``compliance/`` whose name carries the run-id as a token matches, whatever its extension. The standard bundle (fixed basenames such as ``compliance_report.json``) has no run-id, so a real @@ -1030,7 +1128,9 @@ def _artefact_targets_for_run(base: Path, run_id: str) -> List[Path]: } -def _resolve_run_kind_targets(output_dir: str, run_id: str, kind: str) -> List[Path]: +def _resolve_run_kind_targets( + output_dir: str, run_id: str, kind: str, model_names: Sequence[str] = (_DEFAULT_FINAL_MODEL_DIR,) +) -> List[Path]: """Return the on-disk paths the ``--kind`` flag refers to for ``run_id``. Dispatches to a per-kind resolver in :data:`_RUN_KIND_RESOLVERS`. @@ -1043,7 +1143,7 @@ def _resolve_run_kind_targets(output_dir: str, run_id: str, kind: str) -> List[P resolver = _RUN_KIND_RESOLVERS.get(kind) if resolver is None: return [] - return resolver(Path(output_dir), run_id) + return resolver(Path(output_dir), run_id, model_names) _RUN_ID_BOUNDARIES = ("_", "-", ".") @@ -1126,16 +1226,16 @@ def _run_purge_check_policy(args, output_format: str) -> None: ``EXIT_CONFIG_ERROR`` rather than silently degrading to a "no retention block" notice (which the operator would mistake for "no violations"). The file may hold only ``retention:`` - (#1054); see ``_load_retention_policy``. + (#1054); see ``_load_retention_and_config``. """ import yaml as _yaml from forgelm.config import ConfigError - from .._config_load import _load_retention_policy + from .._config_load import _load_retention_and_config try: - retention = _load_retention_policy(getattr(args, "config", None)) + retention, config_loaded = _load_retention_and_config(getattr(args, "config", None)) except (ConfigError, OSError, _yaml.YAMLError) as exc: # OSError covers FileNotFoundError + permission denied; ConfigError # wraps Pydantic ValidationError and YAML parse errors; @@ -1161,7 +1261,7 @@ def _run_purge_check_policy(args, output_format: str) -> None: sys.exit(EXIT_SUCCESS) output_dir = args.output_dir or "." - violations = _scan_retention_violations(retention, output_dir) + violations = _scan_retention_violations(retention, output_dir, _final_model_names(config_loaded)) if output_format == "json": print(dumps_strict({"success": True, "violations": violations, "count": len(violations)}, indent=2)) else: @@ -1179,9 +1279,15 @@ def _run_purge_check_policy(args, output_format: str) -> None: sys.exit(EXIT_SUCCESS) -def _scan_retention_violations(retention, output_dir: str) -> List[Dict[str, Any]]: +def _scan_retention_violations( + retention, output_dir: str, model_names: Sequence[str] = (_DEFAULT_FINAL_MODEL_DIR,) +) -> List[Dict[str, Any]]: """Walk ``output_dir`` and report artefacts past their retention horizon. + *model_names* are the ``training.final_model_dir`` values whose staging siblings + are reported (see :func:`_final_model_names`). ``checkpoint-*`` directories are + deliberately not walked: no ``retention:`` horizon covers them. + Belt-and-suspenders age resolution (design §3.3): 1. **Belt** (preferred): for run-scoped artefacts, the canonical @@ -1209,7 +1315,10 @@ def _scan_retention_violations(retention, output_dir: str) -> List[Dict[str, Any # per-artefact age lookup hits the right audit timestamp. horizons: List[Tuple[str, str, int, Optional[str]]] = [ ("audit_log", audit_log_path, retention.audit_log_retention_days, None), - ("staging_dir", os.path.join(output_dir, "final_model.staging"), retention.staging_ttl_days, None), + *( + ("staging_dir", os.path.join(output_dir, f"{name}.staging"), retention.staging_ttl_days, None) + for name in model_names + ), ( "compliance_bundle", os.path.join(output_dir, "compliance"), @@ -1228,7 +1337,7 @@ def _scan_retention_violations(retention, output_dir: str) -> List[Dict[str, Any # the trainer actually creates since Phase 9 v2) and the raw # documents horizon. Discover both at scan time so the report # reflects every artefact kind the retention block covers. - horizons.extend(_discover_per_run_staging_horizons(output_dir, retention.staging_ttl_days)) + horizons.extend(_discover_per_run_staging_horizons(output_dir, retention.staging_ttl_days, model_names)) horizons.extend(_discover_raw_documents_horizons(output_dir, retention.raw_documents_retention_days)) for kind, path, horizon_days, run_id in horizons: if horizon_days == 0 or not os.path.exists(path): @@ -1269,9 +1378,11 @@ def _parse_iso_timestamp_to_posix(ts: str) -> Optional[float]: return when.timestamp() -def _discover_per_run_staging_horizons(output_dir: str, horizon_days: int) -> List[Tuple[str, str, int, Optional[str]]]: - """Enumerate ``final_model.staging./`` directories under - ``output_dir`` so the retention scan reports each one separately. +def _discover_per_run_staging_horizons( + output_dir: str, horizon_days: int, model_names: Sequence[str] = (_DEFAULT_FINAL_MODEL_DIR,) +) -> List[Tuple[str, str, int, Optional[str]]]: + """Enumerate ``.staging./`` directories next to each model + directory so the retention scan reports each one separately. Returns ``[]`` when the horizon is disabled (``horizon_days == 0``) or the output dir cannot be listed (callers handle the empty @@ -1285,18 +1396,18 @@ def _discover_per_run_staging_horizons(output_dir: str, horizon_days: int) -> Li if horizon_days == 0: return [] discovered: List[Tuple[str, str, int, Optional[str]]] = [] - try: - entries = os.listdir(output_dir) - except OSError: - return [] - for entry in entries: - if not entry.startswith("final_model.staging."): - continue - full = os.path.join(output_dir, entry) - if not os.path.isdir(full): + for name in model_names: + parent, base_name = os.path.split(os.path.join(output_dir, name)) + prefix = f"{base_name}.staging." + try: + entries = os.listdir(parent) + except OSError: continue - run_id = entry[len("final_model.staging.") :] - discovered.append((f"staging_dir[{run_id}]", full, horizon_days, run_id)) + for entry in entries: + full = os.path.join(parent, entry) + if entry.startswith(prefix) and os.path.isdir(full): + run_id = entry[len(prefix) :] + discovered.append((f"staging_dir[{run_id}]", full, horizon_days, run_id)) return discovered @@ -1419,13 +1530,13 @@ def _resolve_artefact_age( def _maybe_load_config(config_path: Optional[str]): """Load the ``ForgeConfig`` from YAML, best effort. - Used by the row-id erasure path, which works without a config (the - corpus-row erasure is intentionally config-agnostic), so a missing / - unreadable / invalid YAML falls back to ``None``. When *config_path* - was explicitly supplied the fallback is logged at ``WARNING`` (not - silent) so an operator relying on it for external-copies detection - sees that the load failed. ``--check-policy`` loads its config - strictly, through ``_load_retention_policy``. + Used by the row-id and run-id erasure paths, which work without a config + (erasure is intentionally config-agnostic), so a missing / unreadable / + invalid YAML falls back to ``None``. When *config_path* was explicitly + supplied the fallback is logged at ``WARNING`` (not silent) so an operator + relying on it for the memorisation check, external-copies detection or a + non-default ``training.final_model_dir`` sees that the load failed. ``--check-policy`` loads its config + strictly, through ``_load_retention_and_config``. """ if not config_path: return None @@ -1441,7 +1552,7 @@ def _maybe_load_config(config_path: Optional[str]): try: return load_config(config_path) - except (ConfigError, OSError, _yaml.YAMLError) as exc: + except (ConfigError, OSError, UnicodeDecodeError, _yaml.YAMLError) as exc: # Best-effort fallback: row-id / run-id paths run config-free. # WARNING, not DEBUG: an operator who explicitly passed --config # (e.g. to surface the external-copies webhook warning) must see @@ -1450,7 +1561,9 @@ def _maybe_load_config(config_path: Optional[str]): # load indistinguishable from "no webhook configured" (a false # compliance signal). logger.warning( - "purge: --config %r could not be loaded (%s); external-copies warning detection is disabled for this run.", + "purge: --config %r could not be loaded (%s); what it supplies is disabled for this run (row mode: " + "the memorisation scan of its training directories and external-copies detection; run mode: " + "its final_model_dir for the staging directory name).", config_path, exc, ) @@ -1558,5 +1671,6 @@ def _run_purge_cmd(args, output_format: str) -> None: "_find_matching_rows", "_atomic_rewrite_dropping_lines", "_detect_warning_conditions", + "_scan_memorisation_artefacts", "_scan_retention_violations", ] diff --git a/forgelm/config.py b/forgelm/config.py index 970656c7..41e211ec 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1620,7 +1620,7 @@ class RetentionConfig(BaseModel): default=7, ge=0, description=( - "Days after which `forgelm purge --check-policy` reports a `final_model.staging./` (or legacy `final_model.staging/`) directory as overdue (aged from the run's first audit event, else the log's first event, else mtime); nothing is deleted. " + "Days after which `forgelm purge --check-policy` reports a `.staging./` (or legacy `.staging/`) directory as overdue (aged from the run's first audit event, else the log's first event, else mtime); nothing is deleted. " "Set to 0 to stop reporting it. This is the canonical staging-retention field; " "the legacy `evaluation.staging_ttl_days` alias was removed in v0.8.0." ), diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index e29bac07..7993bee7 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -1633,6 +1633,581 @@ def test_warning_event_carries_completed_field_set(self, tmp_path: Path) -> None assert "affected_run_ids" in warn_evt +# --------------------------------------------------------------------------- +# Memorisation artefacts (CR-A10-022, CR-A10-042). Weights trained on an erased +# row live in the model directory, in the ``.staging.`` +# sibling a run awaiting human approval leaves, and in ``checkpoint-*`` +# directories; the warning must find them under the layout the trainer really +# writes, with or without ``--config``, and for a non-default ``final_model_dir``. +# --------------------------------------------------------------------------- + +_MISSING_RUN_ID_HINT = "unknown" +_TRAINING_CONFIG = """ +model: + name_or_path: gpt2 + backend: transformers +lora: + r: 8 +training: + trainer_type: sft + output_dir: '{output_dir}' + final_model_dir: {final_model_dir} + num_train_epochs: 1 +data: + dataset_name_or_path: train.jsonl +{extra}""" + + +def _write_training_config( + tmp_path: Path, output_dir: Path, final_model_dir: str = "final_model", extra: str = "" +) -> Path: + config_path = tmp_path / "config.yaml" + config_path.write_text( + _TRAINING_CONFIG.format(output_dir=output_dir, final_model_dir=final_model_dir, extra=extra), encoding="utf-8" + ) + return config_path + + +def _purge_one_row(tmp_path: Path, *, config: Path | None = None, output_dir: Path | None = None) -> list[dict]: + """Erase ``row-1`` from a one-row corpus and return the audit events the erasure wrote. + + Without *output_dir* the corpus directory is used, as it is when the operator omits ``--output-dir``. + """ + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "data" / "train.jsonl" + _seed_corpus(corpus, [{"id": "row-1", "text": "x"}]) + args = _build_args( + row_id="row-1", + corpus=str(corpus), + output_dir=str(output_dir) if output_dir else None, + config=str(config) if config else None, + ) + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + assert ei.value.code == 0 + return _read_audit_events((output_dir or corpus.parent) / "audit_log.jsonl") + + +def _memorisation_event(events: list[dict]) -> dict | None: + return next((e for e in events if e["event"] == "data.erasure_warning_memorisation"), None) + + +class TestMemorisationArtefactDetection: + def test_checkpoints_under_the_configured_output_dir_fire_the_warning(self, tmp_path: Path) -> None: + training_out = tmp_path / "training_out" + (training_out / "checkpoint-100").mkdir(parents=True) + (training_out / "checkpoint-200").mkdir() + config = _write_training_config(tmp_path, training_out) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [ + str(training_out / "checkpoint-100"), + str(training_out / "checkpoint-200"), + ] + assert warning["affected_run_ids"][0].startswith(_MISSING_RUN_ID_HINT) + + def test_default_invocation_finds_the_model_through_the_config(self, tmp_path: Path) -> None: + """No ``--output-dir`` (so the corpus directory is used) and no checkpoints left: the model under the + configured ``training.output_dir`` must still be found.""" + training_out = tmp_path / "checkpoints" + (training_out / "final_model").mkdir(parents=True) + config = _write_training_config(tmp_path, training_out) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(training_out / "final_model")] + + def test_custom_final_model_dir_is_found(self, tmp_path: Path) -> None: + training_out = tmp_path / "ck" + (training_out / "my_model").mkdir(parents=True) + config = _write_training_config(tmp_path, training_out, final_model_dir="my_model") + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=training_out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(training_out / "my_model")] + + def test_nested_final_model_dir_and_its_staging_sibling_are_found(self, tmp_path: Path) -> None: + training_out = tmp_path / "ck" + (training_out / "models" / "final").mkdir(parents=True) + (training_out / "models" / "final.staging.fg-run1").mkdir() + config = _write_training_config(tmp_path, training_out, final_model_dir="models/final") + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [ + str(training_out / "models" / "final"), + str(training_out / "models" / "final.staging.fg-run1"), + ] + assert warning["affected_run_ids"] == ["fg-run1"] + + @pytest.mark.parametrize("model_dir", ["final_model", "my_model"]) + def test_staging_directory_of_a_run_awaiting_approval_is_found(self, tmp_path: Path, model_dir: str) -> None: + """The run exits 4 holding only ``.staging.`` — no model directory, no checkpoint.""" + training_out = tmp_path / "ck" + (training_out / f"{model_dir}.staging.fg-run1").mkdir(parents=True) + config = _write_training_config(tmp_path, training_out, final_model_dir=model_dir) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(training_out / f"{model_dir}.staging.fg-run1")] + assert warning["affected_run_ids"] == ["fg-run1"] + + def test_unscoped_legacy_staging_directory_is_found(self, tmp_path: Path) -> None: + (tmp_path / "ck" / "final_model.staging").mkdir(parents=True) + + warning = _memorisation_event(_purge_one_row(tmp_path, output_dir=tmp_path / "ck")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(tmp_path / "ck" / "final_model.staging")] + + def test_lookalike_names_are_not_reported(self, tmp_path: Path) -> None: + training_out = tmp_path / "ck" + for name in ("final_model.stagingX", "final_model_old", "checkpoint", "my-checkpoint-1", "x.staging.fg-run1"): + (training_out / name).mkdir(parents=True, exist_ok=True) + (training_out / "checkpoint-9").write_text("a file, not a checkpoint directory") + + assert _memorisation_event(_purge_one_row(tmp_path, output_dir=training_out)) is None + + def test_without_config_the_output_dir_is_scanned_for_checkpoints_and_staging(self, tmp_path: Path) -> None: + """``forgelm purge --row-id alice --corpus data/train.jsonl --output-dir out`` with no ``--config``.""" + out = tmp_path / "out" + (out / "checkpoint-1").mkdir(parents=True) + (out / "checkpoint-2").mkdir() + (out / "final_model.staging.fg-run1").mkdir() + + warning = _memorisation_event(_purge_one_row(tmp_path, output_dir=out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [ + str(out / "checkpoint-1"), + str(out / "checkpoint-2"), + str(out / "final_model.staging.fg-run1"), + ] + assert warning["affected_run_ids"] == ["fg-run1"] + + def test_without_config_artefacts_outside_the_output_dir_stay_invisible(self, tmp_path: Path) -> None: + (tmp_path / "training_out" / "checkpoint-100").mkdir(parents=True) + + assert _memorisation_event(_purge_one_row(tmp_path, output_dir=tmp_path / "audit")) is None + + def test_no_artefacts_no_warning_with_config(self, tmp_path: Path, monkeypatch) -> None: + monkeypatch.chdir(tmp_path) # the merge block's default ``./merged_model`` is relative to the cwd + training_out = tmp_path / "training_out" + training_out.mkdir() + config = _write_training_config(tmp_path, training_out, extra="merge:\n enabled: false\n") + + assert _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) is None + + def test_missing_training_directory_is_silent(self, tmp_path: Path, caplog) -> None: + """A config whose run never started has no directory to scan: no warning and no log noise.""" + config = _write_training_config(tmp_path, tmp_path / "never_trained") + + with caplog.at_level("WARNING"): + events = _purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit") + + assert _memorisation_event(events) is None + assert not [r for r in caplog.records if "could not scan" in r.getMessage()] + + def test_pipeline_stage_layouts_are_scanned(self, tmp_path: Path) -> None: + stage_one, stage_two = tmp_path / "stage_one", tmp_path / "stage_two" + (stage_one / "stage_one_model").mkdir(parents=True) + (stage_two / "checkpoint-5").mkdir(parents=True) + pipeline = ( + "pipeline:\n" + f" output_dir: '{tmp_path / 'pipeline_run'}'\n" + " stages:\n" + " - name: sft\n" + " training:\n" + " trainer_type: sft\n" + f" output_dir: '{stage_one}'\n" + " final_model_dir: stage_one_model\n" + " - name: dpo\n" + " training:\n" + " trainer_type: dpo\n" + f" output_dir: '{stage_two}'\n" + ) + config = _write_training_config(tmp_path, tmp_path / "root_unused", extra=pipeline) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [ + str(stage_one / "stage_one_model"), + str(stage_two / "checkpoint-5"), + ] + + def test_merge_output_dir_is_reported_when_the_merge_ran(self, tmp_path: Path) -> None: + merged = tmp_path / "merged_model" + merged.mkdir() + merge = ( + "merge:\n enabled: true\n method: linear\n models:\n - path: ./a\n weight: 0.5\n" + f" - path: ./b\n weight: 0.5\n output_dir: '{merged}'\n" + ) + config = _write_training_config(tmp_path, tmp_path / "never_trained", extra=merge) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(merged)] + + def test_the_same_directory_reached_two_ways_is_listed_once(self, tmp_path: Path) -> None: + """``--output-dir`` is the training directory the config also names.""" + training_out = tmp_path / "ck" + (training_out / "final_model").mkdir(parents=True) + (training_out / "checkpoint-1").mkdir() + config = _write_training_config(tmp_path, training_out) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=training_out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [ + str(training_out / "checkpoint-1"), + str(training_out / "final_model"), + ] + + def test_symlinked_checkpoint_is_reported_where_it_is_named(self, tmp_path: Path) -> None: + """Checkpoints are often symlinked to a scratch volume; the weights are still there, so the warning + lists the link instead of dropping it.""" + scratch = tmp_path / "scratch_volume" / "checkpoint-100" + scratch.mkdir(parents=True) + training_out = tmp_path / "ck" + training_out.mkdir() + try: + (training_out / "checkpoint-100").symlink_to(scratch, target_is_directory=True) + except (OSError, NotImplementedError): + pytest.skip("symlink creation not permitted on this platform") + + warning = _memorisation_event(_purge_one_row(tmp_path, output_dir=training_out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(training_out / "checkpoint-100")] + + def test_unreadable_directory_is_logged_and_the_erasure_still_completes( + self, tmp_path: Path, monkeypatch, caplog + ) -> None: + """A directory that cannot be listed must not abort a purge whose row is already gone, and must not read + as 'nothing found': the operator is told the check is incomplete.""" + training_out = tmp_path / "ck" + (training_out / "checkpoint-1").mkdir(parents=True) + literal = tmp_path / "out" / "final_model" + literal.mkdir(parents=True) + config = _write_training_config(tmp_path, training_out) + real_scandir = os.scandir + + def flaky_scandir(path="."): + if os.fspath(path) == str(training_out): + raise PermissionError(errno.EACCES, "Permission denied", os.fspath(path)) + return real_scandir(path) + + monkeypatch.setattr("forgelm.cli.subcommands._purge.os.scandir", flaky_scandir) + + with caplog.at_level("WARNING"): + events = _purge_one_row(tmp_path, config=config, output_dir=tmp_path / "out") + + assert "data.erasure_completed" in [e["event"] for e in events] + assert [r for r in caplog.records if "could not scan" in r.getMessage() and str(training_out) in r.getMessage()] + warning = _memorisation_event(events) + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(literal)] + + def test_unloadable_config_falls_back_to_the_output_dir_scan(self, tmp_path: Path, caplog) -> None: + bad_config = tmp_path / "broken.yaml" + bad_config.write_text("training: [unterminated", encoding="utf-8") + (tmp_path / "out" / "final_model").mkdir(parents=True) + + with caplog.at_level("WARNING"): + warning = _memorisation_event(_purge_one_row(tmp_path, config=bad_config, output_dir=tmp_path / "out")) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(tmp_path / "out" / "final_model")] + assert [r for r in caplog.records if "could not be loaded" in r.getMessage()] + + def test_output_dir_is_paired_with_the_configs_custom_model_dir(self, tmp_path: Path, monkeypatch) -> None: + """The config names ``final_model_dir: my_model`` but a ``training.output_dir`` that is not reachable from + here (a relative path, or a config copied between hosts); the operator points ``--output-dir`` at the + directory that really holds ``my_model/``.""" + monkeypatch.chdir(tmp_path) + real_out = tmp_path / "ck2" + (real_out / "my_model").mkdir(parents=True) + config = _write_training_config(tmp_path, tmp_path / "not_here", final_model_dir="my_model") + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=real_out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(real_out / "my_model")] + + def test_non_utf8_audit_log_in_a_training_directory_does_not_abort_the_erasure(self, tmp_path: Path) -> None: + """The run-id lookup reads audit logs outside ``--output-dir`` after the row is already gone: a log that + cannot be decoded must leave the chain closed, not stranded at ``data.erasure_requested``.""" + training_out = tmp_path / "ck" + (training_out / "final_model").mkdir(parents=True) + (training_out / "audit_log.jsonl").write_bytes(b'{"event": "x"}\n\xff\xfe not utf-8\n') + config = _write_training_config(tmp_path, training_out) + + events = _purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit") + + assert [e["event"] for e in events][:2] == ["data.erasure_requested", "data.erasure_completed"] + warning = _memorisation_event(events) + assert warning is not None + assert warning["affected_run_ids"][0].startswith(_MISSING_RUN_ID_HINT) + + def test_unexaminable_entry_does_not_hide_its_siblings(self, tmp_path: Path) -> None: + """A symlink loop named ``checkpoint-1`` cannot be stat'ed; the real ``checkpoint-2`` beside it is still + reported.""" + training_out = tmp_path / "ck" + (training_out / "checkpoint-2").mkdir(parents=True) + try: + (training_out / "checkpoint-1").symlink_to(training_out / "checkpoint-1", target_is_directory=True) + except (OSError, NotImplementedError): + pytest.skip("symlink creation not permitted on this platform") + + warning = _memorisation_event(_purge_one_row(tmp_path, output_dir=training_out)) + + assert warning is not None + assert warning["memorisation_artefact_paths"] == [str(training_out / "checkpoint-2")] + + def test_the_staging_name_the_scan_looks_for_is_the_one_the_trainer_writes(self) -> None: + """Drift guard: ``ForgeTrainer`` stages a run next to its model directory. If this literal changes, update + ``_scan_training_tree``, ``_staging_targets_for_run`` and ``_discover_per_run_staging_horizons`` with it.""" + import forgelm.trainer + + source = Path(forgelm.trainer.__file__).read_text(encoding="utf-8") + + assert 'f"{final_path}.staging.{self.audit.run_id}"' in source + + def test_granted_approval_in_the_training_audit_log_names_the_run(self, tmp_path: Path) -> None: + """After ``forgelm approve`` the staging suffix is gone; the run's own audit log, under the configured + training directory, still names it.""" + training_out = tmp_path / "ck" + (training_out / "final_model").mkdir(parents=True) + (training_out / "audit_log.jsonl").write_text( + json.dumps({"event": "human_approval.granted", "run_id": "fg-granted"}) + "\n", encoding="utf-8" + ) + config = _write_training_config(tmp_path, training_out) + + warning = _memorisation_event(_purge_one_row(tmp_path, config=config, output_dir=tmp_path / "audit")) + + assert warning is not None + assert warning["affected_run_ids"] == ["fg-granted"] + + @pytest.mark.parametrize("model_dir", ["final_model", "my_model"]) + def test_staging_naming_is_shared_by_the_scan_the_run_purge_and_the_retention_report( + self, tmp_path: Path, model_dir: str + ) -> None: + """One staging name, three consumers: if any of them drifts from ``.staging.`` + (what ``ForgeTrainer`` writes) a run's staging directory is reported by one and invisible to another.""" + from forgelm.cli.subcommands._purge import ( + _discover_per_run_staging_horizons, + _scan_memorisation_artefacts, + _staging_targets_for_run, + ) + + staging = tmp_path / f"{model_dir}.staging.fg-run1" + staging.mkdir() + config = SimpleNamespace(training=SimpleNamespace(output_dir=str(tmp_path), final_model_dir=model_dir)) + + assert _scan_memorisation_artefacts(str(tmp_path / "elsewhere"), config) == [str(staging)] + assert _staging_targets_for_run(tmp_path, "fg-run1", [model_dir]) == [staging] + assert [d[1] for d in _discover_per_run_staging_horizons(str(tmp_path), 7, [model_dir])] == [str(staging)] + + +class TestRunScopedStagingNamesFollowTheConfig: + def test_run_id_erases_the_staging_directory_of_a_custom_final_model_dir(self, tmp_path: Path) -> None: + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + staging = out / "my_model.staging.fg-x" + staging.mkdir(parents=True) + (staging / "adapter_model.safetensors").write_text("w") + config = _write_training_config(tmp_path, out, final_model_dir="my_model") + + args = _build_args(run_id="fg-x", kind="staging", output_dir=str(out), config=str(config)) + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + + assert ei.value.code == 0 + assert not staging.exists() + + def test_without_the_config_the_custom_staging_directory_is_not_found_and_is_left_alone( + self, tmp_path: Path + ) -> None: + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + staging = out / "my_model.staging.fg-x" + staging.mkdir(parents=True) + + args = _build_args(run_id="fg-x", kind="staging", output_dir=str(out)) + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + + assert ei.value.code == 1 + assert staging.exists() + failed = next(e for e in _read_audit_events(out / "audit_log.jsonl") if e["event"] == "data.erasure_failed") + assert failed["error_class"] == "NoMatchingArtefacts" + + +class TestRunScopedErasureWithAnUnreadableConfig: + def test_non_utf8_config_falls_back_to_the_default_name_and_closes_the_chain(self, tmp_path: Path, caplog) -> None: + """The config is read before ``data.erasure_requested`` is logged and a file that cannot be decoded is + 'cannot be loaded', like any other: the purge continues with the default staging name.""" + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + staging = out / "final_model.staging.fg-x" + staging.mkdir(parents=True) + bad_config = tmp_path / "binary.yaml" + bad_config.write_bytes(b"training:\n output_dir: \xff\xfe\x00\n") + + args = _build_args(run_id="fg-x", kind="staging", output_dir=str(out), config=str(bad_config)) + with caplog.at_level("WARNING"), pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + + assert ei.value.code == 0 + assert not staging.exists() + assert [e["event"] for e in _read_audit_events(out / "audit_log.jsonl")] == [ + "data.erasure_requested", + "data.erasure_completed", + ] + assert [r for r in caplog.records if "could not be loaded" in r.getMessage()] + + def test_artefacts_kind_does_not_load_the_config(self, tmp_path: Path, caplog) -> None: + """Only ``--kind staging`` needs ``final_model_dir``; ``--kind artefacts`` must not warn about a config it + never uses.""" + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + (out / "compliance").mkdir(parents=True) + (out / "compliance" / "report_fg-x.json").write_text("{}") + bad_config = tmp_path / "binary.yaml" + bad_config.write_bytes(b"\xff\xfe\x00") + + args = _build_args(run_id="fg-x", kind="artefacts", output_dir=str(out), config=str(bad_config)) + with caplog.at_level("WARNING"), pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + + assert ei.value.code == 0 + assert not [r for r in caplog.records if "could not be loaded" in r.getMessage()] + + +class TestRetentionReportFollowsTheConfig: + @staticmethod + def _age(path: Path, days: int) -> None: + old = path.stat().st_mtime - days * 86400 + os.utime(path, (old, old)) + + def test_check_policy_reports_staging_next_to_a_custom_final_model_dir(self, tmp_path: Path, capsys) -> None: + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + staging = out / "my_model.staging.fg-old" + staging.mkdir(parents=True) + self._age(staging, 30) + config = _write_training_config( + tmp_path, out, final_model_dir="my_model", extra="retention:\n staging_ttl_days: 7\n" + ) + + args = _build_args(check_policy=True, config=str(config), output_dir=str(out)) + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + + assert ei.value.code == 0 + violations = json.loads(capsys.readouterr().out)["violations"] + assert [(v["artefact_kind"], v["path"]) for v in violations] == [("staging_dir[fg-old]", str(staging))] + + def test_check_policy_with_a_retention_only_file_reports_the_default_staging_name( + self, tmp_path: Path, capsys + ) -> None: + """A retention-only file names no ``training.final_model_dir``, so only ``final_model.staging*`` is known.""" + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + default_staging = out / "final_model.staging.fg-old" + custom_staging = out / "my_model.staging.fg-old" + for staging in (default_staging, custom_staging): + staging.mkdir(parents=True) + self._age(staging, 30) + config = tmp_path / "retention.yaml" + config.write_text("retention:\n staging_ttl_days: 7\n", encoding="utf-8") + + args = _build_args(check_policy=True, config=str(config), output_dir=str(out)) + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + + violations = json.loads(capsys.readouterr().out)["violations"] + assert [v["path"] for v in violations] == [str(default_staging)] + + def test_checkpoints_are_outside_every_retention_horizon(self, tmp_path: Path) -> None: + """No ``retention:`` field covers ``checkpoint-*``; the docs say so and this pins it.""" + from forgelm.cli.subcommands._purge import _scan_retention_violations + from forgelm.config import RetentionConfig + + checkpoint = tmp_path / "checkpoint-1" + staging = tmp_path / "final_model.staging.fg-old" + for artefact in (checkpoint, staging): + artefact.mkdir() + self._age(artefact, 400) + retention = RetentionConfig( + audit_log_retention_days=1, + staging_ttl_days=1, + ephemeral_artefact_retention_days=1, + raw_documents_retention_days=1, + ) + + # The aged staging directory beside it proves the scan ran: only the checkpoint is left out. + assert [v["path"] for v in _scan_retention_violations(retention, str(tmp_path))] == [str(staging)] + + def test_check_policy_follows_pipeline_stage_model_names(self, tmp_path: Path, capsys) -> None: + from forgelm.cli.subcommands._purge import _run_purge_cmd + + out = tmp_path / "ck" + staging = out / "stage_one_model.staging.fg-old" + staging.mkdir(parents=True) + self._age(staging, 30) + extra = ( + "retention:\n staging_ttl_days: 7\n" + "pipeline:\n" + f" output_dir: '{tmp_path / 'pipeline_run'}'\n" + " stages:\n" + " - name: sft\n" + " training:\n" + " trainer_type: sft\n" + f" output_dir: '{out}'\n" + " final_model_dir: stage_one_model\n" + ) + config = _write_training_config(tmp_path, out, extra=extra) + + args = _build_args(check_policy=True, config=str(config), output_dir=str(out)) + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + + violations = json.loads(capsys.readouterr().out)["violations"] + assert [v["path"] for v in violations] == [str(staging)] + + def test_retention_loader_returns_the_full_config_only_for_a_full_file(self, tmp_path: Path) -> None: + from forgelm.cli._config_load import _load_retention_and_config + + full = _write_training_config(tmp_path, tmp_path / "ck", extra="retention:\n staging_ttl_days: 7\n") + only = tmp_path / "retention.yaml" + only.write_text("retention:\n staging_ttl_days: 9\n", encoding="utf-8") + + full_retention, full_config = _load_retention_and_config(str(full)) + only_retention, only_config = _load_retention_and_config(str(only)) + + assert full_retention is not None and full_retention.staging_ttl_days == 7 + assert full_config is not None and full_config.training.output_dir == str(tmp_path / "ck") + assert only_retention is not None and only_retention.staging_ttl_days == 9 + assert only_config is None + assert _load_retention_and_config(None) == (None, None) + + # --------------------------------------------------------------------------- # Finding 5 (LOW) — run-scoped erasure records the PLAIN run_id as target_id # (design §5.3); the module docstring now documents this asymmetry. diff --git a/tools/check_module_size.py b/tools/check_module_size.py index 7761d6a0..100e49c5 100644 --- a/tools/check_module_size.py +++ b/tools/check_module_size.py @@ -493,7 +493,7 @@ class _DeferredSplit: ), ), "forgelm/cli/subcommands/_purge.py": _DeferredSplit( - budget=1224, + budget=1304, deferred_at_loc=1215, budget_history=( "2026-10-09 (Phase 16 S1-S3 review, 1215 -> 1216, +1): the `dumps_strict` import. " @@ -505,6 +505,12 @@ class _DeferredSplit: "`--output-dir` is omitted. The default (the corpus directory) silently pointed the memorisation " "check at a directory that never holds a trained model, so the message has to sit where the " "default is resolved. Taken as a raise rather than a split per decision C-18.", + "2026-10-24 (#102, CR-A10-022 + CR-A10-042, 1224 -> 1304, +80): the memorisation check follows the " + "layouts the trainer writes: the model directory, its `.staging.` sibling and " + "`checkpoint-*` under the output dir, under the loaded config's `training` and `pipeline` stages, and " + "`merge.output_dir`. The run-scoped staging purge and the retention report take their staging name from " + "`training.final_model_dir`; the scan survives an unreadable entry or audit log, because it runs after " + "the row is already erased. Taken as a raise rather than a split per decision C-18.", ), reason=( "GDPR purge: row-id resolution + run-id resolution + retention-policy "