From 44be87562f25e9a43537454614b5fae842a8f287 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 18 Oct 2026 10:16:18 +0300 Subject: [PATCH 1/2] fix(purge): warn when --output-dir falls back to the corpus directory, and document the real memorisation check The purge reference and the user manual both said that the implicit `--output-dir` fallback "emits a WARNING naming the resolved dir". It did not: nothing was logged. The fallback is the corpus's directory, and the memorisation check (`data.erasure_warning_memorisation`) looks for `/final_model/` there, a place a training run rarely writes its model. A default `forgelm purge --row-id X --corpus data/train.jsonl` therefore ended with a completed-erasure record and no memorisation event, so a reader of the audit chain alone would conclude that no model had consumed the row. `--row-id` without `--output-dir` now logs one warning naming the resolved directory and the `final_model/` it will look for, which makes the documented behaviour true. The audit-event catalogue, the purge reference, the GDPR erasure guide, the user manual and the risk-treatment plan (English and Turkish) described the condition as "a final_model/ exists for any run that consumed the corpus"; the code checks one literal directory and cannot tell which runs consumed the corpus. They now say that. Reading `training.output_dir` from `--config` and scanning `final_model*` / `checkpoint-*` is the larger fix and stays open on the issue; it changes the content of an erasure audit event (additively) and wants its own review. Refs #102 Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 7 +++ docs/guides/gdpr_erasure-tr.md | 4 +- docs/guides/gdpr_erasure.md | 4 +- docs/qms/risk_treatment_plan-tr.md | 2 +- docs/qms/risk_treatment_plan.md | 2 +- docs/reference/audit_event_catalog-tr.md | 2 +- docs/reference/audit_event_catalog.md | 2 +- docs/reference/purge_subcommand-tr.md | 4 +- docs/reference/purge_subcommand.md | 4 +- .../usermanuals/en/compliance/gdpr-erasure.md | 4 +- .../usermanuals/tr/compliance/gdpr-erasure.md | 4 +- forgelm/cli/subcommands/_purge.py | 11 ++++ tests/test_gdpr_erasure.py | 50 +++++++++++++++++++ 13 files changed, 84 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d44dd02..584788d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,13 @@ All notable changes to ForgeLM are documented here. ### Fixed +- `forgelm purge --row-id` without `--output-dir` now logs a warning that names the directory it resolved (the corpus's) + and the `final_model/` the memorisation check will look for. The documentation promised that warning but the code never + logged it, and a default run looks for a model next to the corpus, where a training run rarely writes one, so it finished + with a completed-erasure record and no memorisation event. The audit-event catalogue, the purge reference, the GDPR + erasure guide, the user manual and the risk-treatment plan now state the real condition: a `final_model/` directly under + `--output-dir`, not "any run that consumed the corpus" (English and Turkish). Reading `training.output_dir` from + `--config` is still open (#102). - Fix the KTO bullet on the Alignment overview user-manual page (English and Turkish): it named the fields `(prompt, response, good?)`, but the KTO loader reads `prompt`, `completion` and a boolean `label` (#965). diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 64a1bb16..138439d3 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -19,7 +19,7 @@ **Yapmaz:** -- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, corpus'u tüketen herhangi bir koşum için `final_model/` mevcut olduğunda `data.erasure_warning_memorisation` yayar; böylece operatör boşluğu görür. +- Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. - Audit log'un kendisinden entry silmez. Madde 17(3)(b) audit / muhasebe kayıtlarını bir hukuki yükümlülük savunması olarak korur; tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. @@ -96,7 +96,7 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden önce (`--check-policy` salt-okunur; audit event yaymaz) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme bittiğinde | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Yalnız row modunda.** Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. (Run modu bu uyarıyı yüzeylemez — synthetic veri corpus-scoped'dur, run-scoped değil.) | Tüm `completed` field'ları + `synthetic_files` | | `data.erasure_warning_external_copies` | Yüklü config webhook block'u içeriyor | Tüm `completed` field'ları + `webhook_targets` | diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 24941879..06e1a513 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -19,7 +19,7 @@ **Does not:** -- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` exists for any run that consumed the corpus, so the operator sees the gap. +- Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. - Delete entries from the audit log itself. Article 17(3)(b) preserves audit / accounting records as a legal-obligation defence; the tool records the erasure as a new event rather than rewriting history. - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. @@ -96,7 +96,7 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, before any deletion (`--check-policy` is read-only and emits no audit events) | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finishes | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × `final_model/` exists for any run that consumed the corpus. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a `final_model/` directory exists directly under `--output-dir`. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Row mode only.** Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. (Run mode does not surface this warning — synthetic data is corpus-scoped, not run-scoped.) | All `completed` fields + `synthetic_files` | | `data.erasure_warning_external_copies` | Loaded config has a webhook block (both modes). | All `completed` fields + `webhook_targets` | diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md index 1ac0e752..9570b600 100644 --- a/docs/qms/risk_treatment_plan-tr.md +++ b/docs/qms/risk_treatment_plan-tr.md @@ -90,7 +90,7 @@ ekler. |---|---| | Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM, silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` event'ı yayar — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | +| Treatment | ForgeLM, purge'ün `--output-dir`'i altında doğrudan bir `final_model/` varsa `data.erasure_warning_memorisation` event'ı yayar (satırı hangi koşumun tükettiğini bilmez ve varsayılan dizin eğitim çıktısı değil corpus'unkidir) — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | | Residual L × I | High × Med = HIGH (risk **kabul edildi**, bkz. §5) | | Sahip | Data Protection Officer (DPO) | | Review cadence | Madde 17 talebi başına | diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md index c19a9d35..43258b00 100644 --- a/docs/qms/risk_treatment_plan.md +++ b/docs/qms/risk_treatment_plan.md @@ -89,7 +89,7 @@ section §4. |---|---| | Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights | | L × I (inherent) | High × Med = HIGH | -| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when the run that consumed the deleted row has a `final_model/` artefact — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | +| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when a `final_model/` exists directly under the purge's `--output-dir` (it does not know which run consumed the row, and the default directory is the corpus's, not the training output) — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | | Residual L × I | High × Med = HIGH (risk **accepted**, see §5) | | Owner | Data Protection Officer (DPO) | | Review cadence | Per Article 17 request | diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md index 3068d80d..051d863e 100644 --- a/docs/reference/audit_event_catalog-tr.md +++ b/docs/reference/audit_event_catalog-tr.md @@ -78,7 +78,7 @@ Hash zinciri, satır diske düştükten (`flush` + `fsync`) sonra ilerler; kirli | `data.erasure_requested` | Herhangi bir `forgelm purge --row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunur; event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'lenmiş), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. | Tüm `completed` field'ları + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | 17 | diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md index a7a74abd..e5d05c32 100644 --- a/docs/reference/audit_event_catalog.md +++ b/docs/reference/audit_event_catalog.md @@ -78,7 +78,7 @@ The hash chain advances after the line lands on disk (`flush` + `fsync`), so an | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | 17 | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | 17 | | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | 17 | -| `data.erasure_warning_memorisation` | Row erasure × `final_model/` exists for any run that consumed this corpus. | All `completed` fields + `affected_run_ids` | 17 | +| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | 17 | | `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | 17 | diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index de03e4d2..cf111861 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -35,7 +35,7 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | `--row-id ID` | evet | Silinecek identifier. Audit emisyonundan önce `forgelm.cli.subcommands._purge._hash_target_id` üzerinden hash'lenir. | | `--corpus PATH` | evet | Tek bir JSONL dosyası. Dizin modu reddedilir — operatörler kendi script'lerinde döngü kurar. | | `--row-matches {one,all}` | hayır (varsayılan `one`) | `one` >=2 eşleşmede reddeder; `all` her eşleşmeyi siler (operatör niyeti onaylar). | -| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, `forgelm reverse-pii` ile cross-tool korelasyon için operatörün `--output-dir`'i sabitleyebilmesi adına çözülen dizini adlandıran bir WARNING yayar. | +| `--output-dir DIR` | hayır | `--corpus`'un parent'ına default'lar. Per-output-dir salt `/.forgelm_audit_salt`'ta `target_id` hash'leme için okunur. Implicit fallback, operatörün `--output-dir`'i eğitim çıktı dizinine ve `forgelm reverse-pii` ile cross-tool korelasyon için sabitleyebilmesi adına çözülen dizini ve memorisation kontrolünün arayacağı `final_model/`'i adlandıran bir WARNING yayar. | | `--justification TEXT` | hayır | Operatör-sağlanan, her erasure event'ına yazılan sebep. Dahili ticket id'nizi referans verin; subject identifier yapıştırmayın. | | `--dry-run` | hayır | Silmeyi önizler + audit zincirini (`dry_run=true` ile) yayar; disk'e dokunmaz. | @@ -80,7 +80,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | `data.erasure_requested` | Herhangi bir `--row-id` / `--run-id` çağrısının ilk adımı, herhangi bir silmeden ÖNCE. `--check-policy` salt-okunurdur ve event yaymaz. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (row mode'da hash'li), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Başarılı silme tamamlandı. | Tüm `requested` field'ları + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum yok VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut. | Tüm `completed` field'ları + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | | `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index ebbffe38..2c44b85f 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -35,7 +35,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | `--row-id ID` | yes | Identifier to erase. Hashed before audit emission via `forgelm.cli.subcommands._purge._hash_target_id`. | | `--corpus PATH` | yes | Single JSONL file. Directory mode is rejected — operators loop in their own script. | | `--row-matches {one,all}` | no (default `one`) | `one` refuses on >=2 matches; `all` deletes every match (operator confirms intent). | -| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir so the operator can pin `--output-dir` for cross-tool correlation with `forgelm reverse-pii`. | +| `--output-dir DIR` | no | Defaults to the parent of `--corpus`. The per-output-dir salt at `/.forgelm_audit_salt` is read here for `target_id` hashing. The implicit fallback emits a WARNING naming the resolved dir and the `final_model/` the memorisation check will look for, so the operator can pin `--output-dir` to the training output directory and for cross-tool correlation with `forgelm reverse-pii`. | | `--justification TEXT` | no | Operator-supplied reason recorded on every erasure event. Reference your internal ticket id; do not paste subject identifiers. | | `--dry-run` | no | Preview the deletion + emit the audit chain (with `dry_run=true`) without modifying disk. | @@ -80,7 +80,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | `data.erasure_requested` | First step of any `--row-id` / `--run-id` invocation, BEFORE any deletion. `--check-policy` is read-only and emits no events. | `target_kind` ∈ `{row, staging, artefacts}`, `target_id` (hashed in row mode), `salt_source` (row mode), `corpus_path` (row), `output_dir` (run), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finished. | All `requested` fields + `bytes_freed`, `files_modified`, `pre_erasure_line_number` (row mode), `match_count` (row mode) | | `data.erasure_failed` | Disk operation raised, OR no matching row/run, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | -| `data.erasure_warning_memorisation` | Row erasure × `final_model/` exists for any run that consumed this corpus. | All `completed` fields + `affected_run_ids` | +| `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | | `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 6cf43cb9..359a8561 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -42,7 +42,7 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - `data.erasure_requested` — emitted FIRST, before any deletion. - `data.erasure_completed` — emitted LAST after the disk operation succeeded. - `data.erasure_failed` — emitted instead when the disk operation raised, or no matching row/run was found. -- `data.erasure_warning_memorisation` — corpus-row erasure × `final_model/` exists for any run that consumed this corpus (the row is gone from disk but may still be memorised in the trained weights). +- `data.erasure_warning_memorisation` — corpus-row erasure × a `final_model/` exists directly under `--output-dir` (the row is gone from disk but may still be memorised in the trained weights). - `data.erasure_warning_synthetic_data_present` — corpus-row erasure × `synthetic_data*.jsonl` exists in `output_dir`. - `data.erasure_warning_external_copies` — loaded config has a non-empty `webhook` block; downstream consumers may have received notices. @@ -110,7 +110,7 @@ Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of e ::: :::warn -**Running `forgelm purge` without checking for memorisation.** If a `final_model/` exists for any run that consumed the corpus, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap; act on it (re-train without the erased row, or document the residual risk in your DPIA). +**Running `forgelm purge` without checking for memorisation.** If a trained model exists, the row is gone from disk but may still be encoded in the weights. The `data.erasure_warning_memorisation` event surfaces the gap only when a `final_model/` sits directly under `--output-dir` — which defaults to the corpus's directory, so pass your training output directory; act on it (re-train without the erased row, or document the residual risk in your DPIA). ::: :::tip diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index f430e5c0..77e6b09a 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -42,7 +42,7 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - `data.erasure_requested` — herhangi bir silmeden ÖNCE, ilk yayılır. - `data.erasure_completed` — disk operasyonu başarılı sonra son yayılır. - `data.erasure_failed` — disk operasyonu raise ettiğinde veya eşleşen satır/koşum bulunmadığında bunun yerine yayılır. -- `data.erasure_warning_memorisation` — corpus-satır silme × bu corpus'u tüketen herhangi bir koşum için `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). +- `data.erasure_warning_memorisation` — corpus-satır silme × `--output-dir` altında doğrudan bir `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). - `data.erasure_warning_synthetic_data_present` — corpus-satır silme × `output_dir`'de `synthetic_data*.jsonl` mevcut. - `data.erasure_warning_external_copies` — yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. @@ -110,7 +110,7 @@ Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın d ::: :::warn -**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Corpus'u tüketen herhangi bir koşum için `final_model/` mevcutsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yüzeyselleştirir; harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). +**Memorization'ı kontrol etmeden `forgelm purge` çalıştırmak.** Eğitilmiş bir model varsa, satır diskten gitti ama ağırlıklarda hâlâ encoded olabilir. `data.erasure_warning_memorisation` event'ı boşluğu yalnızca `--output-dir` altında doğrudan bir `final_model/` varsa yüzeyselleştirir — `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin; harekete geçin (silinmiş satır olmadan yeniden eğitin veya DPIA'nızda residual riski belgeleyin). ::: :::tip diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 781d9208..97f139aa 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -714,6 +714,17 @@ def _run_purge_row_id(args, output_format: str) -> None: """ _validate_row_id_args(args, output_format) output_dir = args.output_dir or os.path.dirname(os.path.abspath(args.corpus)) or "." + if not args.output_dir: + # The salt, the audit chain and the memorisation check (`/final_model/`) all live under the + # output directory. Defaulting to the corpus directory is almost never where a training run wrote its model, + # so say what was resolved instead of letting the check pass quietly over the wrong directory. + logger.warning( + "--output-dir not given: using the corpus directory %s for the audit log, the salt and the memorisation " + "check (which looks for %s). Pass --output-dir to check your trained model and to " + "correlate with `forgelm reverse-pii`.", + output_dir, + os.path.join(output_dir, "final_model"), + ) os.makedirs(output_dir, exist_ok=True) try: diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index bcd3c6e9..06c27775 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -152,6 +152,56 @@ def test_row_erasure_target_id_is_hashed_not_cleartext(self, tmp_path: Path) -> # --------------------------------------------------------------------------- +class TestImplicitOutputDirWarning: + """`--output-dir` defaults to the corpus directory; the user is told, because the memorisation check looks for + `/final_model/` and a training run rarely writes its model next to the corpus (#102).""" + + def test_the_implicit_directory_is_named_in_a_warning(self, tmp_path: Path, caplog) -> None: + import logging + + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "data" / "train.jsonl" + corpus.parent.mkdir() + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=None) + with caplog.at_level(logging.WARNING, logger="forgelm"): + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + assert ei.value.code == 0 + warned = [r.getMessage() for r in caplog.records if "--output-dir not given" in r.getMessage()] + assert len(warned) == 1 + assert str(corpus.parent) in warned[0] and os.path.join(str(corpus.parent), "final_model") in warned[0] + + def test_an_explicit_output_dir_is_silent(self, tmp_path: Path, caplog) -> None: + import logging + + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "train.jsonl" + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=str(tmp_path)) + with caplog.at_level(logging.WARNING, logger="forgelm"): + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + assert not [r for r in caplog.records if "--output-dir not given" in r.getMessage()] + + def test_no_memorisation_event_without_a_final_model_in_the_resolved_directory(self, tmp_path: Path) -> None: + # The default directory is the corpus's; a model written elsewhere is invisible to the check. The warning + # above is what tells the operator. + from forgelm.cli.subcommands._purge import _run_purge_cmd + + corpus = tmp_path / "data" / "train.jsonl" + corpus.parent.mkdir() + _seed_corpus(corpus, [{"id": "row-A", "text": "x"}]) + (tmp_path / "training_out" / "final_model").mkdir(parents=True) + args = _build_args(row_id="row-A", corpus=str(corpus), output_dir=None) + with pytest.raises(SystemExit): + _run_purge_cmd(args, output_format="json") + events = _read_audit_events(corpus.parent / "audit_log.jsonl") + assert "data.erasure_warning_memorisation" not in [e["event"] for e in events] + + class TestSaltPersistence: def test_salt_file_created_on_first_use_with_mode_0600(self, tmp_path: Path) -> None: from forgelm.cli.subcommands._purge import _resolve_salt From 9d1045c5a4d0b1d90e2d8180ec93f87252455744 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 18 Oct 2026 10:36:46 +0300 Subject: [PATCH 2/2] chore(module-size): raise the purge module's line budget for the --output-dir warning The warning that `forgelm purge --row-id` now logs when --output-dir is omitted adds eight lines to a module on the deferred-split list, and the budget has to equal the measured size. It is a raise rather than a split for the reason the previous raise gave (C-18): the message has to sit where the default is resolved. The history note says why. Refs #102 Co-Authored-By: Claude Sonnet 5.5 --- tools/check_module_size.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tools/check_module_size.py b/tools/check_module_size.py index 35bb0b0d..5fc6af13 100644 --- a/tools/check_module_size.py +++ b/tools/check_module_size.py @@ -488,7 +488,7 @@ class _DeferredSplit: ), ), "forgelm/cli/subcommands/_purge.py": _DeferredSplit( - budget=1216, + budget=1224, deferred_at_loc=1215, budget_history=( "2026-10-09 (Phase 16 S1-S3 review, 1215 -> 1216, +1): the `dumps_strict` import. " @@ -496,6 +496,10 @@ class _DeferredSplit: "strict-JSON guard's hand-picked module list; the guard now scans all of " "`forgelm/`, so they route through the chokepoint like every other writer. " "One line, taken as a raise rather than a split per decision C-18.", + "2026-10-18 (wave-0 triage, #102, 1216 -> 1224, +8): the warning `purge --row-id` logs when " + "`--output-dir` is omitted. The default (the corpus directory) silently pointed the memorisation " + "check at a directory that never holds a trained model, so the message has to sit where the " + "default is resolved. Taken as a raise rather than a split per decision C-18.", ), reason=( "GDPR purge: row-id resolution + run-id resolution + retention-policy "