From 08c26ff0193865504582ef592dbd8a8c123cefdb Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:33:48 +0300 Subject: [PATCH 01/25] docs(gdpr): stop presenting Article 17(3)(b) as a blanket audit-log defence; say what purge --kind artefacts finds Article 17(3)(b) disapplies erasure only to the extent a specific Union or Member State legal obligation requires the processing; it does not preserve an audit log as a class. The guides, reference, user manual, design note, CLI help and code comments said it did. The retention horizons were described as honouring GDPR deadlines and "compliant out of the box"; GDPR sets no fixed period and the defaults are ForgeLM choices. The staging horizon was described as a scheduled cleanup after a reject decision; nothing deletes automatically and the age is counted from the run's first audit event. `purge --kind artefacts` matches compliance filenames that embed the run id, but the exporter writes fixed basenames, so on a real output directory it finds nothing, emits data.erasure_failed (NoMatchingArtefacts) and exits 1. Documented that, and that the legacy final_model.staging/ is only purged when it carries a staging_run_id marker no ForgeLM code writes. The code fix stays with its own finding. Resolves DOC-BLEGAL-004 (and the design's :59/:167/:242/:283 recurrences), DOC-BLEGAL-005, DOC-BLEGAL-064, DOC-BLEGAL-097, DOC-BLEGAL-146, DOC-BFIELD-019, DOC-D1B05-017, DOC-D2B02-025 and the Article 17(3)(b) and retention halves of TECH-T11-002; documents DOC-D1B05-005 (Refs #101). Co-Authored-By: Claude Sonnet 5.5 --- config_template.yaml | 12 +-- docs/design/gdpr_erasure.md | 16 ++-- docs/guides/gdpr_erasure-tr.md | 10 +-- docs/guides/gdpr_erasure.md | 10 +-- docs/reference/configuration-tr.md | 12 ++- docs/reference/configuration.md | 12 ++- docs/reference/purge_subcommand-tr.md | 10 ++- docs/reference/purge_subcommand.md | 10 ++- .../usermanuals/en/compliance/gdpr-erasure.md | 5 +- .../usermanuals/en/reference/configuration.md | 4 +- .../usermanuals/tr/compliance/gdpr-erasure.md | 5 +- .../usermanuals/tr/reference/configuration.md | 4 +- forgelm/cli/_parser.py | 2 +- forgelm/cli/subcommands/_purge.py | 2 +- forgelm/config.py | 14 ++-- tests/test_gdpr_erasure.py | 41 +++++++++++ .../test_usermanual_reference_consistency.py | 73 +++++++++++++++++++ 17 files changed, 186 insertions(+), 56 deletions(-) diff --git a/config_template.yaml b/config_template.yaml index 415cc2c9..823b8e2a 100644 --- a/config_template.yaml +++ b/config_template.yaml @@ -268,12 +268,14 @@ data: # alert_on_drift: true # check_interval_hours: 24 -# Retention horizons — GDPR Article 5(1)(e) "storage limitation" + Article 17 -# "right to erasure". `staging_ttl_days` is the canonical staging-retention -# field (the legacy `evaluation.staging_ttl_days` alias was removed in v0.8.0). +# Retention horizons — operator-set limits that help implement GDPR Article 5(1)(e) +# "storage limitation" and support Article 17 erasure handling; the defaults are +# ForgeLM choices, not legal periods. `staging_ttl_days` is the canonical +# staging-retention field (the legacy `evaluation.staging_ttl_days` alias was +# removed in v0.8.0). # retention: -# audit_log_retention_days: 1825 # ~5 years (Article 12 record-keeping baseline) -# staging_ttl_days: 7 # final_model.staging./ TTL after `forgelm reject` +# audit_log_retention_days: 1825 # ~5 years (ForgeLM default, not a legal period) +# staging_ttl_days: 7 # age at which `purge --check-policy` reports final_model.staging./ overdue # ephemeral_artefact_retention_days: 90 # compliance bundles + audit reports # raw_documents_retention_days: 90 # ingested raw docs (PDF/DOCX/EPUB/TXT/MD) # enforce: "log_only" # reserved: no command acts on it yet ("log_only" | "warn_on_excess" | "block_on_excess") diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 607ea70c..3a0e620a 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -18,7 +18,7 @@ to be forgotten") + Article 5(1)(e) ("storage limitation"). ## 0. Why this document exists -ForgeLM emits training-data corpora, model adapters, audit logs, and derived artefacts that may contain personal data of identifiable EU data subjects. Article 17 of the GDPR gives those subjects the right to request that this data be erased. Article 5(1)(e) requires that any personal data not actively required for the declared purpose is removed by default after a defined retention horizon. +ForgeLM emits training-data corpora, model adapters, audit logs, and derived artefacts that may contain personal data of identifiable EU data subjects. Article 17 of the GDPR gives those subjects the right to request that this data be erased. Article 5(1)(e) requires that personal data be kept in identifiable form no longer than necessary for the purpose; together with Art. 25(2) and recital 39, controllers set time limits for erasure or periodic review. ForgeLM today has no operator-actionable tool for either obligation: @@ -45,9 +45,9 @@ Phase 21 closes the gap with a real implementation. This Phase 20 document spec **Article 17(2)** requires the controller to inform downstream processors / publishers of the erasure when the data has been disclosed. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. -**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation of EU law, which is the entire reason the audit log itself is preserved. +**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation or claim, and the controller has to name it: no provision of EU law protects a training audit log merely because ForgeLM produces it. -The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability) but the original entries that reference the now-erased subject stay in place because the audit log is the legal-obligation record under Art. 17(3)(b). +The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability), and ForgeLM never edits or deletes an entry. Whether the entries that reference the now-erased subject may stay is the controller's decision: Art. 17(3)(b) and (3)(e) cover them only to the extent a specific obligation or claim requires it, and the controller documents which one. --- @@ -56,7 +56,7 @@ The take-away: erasure operates on **training data + derived artefacts**, not on | Artefact | Where it lives | Carries personal data? | Erasure strategy | |---|---|---|---| | **Training corpus rows** | `.jsonl` (operator-supplied) | Yes — the whole point | **Delete row in place** + audit event | -| **Audit log entries** | `/audit_log.jsonl` (append-only) | Indirectly (operator name, run config, possibly file paths that name a subject) | **Never delete original** — append a `data.erasure_completed` event that references the deleted row's `row_id`. Article 17(3)(b) exemption. | +| **Audit log entries** | `/audit_log.jsonl` (append-only) | Indirectly (operator name, run config, possibly file paths that name a subject) | **Never delete original** — append a `data.erasure_completed` event that references the deleted row's `row_id`. The tool never rewrites the chain; keeping the entries is justified only to the extent Art. 17(3)(b) or (3)(e) applies, which the controller documents. | | **Model adapters** | `/final_model/` (PEFT adapter + tokenizer) | **Memorised** training samples — extraction risk for high-overfit models | **Document escalation**: full retraining is the only proper resolution. Tool emits `data.erasure_warning_memorisation` audit event flagging the gap. | | **Final model staging** | `/final_model.staging./` | Same memorisation risk | Operator chooses: delete staging (pre-promotion) or treat as adapters above | | **Compliance artefacts** | `/compliance/*.{json,yaml}` | References to corpus paths (could disclose subject names via filename) | **Regenerate** after erasure (or `forgelm purge --kind artefacts` rewrites them) | @@ -164,9 +164,9 @@ Three sites: For each retention horizon: - Compare artefact `mtime` against `now - _days`. - If older, the artefact has overstayed the declared lawful basis. -- The audit log itself is **never deleted automatically** even when the policy says it has overstayed — Article 17(3)(b) exemption applies and the operator must make the deletion call by hand. The tool only flags. +- The audit log itself is **never deleted automatically** even when the policy says it has overstayed. The tool only flags; the operator makes the deletion call by hand, after checking that no specific Union or Member State obligation (Art. 17(3)(b)) or legal claim (Art. 17(3)(e)) still requires the entries. -The compliance artefacts (`compliance/*.json`) are tied to a specific run; their lifetime is the run's lifetime, not a separate horizon. In-place deletion is supported via `forgelm purge --run-id --kind artefacts` (§4.1) which removes `/compliance/*.json` for the named run while leaving the audit log + final model intact — useful when an operator wants to regenerate the bundle after an Article 17 corpus erasure changed the lineage. Whole-`` deletion remains the operator's manual escalation when the entire run is being scrubbed. +The compliance artefacts (`compliance/*.json`) are tied to a specific run; their lifetime is the run's lifetime, not a separate horizon. In-place deletion is supported via `forgelm purge --run-id --kind artefacts` (§4.1) which is meant to remove `/compliance/*.json` for the named run while leaving the audit log + final model intact (as shipped, the resolver only matches filenames that embed the run id, and the standard bundle uses fixed basenames, so it finds nothing — see the purge reference) — useful when an operator wants to regenerate the bundle after an Article 17 corpus erasure changed the lineage. Whole-`` deletion remains the operator's manual escalation when the entire run is being scrubbed. --- @@ -239,7 +239,7 @@ Recovery path for a step-5 failure (rename fails after the temp file is fully wr | **Re-train the model after corpus erasure** | Operator decision; ForgeLM cannot infer the right downstream pipeline. | | **Distributed corpus deletion** | Single-node only. Multi-node corpus shards live on separate filesystems; the operator runs `forgelm purge --row-id` on each. | | **Webhook receiver cleanup** | The Slack / Teams thread that received `notify_awaiting_approval` is in the operator's chat platform, not in ForgeLM. We log a notice. | -| **Audit log rewriting** | The chain is append-only by construction (Art. 17(3)(b) exemption). Erasure adds an event; never edits an entry. | +| **Audit log rewriting** | The chain is append-only by construction. Erasure adds an event; never edits an entry. Whether the entries may be kept is the controller's call: Art. 17(3)(b) and (3)(e) apply only to the extent a specific obligation or claim requires it. | | **Model weight forgetting** | "Machine unlearning" is an active research area; we do not ship a half-baked impl. We emit `data.erasure_warning_memorisation` so the operator knows the gap is theirs to close. | --- @@ -280,7 +280,7 @@ If the **justification** field includes something that *might* be PII (e.g. oper ### 5.4 Personal-data minimisation across every audit-event field -Article 17(3)(b) preserves the audit log itself, but Article 5(1)(c) (data minimisation) still applies to **what we put in each event**. Phase 21 enumerates every field on every new audit event and classifies it; categories that may carry personal data get a hash / redaction policy at emit time so the persisted log carries the minimum identifying surface needed for accountability. +The audit log itself is append-only, but Article 5(1)(c) (data minimisation) still applies to **what we put in each event**. Phase 21 enumerates every field on every new audit event and classifies it; categories that may carry personal data get a hash / redaction policy at emit time so the persisted log carries the minimum identifying surface needed for accountability. | Event field | Source / shape | Personal-data category | Phase 21 policy | |---|---|---|---| diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 7fa432c1..0be6ad4f 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -12,7 +12,7 @@ **Yapar:** - JSONL eğitim corpus'undan tek bir satırı siler (`--row-id --corpus `). -- Bir koşumun staging dizinini veya compliance bundle'ını siler (`--run-id --kind {staging,artefacts}`). +- Bir koşumun staging dizinini ve adı run id'yi embed eden compliance dosyalarını siler (`--run-id --kind {staging,artefacts}`; `artefacts` standart paketi bulamaz, aşağıya bakın). - Yüklü config'in `retention:` block'una karşı retention-policy ihlallerini raporlar (`--check-policy`). - Her aksiyonu tamper-evident bir audit-event zinciriyle kaydeder — `request → completed` (veya `request → failed`) — böylece adli inceleyici tam olarak ne olduğunu yeniden inşa edebilir. - Operatör tarafından sağlanan satır identifier'larını audit log'a girmeden önce SHA-256 ile hash'ler; böylece zincirin kendisi kişisel-veri sızıntısına dönüşmez (Madde 5(1)(c) veri minimizasyonu). @@ -20,7 +20,7 @@ **Yapmaz:** - Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. -- Audit log'un kendisinden entry silmez. Madde 17(3)(b) audit / muhasebe kayıtlarını bir hukuki yükümlülük savunması olarak korur; tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. +- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır (örneğin sisteminiz için geçerli hale geldiğinde AI Act Md. 19(1) ve 26(6)'daki log saklama yükümlülükleri) ve bu yükümlülüğü adlandırabilmeniz gerekir. - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. @@ -56,9 +56,9 @@ $ forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` -- `--kind staging` — `/final_model.staging./`'i siler (mevcutsa legacy `final_model.staging/` da). -- `--kind artefacts` — ``'i adında embed eden `/compliance/*.json` dosyalarını siler. -- `--kind logs` **bilinçli olarak yok**: audit log'lar append-only Madde 17(3)(b) kayıtları; tool tarafından silinmez. +- `--kind staging` — `/final_model.staging./`'i siler; legacy `final_model.staging/`'i yalnızca içeriği ``'ye eşit bir `staging_run_id` işaretçisi taşıyorsa siler (ForgeLM bu işaretçiyi yazmaz; aksi halde legacy dizini kendiniz silin). +- `--kind artefacts` — ``'i adında token olarak embed eden `/compliance/*.json` dosyalarını siler. **Standart paketi bulamaz:** trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...), bu yüzden normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. +- `--kind logs` **bilinçli olarak yok**: audit log append-only'dir ve tool onu asla silmez (girdilerinin tutulup tutulamayacağına kimin karar verdiği için yukarıdaki ilk listeye bakın). ### 3. Retention policy raporu diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index fd4f7e21..b75caf4c 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -12,7 +12,7 @@ **Does:** - Erases a single row from a JSONL training corpus (`--row-id --corpus `). -- Deletes a run's staging directory or compliance bundle (`--run-id --kind {staging,artefacts}`). +- Deletes a run's staging directory, and compliance files whose names embed the run id (`--run-id --kind {staging,artefacts}`; `artefacts` does not find the standard bundle, see below). - Reports retention-policy violations against the loaded config's `retention:` block (`--check-policy`). - Records every action with a tamper-evident audit-event chain — `request → completed` (or `request → failed`) — so a forensic reviewer can reconstruct exactly what happened. - Hashes operator-supplied row identifiers with SHA-256 before they enter the audit log so the chain itself does not become a personal-data leak (Article 5(1)(c) data minimisation). @@ -20,7 +20,7 @@ **Does not:** - Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. -- Delete entries from the audit log itself. Article 17(3)(b) preserves audit / accounting records as a legal-obligation defence; the tool records the erasure as a new event rather than rewriting history. +- Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history. Whether entries that reference the data subject may be kept is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires the processing (for example the log-retention duties of AI Act Art. 19(1) and Art. 26(6), once they apply to your system), and you have to be able to name that obligation. - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. @@ -56,9 +56,9 @@ $ forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` -- `--kind staging` removes `/final_model.staging./` (and the legacy `final_model.staging/` if present). -- `--kind artefacts` removes `/compliance/*.json` files whose name embeds the ``. -- `--kind logs` is **intentionally absent**: audit logs are append-only Article 17(3)(b) records and are not deleted by the tool. +- `--kind staging` removes `/final_model.staging./`, and the legacy `final_model.staging/` only if it carries a `staging_run_id` marker whose content equals the `` (ForgeLM does not write that marker; otherwise remove the legacy directory yourself). +- `--kind artefacts` removes `/compliance/*.json` files whose name embeds the `` as a token. **It does not find the standard bundle:** the trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, so on a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. Remove the bundle through your storage layer until the resolver is fixed. +- `--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it (see the first list above for who decides whether its entries may be kept). ### 3. Retention-policy report diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index 1bd2349b..879bbd39 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -269,16 +269,20 @@ training: ## `retention` (İsteğe bağlı — GDPR Madde 17 silme ufukları) Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarını -belirler. Ufuklar GDPR Madde 5(1)(e) "saklama sınırlaması" ve Madde 17 -"silme hakkı" tarihlerini onurlandırır. Ufuklar **raporlanır**, zorlanmaz: +belirler. Ufuklar, GDPR Madde 5(1)(e) "saklama sınırlaması"nı (sabit süre yok: +veri yalnızca gerektiği kadar tutulabilir) uygulamaya yardımcı olan ve Madde 17 +kapsamındaki silme işlemlerini destekleyen operatör tarafından belirlenen saklama +sınırlarıdır (talepler gecikmeksizin, Madde 12(3)'teki bir aylık süre içinde +yanıtlanır). Değerleri kendi gereklilik değerlendirmenizden seçin; varsayılanlar +ForgeLM tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: `forgelm purge --check-policy` ufku aşanları listeler ve her zaman 0 ile çıkar; hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `audit_log_retention_days` | int | `1825` (~5 yıl) | `audit_log.jsonl` dosyasının Madde 5(1)(e) kapsamında "geciken" olarak işaretlenmeden önce saklanacağı gün sayısı. `0` süresiz saklamayı belirtir (Madde 17(3)(b) savunması). | -| `staging_ttl_days` | int | `7` | `forgelm reject` kararından sonra `final_model.staging./` dizininin planlı temizlenmeden önce saklanacağı gün sayısı. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | +| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (AI Act'in log saklama yükümlülükleri Md. 19(1) / Md. 26(6), en az altı ay). `0` süresiz saklamayı belirtir; dayandığınız AB veya üye devlet yükümlülüğünü adlandırın. | +| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini "geciken" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | | `ephemeral_artefact_retention_days` | int | `90` | Uyumluluk paketleri, veri denetim raporları ve diğer çalışma kapsamlı türetilmiş artefaktların saklanma süresi (gün). `0` süresiz saklama. | | `raw_documents_retention_days` | int | `90` | İngest edilmiş ham belgelerin (PDF / DOCX / EPUB / TXT / Markdown) operatörün ingestion-output dizininde saklanma süresi (gün). `0` süresiz saklama. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Henüz hiçbir komut onu okumaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa CLI uyarır. | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 1af45a54..3e4b2521 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -269,16 +269,20 @@ across retries. Each retry attempt is logged to the audit trail. ## `retention` (Optional — GDPR Article 17 erasure horizons) Defines maximum retention horizons for compliance, training, and evaluation -artefacts. Horizons honour GDPR Article 5(1)(e) "storage limitation" and -Article 17 "right to erasure" deadlines. The horizons are **reported**, not +artefacts. Horizons are operator-set retention limits that help implement GDPR +Article 5(1)(e) "storage limitation" (no fixed period: data may be kept only as +long as necessary) and support erasure handling under Article 17 (requests are +answered without undue delay, within the Article 12(3) one-month period). Choose +values from your own necessity assessment; the defaults are ForgeLM choices, not +legal periods. The horizons are **reported**, not enforced: `forgelm purge --check-policy` lists what is past them and always exits 0, and nothing is deleted, blocked or scheduled automatically. The `enforce` knob is reserved: no command reads it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `audit_log_retention_days` | int | `1825` (~5 years) | Days to retain `audit_log.jsonl` before flagging it as overdue under Article 5(1)(e). Set to `0` to retain indefinitely (Article 17(3)(b) defence). | -| `staging_ttl_days` | int | `7` | Days to retain `final_model.staging./` after a `forgelm reject` decision before scheduled cleanup. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | +| `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period (the AI Act's log-retention duties are Art. 19(1) / Art. 26(6), at least six months). Set to `0` to retain indefinitely; name the Union or Member State obligation you rely on. | +| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | | `ephemeral_artefact_retention_days` | int | `90` | Days to retain compliance bundles, data audit reports, and other run-scoped derived artefacts. Set to `0` to retain indefinitely. | | `raw_documents_retention_days` | int | `90` | Days to retain ingested raw documents (PDF / DOCX / EPUB / TXT / Markdown) under the operator's ingestion-output directory. Set to `0` to retain indefinitely. | | `enforce` | string | `"log_only"` | Reserved policy mode. No command reads it yet: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and the CLI warns when a config sets either of the last two. | diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index cf111861..b5659ede 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -3,7 +3,7 @@ > **Hedef kitle:** Eğitim corpus'larına ve koşum-kapsamlı artefact'lara karşı GDPR Madde 17 silme taleplerini karşılayan ForgeLM operatörleri ve ortaya çıkan `data.erasure_*` audit zincirini doğrulayan denetçiler. > **Ayna:** [purge_subcommand.md](purge_subcommand.md) -`forgelm purge`, ForgeLM eğitim corpus'ları ve koşum artefact'ları için **GDPR Madde 17 silinme hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 21). Bir satırı, bir koşumun staging dizinini veya bir koşumun compliance bundle'ını atomik olarak siler ve her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. +`forgelm purge`, ForgeLM eğitim corpus'ları ve koşum artefact'ları için **GDPR Madde 17 silinme hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 21). Bir satırı, bir koşumun staging dizinini ve (dosya adları run id'yi taşıyorsa — aşağıdaki `--kind artefacts`'e bakın) bir koşumun compliance dosyalarını atomik olarak siler ve her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. Deployer akışı (DSAR ticket → CLI çağrısı → doğrulama) için bkz. [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure-tr.md). Bu sayfa flag-başına, event-başına referanstır. @@ -47,10 +47,12 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | Kind | Hedef | |---|---| -| `staging` | `/final_model.staging./` (varsa legacy `final_model.staging/` da). | -| `artefacts` | Adı `` embed eden `/compliance/*.json` dosyaları. | +| `staging` | `/final_model.staging./`; legacy `final_model.staging/` yalnızca içinde içeriği ``'ye eşit bir `staging_run_id` dosyası varsa. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | +| `artefacts` | Adı ``'yi bir token olarak (`_`, `-` veya `.` ile sınırlanmış) embed eden `/compliance/*.json` dosyaları. | -`--kind logs` **kasıtlı olarak yok**: audit log'lar append-only Madde 17(3)(b) kayıtlarıdır ve tool tarafından silinmez. +> **`--kind artefacts` standart compliance paketini bulamaz.** Trainer ve exporter sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`; yapılandırıldığında `risk_assessment.json` / `annex_iv_metadata.json`) ve hiçbiri run id içermez. Normal bir çıktı dizininde komut bu yüzden hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar ve 1 ile çıkar; hiçbir şey silmez. Yalnızca adına run id'yi kendiniz yazdığınız dosyalara etki eder. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. + +`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Bir veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü bunları tutmayı gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. ### Retention-policy raporu (`--check-policy`) diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index 2c44b85f..321bebd7 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -3,7 +3,7 @@ > **Audience:** ForgeLM operators honouring GDPR Article 17 erasure requests against training corpora and run-scoped artefacts, plus auditors verifying the resulting `data.erasure_*` audit chain. > **Mirror:** [purge_subcommand-tr.md](purge_subcommand-tr.md) -`forgelm purge` is the operator-facing implementation of the **GDPR Article 17 right-to-erasure** for ForgeLM training corpora and run artefacts (Phase 21). It atomically deletes a row, a run's staging directory, or a run's compliance bundle, and records every step in the tamper-evident `audit_log.jsonl` chain. +`forgelm purge` is the operator-facing implementation of the **GDPR Article 17 right-to-erasure** for ForgeLM training corpora and run artefacts (Phase 21). It atomically deletes a row, or a run's staging directory, and (when the files carry the run id in their name — see `--kind artefacts` below) a run's compliance files, and records every step in the tamper-evident `audit_log.jsonl` chain. For the deployer-flow walkthrough (DSAR ticket → CLI invocation → verification), see [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure.md). This page is the per-flag, per-event reference. @@ -47,10 +47,12 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | Kind | Target | |---|---| -| `staging` | `/final_model.staging./` (and the legacy `final_model.staging/` if present). | -| `artefacts` | `/compliance/*.json` files whose name embeds ``. | +| `staging` | `/final_model.staging./`, and the legacy `final_model.staging/` only when it contains a `staging_run_id` file whose content equals ``. ForgeLM does not write that marker, so remove an unmarked legacy staging directory yourself. | +| `artefacts` | `/compliance/*.json` files whose name embeds `` as a token (delimited by `_`, `-` or `.`). | -`--kind logs` is **intentionally absent**: audit logs are append-only Article 17(3)(b) records and are not deleted by the tool. +> **`--kind artefacts` does not find the standard compliance bundle.** The trainer and exporter write fixed basenames (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`, and `risk_assessment.json` / `annex_iv_metadata.json` when configured), none of which contains a run id. On a normal output directory the command therefore finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`) and exits 1; it deletes nothing. It acts only on files you have named with the run id yourself. Until the resolver is fixed, remove the bundle through your storage layer. + +`--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it. Whether entries that name a data subject may stay is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires keeping them, and you have to be able to name that obligation. ### Retention-policy report (`--check-policy`) diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 359a8561..4ee952ad 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -14,7 +14,7 @@ This page is the operator quick-reference. The deeper deployer-flow walkthrough | Right | Subcommand | What it answers | |---|---|---| | **Article 15** — right of access | `forgelm reverse-pii` | "Do you hold any record of me?" — scans JSONL corpora for the subject's identifier. | -| **Article 17** — right to erasure | `forgelm purge` | "Delete every record of me." — atomically removes a row, a staging directory, or a compliance bundle. | +| **Article 17** — right to erasure | `forgelm purge` | "Delete every record of me." — atomically removes a row or a staging directory, and compliance files whose names embed the run id. | Both subcommands hash the subject's identifier with the **same** per-output-dir salt (`/.forgelm_audit_salt`) before recording the audit row. The `query_hash` field on `data.access_request_query` and the `target_id` field on `data.erasure_*` are byte-identical for the same subject, which lets a reviewer correlate the two without cleartext. @@ -49,7 +49,8 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ ### What it does NOT do - **Re-train models.** Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` so the gap is visible. -- **Delete the audit log.** Article 17(3)(b) preserves audit / accounting records as a legal-obligation defence; the tool records the erasure as a new event rather than rewriting history. +- **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history. Whether entries that name the data subject may be kept is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. +- **Find the standard compliance bundle with `--kind artefacts`.** The trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, and the resolver matches only filenames that embed it. On a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing; remove the bundle through your storage layer. - **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (Article 17(2)). The `data.erasure_warning_external_copies` event is the explicit reminder. ## Article 15 — `forgelm reverse-pii` diff --git a/docs/usermanuals/en/reference/configuration.md b/docs/usermanuals/en/reference/configuration.md index ae2a455b..f997f809 100644 --- a/docs/usermanuals/en/reference/configuration.md +++ b/docs/usermanuals/en/reference/configuration.md @@ -359,8 +359,8 @@ defaults was not recorded. ```yaml retention: - audit_log_retention_days: 1825 # 5 years default (Article 5(1)(e)) - staging_ttl_days: 7 # days to retain staging model after forgelm reject + audit_log_retention_days: 1825 # 5 years: ForgeLM default (project choice; GDPR Article 5(1)(e) sets no fixed period) + staging_ttl_days: 7 # staging dir age that purge --check-policy reports as overdue ephemeral_artefact_retention_days: 90 # compliance bundles, audit reports raw_documents_retention_days: 90 # ingested PDF/DOCX/EPUB/TXT/Markdown enforce: "log_only" # reserved: no command acts on it yet (log_only | warn_on_excess | block_on_excess) diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 77e6b09a..04862ba4 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -14,7 +14,7 @@ Bu sayfa operatör hızlı-referansıdır. Daha derin deployer akışı [GitHub' | Hak | Subcommand | Ne yanıtlar | |---|---|---| | **Madde 15** — erişim hakkı | `forgelm reverse-pii` | "Hakkımda kayıt tutuyor musunuz?" — JSONL corpus'ları subject identifier'ı için tarar. | -| **Madde 17** — silinme hakkı | `forgelm purge` | "Hakkımdaki her kaydı silin." — bir satırı, bir staging dizinini veya bir compliance bundle'ını atomik olarak siler. | +| **Madde 17** — silinme hakkı | `forgelm purge` | "Hakkımdaki her kaydı silin." — bir satırı veya bir staging dizinini, ayrıca adı run id'yi embed eden compliance dosyalarını atomik olarak siler. | İki subcommand da audit satırını kaydetmeden önce subject identifier'ını **aynı** per-output-dir salt (`/.forgelm_audit_salt`) ile hash'ler. `data.access_request_query` üzerindeki `query_hash` ve `data.erasure_*` üzerindeki `target_id` aynı subject için byte-byte aynıdır; reviewer bu sayede ikisini cleartext olmadan korele edebilir. @@ -49,7 +49,8 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( ### Neyi YAPMAZ - **Modelleri yeniden eğitmez.** Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözümdür. Tool `data.erasure_warning_memorisation` yayar; böylece boşluk görünür olur. -- **Audit log'unu silmez.** Madde 17(3)(b) audit / muhasebe kayıtlarını hukuki yükümlülük savunması olarak korur; tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. +- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +- **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca onu embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. - **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. ## Madde 15 — `forgelm reverse-pii` diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 753f164c..128b8c80 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -367,8 +367,8 @@ kaynağı kaydedilmemiştir. ```yaml retention: - audit_log_retention_days: 1825 # varsayılan 5 yıl (Madde 5(1)(e)) - staging_ttl_days: 7 # forgelm reject sonrası staging modelini saklama süresi + audit_log_retention_days: 1825 # 5 yıl: ForgeLM varsayılanı (proje tercihi; GDPR Madde 5(1)(e) sabit süre koymaz) + staging_ttl_days: 7 # purge --check-policy'nin geciken saydığı staging dizin yaşı ephemeral_artefact_retention_days: 90 # uyumluluk paketleri, denetim raporları raw_documents_retention_days: 90 # ingest edilen PDF/DOCX/EPUB/TXT/Markdown enforce: "log_only" # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only | warn_on_excess | block_on_excess) diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 4bd75f23..e3436be5 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1129,7 +1129,7 @@ def _add_purge_subcommand(subparsers) -> None: type=str, default=None, choices=["staging", "artefacts"], - help="Run-scoped erasure target: staging directory OR compliance bundle. `logs` is intentionally absent (audit logs are append-only Article 17(3)(b)).", + help="Run-scoped erasure target: staging directory OR compliance files whose names embed the run id. `logs` is intentionally absent (the audit log is append-only and never deleted by this command).", ) p.add_argument( "--row-matches", diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 97f139aa..3c785f6a 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -88,7 +88,7 @@ _ROW_ID_KEYS: Tuple[str, ...] = ("id", "row_id") # Allowed --kind values per design §4.1. ``logs`` is intentionally -# absent (audit logs are append-only Article 17(3)(b) and never deleted +# absent (the audit log is append-only and never deleted # automatically — the trainer + the tool record the request, the # operator handles the residual disposal manually). _VALID_RUN_KINDS: Tuple[str, ...] = ("staging", "artefacts") diff --git a/forgelm/config.py b/forgelm/config.py index 27f644ca..48677acc 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1588,14 +1588,14 @@ class RetentionConfig(BaseModel): erasure horizons. Top-level retention block (per Phase 20 design `gdpr-erasure-design` - §3 + closure-plan §15.5 v2.5). All four horizons default to values - chosen to be compliant out of the box for typical enterprise EU AI - Act use: audit logs at 5 years (Article 12 record-keeping - obligation × statute-of-limitations buffer), staging at 7 days + §3 + closure-plan §15.5 v2.5). All four horizons default to + ForgeLM's own choices, not legal periods (GDPR Article 5(1)(e) sets + no fixed period): audit logs at 5 years, staging at 7 days (operator gets one work-week to act on a `forgelm reject`), ephemeral artefacts at 90 days (compliance bundle + audit reports have a quarterly review cadence in most QMS), raw documents at 90 days (typical ingestion-window before re-running data audit). + Choose values from your own necessity assessment. Setting any horizon to ``0`` disables the policy for that artefact kind (retain indefinitely). The horizons are *reported*, not enforced: @@ -1612,15 +1612,15 @@ class RetentionConfig(BaseModel): default=1825, ge=0, description=( - "Days to retain `audit_log.jsonl` before flagging it as overdue under Article 5(1)(e). " - "Default 1825 = 5 years. Set to 0 to retain indefinitely (Article 17(3)(b) defence)." + "Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). " + "Default 1825 = 5 years, a ForgeLM choice, not a legal period. Set to 0 to retain indefinitely; name the Union or Member State obligation you rely on." ), ) staging_ttl_days: int = Field( default=7, ge=0, description=( - "Days to retain `final_model.staging./` after a `forgelm reject` decision before scheduled cleanup. " + "Days after which `forgelm purge --check-policy` reports a `final_model.staging./` directory as overdue (aged from the run's first audit event, else mtime); nothing is deleted. " "Set to 0 to retain indefinitely. This is the canonical staging-retention field; " "the legacy `evaluation.staging_ttl_days` alias was removed in v0.8.0." ), diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index 06c27775..22a8b919 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -298,6 +298,47 @@ def test_artefacts_kind_removes_compliance_bundle_for_run(self, tmp_path: Path) # Other run's bundle is untouched. assert (compliance_dir / "compliance_other-run.json").exists() + def test_artefacts_kind_finds_nothing_in_the_standard_fixed_name_bundle(self, tmp_path: Path) -> None: + # CR-A10-002 / DOC-D1B05-005: the exporter writes fixed basenames that + # carry no run id, so ``--kind artefacts`` cannot find the real bundle. + # It fails safe (deletes nothing, exits 1). docs/reference/purge_subcommand.md + # warns about exactly this; when the resolver learns to prove ownership + # this test and that warning change together. + from forgelm.cli.subcommands._purge import _run_purge_cmd + + run_id = "fg-abc123def456" + compliance_dir = tmp_path / "compliance" + compliance_dir.mkdir(parents=True) + bundle = ( + "compliance_report.json", + "training_manifest.yaml", + "data_provenance.json", + "data_governance_report.json", + "annex_iv_metadata.json", + ) + for name in bundle: + (compliance_dir / name).write_text("{}") + + args = _build_args(run_id=run_id, kind="artefacts", output_dir=str(tmp_path)) + with pytest.raises(SystemExit) as ei: + _run_purge_cmd(args, output_format="json") + assert ei.value.code == 1 + assert all((compliance_dir / name).exists() for name in bundle) + failed = [e for e in _read_audit_events(tmp_path / "audit_log.jsonl") if e["event"] == "data.erasure_failed"] + assert [e["error_class"] for e in failed] == ["NoMatchingArtefacts"] + + def test_no_forgelm_writer_creates_the_legacy_staging_marker(self) -> None: + # The docs tell operators to remove an unmarked legacy ``final_model.staging/`` + # themselves because ForgeLM never writes ``staging_run_id``. A writer + # would make that advice stale. + forgelm_dir = Path(__file__).resolve().parent.parent / "forgelm" + writers = [ + str(path.relative_to(forgelm_dir)) + for path in sorted(forgelm_dir.rglob("*.py")) + if "staging_run_id" in path.read_text(encoding="utf-8") + ] + assert writers == [os.path.join("cli", "subcommands", "_purge.py")] + # --------------------------------------------------------------------------- # §7 Test 5 — Audit chain post-erasure still verifies diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 14515fcf..59fa912b 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -54,3 +54,76 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: assert "discard staging" not in text # TR mirror used "staging'i at" ("discard staging"). assert "staging'i at" not in text + + +# --------------------------------------------------------------------------- +# GDPR citations and erasure behaviour (issue #101) +# --------------------------------------------------------------------------- +# +# Article 17(3)(b) disapplies erasure only to the extent a specific Union or +# Member State legal obligation requires the processing; it does not preserve +# an audit log as a class. The retention defaults are ForgeLM choices, not +# legal periods. These pages and code strings used to say otherwise. + +_GDPR_CLAIM_FILES = ( + "forgelm/config.py", + "forgelm/cli/_parser.py", + "forgelm/cli/subcommands/_purge.py", + "config_template.yaml", + "docs/design/gdpr_erasure.md", + "docs/guides/gdpr_erasure.md", + "docs/guides/gdpr_erasure-tr.md", + "docs/reference/configuration.md", + "docs/reference/configuration-tr.md", + "docs/reference/purge_subcommand.md", + "docs/reference/purge_subcommand-tr.md", + "docs/usermanuals/en/compliance/gdpr-erasure.md", + "docs/usermanuals/tr/compliance/gdpr-erasure.md", + "docs/usermanuals/en/reference/configuration.md", + "docs/usermanuals/tr/reference/configuration.md", +) + +_RETIRED_GDPR_CLAIMS = ( + r"17\(3\)\(b\) preserves", + r"17\(3\)\(b\) (?:exemption|defence)", + r"17\(3\)\(b\) records", + r"17\(3\)\(b\) savunması", + r"17\(3\)\(b\) audit / muhasebe", + r"17\(3\)\(b\) kayıt", + r"append-only (?:Article|Madde) 17\(3\)\(b\)", + r"5 years default \(Article 5\(1\)\(e\)\)", + r"varsayılan 5 yıl \(Madde 5\(1\)\(e\)\)", + r"legal-obligation record under", + r"compliant out of the box", + r"Article 12 record-keeping baseline", + r"Horizons honour GDPR", + r"removed by default after a defined retention horizon", + r"after a `forgelm reject` decision before scheduled cleanup", +) + +_PURGE_ARTEFACT_WARNING_FILES = ( + "docs/guides/gdpr_erasure.md", + "docs/guides/gdpr_erasure-tr.md", + "docs/reference/purge_subcommand.md", + "docs/reference/purge_subcommand-tr.md", + "docs/usermanuals/en/compliance/gdpr-erasure.md", + "docs/usermanuals/tr/compliance/gdpr-erasure.md", +) + + +@pytest.mark.parametrize("rel", _GDPR_CLAIM_FILES) +def test_retired_gdpr_claims_stay_out(rel: str) -> None: + text = (_REPO_ROOT / rel).read_text(encoding="utf-8") + found = [pattern for pattern in _RETIRED_GDPR_CLAIMS if re.search(pattern, text)] + assert found == [], f"{rel} still carries retired GDPR wording: {found}" + + +@pytest.mark.parametrize("rel", _PURGE_ARTEFACT_WARNING_FILES) +def test_purge_docs_warn_that_artefacts_misses_the_standard_bundle(rel: str) -> None: + # CR-A10-002 / DOC-D1B05-005: the exporter writes fixed basenames and the + # resolver matches filenames that embed the run id, so the documented + # ``--kind artefacts`` finds nothing on a real output directory. The + # behaviour is pinned in tests/test_gdpr_erasure.py; the docs must say so. + text = (_REPO_ROOT / rel).read_text(encoding="utf-8") + assert "NoMatchingArtefacts" in text, f"{rel} must warn that --kind artefacts finds nothing" + assert "compliance_report.json" in text, f"{rel} must name the fixed-basename bundle" From 810c93153b76d7ece20fb60b76b7202f12ac97da Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:40:03 +0300 Subject: [PATCH 02/25] =?UTF-8?q?docs(gdpr):=20review=20round=201=20?= =?UTF-8?q?=E2=80=94=20finish=20the=20retention=20and=20erasure=20wording?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-ups from the first review round of the Article 17(3)(b) / retention rewrite: - The guide's `retention:` example still carried the retired comments ("Article 12 record-keeping", "one work-week to act on a forgelm reject"); the user-manual human-oversight page and the change-management SOP still said staging_ttl_days "physically prunes" a directory or that it stays "until retention expires". Nothing deletes on age; `purge --check-policy` only reports. - `purge --kind artefacts` removes any file directly under compliance/ whose name contains the run id (not only .json); the reference said .json only. - Run-scoped deletion is not atomic (only row erasure is); the reference and user-manual intros no longer say so, and the exit-code table lists the no-match exit 1. - Keeping audit-log entries needs a documented basis (still necessary for the purpose, or an Article 17(3) exception); 17(3)(b) is one example, not the only route. The AI Act anchors named are Art. 18(1) (documentation, 10 years) and Art. 19(1) / 26(6) (system logs), "as applicable". - Art. 12(3) is extendable by two further months; Turkish "without undue delay" uses the GDPR term. - The design note's `--kind artefacts` allow-list and retention enforcement are marked as specified-not-shipped, with a deviations list in section 12; the purge help text and docstrings no longer promise a "compliance bundle". - Staging age fallback order matches the code (run's first event, log's first event, mtime); the legacy staging directory is reported too. - Tests: the retired-phrase list is whitespace-tolerant and case-insensitive and covers the new files; the legacy-marker test is AST-based and asserts the reader stays read-only. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 35 ++++++++++------- docs/guides/gdpr_erasure-tr.md | 8 ++-- docs/guides/gdpr_erasure.md | 8 ++-- docs/qms/sop_change_management-tr.md | 4 +- docs/qms/sop_change_management.md | 3 +- docs/reference/configuration-tr.md | 16 ++++---- docs/reference/configuration.md | 14 +++---- docs/reference/purge_subcommand-tr.md | 8 ++-- docs/reference/purge_subcommand.md | 8 ++-- .../usermanuals/en/compliance/gdpr-erasure.md | 4 +- .../en/compliance/human-oversight.md | 2 +- .../usermanuals/tr/compliance/gdpr-erasure.md | 4 +- .../tr/compliance/human-oversight.md | 2 +- forgelm/cli/_parser.py | 4 +- forgelm/cli/subcommands/_purge.py | 2 +- forgelm/config.py | 6 +-- tests/test_gdpr_erasure.py | 32 ++++++++++----- .../test_usermanual_reference_consistency.py | 39 +++++++++++++------ 18 files changed, 119 insertions(+), 80 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 3a0e620a..95b901d8 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -45,9 +45,9 @@ Phase 21 closes the gap with a real implementation. This Phase 20 document spec **Article 17(2)** requires the controller to inform downstream processors / publishers of the erasure when the data has been disclosed. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. -**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation or claim, and the controller has to name it: no provision of EU law protects a training audit log merely because ForgeLM produces it. +**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to name it: no provision of EU law protects a training audit log merely because ForgeLM produces it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis. -The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability), and ForgeLM never edits or deletes an entry. Whether the entries that reference the now-erased subject may stay is the controller's decision: Art. 17(3)(b) and (3)(e) cover them only to the extent a specific obligation or claim requires it, and the controller documents which one. +The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability), and ForgeLM never edits or deletes an entry. Whether the entries that reference the now-erased subject may stay is the controller's decision, and the controller documents why: they are still necessary for their purpose, or an Art. 17(3) exception applies (for (3)(b) and (3)(e), only to the extent a specific obligation or claim requires it). --- @@ -56,10 +56,10 @@ The take-away: erasure operates on **training data + derived artefacts**, not on | Artefact | Where it lives | Carries personal data? | Erasure strategy | |---|---|---|---| | **Training corpus rows** | `.jsonl` (operator-supplied) | Yes — the whole point | **Delete row in place** + audit event | -| **Audit log entries** | `/audit_log.jsonl` (append-only) | Indirectly (operator name, run config, possibly file paths that name a subject) | **Never delete original** — append a `data.erasure_completed` event that references the deleted row's `row_id`. The tool never rewrites the chain; keeping the entries is justified only to the extent Art. 17(3)(b) or (3)(e) applies, which the controller documents. | +| **Audit log entries** | `/audit_log.jsonl` (append-only) | Indirectly (operator name, run config, possibly file paths that name a subject) | **Never delete original** — append a `data.erasure_completed` event that references the deleted row's `row_id`. The tool never rewrites the chain; why the entries may be kept (still necessary for their purpose, or an Art. 17(3) exception) is the controller's documented decision. | | **Model adapters** | `/final_model/` (PEFT adapter + tokenizer) | **Memorised** training samples — extraction risk for high-overfit models | **Document escalation**: full retraining is the only proper resolution. Tool emits `data.erasure_warning_memorisation` audit event flagging the gap. | | **Final model staging** | `/final_model.staging./` | Same memorisation risk | Operator chooses: delete staging (pre-promotion) or treat as adapters above | -| **Compliance artefacts** | `/compliance/*.{json,yaml}` | References to corpus paths (could disclose subject names via filename) | **Regenerate** after erasure (or `forgelm purge --kind artefacts` rewrites them) | +| **Compliance artefacts** | `/compliance/*.{json,yaml}` | References to corpus paths (could disclose subject names via filename) | **Regenerate** after erasure (`forgelm purge --kind artefacts` is designed to remove them, but as shipped it does not find the standard bundle — §12) | | **Data audit reports** | `/data_audit_report.json` | `lang_sample` field carries up-to-512-char snippets from random rows (could include subject text) | **Regenerate** post-erasure (or hand-edit the snippets — the report is operator-facing only) | | **Synthetic / GRPO logs** | `/synthetic_data.jsonl`, GRPO completions | Yes if generated from a prompt that named the subject | **Delete row** with the same row-id mechanism as training corpus | | **Webhook history** | (external system; ForgeLM does not store) | Possibly | Out of scope — operator handles in the receiving system | @@ -72,6 +72,8 @@ The four artefact kinds the operator can act on in-tool: `corpus_rows`, `staging Article 5(1)(e) requires personal data to be kept "no longer than is necessary". ForgeLM ships a config block so the operator can declare their retention horizon and the tool can warn / refuse runs that violate it. +> **As shipped:** the horizons are only *reported* by `forgelm purge --check-policy` (read-only, always exit 0). `retention.enforce` is reserved: no command reads it, so nothing warns, refuses or deletes on age. Ages come from the audit log first (the run's first event, else the log's first event) and fall back to `mtime`. Read the warn / refuse and `mtime` passages below as the original design; §12 lists the deviations. + ### 3.1 Prior-state acknowledgement (audited 2026-05-02) Two pre-existing surfaces overlap with this design and must be resolved up front so Phase 21 does not ship a second source of truth: @@ -164,9 +166,9 @@ Three sites: For each retention horizon: - Compare artefact `mtime` against `now - _days`. - If older, the artefact has overstayed the declared lawful basis. -- The audit log itself is **never deleted automatically** even when the policy says it has overstayed. The tool only flags; the operator makes the deletion call by hand, after checking that no specific Union or Member State obligation (Art. 17(3)(b)) or legal claim (Art. 17(3)(e)) still requires the entries. +- The audit log itself is **never deleted automatically** even when the policy says it has overstayed. The tool only flags; the operator makes the deletion call by hand, after checking whether the entries are still necessary for their purpose or a specific Union or Member State obligation (Art. 17(3)(b)) or legal claim (Art. 17(3)(e)) still requires them. -The compliance artefacts (`compliance/*.json`) are tied to a specific run; their lifetime is the run's lifetime, not a separate horizon. In-place deletion is supported via `forgelm purge --run-id --kind artefacts` (§4.1) which is meant to remove `/compliance/*.json` for the named run while leaving the audit log + final model intact (as shipped, the resolver only matches filenames that embed the run id, and the standard bundle uses fixed basenames, so it finds nothing — see the purge reference) — useful when an operator wants to regenerate the bundle after an Article 17 corpus erasure changed the lineage. Whole-`` deletion remains the operator's manual escalation when the entire run is being scrubbed. +The compliance artefacts (`compliance/*.json`) are tied to a specific run; their lifetime is the run's lifetime, not a separate horizon. In-place deletion is specified via `forgelm purge --run-id --kind artefacts` (§4.1) which is meant to remove `/compliance/*.json` for the named run while leaving the audit log + final model intact (as shipped, the resolver only matches filenames that embed the run id, and the standard bundle uses fixed basenames, so it finds nothing — see the purge reference) — useful when an operator wants to regenerate the bundle after an Article 17 corpus erasure changed the lineage. Whole-`` deletion remains the operator's manual escalation when the entire run is being scrubbed. --- @@ -190,7 +192,7 @@ A single invocation does exactly one of the three. Combining flags is a `Config | `--corpus ` | corpus mode | str | Path to **a single JSONL file**. Directory mode is **rejected** in `--row-id` mode — multi-file purges are an operator script (loop over files), not a `forgelm purge` flag, because GDPR Article 17 expects an erasure decision *per row* with its own audit event. An invocation that matches the same id in two files is a "DELETE without WHERE" hazard. | | `--row-matches` | corpus mode optional | enum | `one` (default; refuses if 0 or ≥2 matches in the file) / `all` (deletes every match, requires explicit opt-in for the ambiguity). Without this flag, multi-row matches in a single file abort with `EXIT_CONFIG_ERROR` so the operator confirms intent before bulk delete. | | `--run-id ` | run mode | str | Run id matching `/audit_log.jsonl`'s top-level `run_id`. | -| `--kind {staging,artefacts}` | run mode | enum | Which derived artefacts to erase. `staging` = `final_model.staging.*`; `artefacts` = `compliance/`, `data_audit_report.json`, derived JSONL snapshots. | +| `--kind {staging,artefacts}` | run mode | enum | Which derived artefacts to erase. `staging` = `final_model.staging.*`; `artefacts` = `compliance/`, `data_audit_report.json`, derived JSONL snapshots (design intent; what shipped is in §12). | | `--output-dir ` | run mode | str | Where to look for the run. Default `./` (current dir). | | `--check-policy` | policy mode | bool | Dry-run scan; reports violations against the loaded config's `retention` block. | | `--config ` | policy mode (optional) | str | Config to load for the retention block; defaults to `./forgelm.yaml` then walks up. | @@ -230,7 +232,7 @@ Recovery path for a step-5 failure (rename fails after the temp file is fully wr **Run-scoped erasure** for `kind=staging` uses `shutil.rmtree(staging_path)` after a `os.path.commonpath` check (same defence-in-depth as `_approve.py`'s staging path validator) so an attacker who can rewrite the audit log cannot point us at `/etc`. -**Run-scoped erasure** for `kind=artefacts` deletes a fixed allow-list (`compliance/`, `data_audit_report.json`, `data_governance_report.json`) — never a glob. Allow-list lives in `forgelm.compliance` next to the artefact emitter. +**Run-scoped erasure** for `kind=artefacts` is designed to delete a fixed allow-list (`compliance/`, `data_audit_report.json`, `data_governance_report.json`) — never a glob — kept in `forgelm.compliance` next to the artefact emitter. That allow-list does not exist yet; §12 describes what shipped. ### 4.5 What `forgelm purge` does NOT do @@ -239,7 +241,7 @@ Recovery path for a step-5 failure (rename fails after the temp file is fully wr | **Re-train the model after corpus erasure** | Operator decision; ForgeLM cannot infer the right downstream pipeline. | | **Distributed corpus deletion** | Single-node only. Multi-node corpus shards live on separate filesystems; the operator runs `forgelm purge --row-id` on each. | | **Webhook receiver cleanup** | The Slack / Teams thread that received `notify_awaiting_approval` is in the operator's chat platform, not in ForgeLM. We log a notice. | -| **Audit log rewriting** | The chain is append-only by construction. Erasure adds an event; never edits an entry. Whether the entries may be kept is the controller's call: Art. 17(3)(b) and (3)(e) apply only to the extent a specific obligation or claim requires it. | +| **Audit log rewriting** | The chain is append-only by construction. Erasure adds an event; never edits an entry. Whether the entries may be kept is the controller's documented call: still necessary for their purpose, or an Art. 17(3) exception such as (3)(b) / (3)(e), which applies only to the extent a specific obligation or claim requires it. | | **Model weight forgetting** | "Machine unlearning" is an active research area; we do not ship a half-baked impl. We emit `data.erasure_warning_memorisation` so the operator knows the gap is theirs to close. | --- @@ -322,8 +324,9 @@ in three scopes: 1. **Training corpus row erasure.** `forgelm purge --row-id --corpus ` atomically rewrites the JSONL, dropping the matched row. 2. **Run-scoped artefact erasure.** `forgelm purge --run-id - --kind {staging,artefacts}` deletes the model staging directory or the - compliance bundle for a specific run. + --kind {staging,artefacts}` deletes the model staging directory for a + specific run and the compliance files whose names embed the run id (as + shipped, not the standard bundle; see the purge reference). 3. **Retention policy enforcement.** Configure `retention.*` in your YAML; `forgelm purge --check-policy` lists artefacts that have overstayed the declared horizon. @@ -375,7 +378,7 @@ Phase 21 ships **at least 7 tests** covering the acceptance surface. Every test | 1 | `test_row_erasure_rewrites_jsonl_atomically` | Row matching `row_id=42` removed from `train.jsonl`; other 99 rows byte-identical to pre-erasure. | | 2 | `test_row_erasure_emits_audit_events` | Two new audit events landed: `data.erasure_requested` then `data.erasure_completed`, in that order, with matching `target_id` field. | | 3 | `test_run_scoped_staging_erasure` | `--run-id fg-abc --kind staging` removes `final_model.staging.fg-abc/` recursively; final dir untouched if it exists. | -| 4 | `test_run_scoped_artefacts_erasure` | `--run-id fg-abc --kind artefacts` removes the `compliance/` allow-list; `final_model/` untouched. | +| 4 | `test_run_scoped_artefacts_erasure` | `--run-id fg-abc --kind artefacts` removes `compliance/` files named for the run; `final_model/` untouched (the standard fixed-basename bundle is not found, pinned separately). | | 5 | `test_audit_chain_verifies_post_erasure` | After erasure, `verify_audit_log(path).valid is True`; chain length grew by 2 events. | | 6 | `test_check_policy_reports_violations` | `--check-policy` against a fixture with overstayed artefacts returns the correct violation list (text + JSON). | | 7 | `test_unknown_row_id_clear_error` | Unknown `--row-id` exits 1 with a message that names the file scanned and the id searched. | @@ -465,10 +468,16 @@ The closure plan §15.5 lists three open items that intersect Phase 21: ## 12. Implementation status -The design above shipped end-to-end in v0.5.5: +The design above shipped in v0.5.5, with the deviations listed after this list: - **`RetentionConfig`** (§3): pinned in `forgelm/config.py` as a top-level `retention:` block with five fields (`audit_log_retention_days`, `staging_ttl_days`, `ephemeral_artefact_retention_days`, `raw_documents_retention_days`, `enforce`). `EvaluationConfig.staging_ttl_days` deprecated with `DeprecationWarning` + alias-forwarding to `retention.staging_ttl_days`. - **`forgelm purge` subcommand** (§4): pinned in `forgelm/cli/subcommands/_purge.py` with `--row-id`, `--corpus`, `--run-id`, `--kind`, `--check-policy`, atomic-rewrite + recovery semantics. - **Audit events** (§5): six new events live in `audit_event_catalog.md` — `data.erasure_requested`, `data.erasure_completed`, `data.erasure_failed`, plus three operator-warning events. - **Tests** (§7): `tests/test_gdpr_erasure.py` ships the regression coverage (unknown row id, multi-match refusal, partial-commit recovery, retention horizon enforcement, audit-PII redaction, etc.). - **User-facing docs** (§6, §8): `docs/guides/gdpr_erasure{,-tr}.md` + `docs/usermanuals/{en,tr}/compliance/gdpr-erasure.md` reference the implemented surface. + +**Deviations from the design, as shipped today:** + +- `--kind artefacts` matches `compliance/` filenames that embed the run id as a token. The exporter writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...), so on a standard output directory the command finds nothing: it fails safe (nothing is deleted, `data.erasure_failed` with `NoMatchingArtefacts`, exit 1). The fixed allow-list with an ownership proof that §4.4 describes is open work. +- Retention horizons are reported by `forgelm purge --check-policy` and never enforced; `retention.enforce` is reserved and no command reads it. +- The legacy `final_model.staging/` is purged only when it holds a `staging_run_id` marker equal to the run id. Nothing in ForgeLM writes that marker. diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 0be6ad4f..960693c2 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -20,7 +20,7 @@ **Yapmaz:** - Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. -- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır (örneğin sisteminiz için geçerli hale geldiğinde AI Act Md. 19(1) ve 26(6)'daki log saklama yükümlülükleri) ve bu yükümlülüğü adlandırabilmeniz gerekir. +- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır (log teknik dokümantasyonunuzun parçasıysa AI Act Md. 18(1)'in 10 yıllık dokümantasyon saklaması ya da devreye alınmış yüksek riskli bir sistemin ürettiği loglar için Md. 19(1) ve 26(6)'nın log saklama yükümlülükleri gibi) ve bu yükümlülüğü adlandırabilmeniz gerekir. - Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. @@ -57,7 +57,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` - `--kind staging` — `/final_model.staging./`'i siler; legacy `final_model.staging/`'i yalnızca içeriği ``'ye eşit bir `staging_run_id` işaretçisi taşıyorsa siler (ForgeLM bu işaretçiyi yazmaz; aksi halde legacy dizini kendiniz silin). -- `--kind artefacts` — ``'i adında token olarak embed eden `/compliance/*.json` dosyalarını siler. **Standart paketi bulamaz:** trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...), bu yüzden normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. +- `--kind artefacts` — `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosyayı siler (yalnızca `.json` değil). **Standart paketi bulamaz:** trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...), bu yüzden normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. - `--kind logs` **bilinçli olarak yok**: audit log append-only'dir ve tool onu asla silmez (girdilerinin tutulup tutulamayacağına kimin karar verdiği için yukarıdaki ilk listeye bakın). ### 3. Retention policy raporu @@ -74,8 +74,8 @@ Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical ```yaml retention: - audit_log_retention_days: 1825 # 5 yıl (Madde 12 kayıt-tutma) - staging_ttl_days: 7 # `forgelm reject` üzerine aksiyon almak için bir iş haftası + audit_log_retention_days: 1825 # 5 yıl (ForgeLM varsayılanı, yasal süre değildir) + staging_ttl_days: 7 # purge --check-policy'nin staging dizinini geciken saydığı yaş ephemeral_artefact_retention_days: 90 # üç-aylık review cadence'i raw_documents_retention_days: 90 # data audit yeniden çalıştırmadan önce ingestion penceresi enforce: log_only # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess) diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index b75caf4c..4b298640 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -20,7 +20,7 @@ **Does not:** - Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. -- Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history. Whether entries that reference the data subject may be kept is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires the processing (for example the log-retention duties of AI Act Art. 19(1) and Art. 26(6), once they apply to your system), and you have to be able to name that obligation. +- Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history. Whether entries that reference the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing (such as the 10-year documentation keeping of AI Act Art. 18(1) where the log forms part of your technical documentation, or the log-retention duties of Art. 19(1) and Art. 26(6) for logs a deployed high-risk system generates), and you have to be able to name that obligation. - Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. @@ -57,7 +57,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs ``` - `--kind staging` removes `/final_model.staging./`, and the legacy `final_model.staging/` only if it carries a `staging_run_id` marker whose content equals the `` (ForgeLM does not write that marker; otherwise remove the legacy directory yourself). -- `--kind artefacts` removes `/compliance/*.json` files whose name embeds the `` as a token. **It does not find the standard bundle:** the trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, so on a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. Remove the bundle through your storage layer until the resolver is fixed. +- `--kind artefacts` removes any file directly under `/compliance/` whose name contains the `` bounded by `_`, `-`, `.` or the start/end of the name (not only `.json`). **It does not find the standard bundle:** the trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, so on a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. Remove the bundle through your storage layer until the resolver is fixed. - `--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it (see the first list above for who decides whether its entries may be kept). ### 3. Retention-policy report @@ -74,8 +74,8 @@ Read-only scan. Walks the output directory, compares each artefact's age (canoni ```yaml retention: - audit_log_retention_days: 1825 # 5 years (Article 12 record-keeping) - staging_ttl_days: 7 # one work-week to act on a `forgelm reject` + audit_log_retention_days: 1825 # 5 years (ForgeLM default, not a legal period) + staging_ttl_days: 7 # age at which `purge --check-policy` reports a staging dir as overdue ephemeral_artefact_retention_days: 90 # quarterly review cadence raw_documents_retention_days: 90 # ingestion-window before re-running data audit enforce: log_only # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess) diff --git a/docs/qms/sop_change_management-tr.md b/docs/qms/sop_change_management-tr.md index 8a3d1a4b..76dfc97a 100644 --- a/docs/qms/sop_change_management-tr.md +++ b/docs/qms/sop_change_management-tr.md @@ -106,8 +106,8 @@ CAB'sıdır: atomik rename ile `/final_model/`'e promote olur. 5. Reddedilirse `human_approval.rejected` event yangın eder; model forensic inceleme için `final_model.staging./`'de kalır, - retention süresi dolana kadar (`retention.staging_ttl_days` — - config'te). + biri onu silene kadar (`forgelm purge --check-policy`, onu + `retention.staging_ttl_days`'ten eski olduğunda raporlar). Bu, operatöre SOC 2 CC8.1 kanıtı verir: onay kapısı etkinleştirildiğinde her model promotion kaydedilen onaylayıcı kimliğine atfedilir ve forensic olarak kayıtlıdır. Eğitici ile onaylayıcıyı diff --git a/docs/qms/sop_change_management.md b/docs/qms/sop_change_management.md index cffc7820..e96e9b42 100644 --- a/docs/qms/sop_change_management.md +++ b/docs/qms/sop_change_management.md @@ -102,7 +102,8 @@ in-pipeline CAB: `/final_model/` via atomic rename. 5. If rejected, `human_approval.rejected` event fires; model stays in `final_model.staging./` for forensic review until - retention expires (`retention.staging_ttl_days` in the config). + someone removes it (`forgelm purge --check-policy` reports it once it + is older than `retention.staging_ttl_days`). This gives a deployer SOC 2 CC8.1 evidence: when the approval gate is enabled, every model promotion is attributed to the recorded approver identity and forensically recorded. diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index 879bbd39..f16cc1e1 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -269,20 +269,20 @@ training: ## `retention` (İsteğe bağlı — GDPR Madde 17 silme ufukları) Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarını -belirler. Ufuklar, GDPR Madde 5(1)(e) "saklama sınırlaması"nı (sabit süre yok: -veri yalnızca gerektiği kadar tutulabilir) uygulamaya yardımcı olan ve Madde 17 -kapsamındaki silme işlemlerini destekleyen operatör tarafından belirlenen saklama -sınırlarıdır (talepler gecikmeksizin, Madde 12(3)'teki bir aylık süre içinde -yanıtlanır). Değerleri kendi gereklilik değerlendirmenizden seçin; varsayılanlar -ForgeLM tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: +belirler. Ufuklar, operatörün belirlediği saklama sınırlarıdır. GDPR Madde 5(1)(e) +"saklama sınırlaması"nın (sabit süre yok: veri yalnızca gerektiği kadar tutulabilir) +uygulanmasına yardımcı olur ve Madde 17 kapsamındaki silme işlemlerini destekler +(talepler gereksiz gecikme olmaksızın, Madde 12(3)'teki bir aylık süre içinde, iki ay daha uzatılabilir şekilde yanıtlanır). +Değerleri kendi gereklilik değerlendirmenizden seçin; varsayılanlar ForgeLM +tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: `forgelm purge --check-policy` ufku aşanları listeler ve her zaman 0 ile çıkar; hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (AI Act'in log saklama yükümlülükleri Md. 19(1) / Md. 26(6), en az altı ay). `0` süresiz saklamayı belirtir; dayandığınız AB veya üye devlet yükümlülüğünü adlandırın. | -| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini "geciken" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | +| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (duruma göre AI Act Md. 18(1), dokümantasyon 10 yıl, ve Md. 19(1) / 26(6), sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | +| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve legacy `final_model.staging/`'i) "geciken" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | | `ephemeral_artefact_retention_days` | int | `90` | Uyumluluk paketleri, veri denetim raporları ve diğer çalışma kapsamlı türetilmiş artefaktların saklanma süresi (gün). `0` süresiz saklama. | | `raw_documents_retention_days` | int | `90` | İngest edilmiş ham belgelerin (PDF / DOCX / EPUB / TXT / Markdown) operatörün ingestion-output dizininde saklanma süresi (gün). `0` süresiz saklama. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Henüz hiçbir komut onu okumaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa CLI uyarır. | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 3e4b2521..d0a8e752 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -272,17 +272,17 @@ Defines maximum retention horizons for compliance, training, and evaluation artefacts. Horizons are operator-set retention limits that help implement GDPR Article 5(1)(e) "storage limitation" (no fixed period: data may be kept only as long as necessary) and support erasure handling under Article 17 (requests are -answered without undue delay, within the Article 12(3) one-month period). Choose +answered without undue delay, within the Article 12(3) one-month period, extendable by two further months). Choose values from your own necessity assessment; the defaults are ForgeLM choices, not -legal periods. The horizons are **reported**, not -enforced: `forgelm purge --check-policy` lists what is past them and always -exits 0, and nothing is deleted, blocked or scheduled automatically. The -`enforce` knob is reserved: no command reads it yet. +legal periods. The horizons are **reported**, not enforced: `forgelm purge +--check-policy` lists what is past them and always exits 0, and nothing is +deleted, blocked or scheduled automatically. The `enforce` knob is reserved: no +command reads it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period (the AI Act's log-retention duties are Art. 19(1) / Art. 26(6), at least six months). Set to `0` to retain indefinitely; name the Union or Member State obligation you rely on. | -| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | +| `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period (see AI Act Art. 18(1), documentation kept 10 years, and Art. 19(1) / 26(6), system logs at least six months, as applicable). Set to `0` to retain indefinitely; document why the entries stay necessary or the legal obligation you rely on. | +| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | | `ephemeral_artefact_retention_days` | int | `90` | Days to retain compliance bundles, data audit reports, and other run-scoped derived artefacts. Set to `0` to retain indefinitely. | | `raw_documents_retention_days` | int | `90` | Days to retain ingested raw documents (PDF / DOCX / EPUB / TXT / Markdown) under the operator's ingestion-output directory. Set to `0` to retain indefinitely. | | `enforce` | string | `"log_only"` | Reserved policy mode. No command reads it yet: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and the CLI warns when a config sets either of the last two. | diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index b5659ede..44e3113b 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -3,7 +3,7 @@ > **Hedef kitle:** Eğitim corpus'larına ve koşum-kapsamlı artefact'lara karşı GDPR Madde 17 silme taleplerini karşılayan ForgeLM operatörleri ve ortaya çıkan `data.erasure_*` audit zincirini doğrulayan denetçiler. > **Ayna:** [purge_subcommand.md](purge_subcommand.md) -`forgelm purge`, ForgeLM eğitim corpus'ları ve koşum artefact'ları için **GDPR Madde 17 silinme hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 21). Bir satırı, bir koşumun staging dizinini ve (dosya adları run id'yi taşıyorsa — aşağıdaki `--kind artefacts`'e bakın) bir koşumun compliance dosyalarını atomik olarak siler ve her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. +`forgelm purge`, ForgeLM eğitim corpus'ları ve koşum artefact'ları için **GDPR Madde 17 silinme hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 21). Bir corpus satırını atomik olarak siler; `--run-id` ile bir koşumun staging dizinini (`--kind staging`) veya adı run id'yi embed eden compliance dosyalarını (`--kind artefacts`, aşağıya bakın) siler; her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. Deployer akışı (DSAR ticket → CLI çağrısı → doğrulama) için bkz. [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure-tr.md). Bu sayfa flag-başına, event-başına referanstır. @@ -48,11 +48,11 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | Kind | Hedef | |---|---| | `staging` | `/final_model.staging./`; legacy `final_model.staging/` yalnızca içinde içeriği ``'ye eşit bir `staging_run_id` dosyası varsa. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | -| `artefacts` | Adı ``'yi bir token olarak (`_`, `-` veya `.` ile sınırlanmış) embed eden `/compliance/*.json` dosyaları. | +| `artefacts` | `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosya (yalnızca `.json` değil). | > **`--kind artefacts` standart compliance paketini bulamaz.** Trainer ve exporter sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`; yapılandırıldığında `risk_assessment.json` / `annex_iv_metadata.json`) ve hiçbiri run id içermez. Normal bir çıktı dizininde komut bu yüzden hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar ve 1 ile çıkar; hiçbir şey silmez. Yalnızca adına run id'yi kendiniz yazdığınız dosyalara etki eder. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. -`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Bir veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü bunları tutmayı gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. ### Retention-policy raporu (`--check-policy`) @@ -93,7 +93,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | Kod | Anlamı | |---|---| | 0 | Başarı veya başarılı `--check-policy` raporu (report-not-gate semantik). | -| 1 | Config hatası: bilinmeyen `--row-id`, eksik `--corpus`, karşılıklı dışlayıcı flag kombinasyonu, çakışan `staging_ttl_days` değerleri veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | +| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus`, karşılıklı dışlayıcı flag kombinasyonu, çakışan `staging_ttl_days` değerleri veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | | 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, atomic-rename başarısızlığı. | `--check-policy` asla 3 veya 4 dönmez. Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) eğitim pipeline'ı için ayrılmıştır ve bu subcommand'ın yüzeyinin parçası değildir. diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index 321bebd7..33329512 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -3,7 +3,7 @@ > **Audience:** ForgeLM operators honouring GDPR Article 17 erasure requests against training corpora and run-scoped artefacts, plus auditors verifying the resulting `data.erasure_*` audit chain. > **Mirror:** [purge_subcommand-tr.md](purge_subcommand-tr.md) -`forgelm purge` is the operator-facing implementation of the **GDPR Article 17 right-to-erasure** for ForgeLM training corpora and run artefacts (Phase 21). It atomically deletes a row, or a run's staging directory, and (when the files carry the run id in their name — see `--kind artefacts` below) a run's compliance files, and records every step in the tamper-evident `audit_log.jsonl` chain. +`forgelm purge` is the operator-facing implementation of the **GDPR Article 17 right-to-erasure** for ForgeLM training corpora and run artefacts (Phase 21). It atomically erases a corpus row; with `--run-id` it deletes a run's staging directory (`--kind staging`) or compliance files whose names embed the run id (`--kind artefacts`, see below); and it records every step in the tamper-evident `audit_log.jsonl` chain. For the deployer-flow walkthrough (DSAR ticket → CLI invocation → verification), see [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure.md). This page is the per-flag, per-event reference. @@ -48,11 +48,11 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai | Kind | Target | |---|---| | `staging` | `/final_model.staging./`, and the legacy `final_model.staging/` only when it contains a `staging_run_id` file whose content equals ``. ForgeLM does not write that marker, so remove an unmarked legacy staging directory yourself. | -| `artefacts` | `/compliance/*.json` files whose name embeds `` as a token (delimited by `_`, `-` or `.`). | +| `artefacts` | Any file directly under `/compliance/` whose name contains `` bounded by `_`, `-`, `.` or the start/end of the name (not only `.json`). | > **`--kind artefacts` does not find the standard compliance bundle.** The trainer and exporter write fixed basenames (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`, and `risk_assessment.json` / `annex_iv_metadata.json` when configured), none of which contains a run id. On a normal output directory the command therefore finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`) and exits 1; it deletes nothing. It acts only on files you have named with the run id yourself. Until the resolver is fixed, remove the bundle through your storage layer. -`--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it. Whether entries that name a data subject may stay is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires keeping them, and you have to be able to name that obligation. +`--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it. Whether entries that name the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. ### Retention-policy report (`--check-policy`) @@ -93,7 +93,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | Code | Meaning | |---|---| | 0 | Success, or a successful `--check-policy` report (report-not-gate semantic). | -| 1 | Config error: unknown `--row-id`, missing `--corpus`, mutually-exclusive flag combination, conflicting `staging_ttl_days` values, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | +| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, conflicting `staging_ttl_days` values, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | | 2 | Runtime error: I/O failure, permission denied, atomic-rename failure. | `--check-policy` never returns code 3 or 4. Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are reserved for the training pipeline and are not part of this subcommand's surface. diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 4ee952ad..1c80bef7 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -14,7 +14,7 @@ This page is the operator quick-reference. The deeper deployer-flow walkthrough | Right | Subcommand | What it answers | |---|---|---| | **Article 15** — right of access | `forgelm reverse-pii` | "Do you hold any record of me?" — scans JSONL corpora for the subject's identifier. | -| **Article 17** — right to erasure | `forgelm purge` | "Delete every record of me." — atomically removes a row or a staging directory, and compliance files whose names embed the run id. | +| **Article 17** — right to erasure | `forgelm purge` | "Delete every record of me." — atomically removes a corpus row; with `--run-id` it removes a staging directory or compliance files whose names embed the run id. | Both subcommands hash the subject's identifier with the **same** per-output-dir salt (`/.forgelm_audit_salt`) before recording the audit row. The `query_hash` field on `data.access_request_query` and the `target_id` field on `data.erasure_*` are byte-identical for the same subject, which lets a reviewer correlate the two without cleartext. @@ -49,7 +49,7 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ ### What it does NOT do - **Re-train models.** Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` so the gap is visible. -- **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history. Whether entries that name the data subject may be kept is the controller's decision: Article 17(3)(b) applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. +- **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history. Whether entries that name the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. - **Find the standard compliance bundle with `--kind artefacts`.** The trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, and the resolver matches only filenames that embed it. On a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing; remove the bundle through your storage layer. - **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (Article 17(2)). The `data.erasure_warning_external_copies` event is the explicit reminder. diff --git a/docs/usermanuals/en/compliance/human-oversight.md b/docs/usermanuals/en/compliance/human-oversight.md index 48fa2f2a..bdba2328 100644 --- a/docs/usermanuals/en/compliance/human-oversight.md +++ b/docs/usermanuals/en/compliance/human-oversight.md @@ -95,7 +95,7 @@ ForgeLM does **not** ship a built-in quorum gate. There is no `approval.quorum` ## Timeouts -ForgeLM does **not** time out staged runs. There is no `approval.timeout_hours`, no `human_approval.timeout` event, and no auto-fail clock. A staged run waits indefinitely; if the deployer wants to expire stale staging directories, configure `retention.staging_ttl_days` instead — that wires into the GDPR Article 17 retention pipeline (which physically prunes the directory after the configured horizon, not into the approval gate). +ForgeLM does **not** time out staged runs. There is no `approval.timeout_hours`, no `human_approval.timeout` event, and no auto-fail clock. A staged run waits indefinitely; if the deployer wants to expire stale staging directories, configure `retention.staging_ttl_days` instead — that is a retention report, not an expiry: `forgelm purge --check-policy` lists staging directories older than the horizon, nothing is deleted automatically (remove one with `forgelm purge --run-id --kind staging`), and it never touches the approval gate. ## Inspecting pending runs diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 04862ba4..4e6433e1 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -14,7 +14,7 @@ Bu sayfa operatör hızlı-referansıdır. Daha derin deployer akışı [GitHub' | Hak | Subcommand | Ne yanıtlar | |---|---|---| | **Madde 15** — erişim hakkı | `forgelm reverse-pii` | "Hakkımda kayıt tutuyor musunuz?" — JSONL corpus'ları subject identifier'ı için tarar. | -| **Madde 17** — silinme hakkı | `forgelm purge` | "Hakkımdaki her kaydı silin." — bir satırı veya bir staging dizinini, ayrıca adı run id'yi embed eden compliance dosyalarını atomik olarak siler. | +| **Madde 17** — silinme hakkı | `forgelm purge` | "Hakkımdaki her kaydı silin." — bir corpus satırını atomik olarak siler; `--run-id` ile bir staging dizinini veya adı run id'yi embed eden compliance dosyalarını siler. | İki subcommand da audit satırını kaydetmeden önce subject identifier'ını **aynı** per-output-dir salt (`/.forgelm_audit_salt`) ile hash'ler. `data.access_request_query` üzerindeki `query_hash` ve `data.erasure_*` üzerindeki `target_id` aynı subject için byte-byte aynıdır; reviewer bu sayede ikisini cleartext olmadan korele edebilir. @@ -49,7 +49,7 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( ### Neyi YAPMAZ - **Modelleri yeniden eğitmez.** Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözümdür. Tool `data.erasure_warning_memorisation` yayar; böylece boşluk görünür olur. -- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır: Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. - **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca onu embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. - **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index 98bbbe15..e1c3140e 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -95,7 +95,7 @@ ForgeLM yerleşik bir quorum gate'i **göndermez**. `approval.quorum` alanı yok ## Timeout -ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler; deployer eski staging dizinlerinin sona ermesini istiyorsa bunun yerine `retention.staging_ttl_days`'i konfigüre edin — bu GDPR Madde 17 retention pipeline'ına bağlanır (yapılandırılan horizon'dan sonra dizini fiziksel olarak prune eder, approval gate'e değil). +ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler; deployer eski staging dizinlerinin sona ermesini istiyorsa bunun yerine `retention.staging_ttl_days`'i konfigüre edin — bu bir retention raporudur, süre sonu değildir: `forgelm purge --check-policy` horizon'dan eski staging dizinlerini listeler, hiçbir şey otomatik silinmez (birini `forgelm purge --run-id --kind staging` ile kaldırın) ve approval gate'e asla dokunmaz. ## Bekleyen koşuları inceleme diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index e3436be5..c6397ce9 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1077,7 +1077,7 @@ def _add_purge_subcommand(subparsers) -> None: - ``--row-id --corpus `` — atomic JSONL row erasure with hashed audit event (Article 17 right-to-erasure for training rows). - ``--run-id --kind {staging,artefacts}`` — run-scoped artefact - erasure (staging directory or compliance bundle). + erasure (staging directory or compliance files whose names embed the run id). - ``--check-policy`` — read-only retention-policy violation report against the loaded config's ``retention:`` block. @@ -1092,7 +1092,7 @@ def _add_purge_subcommand(subparsers) -> None: help="GDPR Article 17 erasure: remove corpus rows or run-scoped artefacts.", description=( "Article 17 right-to-erasure: deletes a corpus row, a staging " - "directory, or a run's compliance bundle, and records a tamper-evident " + "directory, or a run's compliance files whose names embed the run id, and records a tamper-evident " "audit-event chain (request → completed / failed). Use --check-policy " "for a read-only retention-violation report against the loaded config." ), diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 3c785f6a..b778c11f 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -7,7 +7,7 @@ erasure with hashed audit event (Article 17 right-to-erasure for training corpus rows). - ``forgelm purge --run-id --kind {staging,artefacts}`` — - run-scoped artefact erasure (staging directory or compliance bundle). + run-scoped artefact erasure (staging directory or compliance files whose names embed the run id). - ``forgelm purge --check-policy [--config ]`` — read-only retention-policy violation report against the loaded config's ``retention:`` block. diff --git a/forgelm/config.py b/forgelm/config.py index 48677acc..b9979b66 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1591,7 +1591,7 @@ class RetentionConfig(BaseModel): §3 + closure-plan §15.5 v2.5). All four horizons default to ForgeLM's own choices, not legal periods (GDPR Article 5(1)(e) sets no fixed period): audit logs at 5 years, staging at 7 days - (operator gets one work-week to act on a `forgelm reject`), + (one work-week before `purge --check-policy` reports a staging dir), ephemeral artefacts at 90 days (compliance bundle + audit reports have a quarterly review cadence in most QMS), raw documents at 90 days (typical ingestion-window before re-running data audit). @@ -1613,14 +1613,14 @@ class RetentionConfig(BaseModel): ge=0, description=( "Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). " - "Default 1825 = 5 years, a ForgeLM choice, not a legal period. Set to 0 to retain indefinitely; name the Union or Member State obligation you rely on." + "Default 1825 = 5 years, a ForgeLM choice, not a legal period. Set to 0 to retain indefinitely; document why the entries stay necessary or the legal obligation you rely on." ), ) staging_ttl_days: int = Field( default=7, ge=0, description=( - "Days after which `forgelm purge --check-policy` reports a `final_model.staging./` directory as overdue (aged from the run's first audit event, else mtime); nothing is deleted. " + "Days after which `forgelm purge --check-policy` reports a `final_model.staging./` (or legacy `final_model.staging/`) directory as overdue (aged from the run's first audit event, else the log's first event, else mtime); nothing is deleted. " "Set to 0 to retain indefinitely. This is the canonical staging-retention field; " "the legacy `evaluation.staging_ttl_days` alias was removed in v0.8.0." ), diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index 22a8b919..56b2e563 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -327,17 +327,31 @@ def test_artefacts_kind_finds_nothing_in_the_standard_fixed_name_bundle(self, tm failed = [e for e in _read_audit_events(tmp_path / "audit_log.jsonl") if e["event"] == "data.erasure_failed"] assert [e["error_class"] for e in failed] == ["NoMatchingArtefacts"] - def test_no_forgelm_writer_creates_the_legacy_staging_marker(self) -> None: + def test_only_the_purge_reader_names_the_legacy_staging_marker(self) -> None: # The docs tell operators to remove an unmarked legacy ``final_model.staging/`` - # themselves because ForgeLM never writes ``staging_run_id``. A writer - # would make that advice stale. + # themselves because ForgeLM never writes ``staging_run_id``. Every use of that + # marker name in the package must be the read-only ownership check; a writer + # anywhere would make the documented advice stale (update docs + this test). + import ast + forgelm_dir = Path(__file__).resolve().parent.parent / "forgelm" - writers = [ - str(path.relative_to(forgelm_dir)) - for path in sorted(forgelm_dir.rglob("*.py")) - if "staging_run_id" in path.read_text(encoding="utf-8") - ] - assert writers == [os.path.join("cli", "subcommands", "_purge.py")] + total = 0 + inside_reader = 0 + reader_calls: set[str] = set() + for path in sorted(forgelm_dir.rglob("*.py")): + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + total += sum(1 for n in ast.walk(tree) if isinstance(n, ast.Constant) and n.value == "staging_run_id") + for fn in ( + n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == "_legacy_staging_owned_by" + ): + inside_reader += sum( + 1 for n in ast.walk(fn) if isinstance(n, ast.Constant) and n.value == "staging_run_id" + ) + reader_calls |= { + n.func.attr for n in ast.walk(fn) if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute) + } + assert total == inside_reader == 1, "the marker name must appear only in _legacy_staging_owned_by" + assert not reader_calls & {"write_text", "write", "touch", "mkdir"}, "the marker reader must stay read-only" # --------------------------------------------------------------------------- diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 59fa912b..08931ae6 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -81,24 +81,39 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: "docs/usermanuals/tr/compliance/gdpr-erasure.md", "docs/usermanuals/en/reference/configuration.md", "docs/usermanuals/tr/reference/configuration.md", + "docs/usermanuals/en/compliance/human-oversight.md", + "docs/usermanuals/tr/compliance/human-oversight.md", + "docs/qms/sop_change_management.md", + "docs/qms/sop_change_management-tr.md", ) _RETIRED_GDPR_CLAIMS = ( - r"17\(3\)\(b\) preserves", - r"17\(3\)\(b\) (?:exemption|defence)", - r"17\(3\)\(b\) records", + r"17\(3\)\(b\)\s+preserves", + r"17\(3\)\(b\)\s+(?:exemption|defence)", + r"17\(3\)\(b\)\s+records", r"17\(3\)\(b\) savunması", - r"17\(3\)\(b\) audit / muhasebe", + r"17\(3\)\(b\)\s+audit\s*/\s*muhasebe", r"17\(3\)\(b\) kayıt", r"append-only (?:Article|Madde) 17\(3\)\(b\)", - r"5 years default \(Article 5\(1\)\(e\)\)", + r"5\s+years\s+default\s+\(Article\s+5\(1\)\(e\)\)", r"varsayılan 5 yıl \(Madde 5\(1\)\(e\)\)", - r"legal-obligation record under", - r"compliant out of the box", - r"Article 12 record-keeping baseline", - r"Horizons honour GDPR", - r"removed by default after a defined retention horizon", - r"after a `forgelm reject` decision before scheduled cleanup", + r"legal-obligation\s+record\s+under", + r"compliant\s+out\s+of\s+the\s+box", + r"Article\s+12\s+record-keeping\s+(?:baseline|\))", + r"Horizons\s+honou?r\s+GDPR", + r"removed\s+by\s+default\s+after\s+a\s+defined\s+retention\s+horizon", + r"after\s+a\s+`forgelm\s+reject`\s+decision\s+before\s+scheduled\s+cleanup", + r"work-week\s+to\s+act\s+on\s+a\s+`forgelm\s+reject`", + r"Madde\s+12\s+kayıt-tutma", + r"run's\s+compliance\s+bundle", + r"staging\s+directory\s+or\s+compliance\s+bundle", + r"physically\s+prunes", + r"fiziksel\s+olarak\s+prune", + r"until\s+retention\s+expires", + r"retention\s+süresi\s+dolana", + r"justified\s+only\s+to\s+the\s+extent", + r"name\s+the\s+Union\s+or\s+Member\s+State\s+obligation", + r"compliance/\*\.json`\s+files\s+whose\s+name\s+embeds", ) _PURGE_ARTEFACT_WARNING_FILES = ( @@ -114,7 +129,7 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: @pytest.mark.parametrize("rel", _GDPR_CLAIM_FILES) def test_retired_gdpr_claims_stay_out(rel: str) -> None: text = (_REPO_ROOT / rel).read_text(encoding="utf-8") - found = [pattern for pattern in _RETIRED_GDPR_CLAIMS if re.search(pattern, text)] + found = [pattern for pattern in _RETIRED_GDPR_CLAIMS if re.search(pattern, text, re.IGNORECASE)] assert found == [], f"{rel} still carries retired GDPR wording: {found}" From a61950cbf785a9a8cd2bbfe07b9c5afdfd77212f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:49:43 +0300 Subject: [PATCH 03/25] =?UTF-8?q?docs(gdpr):=20review=20round=202=20?= =?UTF-8?q?=E2=80=94=20purge=20and=20retention=20docs=20agree=20with=20the?= =?UTF-8?q?=20code?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review round on the erasure and retention wording, checked against the running commands: - Salt rotation: with FORGELM_AUDIT_SECRET set, purge XORs it into the identifier salt, so rotating it changes target_id digests; the user manual said hashes were stable across rotations. The reverse-pii correlation rule now names the salt source. - Downstream notices are GDPR Art. 19 (each recipient), with Art. 17(2) only where the controller made the data public. - `data.erasure_warning_external_copies` fires in row mode only, and only when `--config` is given; run mode never loads the config. - `--check-policy` JSON `age_source` is `audit` or `mtime` (the docs said `audit_genesis`); staging ages come from the run's first audit event. - Retention rows for ephemeral artefacts and raw documents name the directories the scan really covers; `enforce` is "not acted on" (the training CLI warns). - `--output-dir` defaults to the current directory in run mode; `--config` is optional for `--check-policy` (without it nothing is checked); a successful report exits 0 and an unloadable `--config` exits 1. Help text, module docstring and exit-code tables match. - Design note: the retired legal reasoning for the 5-year default is replaced, Art. 19 / 17(2) are cited correctly, and section 12 lists the remaining deviations (no trainer pre-flight gate, raw-documents ageing, config discovery, event scope). - Turkish: genitive in the 17(3)(b) clause, spelled-out "Madde", one sentence where the English has one, "Yapmaz" list reference. - Tests: the retired-phrase list covers the Turkish bundle wording and the old reject-anchored comments, every Article 17(3)(b) mention must sit next to a limiting qualifier (so a reworded claim cannot slip past), and the legacy marker reader is checked against a read-only allowlist. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 32 +++++++++---------- docs/guides/gdpr_erasure-tr.md | 12 +++---- docs/guides/gdpr_erasure.md | 12 +++---- docs/reference/configuration-tr.md | 19 +++++------ docs/reference/configuration.md | 6 ++-- docs/reference/purge_subcommand-tr.md | 16 +++++----- docs/reference/purge_subcommand.md | 12 +++---- .../usermanuals/en/compliance/gdpr-erasure.md | 11 ++++--- .../usermanuals/tr/compliance/gdpr-erasure.md | 15 +++++---- .../tr/compliance/human-oversight.md | 2 +- .../usermanuals/tr/reference/configuration.md | 2 +- forgelm/cli/_parser.py | 6 ++-- forgelm/cli/subcommands/_purge.py | 30 +++++++++-------- forgelm/config.py | 14 ++++---- tests/test_gdpr_erasure.py | 4 ++- .../test_usermanual_reference_consistency.py | 31 ++++++++++++++++-- 16 files changed, 129 insertions(+), 95 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 95b901d8..59e346d3 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -43,7 +43,7 @@ Phase 21 closes the gap with a real implementation. This Phase 20 document spec | (e) Legal obligation | Court order. | (out of scope — operator handles externally; tool just records) | | (f) Child's data offered without parental consent | Subject was a minor at point of collection. | (out of scope — operator handles externally) | -**Article 17(2)** requires the controller to inform downstream processors / publishers of the erasure when the data has been disclosed. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. +**Article 19** requires the controller to tell each recipient the data was disclosed to about the erasure, and **Article 17(2)** adds reasonable steps to inform other controllers where the controller made the data public. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. **Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to name it: no provision of EU law protects a training audit log merely because ForgeLM produces it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis. @@ -72,7 +72,7 @@ The four artefact kinds the operator can act on in-tool: `corpus_rows`, `staging Article 5(1)(e) requires personal data to be kept "no longer than is necessary". ForgeLM ships a config block so the operator can declare their retention horizon and the tool can warn / refuse runs that violate it. -> **As shipped:** the horizons are only *reported* by `forgelm purge --check-policy` (read-only, always exit 0). `retention.enforce` is reserved: no command reads it, so nothing warns, refuses or deletes on age. Ages come from the audit log first (the run's first event, else the log's first event) and fall back to `mtime`. Read the warn / refuse and `mtime` passages below as the original design; §12 lists the deviations. +> **As shipped:** the horizons are only *reported* by `forgelm purge --check-policy` (read-only, always exit 0). `retention.enforce` is reserved: no command acts on it (the training CLI only logs a warning when it is set), so nothing warns, refuses or deletes on age. Ages come from the audit log first (the run's first event, else the log's first event) and fall back to `mtime`. Read the warn / refuse and `mtime` passages below as the original design; §12 lists the deviations. ### 3.1 Prior-state acknowledgement (audited 2026-05-02) @@ -102,16 +102,13 @@ class RetentionConfig(BaseModel): model_config = ConfigDict(extra="forbid") - # Audit log retention horizon. EU AI Act Article 12(2) sets the floor - # at 6 months; Article 11 (technical documentation) extends to 10 years - # post-deployment for high-risk systems. GDPR Article 30 record-keeping - # commonly demands ≥3 years. We default to 1825 days (5 years) as a - # middle-of-the-road horizon that satisfies the AI Act floor with a - # comfortable margin AND covers the GDPR controller-record requirement. - # Operators with stricter compliance (Article 11 deployer obligation, - # ISO 27001 retention contracts) can set this higher; Article 12(2) - # operators with no AI Act obligation can lower to 180 (6 months). - # Set to 0 to disable horizon enforcement entirely (not recommended). + # Audit log retention horizon: the age at which `forgelm purge + # --check-policy` reports `audit_log.jsonl` as overdue. 1825 days + # (5 years) is a ForgeLM choice, not a legal period: GDPR Art. 5(1)(e) + # sets no fixed period. Align it with the obligations that apply to you + # (for example AI Act Art. 18(1), documentation kept 10 years, and + # Art. 19(1) / 26(6), system logs at least six months) and with your + # own necessity assessment. Set to 0 to retain indefinitely. audit_log_retention_days: int = Field(default=1825, ge=0) # Staging directory time-to-live. Article 14 staging is meant to hold a @@ -165,7 +162,7 @@ Three sites: For each retention horizon: - Compare artefact `mtime` against `now - _days`. -- If older, the artefact has overstayed the declared lawful basis. +- If older, the artefact has passed the operator-declared retention horizon; review whether it is still necessary. - The audit log itself is **never deleted automatically** even when the policy says it has overstayed. The tool only flags; the operator makes the deletion call by hand, after checking whether the entries are still necessary for their purpose or a specific Union or Member State obligation (Art. 17(3)(b)) or legal claim (Art. 17(3)(e)) still requires them. The compliance artefacts (`compliance/*.json`) are tied to a specific run; their lifetime is the run's lifetime, not a separate horizon. In-place deletion is specified via `forgelm purge --run-id --kind artefacts` (§4.1) which is meant to remove `/compliance/*.json` for the named run while leaving the audit log + final model intact (as shipped, the resolver only matches filenames that embed the run id, and the standard bundle uses fixed basenames, so it finds nothing — see the purge reference) — useful when an operator wants to regenerate the bundle after an Article 17 corpus erasure changed the lineage. Whole-`` deletion remains the operator's manual escalation when the entire run is being scrubbed. @@ -357,7 +354,7 @@ What ForgeLM does **not** do: - Forgetting from already-trained model weights (this is an active research area called "machine unlearning"; the only proper mitigation today is full retraining without the erased data). -- Pushing erasure notices to downstream processors (Article 17(2)) — that +- Pushing erasure notices to downstream recipients (Article 19; Article 17(2) where the data was made public) — that is your runtime layer's responsibility, not the training toolkit's. See [`docs/guides/gdpr_erasure.md`](../guides/gdpr_erasure.md) for the operator how-to @@ -478,6 +475,9 @@ The design above shipped in v0.5.5, with the deviations listed after this list: **Deviations from the design, as shipped today:** -- `--kind artefacts` matches `compliance/` filenames that embed the run id as a token. The exporter writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...), so on a standard output directory the command finds nothing: it fails safe (nothing is deleted, `data.erasure_failed` with `NoMatchingArtefacts`, exit 1). The fixed allow-list with an ownership proof that §4.4 describes is open work. -- Retention horizons are reported by `forgelm purge --check-policy` and never enforced; `retention.enforce` is reserved and no command reads it. +- `--kind artefacts` matches `compliance/` filenames that embed the run id as a token. The exporter writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...), so on a standard output directory the command finds nothing: it fails safe (nothing is deleted, `data.erasure_failed` with `NoMatchingArtefacts`, exit 1). The fixed allow-list that §4.4 describes, plus an ownership check, is open work. +- Retention horizons are reported by `forgelm purge --check-policy` and never enforced; `retention.enforce` is reserved and no command acts on it (the training CLI logs a warning when it is set). There is no trainer pre-flight gate: nothing exits 3 or refuses a run on age. - The legacy `final_model.staging/` is purged only when it holds a `staging_run_id` marker equal to the run id. Nothing in ForgeLM writes that marker. +- `--check-policy` ages the raw-documents directories (`raw_documents/`, `ingestion_output/`) from the audit log's first event, not per run; it reads `--config` only when given (no `./forgelm.yaml` discovery) and, without a `retention:` block, reports that nothing was checked. `--dry-run` combined with `--check-policy` is not rejected. +- The `data.erasure_warning_external_copies` event is emitted in row mode only, and only when `--config` is given; run mode never loads the config. +- The `recommendation` field listed for the new events in §5.1 is not emitted, and the test table in §7 predates the shipped tests (for example, a dry-run appends audit events, and erasing the same row twice exits 1 with `NoMatchingRow`); `tests/test_gdpr_erasure.py` is the source of truth. diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 960693c2..d0c23475 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -20,8 +20,8 @@ **Yapmaz:** - Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. -- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır (log teknik dokümantasyonunuzun parçasıysa AI Act Md. 18(1)'in 10 yıllık dokümantasyon saklaması ya da devreye alınmış yüksek riskli bir sistemin ürettiği loglar için Md. 19(1) ve 26(6)'nın log saklama yükümlülükleri gibi) ve bu yükümlülüğü adlandırabilmeniz gerekir. -- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. +- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır (log teknik dokümantasyonunuzun parçasıysa AI Act Madde 18(1)'in 10 yıllık dokümantasyon saklaması ya da kullanıma alınmış yüksek riskli bir sistemin ürettiği loglar için Madde 19(1) ile 26(6)'nın log saklama yükümlülükleri gibi) ve bu yükümlülüğü adlandırabilmeniz gerekir. +- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 19 verinin ifşa edildiği her alıcıya bildirim yapılmasını ister; Madde 17(2) ise kontrolör veriyi kamuya açtıysa makul adımlar ekler). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. ## Üç mod @@ -58,7 +58,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs - `--kind staging` — `/final_model.staging./`'i siler; legacy `final_model.staging/`'i yalnızca içeriği ``'ye eşit bir `staging_run_id` işaretçisi taşıyorsa siler (ForgeLM bu işaretçiyi yazmaz; aksi halde legacy dizini kendiniz silin). - `--kind artefacts` — `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosyayı siler (yalnızca `.json` değil). **Standart paketi bulamaz:** trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...), bu yüzden normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. -- `--kind logs` **bilinçli olarak yok**: audit log append-only'dir ve tool onu asla silmez (girdilerinin tutulup tutulamayacağına kimin karar verdiği için yukarıdaki ilk listeye bakın). +- `--kind logs` **bilinçli olarak yok**: audit log append-only'dir ve tool onu asla silmez (girdilerin tutulup tutulamayacağına kimin karar verdiği için yukarıdaki "Yapmaz" listesine bakın). ### 3. Retention policy raporu @@ -66,7 +66,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` -Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. +Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. **Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve `jq '.violations | length'`'e pipe'lar; bu, public exit-code kontratını `0/1/2/3/4` her ForgeLM subcommand'ında tutarlı tutar. @@ -98,7 +98,7 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti | Tüm `requested` field'ları + `error_class`, `error_message` | | `data.erasure_warning_memorisation` | **Yalnız row modunda.** Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut. (Run modu memorisation uyarısı emit etmez — run-scoped `staging` / `artefacts` türleri tanım gereği promote-öncesi.) | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Yalnız row modunda.** Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. (Run modu bu uyarıyı yüzeylemez — synthetic veri corpus-scoped'dur, run-scoped değil.) | Tüm `completed` field'ları + `synthetic_files` | -| `data.erasure_warning_external_copies` | Yüklü config webhook block'u içeriyor | Tüm `completed` field'ları + `webhook_targets` | +| `data.erasure_warning_external_copies` | **Yalnız row modunda, `--config` verildiğinde.** Yüklü config webhook block'u içeriyor. (Run modu `--config` yüklemez.) | Tüm `completed` field'ları + `webhook_targets` | ## Silme sonrası zincir doğrulama @@ -109,7 +109,7 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d | Kod | Anlamı | |---|---| | 0 | Başarı veya başarılı `--check-policy` raporu (rapor-değil-gate semantiği). | -| 1 | Config hatası: bilinmeyen `--row-id`, eksik `--corpus`, mutually-exclusive flag kombinasyonu, çelişen `staging_ttl_days` değerleri veya eksik / okunamayan / Pydantic validation'ı geçemeyen `--check-policy --config `. | +| 1 | Config hatası: bilinmeyen `--row-id`, eksik `--corpus`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), mutually-exclusive flag kombinasyonu veya eksik / okunamayan / Pydantic validation'ı geçemeyen `--check-policy --config `. | | 2 | Runtime hatası: I/O failure, permission denied, atomic-rename failure. | `--check-policy` asla 3 kodu döndürmez. Başarılı bir rapor 0 ile çıkar; verilen `--config`'i yükleme başarısızlığı ise yanıltıcı bir "ihlal yok" raporuna düşmek yerine `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar. CI gate isteyen operatörler ihlal sayısını JSON çıktıdan kendileri hesaplar (design §10 Q5). diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 4b298640..6d3ddf0b 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -21,7 +21,7 @@ - Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. - Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history. Whether entries that reference the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing (such as the 10-year documentation keeping of AI Act Art. 18(1) where the log forms part of your technical documentation, or the log-retention duties of Art. 19(1) and Art. 26(6) for logs a deployed high-risk system generates), and you have to be able to name that obligation. -- Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 17(2)). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. +- Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 19 requires telling each recipient the data was disclosed to; Article 17(2) adds reasonable steps where the controller made the data public). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. ## Three modes @@ -38,7 +38,7 @@ $ forgelm purge \ The tool: -1. Resolves the per-output-dir salt at `/.forgelm_audit_salt` (creates it on first use; mode `0600`; XOR'd with `FORGELM_AUDIT_SECRET[:16]` if the env var is set). **Note:** this XOR feeds **identifier hashing only** — the audit-chain HMAC key is derived independently as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (see [`docs/qms/access_control.md`](../qms/access_control.md) §3.4 / `forgelm/compliance.py:104-114`). The two primitives are intentionally separate. +1. Resolves the per-output-dir salt at `/.forgelm_audit_salt` (creates it on first use; mode `0600`; XOR'd with `FORGELM_AUDIT_SECRET[:16]` if the env var is set). **Note:** this XOR feeds **identifier hashing only** — the audit-chain HMAC key is derived independently as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (see [`docs/qms/access_control.md`](../qms/access_control.md) §3.4 / `AuditLogger.__init__` in `forgelm/compliance.py`). The two primitives are intentionally separate. 2. Writes `data.erasure_requested` to the audit log with the **hashed** target_id (`SHA-256(salt + value)`) and the operator-supplied justification. 3. Locates the matching JSONL row by its `id` (or `row_id`) field — line-number fallback is **rejected** to defend against silent wrong-row deletion on a re-ordered file. ForgeLM does not currently provide an id-population helper (a `forgelm audit --add-row-ids` flag is on the Phase 28 backlog); operators with id-less corpora must pre-populate ids with an operator-side script (e.g. `jq -c 'to_entries | with_entries(...)'` or a one-shot Python loop) before invoking purge. 4. Atomically rewrites the corpus (writes a sibling temp file + `os.replace`); operators get either the full pre-erasure file or the full post-erasure file, never a partial state. @@ -58,7 +58,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs - `--kind staging` removes `/final_model.staging./`, and the legacy `final_model.staging/` only if it carries a `staging_run_id` marker whose content equals the `` (ForgeLM does not write that marker; otherwise remove the legacy directory yourself). - `--kind artefacts` removes any file directly under `/compliance/` whose name contains the `` bounded by `_`, `-`, `.` or the start/end of the name (not only `.json`). **It does not find the standard bundle:** the trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, so on a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing. Remove the bundle through your storage layer until the resolver is fixed. -- `--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it (see the first list above for who decides whether its entries may be kept). +- `--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it (see the "Does not" list above for who decides whether its entries may be kept). ### 3. Retention-policy report @@ -66,7 +66,7 @@ $ forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs ``` -Read-only scan. Walks the output directory, compares each artefact's age (canonical: audit-log genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. +Read-only scan. Walks the output directory, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. **Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves; this keeps the public exit-code contract `0/1/2/3/4` consistent across every ForgeLM subcommand. @@ -98,7 +98,7 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity | All `requested` fields + `error_class`, `error_message` | | `data.erasure_warning_memorisation` | **Row mode only.** Row erasure × a `final_model/` directory exists directly under `--output-dir`. (Run mode does not emit memorisation warnings — the run-scoped `staging` / `artefacts` kinds are by definition pre-promotion.) | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | **Row mode only.** Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. (Run mode does not surface this warning — synthetic data is corpus-scoped, not run-scoped.) | All `completed` fields + `synthetic_files` | -| `data.erasure_warning_external_copies` | Loaded config has a webhook block (both modes). | All `completed` fields + `webhook_targets` | +| `data.erasure_warning_external_copies` | **Row mode only, when `--config` is given.** Loaded config has a webhook block. (Run mode never loads `--config`.) | All `completed` fields + `webhook_targets` | ## Verifying a chain post-erasure @@ -109,7 +109,7 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | Code | Meaning | |---|---| | 0 | Success, or a successful `--check-policy` report (report-not-gate semantic). | -| 1 | Config error: unknown `--row-id`, missing `--corpus`, mutually-exclusive flag combination, conflicting `staging_ttl_days` values, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | +| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | | 2 | Runtime error: I/O failure, permission denied, atomic-rename failure. | `--check-policy` never returns code 3. A successful report exits 0; a failure to load the supplied `--config` exits with `EXIT_CONFIG_ERROR` (non-zero) rather than degrading to a misleading "no violations" report. Operators wiring a CI gate compute the violation count from JSON output themselves (per design §10 Q5). diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index f16cc1e1..d27b4e80 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -269,11 +269,12 @@ training: ## `retention` (İsteğe bağlı — GDPR Madde 17 silme ufukları) Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarını -belirler. Ufuklar, operatörün belirlediği saklama sınırlarıdır. GDPR Madde 5(1)(e) -"saklama sınırlaması"nın (sabit süre yok: veri yalnızca gerektiği kadar tutulabilir) -uygulanmasına yardımcı olur ve Madde 17 kapsamındaki silme işlemlerini destekler -(talepler gereksiz gecikme olmaksızın, Madde 12(3)'teki bir aylık süre içinde, iki ay daha uzatılabilir şekilde yanıtlanır). -Değerleri kendi gereklilik değerlendirmenizden seçin; varsayılanlar ForgeLM +belirler. Ufuklar, GDPR Madde 5(1)(e) "saklama sınırlaması"nın (sabit süre yoktur: +veri yalnızca gerektiği kadar tutulabilir) uygulanmasına yardımcı olan ve Madde 17 +kapsamındaki silme işlemlerini destekleyen, operatör tarafından belirlenen saklama +sınırlarıdır (talepler gereksiz gecikmeden kaçınılarak, en geç Madde 12(3)'teki bir +aylık süre içinde yanıtlanır; bu süre iki ay daha uzatılabilir). Değerleri kendi +gereklilik (necessity) değerlendirmenize göre belirleyin; varsayılanlar ForgeLM tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: `forgelm purge --check-policy` ufku aşanları listeler ve her zaman 0 ile çıkar; hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı @@ -281,11 +282,11 @@ ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (duruma göre AI Act Md. 18(1), dokümantasyon 10 yıl, ve Md. 19(1) / 26(6), sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | +| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (uygulanabildiği ölçüde bkz. AI Act Madde 18(1): dokümantasyon 10 yıl; Madde 19(1) ve 26(6): sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | | `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve legacy `final_model.staging/`'i) "geciken" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | -| `ephemeral_artefact_retention_days` | int | `90` | Uyumluluk paketleri, veri denetim raporları ve diğer çalışma kapsamlı türetilmiş artefaktların saklanma süresi (gün). `0` süresiz saklama. | -| `raw_documents_retention_days` | int | `90` | İngest edilmiş ham belgelerin (PDF / DOCX / EPUB / TXT / Markdown) operatörün ingestion-output dizininde saklanma süresi (gün). `0` süresiz saklama. | -| `enforce` | string | `"log_only"` | Ayrılmış politika modu. Henüz hiçbir komut onu okumaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa CLI uyarır. | +| `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "geciken" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | +| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "geciken" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | +| `enforce` | string | `"log_only"` | Ayrılmış politika modu. Hiçbir komut onu kullanmaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa eğitim ve `--dry-run` uyarır. | > **Kaldırıldı:** `evaluation.staging_ttl_days` (v0.5.5 itibarıyla kullanımdan > kaldırılmıştı) v0.8.0'da kaldırılmıştır. Artık geçerli tek form diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index d0a8e752..e01cdc7f 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -283,9 +283,9 @@ command reads it yet. |-------|------|---------|-------------| | `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period (see AI Act Art. 18(1), documentation kept 10 years, and Art. 19(1) / 26(6), system logs at least six months, as applicable). Set to `0` to retain indefinitely; document why the entries stay necessary or the legal obligation you rely on. | | `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | -| `ephemeral_artefact_retention_days` | int | `90` | Days to retain compliance bundles, data audit reports, and other run-scoped derived artefacts. Set to `0` to retain indefinitely. | -| `raw_documents_retention_days` | int | `90` | Days to retain ingested raw documents (PDF / DOCX / EPUB / TXT / Markdown) under the operator's ingestion-output directory. Set to `0` to retain indefinitely. | -| `enforce` | string | `"log_only"` | Reserved policy mode. No command reads it yet: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and the CLI warns when a config sets either of the last two. | +| `ephemeral_artefact_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/compliance/` and `/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to retain indefinitely. | +| `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to retain indefinitely. | +| `enforce` | string | `"log_only"` | Reserved policy mode. No command acts on it: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and training and `--dry-run` warn when a config sets either of the last two. | > **Removed:** `evaluation.staging_ttl_days` (deprecated as of v0.5.5) was > removed in v0.8.0. `retention.staging_ttl_days` is now the only accepted form. diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index 44e3113b..6b13d95b 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -47,16 +47,16 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir | Kind | Hedef | |---|---| -| `staging` | `/final_model.staging./`; legacy `final_model.staging/` yalnızca içinde içeriği ``'ye eşit bir `staging_run_id` dosyası varsa. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | +| `staging` | `/final_model.staging./`; legacy `final_model.staging/` yalnızca içerdiği `staging_run_id` dosyasının içeriği ``'ye eşitse. ForgeLM bu işaretçiyi yazmaz; işaretsiz bir legacy staging dizinini kendiniz silin. | | `artefacts` | `/compliance/` altında doğrudan bulunan, adı ``'yi `_`, `-`, `.` ya da adın başı/sonu ile sınırlı olarak içeren her dosya (yalnızca `.json` değil). | > **`--kind artefacts` standart compliance paketini bulamaz.** Trainer ve exporter sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`; yapılandırıldığında `risk_assessment.json` / `annex_iv_metadata.json`) ve hiçbiri run id içermez. Normal bir çıktı dizininde komut bu yüzden hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar ve 1 ile çıkar; hiçbir şey silmez. Yalnızca adına run id'yi kendiniz yazdığınız dosyalara etki eder. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. -`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. ### Retention-policy raporu (`--check-policy`) -Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. +Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. **Başarılı policy raporu her zaman 0 ile çıkar** (report-not-gate semantik, design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (1) ile çıkar — eksik, okunamaz veya Pydantic doğrulamasından geçemeyen explicit `--config` non-zero olarak yüzeyselleşir; böylece operatör "loader başarısız" durumunu "ihlal yok" sanmaz. CI gate kuran operatörler `--output-format json` kullanıp `jq '.violations | length'`'e pipe eder. @@ -71,7 +71,7 @@ Per-output-dir salt `/.forgelm_audit_salt`'ta ilk kullanımda oluşt ## `forgelm reverse-pii` ile cross-tool digest korelasyonu -`forgelm purge --row-id ` ve `forgelm reverse-pii --query --salt-source per_dir` **aynı** ``'e (yani aynı `.forgelm_audit_salt`'ı tüketerek) çalıştırıldığında, `data.erasure_*` event'lerindeki `target_id` ile `data.access_request_query` event'indeki `query_hash` byte-byte aynıdır. Compliance reviewer böylece aynı veri sahibi için Madde 17 silme ve Madde 15 erişim taleplerini cleartext identifier'ı asla görmeden korele edebilir. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. +`forgelm purge --row-id ` ve `forgelm reverse-pii --query --salt-source per_dir` **aynı** ``'e (yani aynı `.forgelm_audit_salt`'ı tüketerek) çalıştırıldığında ve purge için `FORGELM_AUDIT_SECRET` ayarlı değilse (`salt_source=per_dir`; ayarlıysa purge onu salt'a XOR'lar ve `salt_source=env_var` kaydeder, bu durumda bunun yerine aynı secret ile `--salt-source env_var` kullanarak sorgulayın), `data.erasure_*` event'lerindeki `target_id` ile `data.access_request_query` event'indeki `query_hash` byte-byte aynıdır. Compliance reviewer böylece aynı veri sahibi için Madde 17 silme ve Madde 15 erişim taleplerini cleartext identifier'ı asla görmeden korele edebilir. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. ## Yayılan audit event'leri @@ -84,7 +84,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum yok VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | | `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | -| `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | +| `data.erasure_warning_external_copies` | **Yalnız row modunda, `--config` verildiğinde.** Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | `forgelm verify-audit /audit_log.jsonl` herhangi bir sayıda erasure event'i sonrası zinciri doğrulamaya devam eder — tool yeni event'ler ekler, eskileri yeniden yazmaz. @@ -93,7 +93,7 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | Kod | Anlamı | |---|---| | 0 | Başarı veya başarılı `--check-policy` raporu (report-not-gate semantik). | -| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus`, karşılıklı dışlayıcı flag kombinasyonu, çakışan `staging_ttl_days` değerleri veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | +| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus`, karşılıklı dışlayıcı flag kombinasyonu veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | | 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, atomic-rename başarısızlığı. | `--check-policy` asla 3 veya 4 dönmez. Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) eğitim pipeline'ı için ayrılmıştır ve bu subcommand'ın yüzeyinin parçası değildir. @@ -145,14 +145,14 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak "path": "/abs/path/outputs/run42/compliance/data_audit_report.json", "age_days": 121.4, "horizon_days": 90, - "age_source": "audit_genesis" + "age_source": "audit" } ], "count": 1 } ``` -`age_source` ∈ `{audit_genesis, mtime}` — `audit_genesis` kanonik yaş (koşumun ilk audit event timestamp'i); `mtime` operatörün farkındalığı için işaretlenmiş filesystem fallback'i. +`age_source` ∈ `{audit, mtime}` — `audit` kanonik yaş (koşumun ilk audit event timestamp'i); `mtime` operatörün farkındalığı için işaretlenmiş filesystem fallback'i. **Hata zarfı (her mod)** — `_output_error_and_exit`: diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index 33329512..48d6ff06 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -56,7 +56,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai ### Retention-policy report (`--check-policy`) -Read-only scan. Walks ``, compares each artefact's age (canonical: audit-log genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. +Read-only scan. Walks ``, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. **Successful policy reports always exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit `EXIT_CONFIG_ERROR` (1) — an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as a non-zero so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves. @@ -71,7 +71,7 @@ The per-output-dir salt at `/.forgelm_audit_salt` is created on firs ## Cross-tool digest correlation with `forgelm reverse-pii` -When `forgelm purge --row-id ` and `forgelm reverse-pii --query --salt-source per_dir` run against the **same** `` (i.e. consume the same `.forgelm_audit_salt`), the `target_id` field on `data.erasure_*` events and the `query_hash` field on `data.access_request_query` events are byte-identical. A compliance reviewer can therefore correlate Article 17 erasures and Article 15 access requests for the same data subject without ever seeing the cleartext identifier. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. +When `forgelm purge --row-id ` and `forgelm reverse-pii --query --salt-source per_dir` run against the **same** `` (i.e. consume the same `.forgelm_audit_salt`) and `FORGELM_AUDIT_SECRET` was unset for the purge (`salt_source=per_dir`; when it was set, the purge XORs it into the salt and records `salt_source=env_var`, so query with `--salt-source env_var` and the same secret instead), the `target_id` field on `data.erasure_*` events and the `query_hash` field on `data.access_request_query` events are byte-identical. A compliance reviewer can therefore correlate Article 17 erasures and Article 15 access requests for the same data subject without ever seeing the cleartext identifier. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. ## Audit events emitted @@ -84,7 +84,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | `data.erasure_failed` | Disk operation raised, OR no matching row/run, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | | `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | -| `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | +| `data.erasure_warning_external_copies` | **Row mode only, when `--config` is given.** Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | `forgelm verify-audit /audit_log.jsonl` continues to validate the chain after any number of erasure events — the tool appends new events, never rewrites old ones. @@ -93,7 +93,7 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | Code | Meaning | |---|---| | 0 | Success, or a successful `--check-policy` report (report-not-gate semantic). | -| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, conflicting `staging_ttl_days` values, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | +| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | | 2 | Runtime error: I/O failure, permission denied, atomic-rename failure. | `--check-policy` never returns code 3 or 4. Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are reserved for the training pipeline and are not part of this subcommand's surface. @@ -145,14 +145,14 @@ For `--dry-run` invocations the same envelope appears with `dry_run: true` and w "path": "/abs/path/outputs/run42/compliance/data_audit_report.json", "age_days": 121.4, "horizon_days": 90, - "age_source": "audit_genesis" + "age_source": "audit" } ], "count": 1 } ``` -`age_source` ∈ `{audit_genesis, mtime}` — `audit_genesis` is the canonical age (run's first audit event timestamp); `mtime` is the filesystem fallback flagged for the operator's awareness. +`age_source` ∈ `{audit, mtime}` — `audit` is the canonical age (run's first audit event timestamp); `mtime` is the filesystem fallback flagged for the operator's awareness. **Failure envelope (any mode)** — `_output_error_and_exit`: diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 1c80bef7..138b5389 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -29,7 +29,7 @@ forgelm purge --row-id "ali@example.com" --corpus train.jsonl \ # Run-scoped erasure forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs -forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs +forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs # finds nothing on the standard bundle, see below # Retention-policy report (read-only) forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs @@ -50,8 +50,9 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - **Re-train models.** Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` so the gap is visible. - **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history. Whether entries that name the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. +- **Remove an unmarked legacy `final_model.staging/`.** `--kind staging` deletes `final_model.staging./`, and the legacy `final_model.staging/` only when it holds a `staging_run_id` file equal to the run id. ForgeLM does not write that marker, so remove an unmarked legacy directory yourself. - **Find the standard compliance bundle with `--kind artefacts`.** The trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, and the resolver matches only filenames that embed it. On a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing; remove the bundle through your storage layer. -- **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (Article 17(2)). The `data.erasure_warning_external_copies` event is the explicit reminder. +- **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (Article 19 requires telling each recipient the data was disclosed to; Article 17(2) adds reasonable steps where the controller made the data public). The `data.erasure_warning_external_copies` event is the explicit reminder. ## Article 15 — `forgelm reverse-pii` @@ -88,9 +89,9 @@ The per-output-dir salt at `/.forgelm_audit_salt` (mode `0600`, atom The two primitives are intentionally separate: -- Rotating `FORGELM_AUDIT_SECRET` rotates **only the chain HMAC key**. Identifier hashes derive from the per-output-dir salt and are **stable across env-var rotations**, so a `forgelm reverse-pii` query mounted on the same `--output-dir` after a rotation still correlates with `forgelm purge` events recorded before it. +- Rotating `FORGELM_AUDIT_SECRET` rotates the chain HMAC key **and**, whenever the variable is set, the identifier-hash salt: `forgelm purge` XORs the first 16 bytes of the secret into the per-output-dir salt and records `salt_source=env_var`, so its `target_id` digests change with the secret. A `forgelm reverse-pii` query correlates with earlier purge events only when it uses the same salt source (`--salt-source env_var` with the same secret, or `--salt-source per_dir` for purges run without the variable). - Inspecting either primitive does not reveal the other. -- The per-output-dir salt is the only input to identifier hashing; it persists for the lifetime of the output dir, isolating the chain HMAC's rotation cadence from the cross-tool digest correlation. +- The per-output-dir salt is always an input to identifier hashing and persists for the lifetime of the output dir; the secret, when set, is a second input. ## Exit codes @@ -99,7 +100,7 @@ Both subcommands follow the project's `0/1/2/3/4` contract: | Code | Meaning | |---|---| | 0 | Success. `forgelm purge --check-policy` exits 0 even when violations are found (report-not-gate semantic). `forgelm reverse-pii` exits 0 even when no matches are found (Article 15 explicitly accepts "no records" as a valid answer). | -| 1 | Config error: bad flags, unreadable corpus, malformed regex, conflicting retention values, unwritable `--audit-dir`. | +| 1 | Config error: bad flags, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded, unwritable `--audit-dir`. | | 2 | Runtime error: I/O failure, permission denied, ReDoS SIGALRM timeout, atomic-rename failure. | Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of either subcommand's surface. diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 4e6433e1..6e6d1d7a 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -29,7 +29,7 @@ forgelm purge --row-id "ali@example.com" --corpus train.jsonl \ # Koşum-kapsamlı silme forgelm purge --run-id fg-abc123def456 --kind staging --output-dir ./outputs -forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs +forgelm purge --run-id fg-abc123def456 --kind artefacts --output-dir ./outputs # standart pakette hiçbir şey bulmaz, aşağıya bakın # Retention-policy raporu (salt-okunur) forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs @@ -49,9 +49,10 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( ### Neyi YAPMAZ - **Modelleri yeniden eğitmez.** Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözümdür. Tool `data.erasure_warning_memorisation` yayar; böylece boşluk görünür olur. -- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca belirli bir AB veya üye devlet hukuku yükümlülüğü işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. -- **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca onu embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. -- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (Madde 17(2)). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. +- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +- **İşaretsiz legacy `final_model.staging/` dizinini silmez.** `--kind staging`, `final_model.staging./` dizinini siler; legacy `final_model.staging/` dizinini yalnızca içinde run id'ye eşit bir `staging_run_id` dosyası varsa siler. ForgeLM bu işaretçiyi yazmaz, bu yüzden işaretsiz bir legacy dizini kendiniz silin. +- **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca run id'yi embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. +- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (Madde 19 verinin ifşa edildiği her alıcıya bildirim yapılmasını ister; Madde 17(2) ise kontrolör veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. ## Madde 15 — `forgelm reverse-pii` @@ -88,9 +89,9 @@ Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kend İki primitif kasıtlı olarak ayrıdır: -- `FORGELM_AUDIT_SECRET` rotasyonu **yalnızca chain HMAC anahtarını** rotate eder. Identifier hash'leri per-output-dir salt'tan türetilir ve **env-var rotasyonlarına karşı stabildir**; rotasyondan sonra aynı `--output-dir` üzerinde mount edilen bir `forgelm reverse-pii` sorgusu, rotasyondan ÖNCE kaydedilmiş `forgelm purge` event'leriyle hâlâ korele eder. +- `FORGELM_AUDIT_SECRET` rotasyonu chain HMAC anahtarını **ve**, değişken ayarlıysa, identifier-hash salt'ını rotate eder: `forgelm purge` secret'ın ilk 16 byte'ını per-output-dir salt'a XOR'lar ve `salt_source=env_var` kaydeder; bu yüzden `target_id` digest'leri secret ile değişir. Bir `forgelm reverse-pii` sorgusu, önceki purge event'leriyle yalnızca aynı salt kaynağını kullandığında korele eder (aynı secret ile `--salt-source env_var` ya da değişken olmadan koşturulmuş purge'lar için `--salt-source per_dir`). - Birini incelemek diğerini ifşa etmez. -- Per-output-dir salt, identifier hash'lemenin tek girdisidir; output dizininin yaşam süresi boyunca kalıcıdır ve chain HMAC'in rotasyon kadansını cross-tool digest korelasyonundan izole eder. +- Per-output-dir salt identifier hash'lemenin her zaman bir girdisidir ve output dizininin yaşam süresi boyunca kalıcıdır; secret, ayarlıysa, ikinci girdidir. ## Exit kodları @@ -99,7 +100,7 @@ Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kend | Kod | Anlamı | |---|---| | 0 | Başarı. İhlal bulunduğunda bile `forgelm purge --check-policy` 0 ile çıkar (report-not-gate semantik). Eşleşme bulunmadığında bile `forgelm reverse-pii` 0 ile çıkar (Madde 15 "kayıt yok"u geçerli yanıt olarak açıkça kabul eder). | -| 1 | Config hatası: kötü flag'ler, okunamaz corpus, hatalı regex, çakışan retention değerleri, yazılamaz `--audit-dir`. | +| 1 | Config hatası: kötü flag'ler, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`, yazılamaz `--audit-dir`. | | 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın da yüzeyinin parçası değildir. diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index e1c3140e..9344a58b 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -95,7 +95,7 @@ ForgeLM yerleşik bir quorum gate'i **göndermez**. `approval.quorum` alanı yok ## Timeout -ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler; deployer eski staging dizinlerinin sona ermesini istiyorsa bunun yerine `retention.staging_ttl_days`'i konfigüre edin — bu bir retention raporudur, süre sonu değildir: `forgelm purge --check-policy` horizon'dan eski staging dizinlerini listeler, hiçbir şey otomatik silinmez (birini `forgelm purge --run-id --kind staging` ile kaldırın) ve approval gate'e asla dokunmaz. +ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler; deployer eski staging dizinlerinin sona ermesini istiyorsa bunun yerine `retention.staging_ttl_days`'i konfigüre edin — bu bir retention raporudur, süre sonu değildir: `forgelm purge --check-policy` horizon'dan eski staging dizinlerini listeler, hiçbir şey otomatik silinmez (birini `forgelm purge --run-id --kind staging` ile kaldırın) ve bu ayar approval gate'e hiçbir şekilde dokunmaz. ## Bekleyen koşuları inceleme diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 128b8c80..794e8cdd 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -368,7 +368,7 @@ kaynağı kaydedilmemiştir. ```yaml retention: audit_log_retention_days: 1825 # 5 yıl: ForgeLM varsayılanı (proje tercihi; GDPR Madde 5(1)(e) sabit süre koymaz) - staging_ttl_days: 7 # purge --check-policy'nin geciken saydığı staging dizin yaşı + staging_ttl_days: 7 # staging dizininin purge --check-policy tarafından geciken sayıldığı yaş ephemeral_artefact_retention_days: 90 # uyumluluk paketleri, denetim raporları raw_documents_retention_days: 90 # ingest edilen PDF/DOCX/EPUB/TXT/Markdown enforce: "log_only" # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only | warn_on_excess | block_on_excess) diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index c6397ce9..a7fe6ec1 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1115,7 +1115,7 @@ def _add_purge_subcommand(subparsers) -> None: mode.add_argument( "--check-policy", action="store_true", - help="Read-only retention-policy violation report. Always exits 0 (report-not-gate).", + help="Read-only retention-policy violation report. A successful report exits 0 (report-not-gate); a --config that cannot be loaded exits 1.", ) p.add_argument( "--corpus", @@ -1143,14 +1143,14 @@ def _add_purge_subcommand(subparsers) -> None: type=str, default=None, metavar="DIR", - help="Output directory containing audit_log.jsonl + the per-output-dir salt file. Defaults to corpus parent dir in row mode; required in run mode.", + help="Output directory containing audit_log.jsonl + the per-output-dir salt file. Defaults to the corpus parent dir in row mode and to the current directory in run mode.", ) p.add_argument( "--config", type=str, default=argparse.SUPPRESS, metavar="PATH", - help="YAML config (optional in row/run mode; required for --check-policy). Detects the webhook block + retention horizons.", + help="YAML config. Row mode: detects the webhook block for the external-copies warning. --check-policy: supplies the `retention:` horizons (without it nothing is checked and the report says so). Run mode does not read it.", ) p.add_argument( "--justification", diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index b778c11f..e66dccd4 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -28,9 +28,9 @@ the chain shows ``request → fail`` instead of ``request → complete``, so an operator scanning the log can spot the unfinished erasure. - ``data.erasure_warning_memorisation`` — corpus row erasure ran - while a ``final_model/`` exists for any run that consumed this - corpus. The row is gone from disk but may still be memorised in - the trained weights. + while a ``final_model/`` directory exists directly under the output + dir (which runs consumed the corpus is not determined). The row is + gone from disk but may still be memorised in the trained weights. - ``data.erasure_warning_synthetic_data_present`` — corpus row erasure ran with synthetic-data snapshots present; the row may have produced derivative snippets the row→snippet mapping no longer connects. @@ -40,10 +40,11 @@ Exit-code contract (per ``docs/standards/error-handling.md``): -- 0 — success or ``--check-policy`` report (gate-not-report semantic - per design §10 Q5). -- 1 — config error (unknown ``row-id``, missing corpus, mutually - exclusive flag combination, conflicting retention values). +- 0 — success or a successful ``--check-policy`` report (report-not-gate + semantic per design §10 Q5). +- 1 — config error (unknown ``row-id``, no matching run-scoped target, + missing corpus, mutually exclusive flag combination, a ``--config`` + that cannot be loaded). - 2 — runtime error (I/O, permission denied, atomic rename failure). """ @@ -386,9 +387,9 @@ def _detect_warning_conditions( Per design §5.1: - - ``data.erasure_warning_memorisation`` when any ``final_model/`` - exists in the operator's output dir tree (a trained model may - have memorised the row). + - ``data.erasure_warning_memorisation`` when a ``final_model/`` + directory exists directly under the operator's output dir (a + trained model may have memorised the row). - ``data.erasure_warning_synthetic_data_present`` when any ``synthetic_data*.jsonl`` exists (the row may have produced derivative snippets). @@ -1010,9 +1011,12 @@ def _legacy_staging_owned_by(legacy_dir: Path, run_id: str) -> bool: def _artefact_targets_for_run(base: Path, run_id: str) -> List[Path]: """Return ``compliance/`` files whose name embeds ``run_id`` as a token. - Compliance bundle filenames embed the run-id; we accept both - ``compliance_.json`` and ``annex_iv_.json``. The - token-boundary check (`_filename_contains_run_id`) defends against + Any regular file directly under ``compliance/`` whose name carries + the run-id as a token is matched, whatever its extension. The + standard bundle ForgeLM writes uses fixed basenames + (``compliance_report.json`` ...) with no run-id, so it is never matched + (pinned by ``test_artefacts_kind_finds_nothing_in_the_standard_fixed_name_bundle``). + The token-boundary check (`_filename_contains_run_id`) defends against short-run-id false positives where a substring match would accidentally pick up files belonging to a different run. """ diff --git a/forgelm/config.py b/forgelm/config.py index b9979b66..a2eec696 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1603,7 +1603,7 @@ class RetentionConfig(BaseModel): exits 0; nothing is deleted, blocked or scheduled automatically. ``enforce`` is reserved: no command acts on it yet, so ``warn_on_excess`` and ``block_on_excess`` behave like ``log_only`` - (the CLI logs a warning when a config sets either). + (training and ``--dry-run`` log a warning when a config sets either). """ model_config = ConfigDict(extra="forbid") @@ -1629,25 +1629,25 @@ class RetentionConfig(BaseModel): default=90, ge=0, description=( - "Days to retain compliance bundles, data audit reports, and other run-scoped derived artefacts. " - "Set to 0 to retain indefinitely." + "Age in days after which `forgelm purge --check-policy` reports `/compliance/` and " + "`/data_audit_report.json` as overdue; nothing is deleted. Set to 0 to retain indefinitely." ), ) raw_documents_retention_days: int = Field( default=90, ge=0, description=( - "Days to retain ingested raw documents (PDF / DOCX / EPUB / TXT / Markdown) under " - "the operator's ingestion-output directory. Set to 0 to retain indefinitely. " + "Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or " + "`/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue; nothing is deleted. Set to 0 to retain indefinitely. " "Closes ghost-features GH-023 (was nested as `ingestion.retention.raw_documents.ttl_days`; now top-level)." ), ) enforce: Literal["log_only", "warn_on_excess", "block_on_excess"] = Field( default="log_only", description=( - "Reserved: no command reads it yet. Retention horizons are only reported, read-only, by `forgelm purge " + "Reserved: no command acts on it. Retention horizons are only reported, read-only, by `forgelm purge " "--check-policy` (always exit 0); nothing is deleted, warned about or blocked automatically. " - "`warn_on_excess` and `block_on_excess` are accepted but behave like `log_only`; the CLI warns when set." + "`warn_on_excess` and `block_on_excess` are accepted but behave like `log_only`; training and `--dry-run` warn when set." ), ) diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index 56b2e563..3e34ff42 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -351,7 +351,9 @@ def test_only_the_purge_reader_names_the_legacy_staging_marker(self) -> None: n.func.attr for n in ast.walk(fn) if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute) } assert total == inside_reader == 1, "the marker name must appear only in _legacy_staging_owned_by" - assert not reader_calls & {"write_text", "write", "touch", "mkdir"}, "the marker reader must stay read-only" + assert reader_calls <= {"is_file", "exists", "read_text", "strip"}, ( + f"the marker reader must stay read-only: {reader_calls}" + ) # --------------------------------------------------------------------------- diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 08931ae6..042679d2 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -99,7 +99,7 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"varsayılan 5 yıl \(Madde 5\(1\)\(e\)\)", r"legal-obligation\s+record\s+under", r"compliant\s+out\s+of\s+the\s+box", - r"Article\s+12\s+record-keeping\s+(?:baseline|\))", + r"years\s+\(Article\s+12\s+record-keeping\)|Article\s+12\s+record-keeping\s+baseline", r"Horizons\s+honou?r\s+GDPR", r"removed\s+by\s+default\s+after\s+a\s+defined\s+retention\s+horizon", r"after\s+a\s+`forgelm\s+reject`\s+decision\s+before\s+scheduled\s+cleanup", @@ -111,9 +111,18 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"fiziksel\s+olarak\s+prune", r"until\s+retention\s+expires", r"retention\s+süresi\s+dolana", - r"justified\s+only\s+to\s+the\s+extent", - r"name\s+the\s+Union\s+or\s+Member\s+State\s+obligation", r"compliance/\*\.json`\s+files\s+whose\s+name\s+embeds", + r"staging\s+directory,?\s+or\s+(?:the\s+|a\s+)?compliance\s+bundle", + r"compliance\s+bundle'ı(?:nı)?\s+(?:siler|atomik)", + r"compliance/\*\.json`\s+dosyalar", + r"retain\s+staging\s+model\s+after\s+forgelm\s+reject", + r"forgelm\s+reject\s+sonrası\s+staging", + r"TTL\s+after\s+`forgelm\s+reject`", + r"planlı\s+temizlenmeden", + r"bir\s+iş\s+haftası", + r"onurlandırır", + r"17\(3\)\(b\)\s+protects", + r"17\(3\)\(b\)[^.\n]{0,40}\bkorur", ) _PURGE_ARTEFACT_WARNING_FILES = ( @@ -133,6 +142,22 @@ def test_retired_gdpr_claims_stay_out(rel: str) -> None: assert found == [], f"{rel} still carries retired GDPR wording: {found}" +# Positive invariant, so a reworded version of the retired claim cannot slip past the phrase list: every mention of +# Article 17(3)(b) must sit next to a qualifier that limits it ("to the extent", "still requires", Turkish "ölçüde"). +_CONDITIONAL_17_3_B = re.compile(r"to\s+the\s+extent|only\s+to\s|still\s+requires|ölçüde", re.IGNORECASE) + + +@pytest.mark.parametrize("rel", _GDPR_CLAIM_FILES) +def test_every_article_17_3_b_mention_is_conditional(rel: str) -> None: + text = (_REPO_ROOT / rel).read_text(encoding="utf-8") + unqualified = [ + text[max(0, m.start() - 60) : m.end() + 60] + for m in re.finditer(r"17\(3\)\(b\)", text) + if not _CONDITIONAL_17_3_B.search(text[max(0, m.start() - 250) : m.end() + 250]) + ] + assert unqualified == [], f"{rel}: Article 17(3)(b) mentioned without a limiting qualifier: {unqualified}" + + @pytest.mark.parametrize("rel", _PURGE_ARTEFACT_WARNING_FILES) def test_purge_docs_warn_that_artefacts_misses_the_standard_bundle(rel: str) -> None: # CR-A10-002 / DOC-D1B05-005: the exporter writes fixed basenames and the From 452c8414051461bf2be7d0bde63d9027833b0c99 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:50:50 +0300 Subject: [PATCH 04/25] docs(pii): cite the GDPR articles that actually apply to the detector set and the severity tiers The regex detector set and the severity tier table were attributed to GDPR provisions that do not say that. Art. 9 lists special categories of data (national identification numbers are not among them; Art. 87 addresses them), and Art. 10 covers criminal-conviction data. The identifiers are personal data because of Art. 4(1). "GDPR-mandated structured identifiers" in three code comments claimed a mandate GDPR does not make. - The tier table cites Art. 87 and Art. 32 plus ENISA for national IDs (data_audit guide EN and TR, `PII_SEVERITY` comment). - The `--pii-ml` page and the data-audit guide describe the detector set as the common structured identifiers that make data personal under Art. 4(1) (see also Art. 87) instead of "what Article 10 cares about". - `_pii_regex.py`, `_optional.py` and `pyproject.toml` comments say "common structured identifiers". - The retired-phrase test covers all of these files. Resolves DOC-BLEGAL-068, DOC-BLEGAL-126, DOC-BLEGAL-179, DOC-BLEGAL-180, DOC-D3B07-044 and the citation half of TECH-T11-002 (Refs #101). Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/data_audit-tr.md | 8 +++++--- docs/guides/data_audit.md | 8 +++++--- docs/usermanuals/en/data/pii-ml.md | 2 +- docs/usermanuals/tr/data/pii-ml.md | 2 +- forgelm/data_audit/_optional.py | 2 +- forgelm/data_audit/_pii_regex.py | 4 ++-- forgelm/data_audit/_types.py | 4 ++-- pyproject.toml | 2 +- tests/test_usermanual_reference_consistency.py | 15 +++++++++++++++ 9 files changed, 33 insertions(+), 14 deletions(-) diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 74386f96..06a90272 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -161,7 +161,8 @@ kötü?* sorusunda sıfır rehberlik veriyor. Faz 11.5 yanına bir ``` Tier tablosu konsensüs regülatif ağırlıklandırmadır (finansal -kimlikler için PCI-DSS; devlet kimlikleri için GDPR Md. 9 + ENISA). +kimlikler için PCI-DSS; devlet kimlikleri için GDPR Madde 87 (ulusal kimlik +numaraları) ve Madde 32 + ENISA). `critical` katman artık komutun kendisini kapılar (sonraki bölüm); `high`'ta da durmak isteyen pipeline'lar `pii_severity.worst_tier`'i okumalı ve açık bir review olmadan yayınlamayı reddetmelidir. @@ -406,8 +407,9 @@ install ile uyumlu kalır. ### ML-NER PII adaptörü — `--pii-ml` (Faz 12.5, opt-in) -Varsayılan regex detector, EU AI Act Madde 10'un önemsediği yapılandırılmış -identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. +Varsayılan regex detector, yaygın yapılandırılmış identifier'ları (email, +telefon, IBAN, kredi kartı, ulusal kimlik) kapsar; bunlar GDPR Madde 4(1) +kapsamında kişisel veridir. Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier kategorilerini ekleyen opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 81dd7e04..450af127 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -151,7 +151,8 @@ The flat `pii_summary` map gives compliance reviewers no guidance on ``` The tier table is consensus regulatory weighting (PCI-DSS for financial -identifiers; GDPR Art. 9 + ENISA for government IDs). The `critical` +identifiers; GDPR Art. 87 (national identification numbers) and Art. 32 +plus ENISA for government IDs). The `critical` tier now gates the command itself (next section); pipelines that also want to stop at `high` should read `pii_severity.worst_tier` and refuse to publish without explicit review. @@ -400,8 +401,9 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. ### ML-NER PII adapter — `--pii-ml` (Phase 12.5, opt-in) -The default regex detector covers the structured identifiers EU AI Act -Article 10 cares about (email, phone, IBAN, credit card, national IDs). +The default regex detector covers the common structured identifiers +(email, phone, IBAN, credit card, national IDs), which are personal data +under GDPR Art. 4(1). Phase 12.5 adds an opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adapter that layers ML-NER on top — adding the unstructured-identifier categories regex inherently misses: `person`, `organization`, `location`. diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index db09c611..e49a2068 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -5,7 +5,7 @@ description: Optional Microsoft Presidio adapter that adds person / organization # ML-NER PII Detection — `--pii-ml` -The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the *structured* identifiers GDPR Article 10 cares about (email, phone, IBAN, credit card, national IDs). Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. +The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the *structured* identifiers that make data personal under GDPR Article 4(1) (email, phone, IBAN, credit card, national IDs; see also Article 87). Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. This is an **opt-in** layer. The default audit + ingest behaviour is unchanged when `--pii-ml` is not passed. diff --git a/docs/usermanuals/tr/data/pii-ml.md b/docs/usermanuals/tr/data/pii-ml.md index 556d7cdc..59da063b 100644 --- a/docs/usermanuals/tr/data/pii-ml.md +++ b/docs/usermanuals/tr/data/pii-ml.md @@ -5,7 +5,7 @@ description: Regex PII katmanının üzerine person / organization / location te # ML-NER PII Tespiti — `--pii-ml` -Varsayılan [PII Maskeleme](#/data/pii-masking) katmanı regex-anchored'dır ve GDPR Madde 10'un önemsediği *yapılandırılmış* identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlikler) kapsar. Presidio NER, regex'in doğal olarak kaçırdığı *yapılandırılmamış* identifier'ları ekler: kişi isimleri, organizasyon isimleri, coğrafi yerler. +Varsayılan [PII Maskeleme](#/data/pii-masking) katmanı regex-anchored'dır ve GDPR Madde 4(1) kapsamında veriyi kişisel yapan *yapılandırılmış* identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlikler; ayrıca Madde 87) kapsar. Presidio NER, regex'in doğal olarak kaçırdığı *yapılandırılmamış* identifier'ları ekler: kişi isimleri, organizasyon isimleri, coğrafi yerler. Bu katman **opt-in**'dir. `--pii-ml` geçilmediğinde varsayılan audit + ingest davranışı değişmez. diff --git a/forgelm/data_audit/_optional.py b/forgelm/data_audit/_optional.py index d8becea5..20ff67e1 100644 --- a/forgelm/data_audit/_optional.py +++ b/forgelm/data_audit/_optional.py @@ -66,7 +66,7 @@ # the regex detector so the default audit keeps its zero-extra-deps # guarantee. Presidio + spaCy are heavyweight (~50 MB model download), # so we stay strictly opt-in and fail soft when missing — the regex set -# already covers the GDPR-mandated structured identifiers (email, phone, +# already covers the common structured identifiers (email, phone, # IBAN, credit card, national IDs) that the audit's compliance contract # is built around. ML adds the *unstructured* identifiers regex inherently # misses: person names, organizations, locations. diff --git a/forgelm/data_audit/_pii_regex.py b/forgelm/data_audit/_pii_regex.py index 70937752..775b821c 100644 --- a/forgelm/data_audit/_pii_regex.py +++ b/forgelm/data_audit/_pii_regex.py @@ -1,8 +1,8 @@ """Regex-based PII detector + masker. -Prefix-anchored / shape-anchored patterns covering the GDPR-mandated +Prefix-anchored / shape-anchored patterns covering the common structured identifiers (email, phone, IBAN, credit card, national IDs). -These are the categories every audit *must* surface; the optional +These are the categories every audit surfaces by default; the optional Presidio ML-NER adapter (:mod:`forgelm.data_audit._pii_ml`) layers on top to pick up unstructured identifiers (person names, organizations, locations) which regex inherently misses. diff --git a/forgelm/data_audit/_types.py b/forgelm/data_audit/_types.py index c63e4512..5eb83299 100644 --- a/forgelm/data_audit/_types.py +++ b/forgelm/data_audit/_types.py @@ -23,8 +23,8 @@ # guidance to a compliance reviewer staring at the audit JSON — they have to # remember by hand that ``credit_card`` is materially worse than ``phone``. # The tiers below encode the consensus regulatory weighting (PCI-DSS for -# financial; GDPR Art. 9 + ENISA for identifiers) so the audit surfaces a -# one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when local +# financial; GDPR Art. 87 + Art. 32 and ENISA for national IDs) so the audit +# surfaces a one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when local # law disagrees — the audit honours whatever map is loaded at call time. PII_SEVERITY: Dict[str, str] = { # Financial / fully reversible identity theft -> highest weight. diff --git a/pyproject.toml b/pyproject.toml index 6c754305..5db58fed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -172,7 +172,7 @@ ingestion-scale = [ # categories (person / organization / location) that the regex detector # inherently misses. Heavyweight install (≈ 50 MB spaCy model on first # analyzer build), so opt-in only — the regex set already covers the -# GDPR-mandated structured identifiers (email, phone, IBAN, credit card, +# common structured identifiers (email, phone, IBAN, credit card, # national IDs). ingestion-pii-ml = [ "presidio-analyzer>=2.2.0,<3.0.0", diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 042679d2..4999d563 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -85,6 +85,14 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: "docs/usermanuals/tr/compliance/human-oversight.md", "docs/qms/sop_change_management.md", "docs/qms/sop_change_management-tr.md", + "forgelm/data_audit/_types.py", + "forgelm/data_audit/_pii_regex.py", + "forgelm/data_audit/_optional.py", + "pyproject.toml", + "docs/guides/data_audit.md", + "docs/guides/data_audit-tr.md", + "docs/usermanuals/en/data/pii-ml.md", + "docs/usermanuals/tr/data/pii-ml.md", ) _RETIRED_GDPR_CLAIMS = ( @@ -123,6 +131,13 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"onurlandırır", r"17\(3\)\(b\)\s+protects", r"17\(3\)\(b\)[^.\n]{0,40}\bkorur", + # The PII tier table and detector set are not a GDPR-mandated list: Art. 9 covers special categories (not + # national IDs, which Art. 87 addresses) and Art. 10 covers criminal-conviction data. + r"GDPR\s+Art\.\s+9\s*\+\s*ENISA", + r"GDPR\s+Md\.\s+9\s*\+\s*ENISA", + r"Article\s+10\s+cares", + r"Madde\s+10'un\s+önemsediği", + r"GDPR-mandated", ) _PURGE_ARTEFACT_WARNING_FILES = ( From 3428631052e10d65c0a07e197611bc8d5996dfaa Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:51:54 +0300 Subject: [PATCH 05/25] docs(gdpr): make the reverse-pii example runnable, point names to --pii-ml, fix the site card and retention hints MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - GDPR / KVKK page (EN, TR): the access-request example now passes `--output-dir ./outputs` and shows the real zero-match line, "No matches for identifier hash <16 hex>… (literal, plaintext mode) in N file(s)." It says what `--output-dir` controls (the per-directory salt and the `data.access_request_query` event, which join `forgelm purge`'s chain when the directory matches; without it both land next to the first corpus file with a warning) and that match snippets are raw corpus text. Names, organisations and locations are not masked; `forgelm audit --pii-ml` can detect them (Presidio NER, detection only, needs the `ingestion-pii-ml` extra and a spaCy model). - Site, six languages plus the HTML fallback: the home GDPR card no longer says the two commands are "Article 17 erasure" and "Article 15 access". It says `reverse-pii` locates a subject's rows (an input to an Article 15 response), `purge` removes them from training corpora (one step of Article 17 erasure; trained weights are not changed), and both are audited. The wizard's retention hints no longer say the 5-year default "matches EU AI Act guidance" or that the staging horizon keeps "rejected models"; they say what `forgelm purge --check-policy` reports. The German, French, Spanish and Chinese strings were written without a native review, as in the earlier six-language site pass; English and Turkish are authoritative. - Tests: the retired-phrase list covers the original GDPR page claims (EN and TR), the example is pinned to its real command and output line, and the site tests keep the old card and hint wording out of every locale. Resolves CR-A11-001 (with the earlier page rewrite), DOC-D3B02-047, DOC-BLEGAL-047 and, as already fixed on development, DOC-D3B02-001, -007, -008, -009, -010, -024, DOC-BYAML-004, DOC-BLEGAL-041 and DOC-BLEGAL-050 (Refs #101). Co-Authored-By: Claude Sonnet 5.5 --- docs/usermanuals/en/compliance/gdpr.md | 7 +-- docs/usermanuals/tr/compliance/gdpr.md | 7 +-- site/index.html | 2 +- site/js/translations.js | 36 ++++++++-------- tests/site/site_translations.test.cjs | 23 +++++++++- .../test_usermanual_reference_consistency.py | 43 +++++++++++++++++++ 6 files changed, 91 insertions(+), 27 deletions(-) diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index d0f37a87..d60ca980 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -31,7 +31,7 @@ PII masking at ingest replaces the identifier formats ForgeLM detects with a pla $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. See [PII Masking](#/data/pii-masking). +The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. To find names, organisations and locations, run `forgelm audit --pii-ml`: it adds Presidio NER **detection** (it reports them, it does not mask them) and needs the `ingestion-pii-ml` extra and a spaCy model. See [PII Masking](#/data/pii-masking) and [ML-NER PII](#/data/pii-ml). ### Accountability (Art. 5(2)) @@ -66,10 +66,11 @@ Most-common request types and how ForgeLM helps: Search your training data for the person's identifier: ```shell -$ forgelm reverse-pii --query "ali@example.com" data/*.jsonl +$ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.jsonl +No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -The scan reports the lines where the identifier appears; zero matches is a valid answer. It finds what is literally in the files you point it at. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. +The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed hash is the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning. It finds what is literally in the files you point it at. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. ### Right to erasure (Art. 17) diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index 152a5c50..461bebe2 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -31,7 +31,7 @@ Ingest'te PII maskeleme, ForgeLM'in tespit ettiği tanımlayıcı biçimlerini b $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. Bkz. [PII Maskeleme](#/data/pii-masking). +Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. İsimleri, kurumları ve yerleri bulmak için `forgelm audit --pii-ml` çalıştırın: Presidio NER **tespiti** ekler (raporlar, maskelemez) ve `ingestion-pii-ml` extra'sını ile bir spaCy modeli gerektirir. Bkz. [PII Maskeleme](#/data/pii-masking) ve [ML-NER PII](#/data/pii-ml). ### Hesap verebilirlik (Md. 5(2)) @@ -66,10 +66,11 @@ En sık talep tipleri ve ForgeLM nasıl yardım eder: Kişinin tanımlayıcısını eğitim verinizde arayın: ```shell -$ forgelm reverse-pii --query "ali@example.com" data/*.jsonl +$ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.jsonl +No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -Tarama, tanımlayıcının geçtiği satırları raporlar; sıfır eşleşme geçerli bir cevaptır. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. +Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan hash, audit event'inin kaydettiği tuzlu tanımlayıcı hash'idir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. ### Silme hakkı (Md. 17) diff --git a/site/index.html b/site/index.html index dce4c389..e888ec77 100644 --- a/site/index.html +++ b/site/index.html @@ -536,7 +536,7 @@

ISO 27001 / SOC 2 alignment

GDPR rights tooling

-

Article 17 erasure with forgelm purge and Article 15 access with forgelm reverse-pii, both wired into the audit log.

+

Tooling for GDPR data-subject requests: forgelm reverse-pii locates a subject's rows (an input to an Article 15 response); forgelm purge removes them from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

forgelm purge forgelm reverse-pii diff --git a/site/js/translations.js b/site/js/translations.js index ac982eb6..2b728f11 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -390,7 +390,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 alignment", "home.cap.iso.body": "Audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor asks for. Software is aligned, not certified.", "home.cap.gdpr.title": "GDPR rights tooling", - "home.cap.gdpr.body": "Article 17 erasure with forgelm purge and Article 15 access with forgelm reverse-pii, both wired into the audit log.", + "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii locates a subject's rows (an input to an Article 15 response); forgelm purge removes them from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", "home.cap.see_all": "See the full feature matrix", "home.yaml.eyebrow": "A real config — not a screenshot", "home.yaml.title": "One YAML, end-to-end run", @@ -496,7 +496,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 hizalaması", "home.cap.iso.body": "Deployer denetçisinin sorduğu audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtları. Yazılım sertifikalı değil, hizalanmıştır.", "home.cap.gdpr.title": "GDPR hakları araçları", - "home.cap.gdpr.body": "forgelm purge ile Madde 17 silme ve forgelm reverse-pii ile Madde 15 erişim — her ikisi de audit log'a bağlı.", + "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii bir öznenin satırlarını bulur (Madde 15 yanıtının bir girdisi); forgelm purge bunları eğitim corpus'larından siler (Madde 17 silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", "home.cap.see_all": "Tüm özellik matrisini gör", "home.yaml.eyebrow": "Gerçek bir konfigürasyon — ekran görüntüsü değil", "home.yaml.title": "Bir YAML, uçtan uca koşu", @@ -602,7 +602,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 Alignment", "home.cap.iso.body": "Audit-Trail, Change-Management, Daten-Lineage und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Software ist aligned, nicht zertifiziert.", "home.cap.gdpr.title": "DSGVO-Rechtetooling", - "home.cap.gdpr.body": "Artikel-17-Löschung mit forgelm purge und Artikel-15-Auskunft mit forgelm reverse-pii — beide ans Audit-Log angebunden.", + "home.cap.gdpr.body": "Werkzeuge für DSGVO-Betroffenenanfragen: forgelm reverse-pii findet die Zeilen einer betroffenen Person (ein Baustein der Antwort nach Artikel 15); forgelm purge entfernt sie aus Trainingskorpora (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Beides wird im Audit-Log festgehalten.", "home.cap.see_all": "Vollständige Funktionsmatrix anzeigen", "home.yaml.eyebrow": "Eine echte Konfiguration — kein Screenshot", "home.yaml.title": "Ein YAML, ein Ende-zu-Ende-Lauf", @@ -708,7 +708,7 @@ "home.cap.iso.title": "Alignement ISO 27001 / SOC 2", "home.cap.iso.body": "Piste d'audit, gestion du changement, lignée des données et preuves de chaîne logistique demandées par l'auditeur du déployeur. Le logiciel est aligné, pas certifié.", "home.cap.gdpr.title": "Outils de droits RGPD", - "home.cap.gdpr.body": "Effacement Article 17 avec forgelm purge et accès Article 15 avec forgelm reverse-pii — les deux connectés au journal d'audit.", + "home.cap.gdpr.body": "Outils pour les demandes de personnes concernées au titre du RGPD : forgelm reverse-pii localise les lignes d'une personne (un élément de la réponse à l'article 15) ; forgelm purge les supprime des corpus d'entraînement (une étape de l'effacement de l'article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", "home.cap.see_all": "Voir la matrice complète des fonctionnalités", "home.yaml.eyebrow": "Une vraie configuration — pas une capture d'écran", "home.yaml.title": "Un YAML, exécution de bout en bout", @@ -814,7 +814,7 @@ "home.cap.iso.title": "Alineación con ISO 27001 / SOC 2", "home.cap.iso.body": "Pista de auditoría, gestión del cambio, linaje de datos y evidencias de cadena de suministro que pide el auditor del implementador. El software está alineado, no certificado.", "home.cap.gdpr.title": "Herramientas de derechos RGPD", - "home.cap.gdpr.body": "Borrado del Artículo 17 con forgelm purge y acceso del Artículo 15 con forgelm reverse-pii — ambos conectados al log de auditoría.", + "home.cap.gdpr.body": "Herramientas para solicitudes de interesados del RGPD: forgelm reverse-pii localiza las filas de un sujeto (un insumo para la respuesta del Artículo 15); forgelm purge las elimina de los corpus de entrenamiento (un paso del borrado del Artículo 17; los pesos entrenados no cambian). Ambos quedan registrados en el log de auditoría.", "home.cap.see_all": "Ver la matriz completa de características", "home.yaml.eyebrow": "Una configuración real — no una captura", "home.yaml.title": "Un YAML, ejecución de extremo a extremo", @@ -920,7 +920,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 对齐", "home.cap.iso.body": "部署方审计员要求的审计轨迹、变更管理、数据沿袭和供应链证据。软件是对齐的,不是认证的。", "home.cap.gdpr.title": "GDPR 权利工具", - "home.cap.gdpr.body": "通过 forgelm purge 实现第 17 条删除,通过 forgelm reverse-pii 实现第 15 条访问 — 两者都接入审计日志。", + "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 定位某数据主体的行(第 15 条答复的输入之一),forgelm purge 将这些行从训练语料中删除(第 17 条删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", "home.cap.see_all": "查看完整功能矩阵", "home.yaml.eyebrow": "真实配置 — 不是截图", "home.yaml.title": "一个 YAML,端到端运行", @@ -1813,9 +1813,9 @@ "wizard.retention.title": "Retention horizons (GDPR Art. 5+17)", "wizard.retention.hint": "How long ForgeLM keeps audit logs, staging models, and ephemeral artefacts.", "wizard.retention.audit_log.label": "Audit log retention (days)", - "wizard.retention.audit_log.hint": "How long to keep audit_log.jsonl. Default 1825 (5 years) matches EU AI Act guidance.", + "wizard.retention.audit_log.hint": "How long before forgelm purge --check-policy reports audit_log.jsonl as overdue. Default 1825 (5 years) is a ForgeLM choice, not a legal period.", "wizard.retention.staging.label": "Staging horizon (days)", - "wizard.retention.staging.hint": "How long to keep rejected models in final_model.staging/. Default 7.", + "wizard.retention.staging.hint": "Age at which forgelm purge --check-policy reports a final_model.staging/ directory as overdue; nothing is deleted automatically. Default 7.", "wizard.retention.ephemeral.label": "Ephemeral artefacts (days)", "wizard.retention.ephemeral.hint": "How long to keep intermediate checkpoints, eval samples. Default 90.", "wizard.retention.raw_documents.label": "Raw documents (days)", @@ -2211,9 +2211,9 @@ "wizard.retention.title": "Saklama ufukları (GDPR Madde 5+17)", "wizard.retention.hint": "ForgeLM audit log, staging model ve geçici artefact'ları ne kadar tutar.", "wizard.retention.audit_log.label": "Audit log saklama (gün)", - "wizard.retention.audit_log.hint": "audit_log.jsonl ne kadar tutulsun. Varsayılan 1825 (5 yıl) EU AI Act rehberiyle uyumlu.", + "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra geciken olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", "wizard.retention.staging.label": "Staging ufku (gün)", - "wizard.retention.staging.hint": "final_model.staging/ içinde reddedilen modeller ne kadar tutulsun. Varsayılan 7.", + "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging/ dizinini hangi yaşta geciken olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", "wizard.retention.ephemeral.label": "Geçici artefact'lar (gün)", "wizard.retention.ephemeral.hint": "Ara checkpoint'ler, eval örnekleri ne kadar tutulsun. Varsayılan 90.", "wizard.retention.raw_documents.label": "Ham dokümanlar (gün)", @@ -2605,9 +2605,9 @@ "wizard.retention.title": "Aufbewahrungsfristen (DSGVO Art. 5+17)", "wizard.retention.hint": "Wie lange ForgeLM Audit-Logs, Staging-Modelle, ephemere Artefakte aufbewahrt.", "wizard.retention.audit_log.label": "Audit-Log Aufbewahrung (Tage)", - "wizard.retention.audit_log.hint": "Wie lange audit_log.jsonl behalten. Default 1825 (5 Jahre) entspricht EU AI Act-Vorgabe.", + "wizard.retention.audit_log.hint": "Nach wie vielen Tagen forgelm purge --check-policy audit_log.jsonl als überfällig meldet. Default 1825 (5 Jahre) ist eine ForgeLM-Wahl, keine gesetzliche Frist.", "wizard.retention.staging.label": "Staging-Horizont (Tage)", - "wizard.retention.staging.hint": "Wie lange abgelehnte Modelle in final_model.staging/ behalten. Default 7.", + "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging/-Verzeichnis als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", "wizard.retention.ephemeral.label": "Ephemere Artefakte (Tage)", "wizard.retention.ephemeral.hint": "Wie lange Zwischen-Checkpoints, Eval-Samples behalten. Default 90.", "wizard.retention.raw_documents.label": "Rohdokumente (Tage)", @@ -2999,9 +2999,9 @@ "wizard.retention.title": "Horizons de rétention (RGPD Art. 5+17)", "wizard.retention.hint": "Combien de temps ForgeLM garde audit logs, modèles staging, artefacts éphémères.", "wizard.retention.audit_log.label": "Rétention audit log (jours)", - "wizard.retention.audit_log.hint": "Combien de temps garder audit_log.jsonl. Default 1825 (5 ans) match EU AI Act.", + "wizard.retention.audit_log.hint": "Après combien de jours forgelm purge --check-policy signale audit_log.jsonl comme dépassé. Default 1825 (5 ans) est un choix de ForgeLM, pas un délai légal.", "wizard.retention.staging.label": "Horizon staging (jours)", - "wizard.retention.staging.hint": "Combien de temps garder modèles rejetés dans final_model.staging/. Default 7.", + "wizard.retention.staging.hint": "À partir de quel âge forgelm purge --check-policy signale un répertoire final_model.staging/ comme dépassé ; rien n'est supprimé automatiquement. Default 7.", "wizard.retention.ephemeral.label": "Artefacts éphémères (jours)", "wizard.retention.ephemeral.hint": "Combien de temps garder checkpoints intermédiaires, samples eval. Default 90.", "wizard.retention.raw_documents.label": "Documents bruts (jours)", @@ -3393,9 +3393,9 @@ "wizard.retention.title": "Horizontes de retención (RGPD Art. 5+17)", "wizard.retention.hint": "Cuánto tiempo ForgeLM guarda audit logs, modelos staging, artefactos efímeros.", "wizard.retention.audit_log.label": "Retención audit log (días)", - "wizard.retention.audit_log.hint": "Cuánto guardar audit_log.jsonl. Default 1825 (5 años) coincide con EU AI Act.", + "wizard.retention.audit_log.hint": "Tras cuántos días forgelm purge --check-policy informa audit_log.jsonl como vencido. Default 1825 (5 años) es una elección de ForgeLM, no un plazo legal.", "wizard.retention.staging.label": "Horizonte staging (días)", - "wizard.retention.staging.hint": "Cuánto guardar modelos rechazados en final_model.staging/. Default 7.", + "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy informa un directorio final_model.staging/ como vencido; no se borra nada automáticamente. Default 7.", "wizard.retention.ephemeral.label": "Artefactos efímeros (días)", "wizard.retention.ephemeral.hint": "Cuánto guardar checkpoints intermedios, samples eval. Default 90.", "wizard.retention.raw_documents.label": "Documentos crudos (días)", @@ -3787,9 +3787,9 @@ "wizard.retention.title": "保留期限(GDPR 第 5+17 条)", "wizard.retention.hint": "ForgeLM 保留 audit log、staging 模型、临时 artefact 多长时间。", "wizard.retention.audit_log.label": "Audit log 保留(天)", - "wizard.retention.audit_log.hint": "audit_log.jsonl 保留多久。默认 1825(5 年)符合 EU AI Act。", + "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已过期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", "wizard.retention.staging.label": "Staging 期限(天)", - "wizard.retention.staging.hint": "final_model.staging/ 中被拒绝的模型保留多久。默认 7。", + "wizard.retention.staging.hint": "final_model.staging/ 目录存在多少天后,forgelm purge --check-policy 会将其报告为已过期;不会自动删除任何内容。默认 7。", "wizard.retention.ephemeral.label": "临时 artefact(天)", "wizard.retention.ephemeral.hint": "中间 checkpoint、eval 样本保留多久。默认 90。", "wizard.retention.raw_documents.label": "原始文档(天)", diff --git a/tests/site/site_translations.test.cjs b/tests/site/site_translations.test.cjs index 4cabd82b..b1b6df55 100644 --- a/tests/site/site_translations.test.cjs +++ b/tests/site/site_translations.test.cjs @@ -38,9 +38,11 @@ test('claims the code does not back stay out of the site copy', () => { en: [/dual-control/i, /\bauto-deletes\b/, /run fails \(exit 3\)/, /satisfying the access-request/, /Slack, Teams, Discord, or any HTTP/, /Slack \/ Teams \/ Discord/, /enforced upstream/, /GDPR-compliant TTLs/, /SHA-256 \+ size manifest/, /6 quant levels and SHA-256 manifest/, /ZeRO-2 \/ ZeRO-3 presets/, - /PII redacted at ingest/, /GGUF export at q4_k_m/], + /PII redacted at ingest/, /GGUF export at q4_k_m/, /both wired into the audit log/, /matches EU AI Act guidance/, + /rejected models in final_model\.staging/], tr: [/ikili[- ]kontrol/i, /otomatik siler/, /GDPR uyumlu TTL/, /erişim-talebi tarafını karşılar/, - /Slack, Teams, Discord veya/, /Slack \/ Teams \/ Discord/, /upstream'de uygulanır/], + /Slack, Teams, Discord veya/, /Slack \/ Teams \/ Discord/, /upstream'de uygulanır/, /her ikisi de audit log'a bağlı/, + /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/], }; const found = []; for (const [locale, patterns] of Object.entries(retired)) { @@ -55,5 +57,22 @@ test('claims the code does not back stay out of the site copy', () => { assert.deepEqual(found, []); for (const [locale, table] of Object.entries(translations)) { assert.match(table['wizard.retention.enforce.hint'], /--check-policy/, `${locale}: retention is only reported`); + for (const key of ['wizard.retention.audit_log.hint', 'wizard.retention.staging.hint']) { + assert.match(table[key], /--check-policy/, `${locale} ${key}: a horizon is only reported, not enforced`); + } + } +}); + +// The home-page GDPR card says what the two commands are for (an input to an Article 15 response, one step of an +// Article 17 erasure) and that trained weights are untouched. Every locale keeps the weights caveat. +test('the home GDPR card keeps the trained-weights caveat in every locale', () => { + const { translations } = loadWizard(); + const caveat = { en: /trained weights are not changed/, tr: /eğitilmiş ağırlıklar değişmez/, + de: /Gewichte bleiben unverändert/, fr: /poids entraînés ne changent pas/, es: /pesos entrenados no cambian/, + zh: /权重不会改变/ }; + for (const [locale, pattern] of Object.entries(caveat)) { + assert.match(translations[locale]['home.cap.gdpr.body'], pattern, `${locale}: home.cap.gdpr.body`); } + const html = fs.readFileSync(path.join(root, 'site', 'index.html'), 'utf8'); + assert.match(html, /trained weights are not changed/, 'index.html static fallback'); }); diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 4999d563..7f2545e2 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -93,6 +93,8 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: "docs/guides/data_audit-tr.md", "docs/usermanuals/en/data/pii-ml.md", "docs/usermanuals/tr/data/pii-ml.md", + "docs/usermanuals/en/compliance/gdpr.md", + "docs/usermanuals/tr/compliance/gdpr.md", ) _RETIRED_GDPR_CLAIMS = ( @@ -138,6 +140,22 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"Article\s+10\s+cares", r"Madde\s+10'un\s+önemsediği", r"GDPR-mandated", + # The GDPR / KVKK page (CR-A11-001): masking is a fixed regex set, not anonymisation, and no audit report can + # confirm that a model holds nothing about a person. + r"identifiable\s+subjects\s+are\s+removed", + r"no\s+specific\s+person's\s+data\s+is\s+recoverable", + r"audit\s+report\s+confirms\s+this", + r"tanımlanabilir\s+özneler\s+kaldırılmıştır", + r"geri\s+çıkarılamaz", + r"Audit\s+raporu\s+bunu\s+teyit", + r"international_transfers", + r"No\s+matches\s+found\s+in\s+masked", + r"eşleşme\s+bulunamadı", + r"auto-delete\s+originals", + r"records\s+the\s+intended\s+TTL", + r"SHA-256\s+manifest\s+for\s+tamper", + r"Source\s+attribution\s+\(which\s+document", + r"pii_mask:", ) _PURGE_ARTEFACT_WARNING_FILES = ( @@ -182,3 +200,28 @@ def test_purge_docs_warn_that_artefacts_misses_the_standard_bundle(rel: str) -> text = (_REPO_ROOT / rel).read_text(encoding="utf-8") assert "NoMatchingArtefacts" in text, f"{rel} must warn that --kind artefacts finds nothing" assert "compliance_report.json" in text, f"{rel} must name the fixed-basename bundle" + + +@pytest.mark.parametrize("lang", ["en", "tr"]) +class TestGdprPageStaysExecutable: + """CR-A11-001: the GDPR / KVKK page shows commands an operator can run and says what they find.""" + + @staticmethod + def _page(lang: str) -> str: + return (_REPO_ROOT / "docs" / "usermanuals" / lang / "compliance" / "gdpr.md").read_text(encoding="utf-8") + + def test_reverse_pii_example_pins_the_output_dir_and_shows_the_real_output_line(self, lang: str) -> None: + # Without --output-dir the command writes the per-directory salt and an audit event next to the first corpus + # file; the real zero-match line is "No matches for identifier hash <16 hex>… (literal, plaintext mode) in N file(s)." + text = self._page(lang) + assert re.search( + r"forgelm reverse-pii --query \"ali@example\.com\" --output-dir \./outputs data/\*\.jsonl", text + ) + assert re.search( + r"No matches for identifier hash [0-9a-f]{16}… \(literal, plaintext mode\) in \d+ file\(s\)\.", text + ) + + def test_names_point_to_the_detection_only_pii_ml_layer(self, lang: str) -> None: + text = self._page(lang) + assert "forgelm audit --pii-ml" in text and "#/data/pii-ml" in text + assert "ingestion-pii-ml" in text From 41025087f6f5a3944b3f89660b63c198520db04b Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:53:31 +0300 Subject: [PATCH 06/25] test(pii): ban the wrong GDPR attributions of the PII detector set broadly, not one sentence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first guard for the Article 9 / Article 10 / "GDPR-mandated" attributions matched only the exact retired strings, so "GDPR Article 9 + ENISA", "GDPR Madde 9", "Madde 10 ... önemsediği" with a curly apostrophe or "GDPR mandates these identifiers" would pass. The eight PII files now have their own list with broad patterns (any Art. 9 / Madde 9 mention, GDPR "mandat*" or "requir*", "regulation-mandated", Art. 10 "cares / concerned / önemsediği"), since nothing in them has a legitimate reason to cite Art. 9. Also reflows the two edited paragraphs. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/data_audit-tr.md | 3 +- docs/guides/data_audit.md | 3 +- forgelm/data_audit/_types.py | 4 +- .../test_usermanual_reference_consistency.py | 44 ++++++++++++------- 4 files changed, 33 insertions(+), 21 deletions(-) diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 06a90272..10277c10 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -162,8 +162,7 @@ kötü?* sorusunda sıfır rehberlik veriyor. Faz 11.5 yanına bir Tier tablosu konsensüs regülatif ağırlıklandırmadır (finansal kimlikler için PCI-DSS; devlet kimlikleri için GDPR Madde 87 (ulusal kimlik -numaraları) ve Madde 32 + ENISA). -`critical` katman artık komutun kendisini kapılar (sonraki bölüm); +numaraları) ve Madde 32 + ENISA). `critical` katman artık komutun kendisini kapılar (sonraki bölüm); `high`'ta da durmak isteyen pipeline'lar `pii_severity.worst_tier`'i okumalı ve açık bir review olmadan yayınlamayı reddetmelidir. diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 450af127..e71bec1b 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -152,8 +152,7 @@ The flat `pii_summary` map gives compliance reviewers no guidance on The tier table is consensus regulatory weighting (PCI-DSS for financial identifiers; GDPR Art. 87 (national identification numbers) and Art. 32 -plus ENISA for government IDs). The `critical` -tier now gates the command itself (next section); pipelines that also +plus ENISA for government IDs). The `critical` tier now gates the command itself (next section); pipelines that also want to stop at `high` should read `pii_severity.worst_tier` and refuse to publish without explicit review. diff --git a/forgelm/data_audit/_types.py b/forgelm/data_audit/_types.py index 5eb83299..b6b7be97 100644 --- a/forgelm/data_audit/_types.py +++ b/forgelm/data_audit/_types.py @@ -24,8 +24,8 @@ # remember by hand that ``credit_card`` is materially worse than ``phone``. # The tiers below encode the consensus regulatory weighting (PCI-DSS for # financial; GDPR Art. 87 + Art. 32 and ENISA for national IDs) so the audit -# surfaces a one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when local -# law disagrees — the audit honours whatever map is loaded at call time. +# surfaces a one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when +# local law disagrees — the audit honours whatever map is loaded at call time. PII_SEVERITY: Dict[str, str] = { # Financial / fully reversible identity theft -> highest weight. "credit_card": "critical", diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 7f2545e2..9df61725 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -85,14 +85,6 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: "docs/usermanuals/tr/compliance/human-oversight.md", "docs/qms/sop_change_management.md", "docs/qms/sop_change_management-tr.md", - "forgelm/data_audit/_types.py", - "forgelm/data_audit/_pii_regex.py", - "forgelm/data_audit/_optional.py", - "pyproject.toml", - "docs/guides/data_audit.md", - "docs/guides/data_audit-tr.md", - "docs/usermanuals/en/data/pii-ml.md", - "docs/usermanuals/tr/data/pii-ml.md", "docs/usermanuals/en/compliance/gdpr.md", "docs/usermanuals/tr/compliance/gdpr.md", ) @@ -133,13 +125,6 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"onurlandırır", r"17\(3\)\(b\)\s+protects", r"17\(3\)\(b\)[^.\n]{0,40}\bkorur", - # The PII tier table and detector set are not a GDPR-mandated list: Art. 9 covers special categories (not - # national IDs, which Art. 87 addresses) and Art. 10 covers criminal-conviction data. - r"GDPR\s+Art\.\s+9\s*\+\s*ENISA", - r"GDPR\s+Md\.\s+9\s*\+\s*ENISA", - r"Article\s+10\s+cares", - r"Madde\s+10'un\s+önemsediği", - r"GDPR-mandated", # The GDPR / KVKK page (CR-A11-001): masking is a fixed regex set, not anonymisation, and no audit report can # confirm that a model holds nothing about a person. r"identifiable\s+subjects\s+are\s+removed", @@ -158,6 +143,28 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"pii_mask:", ) +# The PII tier table and the regex detector set are not a GDPR-mandated list: Art. 9 covers special categories (not +# national IDs, which Art. 87 addresses), Art. 10 covers criminal-conviction data, and the identifiers are personal +# data because of Art. 4(1). Nothing in these files has a legitimate reason to cite Art. 9 or to say GDPR mandates +# or requires the detector set, so the guard bans the citation broadly instead of one exact old sentence. +_PII_CITATION_FILES = ( + "forgelm/data_audit/_types.py", + "forgelm/data_audit/_pii_regex.py", + "forgelm/data_audit/_optional.py", + "pyproject.toml", + "docs/guides/data_audit.md", + "docs/guides/data_audit-tr.md", + "docs/usermanuals/en/data/pii-ml.md", + "docs/usermanuals/tr/data/pii-ml.md", +) + +_RETIRED_PII_CITATIONS = ( + r"(?:Art(?:icle|\.)?|Madde|Md\.)\s*9\b", + r"GDPR[\s\u2010-\u2015-]*(?:mandat|requir)", + r"regulat\w*[\s\u2010-\u2015-]*mandated", + r"(?:Article|Madde)\s+10\b[^.\n]{0,60}(?:cares|concerned|önemsediği)", +) + _PURGE_ARTEFACT_WARNING_FILES = ( "docs/guides/gdpr_erasure.md", "docs/guides/gdpr_erasure-tr.md", @@ -180,6 +187,13 @@ def test_retired_gdpr_claims_stay_out(rel: str) -> None: _CONDITIONAL_17_3_B = re.compile(r"to\s+the\s+extent|only\s+to\s|still\s+requires|ölçüde", re.IGNORECASE) +@pytest.mark.parametrize("rel", _PII_CITATION_FILES) +def test_retired_pii_citations_stay_out(rel: str) -> None: + text = (_REPO_ROOT / rel).read_text(encoding="utf-8") + found = [pattern for pattern in _RETIRED_PII_CITATIONS if re.search(pattern, text, re.IGNORECASE)] + assert found == [], f"{rel} still attributes the PII detector set or tiers to the wrong GDPR provision: {found}" + + @pytest.mark.parametrize("rel", _GDPR_CLAIM_FILES) def test_every_article_17_3_b_mention_is_conditional(rel: str) -> None: text = (_REPO_ROOT / rel).read_text(encoding="utf-8") From 0d0ecfe05fba66c323667d6855423cff2a0c50ad Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 01:53:31 +0300 Subject: [PATCH 07/25] fix(docs-guard): flag unrecognised top-level YAML blocks in the user manual; correct the chat and GPU-cost examples `tools/check_usermanual_schema_drift.py` skipped every top-level key that is not a ForgeConfig block, so a page could show a `chat:` or `output:` block that Pydantic rejects with `extra_forbidden` while the guard stayed green (CR-A17a-007). It now reports such blocks too, unless the (page, exact key set) pair is listed with a reason in one of two tables: - `_NON_FORGELM_YAML` for another tool's YAML (GitHub Actions, GitLab CI, docker-compose, a vLLM server config, the pip-audit ignore file); - `_KNOWN_OPEN` for a fabricated block a tracked issue still owns (the troubleshooting `audit:` block and the custom `gpu_profiles:` block). Rows are keyed by page path relative to the language directory, so the English and Turkish copies share one row and a line shift never invalidates it. A row that matches no snippet in every language is itself a failure, so the tables cannot rot into an allowlist. Tests plant a synthetic manual tree and pin each rule, plus the real corpus. The three fabricated blocks the finding names are corrected in English and Turkish: - `forgelm chat` reads no YAML config. The `chat:` block (temperature, top-p, max tokens, system prompt, history file) is replaced by the real flags (`--system`, `--temperature`, `--max-new-tokens`, `--no-stream`, `--load-in-4bit/-8bit`), and the page says there is no top-p setting or persisted history. - The GPU-cost page used an `output.cost_tracking` block that does not exist. The hourly rate is `training.gpu_cost_per_hour` (when unset, the built-in price table is used and the run records `cost_source`); the multi-GPU example uses the same field; the alert/halt thresholds are described as roadmap prose instead of a YAML block. Resolves CR-A17a-007 (with the gdpr.md block removed by the earlier page rewrite) (Refs #101). Co-Authored-By: Claude Sonnet 5.5 --- docs/usermanuals/en/deployment/chat.md | 19 +- docs/usermanuals/en/operations/gpu-cost.md | 49 ++--- docs/usermanuals/tr/deployment/chat.md | 19 +- docs/usermanuals/tr/operations/gpu-cost.md | 43 +--- tests/test_check_usermanual_schema_drift.py | 124 ++++++++++- tools/check_usermanual_schema_drift.py | 219 +++++++++++++++++--- 6 files changed, 356 insertions(+), 117 deletions(-) diff --git a/docs/usermanuals/en/deployment/chat.md b/docs/usermanuals/en/deployment/chat.md index c02716a7..132b7246 100644 --- a/docs/usermanuals/en/deployment/chat.md +++ b/docs/usermanuals/en/deployment/chat.md @@ -37,14 +37,17 @@ yolunu izleyebilirsiniz... ## Configuration -```yaml -chat: - default_temperature: 0.7 - default_top_p: 0.9 - default_max_tokens: 1024 - default_system_prompt: "You are a helpful assistant." - history_file: "~/.forgelm/chat-history" # persisted across sessions -``` +`forgelm chat` reads no YAML config and has no `chat:` block: it is set with command-line flags and the slash commands above. + +| Setting | Flag | Default | +|---|---|---| +| Initial system prompt | `--system PROMPT` (change it later with `/system`) | none | +| Sampling temperature | `--temperature` (change it later with `/temperature`) | `0.7` | +| Maximum tokens per reply | `--max-new-tokens` | `512` | +| Streaming output | `--no-stream` turns it off | on | +| Quantised loading | `--load-in-4bit` or `--load-in-8bit` | off | + +There is no top-p setting and no history file that persists across sessions; write a conversation out with `/save`. ## Loading a checkpoint diff --git a/docs/usermanuals/en/operations/gpu-cost.md b/docs/usermanuals/en/operations/gpu-cost.md index 504b0466..4695a74b 100644 --- a/docs/usermanuals/en/operations/gpu-cost.md +++ b/docs/usermanuals/en/operations/gpu-cost.md @@ -6,12 +6,12 @@ description: Auto-detect across 16 GPU profiles and track per-run cost against y # GPU Cost Estimation > **Status:** GPU detection + per-run duration + audit-log stamping ship -> today; the config-driven `cost_tracking:` block (rate tables, alert / -> halt thresholds) is **tracked on the roadmap and not yet implemented** — -> `forgelm/config.py` has no `output.cost_tracking` surface. Examples below -> that show `cost_tracking:` fields are forward-looking placeholders — set -> hourly rates manually until the YAML surface lands. The deferral is -> tracked in the [risks-and-decisions roadmap on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). +> today, and `training.gpu_cost_per_hour` is the one cost setting. A +> config-driven `cost_tracking:` block (per-GPU rate tables, alert / halt +> thresholds) is **tracked on the roadmap and not yet implemented** — +> `forgelm/config.py` has no `output.cost_tracking` surface, so no example on +> this page uses one. The deferral is tracked in the +> [risks-and-decisions roadmap on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). ForgeLM detects the GPU you're running on, looks up its profile (memory, compute, typical hourly rate), and tracks per-run cost. After every run, the audit log records exactly how much GPU time was used and what it cost. @@ -36,19 +36,11 @@ If your GPU isn't recognised, ForgeLM logs a warning and falls back to a generic ## Configuring your hourly rate ```yaml -output: - cost_tracking: - enabled: true - rate_per_hour: - A100_80GB: 1.10 # USD per hour - A100_40GB: 0.85 - H100_80GB: 2.40 - RTX_4090: 0.50 # rough cost-of-electricity - default: 1.00 # for unmatched GPUs - currency: "USD" +training: + gpu_cost_per_hour: 1.10 # USD per hour for the training GPU ``` -Once the `cost_tracking:` surface lands, set this once per project and ForgeLM will use the matched rate for cost reporting. Until then, the block is a forward-looking placeholder (see the status banner above). +Set `training.gpu_cost_per_hour` to your real rate. When it is unset, ForgeLM falls back to its built-in price table for the detected GPU; the run's `resource_usage` records which one it used (`cost_source`: `user_config` or `auto_detected`). There is one rate per run: per-GPU-model rate tables are not supported. ## Output @@ -92,29 +84,16 @@ The calibration approach is typically within 20% of actual. ## Multi-GPU and distributed -For multi-GPU training, ForgeLM multiplies the per-GPU rate by the GPU count: - -```yaml -output: - cost_tracking: - rate_per_hour: - A100_80GB: 1.10 -``` +For multi-GPU training, ForgeLM multiplies the single `training.gpu_cost_per_hour` rate by the GPU count (`gpu_hours` is the training time times the GPU count): A 4×A100 run for 2 hours = 4 × 2 × $1.10 = $8.80, regardless of whether you use ZeRO or FSDP. ## Cost alerts (roadmap — not yet implemented) -For runs that may run away, the planned `cost_tracking` block will support -threshold-based alerts and halts: - -```yaml -# roadmap placeholder — not yet honoured by forgelm/config.py -output: - cost_tracking: - alert_threshold_usd: 50.0 # webhook fires when crossed - halt_threshold_usd: 200.0 # training stops -``` +For runs that may run away, the planned `cost_tracking` block would support +threshold-based alerts (a webhook fires when an alert threshold is crossed) and +halts (training stops at a halt threshold). No such setting exists in +`forgelm/config.py` today. When implemented, the alert will fire the configured webhook (see [Webhooks](#/operations/webhooks)) — useful in CI to catch a misconfigured run before it spends a week of budget overnight. Until then, monitor cost manually via the audit log + a budget-side guardrail in your scheduler. diff --git a/docs/usermanuals/tr/deployment/chat.md b/docs/usermanuals/tr/deployment/chat.md index 9ef6f7ec..4054ae7d 100644 --- a/docs/usermanuals/tr/deployment/chat.md +++ b/docs/usermanuals/tr/deployment/chat.md @@ -37,14 +37,17 @@ Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... ## Konfigürasyon -```yaml -chat: - default_temperature: 0.7 - default_top_p: 0.9 - default_max_tokens: 1024 - default_system_prompt: "Yardımsever bir asistansın." - history_file: "~/.forgelm/chat-history" # oturumlar arası kalıcı -``` +`forgelm chat` YAML config okumaz ve `chat:` bloğu yoktur: komut satırı flag'leri ve yukarıdaki slash komutlarıyla ayarlanır. + +| Ayar | Flag | Varsayılan | +|---|---|---| +| Başlangıç system prompt'u | `--system PROMPT` (sonra `/system` ile değiştirin) | yok | +| Sampling temperature | `--temperature` (sonra `/temperature` ile değiştirin) | `0.7` | +| Yanıt başına maksimum token | `--max-new-tokens` | `512` | +| Streaming çıktı | `--no-stream` kapatır | açık | +| Quantize yükleme | `--load-in-4bit` veya `--load-in-8bit` | kapalı | + +top-p ayarı ve oturumlar arası kalıcı bir history dosyası yoktur; bir konuşmayı `/save` ile yazın. ## Checkpoint yükleme diff --git a/docs/usermanuals/tr/operations/gpu-cost.md b/docs/usermanuals/tr/operations/gpu-cost.md index 80f07a57..a1b33482 100644 --- a/docs/usermanuals/tr/operations/gpu-cost.md +++ b/docs/usermanuals/tr/operations/gpu-cost.md @@ -6,12 +6,12 @@ description: 16 GPU profilinde otomatik tespit ve saatlik tarifenize göre koşu # GPU Maliyet Tahmini > **Durum:** GPU tespiti + koşu süresi + audit-log damgalama bugün -> gönderiliyor; konfigürasyon-tabanlı `cost_tracking:` bloğu (tarife +> gönderiliyor ve `training.gpu_cost_per_hour` tek maliyet ayarıdır. +> Konfigürasyon-tabanlı bir `cost_tracking:` bloğu (GPU başına tarife > tabloları, alarm / durdurma eşikleri) **yol haritasında ve henüz > uygulanmadı** — `forgelm/config.py`'de `output.cost_tracking` yüzeyi -> yok. Aşağıdaki `cost_tracking:` örnekleri ileriye dönük yer -> tutuculardır — YAML yüzeyi inene kadar saatlik tarifeleri manuel -> ayarlayın. Erteleme için bkz. [GitHub'daki risks-and-decisions yol haritası](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). +> yok, bu yüzden bu sayfadaki hiçbir örnek onu kullanmaz. Erteleme için bkz. +> [GitHub'daki risks-and-decisions yol haritası](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). ForgeLM, üzerinde koştuğunuz GPU'yu tespit eder, profilini (bellek, compute, tipik saatlik tarife) bulur ve koşu başı maliyeti izler. Her koşunun ardından audit log tam olarak ne kadar GPU zamanı kullanıldığını ve maliyetini kaydeder. @@ -36,19 +36,11 @@ GPU'nuz tanınmazsa ForgeLM uyarı bırakır ve generic profile fallback yapar ( ## Saatlik tarife konfigürasyonu ```yaml -output: - cost_tracking: - enabled: true - rate_per_hour: - A100_80GB: 1.10 # saatlik USD - A100_40GB: 0.85 - H100_80GB: 2.40 - RTX_4090: 0.50 # kabaca elektrik maliyeti - default: 1.00 # eşleşmeyen GPU'lar için - currency: "USD" +training: + gpu_cost_per_hour: 1.10 # eğitim GPU'su için saatlik USD ``` -`cost_tracking:` yüzeyi indiğinde, bunu proje başına bir kez ayarlayın ve ForgeLM maliyet raporlama için eşleşen tarifeyi kullanacak. O zamana kadar blok ileriye dönük bir yer tutucudur (yukarıdaki durum bandına bakın). +`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config` veya `auto_detected`). Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. ## Çıktı @@ -92,29 +84,16 @@ Kalibrasyon yaklaşımı genelde gerçeğin %20 içindedir. ## Çoklu-GPU ve dağıtık -Çoklu-GPU eğitiminde ForgeLM GPU başı tarifeyi GPU sayısıyla çarpar: - -```yaml -output: - cost_tracking: - rate_per_hour: - A100_80GB: 1.10 -``` +Çoklu-GPU eğitiminde ForgeLM tek `training.gpu_cost_per_hour` tarifesini GPU sayısıyla çarpar (`gpu_hours`, eğitim süresi çarpı GPU sayısıdır): 4×A100 koşusu 2 saat = 4 × 2 × $1.10 = $8.80; ZeRO veya FSDP kullanmak fark etmez. ## Maliyet alarmları (yol haritası — henüz uygulanmadı) Kontrolden çıkabilecek koşular için planlanan `cost_tracking` bloğu eşik -tabanlı alarm + durdurma destekleyecek: - -```yaml -# yol haritası yer tutucusu — forgelm/config.py tarafından henüz honure edilmez -output: - cost_tracking: - alert_threshold_usd: 50.0 # geçildiğinde webhook fırlat - halt_threshold_usd: 200.0 # eğitim durur -``` +tabanlı alarmı (alarm eşiği aşılınca webhook tetiklenir) ve durdurmayı +(durdurma eşiğinde eğitim durur) destekleyecekti. Böyle bir ayar bugün +`forgelm/config.py`'de yoktur. İmplemente edildiğinde, alarm konfigüre webhook'u fırlatır (bkz. [Webhook'lar](#/operations/webhooks)) — yanlış konfigüre koşunun gece bütçeyi haftalık bitirmesini CI'da yakalamak için faydalı. O zamana kadar maliyeti audit log + zamanlayıcınızdaki bütçe-tarafı koruyucu ile manuel izleyin. diff --git a/tests/test_check_usermanual_schema_drift.py b/tests/test_check_usermanual_schema_drift.py index bffc7c61..bc54a7f8 100644 --- a/tests/test_check_usermanual_schema_drift.py +++ b/tests/test_check_usermanual_schema_drift.py @@ -211,10 +211,10 @@ def test_full_triplet_snippet_is_skipped(self, tool, synthetic_schema): ) assert tool.check_snippet(snippet, synthetic_schema, top) == [] - def test_unrecognised_top_level_key_is_out_of_scope(self, tool, synthetic_schema): - # A yaml block unrelated to ForgeConfig (e.g. a docker-compose or - # deepspeed example) must not be flagged just because its keys - # happen not to match anything. + def test_unrecognised_top_level_key_is_not_a_nested_key_drift(self, tool, synthetic_schema): + # check_snippet only walks the keys *inside* recognised ForgeConfig + # blocks; an unrecognised top-level block is reported by + # unknown_top_level_keys / scan (see TestUnknownTopLevelBlocks). top = synthetic_schema["ForgeConfig"] snippet = self._snippet(tool, "services:\n train:\n image: x\n") assert tool.check_snippet(snippet, synthetic_schema, top) == [] @@ -235,6 +235,122 @@ def test_unparseable_yaml_is_not_this_guards_concern(self, tool, synthetic_schem assert tool.check_snippet(snippet, synthetic_schema, top) == [] +class TestUnknownTopLevelBlocks: + """CR-A17a-007: a fragment may not use a top-level key ForgeConfig rejects.""" + + @staticmethod + def _snippet(tool, body: str): + return tool.Snippet(path=Path("doc.md"), line_start=1, body=body) + + @staticmethod + def _write(root: Path, lang: str, page: str, body: str) -> None: + path = root / lang / page + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f"# Page\n\n```yaml\n{body}```\n", encoding="utf-8") + + def test_fabricated_top_level_block_is_reported(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + keys = tool.unknown_top_level_keys(self._snippet(tool, "chat:\n default_temperature: 0.7\n"), top) + assert keys == frozenset({"chat"}) + + def test_known_blocks_and_nested_drift_are_not_unknown_blocks(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + assert tool.unknown_top_level_keys(self._snippet(tool, "training:\n bogus: 1\n"), top) == frozenset() + + def test_mixed_snippet_reports_only_the_unknown_keys(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + body = "training:\n learning_rate: 0.1\noutput:\n cost_tracking: {}\n" + assert tool.unknown_top_level_keys(self._snippet(tool, body), top) == frozenset({"output"}) + + def test_full_config_and_unparseable_and_non_mapping_snippets_are_skipped(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + full = "model:\n name_or_path: x\ntraining: {}\ndata:\n dataset_name_or_path: y\nchat: {}\n" + assert tool.unknown_top_level_keys(self._snippet(tool, full), top) == frozenset() + assert tool.unknown_top_level_keys(self._snippet(tool, "chat:\n - [unbalanced\n"), top) == frozenset() + assert tool.unknown_top_level_keys(self._snippet(tool, "- chat\n- output\n"), top) == frozenset() + + def test_scan_flags_an_unlisted_block_in_both_languages(self, tool, synthetic_schema, tmp_path): + for lang in ("en", "tr"): + self._write(tmp_path, lang, "deployment/chat.md", "chat:\n default_temperature: 0.7\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, {}) + assert [(u.path.relative_to(tmp_path).as_posix(), sorted(u.keys)) for u in result.unknown_blocks] == [ + ("en/deployment/chat.md", ["chat"]), + ("tr/deployment/chat.md", ["chat"]), + ] + assert not result.clean + + def test_a_listed_block_passes_in_every_language_and_survives_a_line_shift(self, tool, synthetic_schema, tmp_path): + rows = {("operations/cicd.md", frozenset({"jobs", "name"})): "GitHub Actions workflow"} + self._write(tmp_path, "en", "operations/cicd.md", "name: ci\njobs: {}\n") + (tmp_path / "tr" / "operations").mkdir(parents=True) + (tmp_path / "tr" / "operations" / "cicd.md").write_text( + "Intro line\n\nMore text\n\n```yaml\nname: ci\njobs: {}\n```\n", encoding="utf-8" + ) + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert result.clean + + def test_known_open_rows_behave_like_non_forgelm_rows(self, tool, synthetic_schema, tmp_path): + rows = {("operations/troubleshooting.md", frozenset({"audit"})): "tracked elsewhere"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "operations/troubleshooting.md", "audit:\n quality_filter: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, rows) + assert result.clean + + def test_a_row_must_match_the_exact_key_set(self, tool, synthetic_schema, tmp_path): + rows = {("operations/cicd.md", frozenset({"jobs"})): "GitHub Actions workflow"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "operations/cicd.md", "jobs: {}\nchat: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert [sorted(u.keys) for u in result.unknown_blocks] == [["chat", "jobs"], ["chat", "jobs"]] + assert [row for _, row, _ in result.stale_rows] == [("operations/cicd.md", frozenset({"jobs"}))] + + def test_a_row_that_matches_no_snippet_is_stale(self, tool, synthetic_schema, tmp_path): + rows = {("operations/gpu-cost.md", frozenset({"gpu_profiles"})): "tracked elsewhere"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "operations/gpu-cost.md", "training:\n learning_rate: 0.1\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, rows) + assert [(name, row) for name, row, _ in result.stale_rows] == [ + ("_KNOWN_OPEN", ("operations/gpu-cost.md", frozenset({"gpu_profiles"}))) + ] + + def test_a_row_matched_in_only_one_language_is_stale(self, tool, synthetic_schema, tmp_path): + rows = {("operations/gpu-cost.md", frozenset({"gpu_profiles"})): "tracked elsewhere"} + self._write(tmp_path, "en", "operations/gpu-cost.md", "gpu_profiles:\n custom: []\n") + self._write(tmp_path, "tr", "operations/gpu-cost.md", "training:\n learning_rate: 0.1\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, rows) + assert len(result.stale_rows) == 1 + + def test_main_strict_exits_one_on_an_unlisted_block_and_zero_in_advisory_mode( + self, tool, synthetic_schema, tmp_path, monkeypatch, capsys + ): + config_path = tmp_path / "config.py" + config_path.write_text(_SYNTHETIC_CONFIG, encoding="utf-8") + root = tmp_path / "usermanuals" + for lang in ("en", "tr"): + self._write(root, lang, "deployment/chat.md", "chat:\n default_temperature: 0.7\n") + monkeypatch.setattr(tool, "CONFIG_PATH", config_path) + monkeypatch.setattr(tool, "USERMANUAL_ROOT", root) + monkeypatch.setattr(tool, "REPO_ROOT", tmp_path) + assert tool.main(["--strict"]) == 1 + out = capsys.readouterr().out + assert "usermanuals/en/deployment/chat.md:3 chat" in out and "_NON_FORGELM_YAML" in out + assert tool.main([]) == 0 + + def test_every_real_exemption_row_is_matched_in_every_language(self, tool): + schema = tool.build_schema_map(tool.CONFIG_PATH) + result = tool.scan( + tool.USERMANUAL_ROOT, schema, schema["ForgeConfig"], tool._NON_FORGELM_YAML, tool._KNOWN_OPEN + ) + assert result.stale_rows == [] and result.unknown_blocks == [] + + def test_real_exemption_tables_carry_a_reason_and_do_not_overlap(self, tool): + for table in (tool._NON_FORGELM_YAML, tool._KNOWN_OPEN): + assert all(reason.strip() for reason in table.values()) + assert not set(tool._NON_FORGELM_YAML) & set(tool._KNOWN_OPEN) + # A fabricated block must never be filed as "another tool's YAML". + assert not any({"chat", "output"} & keys for _, keys in tool._NON_FORGELM_YAML) + + class TestRealSchemaSmoke: """Regression-pins the AST walker's shape against the real schema, and separately pins the zero-drift invariant that CI enforces with diff --git a/tools/check_usermanual_schema_drift.py b/tools/check_usermanual_schema_drift.py index 27928df2..bdac85bd 100644 --- a/tools/check_usermanual_schema_drift.py +++ b/tools/check_usermanual_schema_drift.py @@ -47,6 +47,25 @@ which is what keeps this guard from false-flagging genuinely free-form fields like ``MergeConfig.models: List[Dict[str, Any]]``. +A fragment may also carry a top-level key that is not a ``ForgeConfig`` +block at all — ``chat:``, ``output:``, ``gpu_profiles:``. Those used to +be skipped as "someone else's YAML", which let a reader copy a block that +Pydantic rejects with ``extra_forbidden`` while this guard stayed green. +They are now flagged too, unless the (page, key set) pair is listed with +a written reason in one of two tables below: + +- ``_NON_FORGELM_YAML`` — the snippet really is another tool's YAML (a + GitHub Actions workflow, a docker-compose file, a vLLM server config). +- ``_KNOWN_OPEN`` — the block is fabricated or not implemented and a + tracked issue owns the fix. The row is deleted in the same change that + corrects the page. + +Both tables are keyed by the page path relative to its language directory +plus the exact set of unknown top-level keys, so the English and Turkish +copies share one row and a line shift never invalidates it. A row that +matches nothing (the snippet was fixed, moved or removed) is itself a +failure, so the tables cannot rot into a blanket allowlist. + Deliberately out of scope (would need real Pydantic validation, not an AST key-path walk, to check safely): value type/range checks, `Literal` choice validation, cross-field validators. This guard only answers @@ -57,8 +76,8 @@ that ``forgelm/`` honours): - ``0`` — every checked key path resolves against the schema. -- ``1`` — at least one fabricated key path found (strict mode), or - invalid arguments. +- ``1`` — at least one fabricated key path, unrecognised top-level block + or stale exemption row found (strict mode), or invalid arguments. CI wiring: this guard runs in ``.github/workflows/ci.yml`` with ``--strict`` — any fabricated key path fails the build. The @@ -83,7 +102,7 @@ import sys from dataclasses import dataclass from pathlib import Path -from typing import Dict, List, Optional, Tuple +from typing import Dict, FrozenSet, List, Optional, Set, Tuple try: import yaml @@ -113,6 +132,48 @@ # reports on the same line. _FULL_CONFIG_KEYS = frozenset({"model", "training", "data"}) +# (page relative to its language directory, exact set of unrecognised +# top-level keys) -> why the block is allowed to stay. +_PageKeys = Tuple[str, FrozenSet[str]] + +# Snippets that are another tool's YAML, not a ForgeLM config. +_NON_FORGELM_YAML: Dict[_PageKeys, str] = { + ("compliance/human-approval-gate.md", frozenset({"env", "steps"})): "GitHub Actions workflow excerpt", + ( + "deployment/deploy-targets.md", + frozenset( + { + "chat_template", + "dtype", + "gpu_memory_utilization", + "max_model_len", + "served_model_name", + "tokenizer_mode", + "trust_remote_code", + } + ), + ): "vLLM server config that `forgelm deploy --target vllm` generates", + ("operations/cicd.md", frozenset({"jobs", "name"})): "GitHub Actions workflow", + ("operations/cicd.md", frozenset({"audit", "deploy", "stages", "train"})): "GitLab CI pipeline", + ("operations/cicd.md", frozenset({"concurrency"})): "GitHub Actions concurrency block", + ("operations/docker.md", frozenset({"services", "version", "volumes"})): "docker-compose file", + ("operations/supply-chain.md", frozenset({"ignores"})): "ignore file read by tools/check_pip_audit.py --ignores", + ("reference/exit-codes.md", frozenset({"train"})): "GitLab CI job", +} + +# Fabricated or unimplemented blocks that a tracked issue still owns. Delete +# the row in the same change that corrects the page (a stale row fails). +_KNOWN_OPEN: Dict[_PageKeys, str] = { + ( + "operations/troubleshooting.md", + frozenset({"audit"}), + ): "#425: documents an `audit.quality_filter.skip` block that ForgeConfig rejects", + ( + "operations/gpu-cost.md", + frozenset({"gpu_profiles"}), + ): "#427: documents custom GPU profiles that do not exist", +} + @dataclass(frozen=True) class FieldType: @@ -131,6 +192,29 @@ class KeyDrift: key_path: str +@dataclass(frozen=True) +class UnknownBlock: + """Top-level keys of a usermanual YAML fragment that are not ForgeConfig blocks.""" + + path: Path + line: int + keys: FrozenSet[str] + + +@dataclass(frozen=True) +class ScanResult: + """Everything one pass over a usermanual tree found.""" + + drifts: List[KeyDrift] + unknown_blocks: List[UnknownBlock] + stale_rows: List[Tuple[str, _PageKeys, str]] # (table name, row key, reason) + snippet_count: int + + @property + def clean(self) -> bool: + return not (self.drifts or self.unknown_blocks or self.stale_rows) + + def _unwrap_subscript(node: ast.Subscript) -> Tuple[Optional[str], List[ast.AST]]: """Return ``(base_name, slice_elements)`` for a subscript annotation. @@ -283,25 +367,48 @@ def _walk_value( return drift -def check_snippet( - snippet: "Snippet", schema: Dict[str, Dict[str, FieldType]], top_level: Dict[str, FieldType] -) -> List[KeyDrift]: - """Check one fenced yaml snippet; return the key-path drift found.""" +def _parse_fragment(snippet: "Snippet") -> Optional[dict]: + """Return the parsed mapping of a fragment this guard checks, else ``None``. + + ``None`` covers unparseable YAML (``check_yaml_snippets.py`` reports parse + errors on full configs), non-mapping documents, and full configs that + ``check_yaml_snippets.py`` already validates with a real ``ForgeConfig(**data)``. + """ try: parsed = yaml.safe_load(snippet.body) except yaml.YAMLError: - return [] # not this guard's concern — check_yaml_snippets covers parse errors on full configs - if not isinstance(parsed, dict): + return None + if not isinstance(parsed, dict) or _FULL_CONFIG_KEYS.issubset(parsed.keys()): + return None + return parsed + + +def unknown_top_level_keys(snippet: "Snippet", top_level: Dict[str, FieldType]) -> FrozenSet[str]: + """Return the string top-level keys of ``snippet`` that are not ForgeConfig blocks.""" + parsed = _parse_fragment(snippet) + if parsed is None: + return frozenset() + return frozenset(key for key in parsed if isinstance(key, str) and key not in top_level) + + +def check_snippet( + snippet: "Snippet", schema: Dict[str, Dict[str, FieldType]], top_level: Dict[str, FieldType] +) -> List[KeyDrift]: + """Check one fenced yaml snippet; return the key-path drift found. + + Only the keys *inside* recognised ForgeConfig blocks are walked here; an + unrecognised top-level block is :func:`unknown_top_level_keys`'s concern. + """ + parsed = _parse_fragment(snippet) + if parsed is None: return [] - if _FULL_CONFIG_KEYS.issubset(parsed.keys()): - return [] # fully covered by check_yaml_snippets.py's real Pydantic validation drifts: List[KeyDrift] = [] for key, sub_value in parsed.items(): if not isinstance(key, str): continue field_type = top_level.get(key) if field_type is None or field_type.nested_class is None: - continue # not a recognised ForgeConfig top-level block — out of scope + continue # unrecognised or opaque top-level block — see unknown_top_level_keys for key_path in _walk_value(sub_value, field_type.nested_class, schema, key): drifts.append(KeyDrift(path=snippet.path, line=snippet.line_start, key_path=key_path)) return drifts @@ -312,6 +419,46 @@ def walk_usermanuals(root: Path) -> List[Path]: return [p for p in sorted(root.rglob("*.md")) if p.is_file()] +def _language_and_page(root: Path, md_path: Path) -> Tuple[str, str]: + """Split ``//
/.md`` into ``(lang, "
/.md")``.""" + parts = md_path.relative_to(root).parts + return parts[0], "/".join(parts[1:]) + + +def scan( + root: Path, + schema: Dict[str, Dict[str, FieldType]], + top_level: Dict[str, FieldType], + non_forgelm: Dict[_PageKeys, str], + known_open: Dict[_PageKeys, str], +) -> ScanResult: + """Walk every fenced yaml block under ``root`` and classify what it contains.""" + drifts: List[KeyDrift] = [] + unknown: List[UnknownBlock] = [] + matched: Set[Tuple[str, str, FrozenSet[str]]] = set() + languages: Set[str] = set() + snippet_count = 0 + for md_path in walk_usermanuals(root): + language, page = _language_and_page(root, md_path) + languages.add(language) + for snippet in extract_yaml_snippets(md_path): + snippet_count += 1 + drifts.extend(check_snippet(snippet, schema, top_level)) + keys = unknown_top_level_keys(snippet, top_level) + if not keys: + continue + if (page, keys) in non_forgelm or (page, keys) in known_open: + matched.add((language, page, keys)) + else: + unknown.append(UnknownBlock(path=snippet.path, line=snippet.line_start, keys=keys)) + stale: List[Tuple[str, _PageKeys, str]] = [] + for table_name, table in (("_NON_FORGELM_YAML", non_forgelm), ("_KNOWN_OPEN", known_open)): + for (page, keys), reason in table.items(): + if any((language, page, keys) not in matched for language in sorted(languages)): + stale.append((table_name, (page, keys), reason)) + return ScanResult(drifts=drifts, unknown_blocks=unknown, stale_rows=stale, snippet_count=snippet_count) + + def _build_arg_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description=( @@ -348,31 +495,43 @@ def main(argv: Optional[List[str]] = None) -> int: print("check_usermanual_schema_drift: could not locate ForgeConfig in forgelm/config.py.", file=sys.stderr) return 1 - all_drift: List[KeyDrift] = [] - snippet_count = 0 - for md_path in walk_usermanuals(USERMANUAL_ROOT): - for snippet in extract_yaml_snippets(md_path): - snippet_count += 1 - drifts = check_snippet(snippet, schema, top_level) - if drifts: - all_drift.extend(drifts) + result = scan(USERMANUAL_ROOT, schema, top_level, _NON_FORGELM_YAML, _KNOWN_OPEN) - if all_drift: - print(f"FAIL: {len(all_drift)} fabricated schema key(s) found in docs/usermanuals/ YAML examples.") - for d in all_drift: - rel = d.path.relative_to(REPO_ROOT) if d.path.is_absolute() else d.path - print(f" {rel}:{d.line} {d.key_path} (no such field on the ForgeConfig schema)") - if args.strict: - return 1 - return 0 + if not result.clean: + _report(result) + return 1 if args.strict else 0 if not args.quiet: print( - f"OK: {snippet_count} yaml block(s) under docs/usermanuals/ checked; " - f"every recognised ForgeConfig key path resolves against the schema." + f"OK: {result.snippet_count} yaml block(s) under docs/usermanuals/ checked; " + f"every recognised ForgeConfig key path resolves against the schema and every " + f"other top-level block is another tool's YAML." ) return 0 +def _report(result: ScanResult) -> None: + """Print every finding of a failed scan.""" + if result.drifts: + print(f"FAIL: {len(result.drifts)} fabricated schema key(s) found in docs/usermanuals/ YAML examples.") + for d in result.drifts: + rel = d.path.relative_to(REPO_ROOT) if d.path.is_absolute() else d.path + print(f" {rel}:{d.line} {d.key_path} (no such field on the ForgeConfig schema)") + if result.unknown_blocks: + print( + f"FAIL: {len(result.unknown_blocks)} YAML block(s) in docs/usermanuals/ use top-level keys ForgeConfig rejects." + ) + for u in result.unknown_blocks: + rel = u.path.relative_to(REPO_ROOT) if u.path.is_absolute() else u.path + print( + f" {rel}:{u.line} {', '.join(sorted(u.keys))} (not a ForgeConfig top-level block: correct the " + f"example, or list it in _NON_FORGELM_YAML with a reason if it is another tool's YAML)" + ) + if result.stale_rows: + print(f"FAIL: {len(result.stale_rows)} exemption row(s) match no YAML block in every language.") + for table_name, (page, keys), reason in result.stale_rows: + print(f" {table_name}: {page} {', '.join(sorted(keys))} ({reason}) — delete the row") + + if __name__ == "__main__": sys.exit(main()) From 8bf2d08d56b729ea78056d242c2b5f994c718928 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:00:47 +0300 Subject: [PATCH 08/25] docs(pii): the severity tiers are ForgeLM's weighting, and an identifier is personal data only when it relates to a person Review follow-up on the PII citations. No regulator ranks identifiers, PCI DSS is an industry standard that covers card numbers (not IBANs), and Art. 87 addresses national identification numbers without expressing severity, so the guide (EN, TR) and the `PII_SEVERITY` comment now call the table ForgeLM's default weighting and cite each provision for what it actually says. The detector set is personal data under Art. 4(1) only when a match relates to an identified or identifiable natural person; a company's IBAN or switchboard number or a test card number is not (recital 14). The retired-phrase guard bans "consensus regulatory" wording too. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/data_audit-tr.md | 19 ++++++++++++++----- docs/guides/data_audit.md | 18 +++++++++++++----- forgelm/data_audit/_types.py | 11 ++++++----- .../test_usermanual_reference_consistency.py | 2 ++ 4 files changed, 35 insertions(+), 15 deletions(-) diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 10277c10..4095d689 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -160,9 +160,16 @@ kötü?* sorusunda sıfır rehberlik veriyor. Faz 11.5 yanına bir } ``` -Tier tablosu konsensüs regülatif ağırlıklandırmadır (finansal -kimlikler için PCI-DSS; devlet kimlikleri için GDPR Madde 87 (ulusal kimlik -numaraları) ve Madde 32 + ENISA). `critical` katman artık komutun kendisini kapılar (sonraki bölüm); +Tier tablosu ForgeLM'in varsayılan ağırlıklandırmasıdır; herhangi bir +mevzuattan alınmış bir sınıflandırma değildir. Kart numaraları ve IBAN'lar +doğrudan dolandırıcılıkta kullanılabildikleri için `critical` katmandadır +(PCI DSS, yasa değil sektör standardı, kart numaralarını kapsar; IBAN'lar aynı +katmanda ForgeLM'in kendi değerlendirmesiyle yer alır). Ulusal kimlik +numaraları değiştirilmesi zor olduğu ve bazı ulusal mevzuatlar kullanımlarını +kısıtladığı için `high` katmandadır (GDPR Madde 87, üye devletlerin ulusal +kimlik numarasının işlenme koşullarını belirlemesine izin verir). GDPR hiçbir +tanımlayıcıyı sıralamaz: Madde 32 riske uygun güvenlik ister; bu tablo o +değerlendirmeye başlamanıza yardım eder, yerini tutmaz. `critical` katman artık komutun kendisini kapılar (sonraki bölüm); `high`'ta da durmak isteyen pipeline'lar `pii_severity.worst_tier`'i okumalı ve açık bir review olmadan yayınlamayı reddetmelidir. @@ -407,8 +414,10 @@ install ile uyumlu kalır. ### ML-NER PII adaptörü — `--pii-ml` (Faz 12.5, opt-in) Varsayılan regex detector, yaygın yapılandırılmış identifier'ları (email, -telefon, IBAN, kredi kartı, ulusal kimlik) kapsar; bunlar GDPR Madde 4(1) -kapsamında kişisel veridir. +telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. Her biri, kimliği belirli +veya belirlenebilir bir gerçek kişiye ilişkin olduğunda GDPR Madde 4(1) +kapsamında kişisel veridir; bir şirketin IBAN'ı, santral numarası veya test +kart numarası değildir (Gerekçe 14). Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier kategorilerini ekleyen opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index e71bec1b..8ea429d4 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -150,9 +150,15 @@ The flat `pii_summary` map gives compliance reviewers no guidance on } ``` -The tier table is consensus regulatory weighting (PCI-DSS for financial -identifiers; GDPR Art. 87 (national identification numbers) and Art. 32 -plus ENISA for government IDs). The `critical` tier now gates the command itself (next section); pipelines that also +The tier table is ForgeLM's default weighting, not a classification taken +from any regulation. Card numbers and IBANs are `critical` because they are +directly usable for fraud (PCI DSS, an industry standard, covers card +numbers; IBANs share the tier as ForgeLM's own judgement). National +identifiers are `high` because they are hard to change and some national +laws restrict their use (GDPR Art. 87 lets Member States set conditions for +processing a national identification number). GDPR itself ranks no +identifier: Art. 32 asks for security appropriate to the risk, an assessment +this table helps you start and does not replace. The `critical` tier now gates the command itself (next section); pipelines that also want to stop at `high` should read `pii_severity.worst_tier` and refuse to publish without explicit review. @@ -401,8 +407,10 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. ### ML-NER PII adapter — `--pii-ml` (Phase 12.5, opt-in) The default regex detector covers the common structured identifiers -(email, phone, IBAN, credit card, national IDs), which are personal data -under GDPR Art. 4(1). +(email, phone, IBAN, credit card, national IDs). Each is personal data under +GDPR Art. 4(1) when it relates to an identified or identifiable natural +person; a company's IBAN or switchboard number, or a test card number, is not +(recital 14). Phase 12.5 adds an opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adapter that layers ML-NER on top — adding the unstructured-identifier categories regex inherently misses: `person`, `organization`, `location`. diff --git a/forgelm/data_audit/_types.py b/forgelm/data_audit/_types.py index b6b7be97..439d2cf7 100644 --- a/forgelm/data_audit/_types.py +++ b/forgelm/data_audit/_types.py @@ -22,12 +22,13 @@ # Phase 11.5: PII severity grading. A flat ``{type: count}`` map gives zero # guidance to a compliance reviewer staring at the audit JSON — they have to # remember by hand that ``credit_card`` is materially worse than ``phone``. -# The tiers below encode the consensus regulatory weighting (PCI-DSS for -# financial; GDPR Art. 87 + Art. 32 and ENISA for national IDs) so the audit -# surfaces a one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when -# local law disagrees — the audit honours whatever map is loaded at call time. +# The tiers below are ForgeLM's default weighting, not a regulatory +# classification (PCI DSS protects card numbers; GDPR Art. 87 lets Member +# States restrict national identification numbers), so the audit surfaces a +# one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when local law +# disagrees — the audit honours whatever map is loaded at call time. PII_SEVERITY: Dict[str, str] = { - # Financial / fully reversible identity theft -> highest weight. + # Financial identifiers -> highest weight: directly usable for fraud. "credit_card": "critical", "iban": "critical", # Government-issued identifiers -> high. Tied to a specific person and diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 9df61725..308241c3 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -163,6 +163,8 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"GDPR[\s\u2010-\u2015-]*(?:mandat|requir)", r"regulat\w*[\s\u2010-\u2015-]*mandated", r"(?:Article|Madde)\s+10\b[^.\n]{0,60}(?:cares|concerned|önemsediği)", + r"consensus\s+regulatory", + r"konsensüs\s+regülatif", ) _PURGE_ARTEFACT_WARNING_FILES = ( From 5f4b846e28b7e403fb034f62bb05a6e5c338a88f Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:00:47 +0300 Subject: [PATCH 09/25] =?UTF-8?q?docs(gdpr):=20review=20follow-up=20?= =?UTF-8?q?=E2=80=94=20the=20access/erasure=20walkthrough=20is=20consisten?= =?UTF-8?q?t,=20and=20the=20site=20no=20longer=20overclaims?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked the GDPR page and the site against the running commands: - The page's own purge command now passes the same `--output-dir ./outputs` as the reverse-pii example (without it the two commands wrote separate audit chains and salts); the printed hash is a 16-hex prefix; with a `data/*.jsonl` glob the implicit fallback would put `audit_log.jsonl` among the corpus files, so the page says always pass `--output-dir`. - `forgelm audit --pii-ml` reports per-category counts per split, not which rows or names, and masks nothing; Turkish needs `--pii-ml-language xx` (`xx_ent_wiki_sm`). The `--pii-ml-language` help and the pii-ml page no longer suggest `tr` or an `ingest --pii-ml` flag that do not exist. - The guide says the purge and reverse-pii digests match when the same value is passed to `--row-id` and `--query`. - Site, six languages plus the HTML fallbacks: the home card says reverse-pii finds the rows that contain an identifier you supply; the access card no longer says it surfaces "every row of every split" and that a deletion links later runs "forensically" (no code does); it says reverse-pii is an input to an Article 15 response that must also give a copy of the data and the Art. 15(1)(a)-(h) information, and that purge is one step of an erasure that leaves trained weights, source documents and other copies unchanged. The wizard's retention hints describe what `--check-policy` reports for the section, ephemeral artefacts, raw documents and per-run staging directories. German, French, Spanish and Chinese were written without a native review. - Tests: retired phrases for the new cards and hints in every locale, the weights caveat on the erasure card in every locale, and the static fallback must equal the English string. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/gdpr_erasure-tr.md | 3 +- docs/guides/gdpr_erasure.md | 5 +- docs/usermanuals/en/compliance/gdpr.md | 6 +- docs/usermanuals/en/data/pii-ml.md | 4 +- docs/usermanuals/tr/compliance/gdpr.md | 6 +- docs/usermanuals/tr/data/pii-ml.md | 4 +- forgelm/cli/_parser.py | 5 +- site/compliance.html | 4 +- site/features.html | 2 +- site/index.html | 2 +- site/js/translations.js | 98 +++++++++++++------------- tests/site/site_translations.test.cjs | 17 +++-- 12 files changed, 84 insertions(+), 72 deletions(-) diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index d0c23475..5c9560cb 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -145,7 +145,8 @@ için kullandığı aynı per-output-dir salt'ı yeniden kullanılır. Madde 15 erişim talepleri kendisi de subject'in verisini audit log'a sızdırmamalıdır. Salt'lı form, compliance reviewer'ın aynı subject için Madde 17 (purge) ve Madde 15 (reverse-pii) olaylarını cleartext -görmeden ilişkilendirmesini sağlar (digest'ler eşleşir). +görmeden ilişkilendirmesini sağlar (`--row-id` ve `--query`'ye aynı değeri +verdiğinizde digest'ler eşleşir). **Identifier tipleri** (`--type`): `literal` (varsayılan), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 6d3ddf0b..481f9447 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -143,8 +143,9 @@ that `forgelm purge` uses for its `target_id` field — Article 15 access requests must not themselves leak the subject's data into the audit log. The salted form is a stable per-identifier fingerprint that lets a compliance reviewer correlate Article 17 (purge) and -Article 15 (reverse-pii) events for the same subject (the digests -match) without seeing the cleartext. +Article 15 (reverse-pii) events for the same subject (the digests match +when you pass the same value to `--row-id` and `--query`) without seeing +the cleartext. **Identifier types** (`--type`): `literal` (default), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. All diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index d60ca980..bc83b452 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -31,7 +31,7 @@ PII masking at ingest replaces the identifier formats ForgeLM detects with a pla $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. To find names, organisations and locations, run `forgelm audit --pii-ml`: it adds Presidio NER **detection** (it reports them, it does not mask them) and needs the `ingestion-pii-ml` extra and a spaCy model. See [PII Masking](#/data/pii-masking) and [ML-NER PII](#/data/pii-ml). +The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. To measure how many names, organisations and locations remain, run `forgelm audit --pii-ml`: it adds Presidio NER **detection**, which reports per-category counts for each split (not which rows or which names) and masks nothing. It needs the `ingestion-pii-ml` extra and a spaCy model (for a language without a dedicated model, such as Turkish, `--pii-ml-language xx` uses `xx_ent_wiki_sm`). To find a known name, search for it with `forgelm reverse-pii`. See [PII Masking](#/data/pii-masking) and [ML-NER PII](#/data/pii-ml). ### Accountability (Art. 5(2)) @@ -70,14 +70,14 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed hash is the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning. It finds what is literally in the files you point it at. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. +The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed value is the first 16 hex digits of the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning; with a `data/*.jsonl` glob that puts `audit_log.jsonl` among your corpus files, so always pass `--output-dir`. It finds what is literally in the files you point it at. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. ### Right to erasure (Art. 17) "Delete my data." If a person's data was in your training set: -1. Find the rows with `forgelm reverse-pii`, then remove them with `forgelm purge --row-id --corpus `; every erasure is written to the audit log. `--row-id` is matched against the row's `id` or `row_id` field and never against a line number, and `forgelm ingest` does not write either field, so give every row a stable id **before you train** (see below). A corpus without ids cannot be purged row by row. +1. Find the rows with `forgelm reverse-pii`, then remove them with `forgelm purge --row-id --corpus --output-dir ./outputs` (the same directory as `reverse-pii`; use your training output directory so purge can also warn about a trained `final_model/`); every erasure is written to the audit log. `--row-id` is matched against the row's `id` or `row_id` field and never against a line number, and `forgelm ingest` does not write either field, so give every row a stable id **before you train** (see below). A corpus without ids cannot be purged row by row. 2. Source documents may still hold it — drop them from your raw store and re-ingest if necessary. 3. The model may have memorised some details — see "model-level erasure" below. diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index e49a2068..99d18c43 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -5,9 +5,9 @@ description: Optional Microsoft Presidio adapter that adds person / organization # ML-NER PII Detection — `--pii-ml` -The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the *structured* identifiers that make data personal under GDPR Article 4(1) (email, phone, IBAN, credit card, national IDs; see also Article 87). Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. +The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the common *structured* identifiers (email, phone, IBAN, credit card, national IDs). They are personal data under GDPR Article 4(1) whenever they relate to an identifiable natural person; Article 87 additionally lets Member States restrict how national identification numbers are processed. Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. -This is an **opt-in** layer. The default audit + ingest behaviour is unchanged when `--pii-ml` is not passed. +This is an **opt-in** layer. The default audit behaviour is unchanged when `--pii-ml` is not passed (`forgelm ingest` has no `--pii-ml` flag). ## When to use it diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index 461bebe2..ad20c135 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -31,7 +31,7 @@ Ingest'te PII maskeleme, ForgeLM'in tespit ettiği tanımlayıcı biçimlerini b $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. İsimleri, kurumları ve yerleri bulmak için `forgelm audit --pii-ml` çalıştırın: Presidio NER **tespiti** ekler (raporlar, maskelemez) ve `ingestion-pii-ml` extra'sını ile bir spaCy modeli gerektirir. Bkz. [PII Maskeleme](#/data/pii-masking) ve [ML-NER PII](#/data/pii-ml). +Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. Geriye kaç isim, kurum ve yer kaldığını ölçmek için `forgelm audit --pii-ml` çalıştırın: Presidio NER **tespiti** ekler; bu tespit her split için kategori başına sayıları raporlar (hangi satırların ya da hangi isimlerin olduğunu değil) ve hiçbir şeyi maskelemez. `ingestion-pii-ml` extra'sını ve bir spaCy modelini gerektirir (Türkçe gibi özel modeli olmayan diller için `--pii-ml-language xx`, `xx_ent_wiki_sm` kullanır). Bilinen bir ismi bulmak için onu `forgelm reverse-pii` ile arayın. Bkz. [PII Maskeleme](#/data/pii-masking) ve [ML-NER PII](#/data/pii-ml). ### Hesap verebilirlik (Md. 5(2)) @@ -70,14 +70,14 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan hash, audit event'inin kaydettiği tuzlu tanımlayıcı hash'idir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. +Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan değer, audit event'inin kaydettiği tuzlu tanımlayıcı hash'inin ilk 16 hex hanesidir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar; `data/*.jsonl` gibi bir glob ile bu, `audit_log.jsonl`'i corpus dosyalarınızın arasına koyar, bu yüzden her zaman `--output-dir` verin. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. ### Silme hakkı (Md. 17) "Verilerimi silin." Bir kişinin verisi eğitim setinizdeydi: -1. Satırları `forgelm reverse-pii` ile bulun, sonra `forgelm purge --row-id --corpus ` ile kaldırın; her silme audit log'a yazılır. `--row-id` satırın `id` veya `row_id` alanıyla eşleştirilir, satır numarasıyla asla; `forgelm ingest` bu alanlardan hiçbirini yazmaz. Bu yüzden **eğitimden önce** her satıra kalıcı bir id verin (aşağıya bakın). Id'si olmayan bir corpus satır satır silinemez. +1. Satırları `forgelm reverse-pii` ile bulun, sonra `forgelm purge --row-id --corpus --output-dir ./outputs` ile kaldırın (`reverse-pii` ile aynı dizin; purge'ün eğitilmiş bir `final_model/` için de uyarabilmesi adına eğitim çıktı dizininizi kullanın); her silme audit log'a yazılır. `--row-id` satırın `id` veya `row_id` alanıyla eşleştirilir, satır numarasıyla asla; `forgelm ingest` bu alanlardan hiçbirini yazmaz. Bu yüzden **eğitimden önce** her satıra kalıcı bir id verin (aşağıya bakın). Id'si olmayan bir corpus satır satır silinemez. 2. Ham kaynak dokümanlar hâlâ içerebilir — ham deponuzdan düşürün ve gerekirse yeniden ingest edin. 3. Model bazı detayları ezberlemiş olabilir — aşağıdaki "model-seviyesi silme"ye bakın. diff --git a/docs/usermanuals/tr/data/pii-ml.md b/docs/usermanuals/tr/data/pii-ml.md index 59da063b..f7df352f 100644 --- a/docs/usermanuals/tr/data/pii-ml.md +++ b/docs/usermanuals/tr/data/pii-ml.md @@ -5,9 +5,9 @@ description: Regex PII katmanının üzerine person / organization / location te # ML-NER PII Tespiti — `--pii-ml` -Varsayılan [PII Maskeleme](#/data/pii-masking) katmanı regex-anchored'dır ve GDPR Madde 4(1) kapsamında veriyi kişisel yapan *yapılandırılmış* identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlikler; ayrıca Madde 87) kapsar. Presidio NER, regex'in doğal olarak kaçırdığı *yapılandırılmamış* identifier'ları ekler: kişi isimleri, organizasyon isimleri, coğrafi yerler. +Varsayılan [PII Maskeleme](#/data/pii-masking) katmanı regex-anchored'dır ve yaygın *yapılandırılmış* tanımlayıcıları (e-posta, telefon, IBAN, kredi kartı, ulusal kimlik numaraları) kapsar. Bunlar, kimliği belirli veya belirlenebilir bir gerçek kişiye ilişkin olduklarında GDPR Madde 4(1) kapsamında kişisel veridir; Madde 87 ayrıca üye devletlerin ulusal kimlik numaralarının işlenme koşullarını belirlemesine izin verir. Presidio NER, regex'in doğal olarak kaçırdığı *yapılandırılmamış* identifier'ları ekler: kişi isimleri, organizasyon isimleri, coğrafi yerler. -Bu katman **opt-in**'dir. `--pii-ml` geçilmediğinde varsayılan audit + ingest davranışı değişmez. +Bu katman **opt-in**'dir. `--pii-ml` geçilmediğinde varsayılan audit davranışı değişmez (`forgelm ingest`'in `--pii-ml` flag'i yoktur). ## Ne zaman kullanılır diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index a7fe6ec1..56804e2c 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -628,8 +628,9 @@ def _add_audit_subcommand(subparsers) -> None: "Phase 12.5: language code passed to Presidio's NLP engine when --pii-ml is on " "(default: 'en'). Presidio raises a typed exception if no engine is registered for " "the requested language — surface it to the operator instead of silently " - "running an English NER on a non-English corpus. Set to e.g. 'tr' on a Turkish " - "corpus AND make sure the matching spaCy model is installed." + "running an English NER on a non-English corpus. Set to e.g. 'de' on a German " + "corpus, or 'xx' (multilingual fallback) for a language without a dedicated spaCy " + "model such as Turkish, AND make sure the matching spaCy model is installed." ), ) p.add_argument( diff --git a/site/compliance.html b/site/compliance.html index b4846663..fc14b304 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -202,14 +202,14 @@

PII redaction at ingest

GDPR Art. 15

Right of access

-

forgelm reverse-pii scans masked JSONL corpora for plaintext residuals or salted SHA-256 digests of a data-subject identifier and reports every row of every split where that subject appears. It finds only what is literally in the files you scan: raw sources and the trained model are not searched, and handling an access request stays your process.

+

forgelm reverse-pii scans JSONL corpora for plaintext occurrences, or for salted SHA-256 digests written by an external masking pipeline, of a data-subject identifier and reports every line, in the files you scan, that contains it. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.

forgelm reverse-pii
GDPR Art. 17

Right to erasure

-

forgelm purge deletes a data-subject's rows from training corpora and stamps data.erasure_requested + data.erasure_completed events into the append-only audit log so post-deletion runs are forensically linked to the erasure event.

+

forgelm purge deletes rows by id from a training corpus and records data.erasure_requested + data.erasure_completed events in the append-only audit log. It is one step of an Article 17 erasure: trained weights, source documents and copies elsewhere are not changed.

forgelm purge
diff --git a/site/features.html b/site/features.html index 65307521..4e26d200 100644 --- a/site/features.html +++ b/site/features.html @@ -420,7 +420,7 @@

GDPR Article 17 erasure

GDPR Article 15 access

-

forgelm reverse-pii scans masked JSONL corpora for forensic substring or salted-digest matches that surface every row in which a data-subject identifier appears.

+

forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.

diff --git a/site/index.html b/site/index.html index e888ec77..7bb24214 100644 --- a/site/index.html +++ b/site/index.html @@ -536,7 +536,7 @@

ISO 27001 / SOC 2 alignment

GDPR rights tooling

-

Tooling for GDPR data-subject requests: forgelm reverse-pii locates a subject's rows (an input to an Article 15 response); forgelm purge removes them from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

+

Tooling for GDPR data-subject requests: forgelm reverse-pii finds the rows that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

forgelm purge forgelm reverse-pii diff --git a/site/js/translations.js b/site/js/translations.js index 2b728f11..c507181f 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -390,7 +390,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 alignment", "home.cap.iso.body": "Audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor asks for. Software is aligned, not certified.", "home.cap.gdpr.title": "GDPR rights tooling", - "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii locates a subject's rows (an input to an Article 15 response); forgelm purge removes them from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", + "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii finds the rows that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", "home.cap.see_all": "See the full feature matrix", "home.yaml.eyebrow": "A real config — not a screenshot", "home.yaml.title": "One YAML, end-to-end run", @@ -496,7 +496,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 hizalaması", "home.cap.iso.body": "Deployer denetçisinin sorduğu audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtları. Yazılım sertifikalı değil, hizalanmıştır.", "home.cap.gdpr.title": "GDPR hakları araçları", - "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii bir öznenin satırlarını bulur (Madde 15 yanıtının bir girdisi); forgelm purge bunları eğitim corpus'larından siler (Madde 17 silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", + "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 yanıtının bir girdisi); forgelm purge satırları eğitim corpus'larından siler (Madde 17 silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", "home.cap.see_all": "Tüm özellik matrisini gör", "home.yaml.eyebrow": "Gerçek bir konfigürasyon — ekran görüntüsü değil", "home.yaml.title": "Bir YAML, uçtan uca koşu", @@ -602,7 +602,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 Alignment", "home.cap.iso.body": "Audit-Trail, Change-Management, Daten-Lineage und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Software ist aligned, nicht zertifiziert.", "home.cap.gdpr.title": "DSGVO-Rechtetooling", - "home.cap.gdpr.body": "Werkzeuge für DSGVO-Betroffenenanfragen: forgelm reverse-pii findet die Zeilen einer betroffenen Person (ein Baustein der Antwort nach Artikel 15); forgelm purge entfernt sie aus Trainingskorpora (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Beides wird im Audit-Log festgehalten.", + "home.cap.gdpr.body": "Werkzeuge für DSGVO-Betroffenenanfragen: forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Artikel 15); forgelm purge entfernt Zeilen aus Trainingskorpora (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Beides wird im Audit-Log festgehalten.", "home.cap.see_all": "Vollständige Funktionsmatrix anzeigen", "home.yaml.eyebrow": "Eine echte Konfiguration — kein Screenshot", "home.yaml.title": "Ein YAML, ein Ende-zu-Ende-Lauf", @@ -708,7 +708,7 @@ "home.cap.iso.title": "Alignement ISO 27001 / SOC 2", "home.cap.iso.body": "Piste d'audit, gestion du changement, lignée des données et preuves de chaîne logistique demandées par l'auditeur du déployeur. Le logiciel est aligné, pas certifié.", "home.cap.gdpr.title": "Outils de droits RGPD", - "home.cap.gdpr.body": "Outils pour les demandes de personnes concernées au titre du RGPD : forgelm reverse-pii localise les lignes d'une personne (un élément de la réponse à l'article 15) ; forgelm purge les supprime des corpus d'entraînement (une étape de l'effacement de l'article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", + "home.cap.gdpr.body": "Outils pour les demandes de personnes concernées au titre du RGPD : forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, Article 15) ; forgelm purge supprime des lignes des corpus d'entraînement (une étape de l'effacement, Article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", "home.cap.see_all": "Voir la matrice complète des fonctionnalités", "home.yaml.eyebrow": "Une vraie configuration — pas une capture d'écran", "home.yaml.title": "Un YAML, exécution de bout en bout", @@ -814,7 +814,7 @@ "home.cap.iso.title": "Alineación con ISO 27001 / SOC 2", "home.cap.iso.body": "Pista de auditoría, gestión del cambio, linaje de datos y evidencias de cadena de suministro que pide el auditor del implementador. El software está alineado, no certificado.", "home.cap.gdpr.title": "Herramientas de derechos RGPD", - "home.cap.gdpr.body": "Herramientas para solicitudes de interesados del RGPD: forgelm reverse-pii localiza las filas de un sujeto (un insumo para la respuesta del Artículo 15); forgelm purge las elimina de los corpus de entrenamiento (un paso del borrado del Artículo 17; los pesos entrenados no cambian). Ambos quedan registrados en el log de auditoría.", + "home.cap.gdpr.body": "Herramientas para solicitudes de interesados del RGPD: forgelm reverse-pii encuentra las filas que contienen un identificador que usted proporciona (un elemento de la respuesta a la solicitud de acceso del Artículo 15); forgelm purge elimina filas de los corpus de entrenamiento (un paso de la supresión del Artículo 17; los pesos entrenados no cambian). Ambos quedan registrados en el log de auditoría.", "home.cap.see_all": "Ver la matriz completa de características", "home.yaml.eyebrow": "Una configuración real — no una captura", "home.yaml.title": "Un YAML, ejecución de extremo a extremo", @@ -920,7 +920,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 对齐", "home.cap.iso.body": "部署方审计员要求的审计轨迹、变更管理、数据沿袭和供应链证据。软件是对齐的,不是认证的。", "home.cap.gdpr.title": "GDPR 权利工具", - "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 定位某数据主体的行(第 15 条答复的输入之一),forgelm purge 将这些行从训练语料中删除(第 17 条删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", + "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 查找包含你所提供标识符的行(第 15 条答复的输入之一),forgelm purge 从训练语料中删除行(第 17 条删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", "home.cap.see_all": "查看完整功能矩阵", "home.yaml.eyebrow": "真实配置 — 不是截图", "home.yaml.title": "一个 YAML,端到端运行", @@ -1035,7 +1035,7 @@ "features.gov.purge.title": "GDPR Article 17 erasure", "features.gov.purge.body": "forgelm purge drops a data-subject's rows from training corpora and stamps the deletion in the append-only audit log.", "features.gov.access.title": "GDPR Article 15 access", - "features.gov.access.body": "forgelm reverse-pii scans masked JSONL corpora for forensic substring or salted-digest matches that surface every row in which a data-subject identifier appears.", + "features.gov.access.body": "forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.", "features.gov.annex.title": "Annex IV verifier", "features.gov.annex.body": "forgelm verify-annex-iv validates the Annex IV §1-9 schema completeness and the bundle's manifest_hash for tampering detection. (Audit-chain integrity is the separate forgelm verify-audit command.)", "features.gov.gguf.title": "GGUF integrity verifier", @@ -1140,7 +1140,7 @@ "features.gov.purge.title": "GDPR Madde 17 silme", "features.gov.purge.body": "forgelm purge bir veri sahibinin satırlarını eğitim corpus'undan düşürür ve silmeyi append-only audit log'a damgalar.", "features.gov.access.title": "GDPR Madde 15 erişim", - "features.gov.access.body": "forgelm reverse-pii maskelenmiş JSONL corpus'larını tarayarak veri sahibi tanımlayıcısının düz-metin kalıntılarına ya da tuzlanmış-digest eşleşmelerine bakar ve geçtiği her satırı yüzeye çıkarır.", + "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz-metin ya da tuzlanmış-digest eşleşmelerine bakar ve taranan dosyalarda onu içeren satırları listeler.", "features.gov.annex.title": "Annex IV doğrulayıcı", "features.gov.annex.body": "forgelm verify-annex-iv Annex IV §1-9 şema tamlığını ve bundle'ın manifest_hash'ini tampering tespiti için doğrular. (Audit-chain bütünlüğü ayrı bir forgelm verify-audit komutudur.)", "features.gov.gguf.title": "GGUF bütünlük doğrulayıcı", @@ -1811,15 +1811,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API key env var", "wizard.synthetic.api_key_env.hint": "Environment variable name carrying the teacher API key. Empty = local teacher model.", "wizard.retention.title": "Retention horizons (GDPR Art. 5+17)", - "wizard.retention.hint": "How long ForgeLM keeps audit logs, staging models, and ephemeral artefacts.", + "wizard.retention.hint": "Ages at which forgelm purge --check-policy reports artefacts as overdue; nothing is deleted.", "wizard.retention.audit_log.label": "Audit log retention (days)", "wizard.retention.audit_log.hint": "How long before forgelm purge --check-policy reports audit_log.jsonl as overdue. Default 1825 (5 years) is a ForgeLM choice, not a legal period.", "wizard.retention.staging.label": "Staging horizon (days)", - "wizard.retention.staging.hint": "Age at which forgelm purge --check-policy reports a final_model.staging/ directory as overdue; nothing is deleted automatically. Default 7.", + "wizard.retention.staging.hint": "Age at which forgelm purge --check-policy reports a final_model.staging.*/ (or legacy final_model.staging/) directory as overdue; nothing is deleted automatically. Default 7.", "wizard.retention.ephemeral.label": "Ephemeral artefacts (days)", - "wizard.retention.ephemeral.hint": "How long to keep intermediate checkpoints, eval samples. Default 90.", + "wizard.retention.ephemeral.hint": "Age at which forgelm purge --check-policy reports compliance/ and data_audit_report.json as overdue. Default 90.", "wizard.retention.raw_documents.label": "Raw documents (days)", - "wizard.retention.raw_documents.hint": "How long to keep raw PDFs / DOCX after ingestion. Default 90.", + "wizard.retention.raw_documents.hint": "Age at which forgelm purge --check-policy reports raw_documents/ or ingestion_output/ as overdue. Default 90.", "wizard.retention.enforce.label": "Policy mode (reserved)", "wizard.retention.enforce.hint": "Reserved: no ForgeLM command acts on this yet, so all three modes behave the same. Retention horizons are only reported by forgelm purge --check-policy; nothing is deleted automatically.", "wizard.retention.enforce.log_only": "log_only — default (no command acts on it)", @@ -2209,15 +2209,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API key env değişkeni", "wizard.synthetic.api_key_env.hint": "Teacher API anahtarını taşıyan ortam değişkeni adı. Boş = yerel teacher modeli.", "wizard.retention.title": "Saklama ufukları (GDPR Madde 5+17)", - "wizard.retention.hint": "ForgeLM audit log, staging model ve geçici artefact'ları ne kadar tutar.", + "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları geciken olarak raporladığı yaşlar; hiçbir şey silinmez.", "wizard.retention.audit_log.label": "Audit log saklama (gün)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra geciken olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", "wizard.retention.staging.label": "Staging ufku (gün)", - "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging/ dizinini hangi yaşta geciken olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", + "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging.*/ (veya legacy final_model.staging/) dizinini hangi yaşta geciken olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", "wizard.retention.ephemeral.label": "Geçici artefact'lar (gün)", - "wizard.retention.ephemeral.hint": "Ara checkpoint'ler, eval örnekleri ne kadar tutulsun. Varsayılan 90.", + "wizard.retention.ephemeral.hint": "forgelm purge --check-policy compliance/ ve data_audit_report.json'u hangi yaşta geciken olarak raporlasın. Varsayılan 90.", "wizard.retention.raw_documents.label": "Ham dokümanlar (gün)", - "wizard.retention.raw_documents.hint": "Ingestion sonrası ham PDF / DOCX ne kadar tutulsun. Varsayılan 90.", + "wizard.retention.raw_documents.hint": "forgelm purge --check-policy raw_documents/ veya ingestion_output/ dizinini hangi yaşta geciken olarak raporlasın. Varsayılan 90.", "wizard.retention.enforce.label": "Politika modu (ayrılmış)", "wizard.retention.enforce.hint": "Ayrılmış: hiçbir ForgeLM komutu bunu henüz kullanmaz, bu yüzden üç mod da aynı davranır. Saklama ufukları yalnızca forgelm purge --check-policy ile raporlanır; hiçbir şey otomatik silinmez.", "wizard.retention.enforce.log_only": "log_only — varsayılan (hiçbir komut kullanmaz)", @@ -2603,15 +2603,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API-Key Env-Variable", "wizard.synthetic.api_key_env.hint": "Env-Variable mit dem Teacher-API-Key. Leer = lokales Teacher-Modell.", "wizard.retention.title": "Aufbewahrungsfristen (DSGVO Art. 5+17)", - "wizard.retention.hint": "Wie lange ForgeLM Audit-Logs, Staging-Modelle, ephemere Artefakte aufbewahrt.", + "wizard.retention.hint": "Alter, ab dem forgelm purge --check-policy Artefakte als überfällig meldet; nichts wird gelöscht.", "wizard.retention.audit_log.label": "Audit-Log Aufbewahrung (Tage)", "wizard.retention.audit_log.hint": "Nach wie vielen Tagen forgelm purge --check-policy audit_log.jsonl als überfällig meldet. Default 1825 (5 Jahre) ist eine ForgeLM-Wahl, keine gesetzliche Frist.", "wizard.retention.staging.label": "Staging-Horizont (Tage)", - "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging/-Verzeichnis als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", + "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging.*/-Verzeichnis (oder das ältere final_model.staging/) als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", "wizard.retention.ephemeral.label": "Ephemere Artefakte (Tage)", - "wizard.retention.ephemeral.hint": "Wie lange Zwischen-Checkpoints, Eval-Samples behalten. Default 90.", + "wizard.retention.ephemeral.hint": "Ab welchem Alter forgelm purge --check-policy compliance/ und data_audit_report.json als überfällig meldet. Default 90.", "wizard.retention.raw_documents.label": "Rohdokumente (Tage)", - "wizard.retention.raw_documents.hint": "Wie lange rohe PDFs/DOCX nach Ingestion behalten. Default 90.", + "wizard.retention.raw_documents.hint": "Ab welchem Alter forgelm purge --check-policy raw_documents/ oder ingestion_output/ als überfällig meldet. Default 90.", "wizard.retention.enforce.label": "Richtlinienmodus (reserviert)", "wizard.retention.enforce.hint": "Reserviert: Noch wertet kein ForgeLM-Befehl dies aus, daher verhalten sich alle drei Modi gleich. Aufbewahrungshorizonte meldet nur forgelm purge --check-policy; nichts wird automatisch gelöscht.", "wizard.retention.enforce.log_only": "log_only — Default (kein Befehl wertet es aus)", @@ -2997,15 +2997,15 @@ "wizard.synthetic.api_key_env.label": "Variable env clé API teacher", "wizard.synthetic.api_key_env.hint": "Nom variable env portant clé API teacher. Vide = modèle teacher local.", "wizard.retention.title": "Horizons de rétention (RGPD Art. 5+17)", - "wizard.retention.hint": "Combien de temps ForgeLM garde audit logs, modèles staging, artefacts éphémères.", + "wizard.retention.hint": "Âges à partir desquels forgelm purge --check-policy signale des artefacts comme dépassés ; rien n'est supprimé.", "wizard.retention.audit_log.label": "Rétention audit log (jours)", "wizard.retention.audit_log.hint": "Après combien de jours forgelm purge --check-policy signale audit_log.jsonl comme dépassé. Default 1825 (5 ans) est un choix de ForgeLM, pas un délai légal.", "wizard.retention.staging.label": "Horizon staging (jours)", - "wizard.retention.staging.hint": "À partir de quel âge forgelm purge --check-policy signale un répertoire final_model.staging/ comme dépassé ; rien n'est supprimé automatiquement. Default 7.", + "wizard.retention.staging.hint": "À partir de quel âge forgelm purge --check-policy signale un répertoire final_model.staging.*/ (ou l'ancien final_model.staging/) comme dépassé ; rien n'est supprimé automatiquement. Default 7.", "wizard.retention.ephemeral.label": "Artefacts éphémères (jours)", - "wizard.retention.ephemeral.hint": "Combien de temps garder checkpoints intermédiaires, samples eval. Default 90.", + "wizard.retention.ephemeral.hint": "À partir de quel âge forgelm purge --check-policy signale compliance/ et data_audit_report.json comme dépassés. Default 90.", "wizard.retention.raw_documents.label": "Documents bruts (jours)", - "wizard.retention.raw_documents.hint": "Combien de temps garder PDFs/DOCX bruts après ingestion. Default 90.", + "wizard.retention.raw_documents.hint": "À partir de quel âge forgelm purge --check-policy signale raw_documents/ ou ingestion_output/ comme dépassés. Default 90.", "wizard.retention.enforce.label": "Mode de politique (réservé)", "wizard.retention.enforce.hint": "Réservé : aucune commande ForgeLM n'agit encore dessus, donc les trois modes se comportent de la même façon. Les horizons de rétention ne sont signalés que par forgelm purge --check-policy ; rien n'est supprimé automatiquement.", "wizard.retention.enforce.log_only": "log_only — défaut (aucune commande n'en tient compte)", @@ -3391,15 +3391,15 @@ "wizard.synthetic.api_key_env.label": "Variable env clave API teacher", "wizard.synthetic.api_key_env.hint": "Nombre variable env con clave API teacher. Vacío = modelo teacher local.", "wizard.retention.title": "Horizontes de retención (RGPD Art. 5+17)", - "wizard.retention.hint": "Cuánto tiempo ForgeLM guarda audit logs, modelos staging, artefactos efímeros.", + "wizard.retention.hint": "Antigüedades a partir de las cuales forgelm purge --check-policy reporta artefactos como vencidos; no se borra nada.", "wizard.retention.audit_log.label": "Retención audit log (días)", - "wizard.retention.audit_log.hint": "Tras cuántos días forgelm purge --check-policy informa audit_log.jsonl como vencido. Default 1825 (5 años) es una elección de ForgeLM, no un plazo legal.", + "wizard.retention.audit_log.hint": "Tras cuántos días forgelm purge --check-policy reporta audit_log.jsonl como vencido. Default 1825 (5 años) es una elección de ForgeLM, no un plazo legal.", "wizard.retention.staging.label": "Horizonte staging (días)", - "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy informa un directorio final_model.staging/ como vencido; no se borra nada automáticamente. Default 7.", + "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta un directorio final_model.staging.*/ (o el antiguo final_model.staging/) como vencido; no se borra nada automáticamente. Default 7.", "wizard.retention.ephemeral.label": "Artefactos efímeros (días)", - "wizard.retention.ephemeral.hint": "Cuánto guardar checkpoints intermedios, samples eval. Default 90.", + "wizard.retention.ephemeral.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta compliance/ y data_audit_report.json como vencidos. Default 90.", "wizard.retention.raw_documents.label": "Documentos crudos (días)", - "wizard.retention.raw_documents.hint": "Cuánto guardar PDFs/DOCX crudos tras ingestion. Default 90.", + "wizard.retention.raw_documents.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta raw_documents/ o ingestion_output/ como vencidos. Default 90.", "wizard.retention.enforce.label": "Modo de política (reservado)", "wizard.retention.enforce.hint": "Reservado: ningún comando de ForgeLM actúa aún sobre esto, así que los tres modos se comportan igual. Los horizontes de retención solo los informa forgelm purge --check-policy; nada se elimina automáticamente.", "wizard.retention.enforce.log_only": "log_only — default (ningún comando lo usa)", @@ -3785,15 +3785,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API 密钥环境变量", "wizard.synthetic.api_key_env.hint": "承载 teacher API 密钥的环境变量名。空 = 本地 teacher 模型。", "wizard.retention.title": "保留期限(GDPR 第 5+17 条)", - "wizard.retention.hint": "ForgeLM 保留 audit log、staging 模型、临时 artefact 多长时间。", + "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已过期的时长;不会删除任何内容。", "wizard.retention.audit_log.label": "Audit log 保留(天)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已过期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", "wizard.retention.staging.label": "Staging 期限(天)", - "wizard.retention.staging.hint": "final_model.staging/ 目录存在多少天后,forgelm purge --check-policy 会将其报告为已过期;不会自动删除任何内容。默认 7。", + "wizard.retention.staging.hint": "final_model.staging.*/(或旧的 final_model.staging/)目录存在多少天后,forgelm purge --check-policy 会将其报告为已过期;不会自动删除任何内容。默认 7。", "wizard.retention.ephemeral.label": "临时 artefact(天)", - "wizard.retention.ephemeral.hint": "中间 checkpoint、eval 样本保留多久。默认 90。", + "wizard.retention.ephemeral.hint": "compliance/ 和 data_audit_report.json 存在多少天后,forgelm purge --check-policy 会将其报告为已过期。默认 90。", "wizard.retention.raw_documents.label": "原始文档(天)", - "wizard.retention.raw_documents.hint": "ingestion 后原始 PDF/DOCX 保留多久。默认 90。", + "wizard.retention.raw_documents.hint": "raw_documents/ 或 ingestion_output/ 存在多少天后,forgelm purge --check-policy 会将其报告为已过期。默认 90。", "wizard.retention.enforce.label": "策略模式(保留)", "wizard.retention.enforce.hint": "保留项:目前没有任何 ForgeLM 命令使用它,因此三种模式行为相同。保留期限仅由 forgelm purge --check-policy 报告;不会自动删除任何内容。", "wizard.retention.enforce.log_only": "log_only — 默认(没有命令使用它)", @@ -3870,9 +3870,9 @@ "compliance.gdpr.title": "PII redaction at ingest", "compliance.gdpr.body": "--pii-mask redacts emails, phones, credit cards (Luhn), IBAN, national IDs (TR/DE/FR/US-SSN) before chunks land in JSONL. It reduces what reaches training and supports data minimisation (Article 5(1)(c)); names and free text are not detected, so it is an aid, not a guarantee.", "compliance.gdpr15.title": "Right of access", - "compliance.gdpr15.body": "forgelm reverse-pii scans masked JSONL corpora for plaintext residuals or salted SHA-256 digests of a data-subject identifier and reports every row of every split where that subject appears. It finds only what is literally in the files you scan: raw sources and the trained model are not searched, and handling an access request stays your process.", + "compliance.gdpr15.body": "forgelm reverse-pii scans JSONL corpora for plaintext occurrences, or for salted SHA-256 digests written by an external masking pipeline, of a data-subject identifier and reports every line, in the files you scan, that contains it. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.", "compliance.gdpr17.title": "Right to erasure", - "compliance.gdpr17.body": "forgelm purge deletes a data-subject's rows from training corpora and stamps data.erasure_requested + data.erasure_completed events into the append-only audit log so post-deletion runs are forensically linked to the erasure event.", + "compliance.gdpr17.body": "forgelm purge deletes rows by id from a training corpus and records data.erasure_requested + data.erasure_completed events in the append-only audit log. It is one step of an Article 17 erasure: trained weights, source documents and copies elsewhere are not changed.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.title": "Aligned, not certified", "compliance.iso.lead": "Software cannot be ISO 27001 or SOC 2 certified — only organisations can. ForgeLM is aligned with the controls an auditor asks the deployer about: it produces the audit-trail, change-management, data-lineage, and supply-chain evidence on every run, ready to drop into your audit binder.", @@ -3929,9 +3929,9 @@ "compliance.gdpr.title": "Ingest'te PII redaksiyonu", "compliance.gdpr.body": "--pii-mask chunk'lar JSONL'a düşmeden e-posta, telefon, kredi kartı (Luhn), IBAN, ulusal kimlik (TR/DE/FR/US-SSN) maskeler. Eğitime ulaşanı azaltır ve veri minimizasyonunu (Madde 5(1)(c)) destekler; isimler ve serbest metin tespit edilmez, bu yüzden bir yardımcıdır, garanti değildir.", "compliance.gdpr15.title": "Erişim hakkı", - "compliance.gdpr15.body": "forgelm reverse-pii maskelenmiş JSONL corpus'larında bir veri sahibi tanımlayıcısının düz-metin kalıntılarını veya tuzlanmış SHA-256 digest'lerini arar ve o öznenin geçtiği her split'in her satırını raporlar. Yalnızca taradığınız dosyalarda harfiyen bulunanı bulur: ham kaynaklar ve eğitilmiş model aranmaz; bir erişim talebini yürütmek sizin sürecinizde kalır.", + "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz-metin geçişlerini veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", "compliance.gdpr17.title": "Silme hakkı", - "compliance.gdpr17.body": "forgelm purge bir veri sahibinin satırlarını eğitim corpus'undan siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını damgalar; silme sonrası çalışmalar adli olarak silme olayına bağlanır.", + "compliance.gdpr17.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını kaydeder. Madde 17 silmenin bir adımıdır: eğitilmiş ağırlıklar, kaynak dokümanlar ve başka yerlerdeki kopyalar değişmez.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.title": "Hizalanmış, sertifikalı değil", "compliance.iso.lead": "Yazılım ISO 27001 veya SOC 2 sertifikalı olamaz — sadece organizasyonlar olabilir. ForgeLM, denetçinin deployer'a sorduğu kontrollerle hizalıdır: her çalışmada audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtlarını üretir; doğrudan audit klasörünüze düşer.", @@ -4452,9 +4452,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.de, { "compliance.gdpr15.title": "Auskunftsrecht", - "compliance.gdpr15.body": "forgelm reverse-pii durchsucht maskierte JSONL-Korpora nach Klartextresten oder gesalzenen SHA-256-Digests einer Betroffenenkennung und meldet jede Zeile jedes Splits, in der die betroffene Person erscheint. Es findet nur, was wörtlich in den gescannten Dateien steht: Rohquellen und das trainierte Modell werden nicht durchsucht, und die Bearbeitung eines Auskunftsersuchens bleibt Ihr Prozess.", + "compliance.gdpr15.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext-Vorkommen oder, bei einer externen Maskierungs-Pipeline, nach gesalzenen SHA-256-Digests einer Betroffenenkennung und meldet jede Zeile der gescannten Dateien, die sie enthält. Das ist ein Baustein Ihrer Auskunft nach Artikel 15, die außerdem eine Datenkopie und die Informationen nach Artikel 15 Abs. 1 Buchst. a bis h enthalten muss; Rohquellen und das trainierte Modell werden nicht durchsucht.", "compliance.gdpr17.title": "Recht auf Löschung", - "compliance.gdpr17.body": "forgelm purge löscht die Zeilen einer betroffenen Person aus Trainingskorpora und stempelt data.erasure_requested + data.erasure_completed-Ereignisse in das Append-only-Audit-Log, sodass Läufe nach der Löschung forensisch mit dem Löschereignis verknüpft sind.", + "compliance.gdpr17.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert data.erasure_requested + data.erasure_completed im Append-only-Audit-Log. Es ist ein Schritt einer Löschung nach Artikel 17: trainierte Gewichte, Quelldokumente und Kopien an anderen Orten bleiben unverändert.", "compliance.iso.title": "Aligned, nicht zertifiziert", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "Software kann nicht ISO 27001 oder SOC 2 zertifiziert werden — nur Organisationen. ForgeLM ist aligned mit den Kontrollen, nach denen ein Auditor den Deployer fragt: Es produziert die Audit-Trail-, Change-Management-, Daten-Lineage- und Supply-Chain-Nachweise auf jedem Lauf, bereit zur Aufnahme in Ihren Audit-Ordner.", @@ -4470,7 +4470,7 @@ "features.gov.heading": "Compliance- & DSGVO-Tooling", "features.gov.body": "EU-AI-Act-Artikel 9-17, DSGVO-Artikel 15 + 17, ISO-27001- / SOC-2-Type-II-Alignment-Nachweise.", "features.gov.access.title": "DSGVO-Artikel-15-Auskunft", - "features.gov.access.body": "forgelm reverse-pii durchsucht maskierte JSONL-Korpora nach forensischen Substring- oder gesalzenen Digest-Treffern und bringt jede Zeile zum Vorschein, in der eine Betroffenenkennung erscheint.", + "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern einer Betroffenenkennung und listet die Zeilen der gescannten Dateien auf, die sie enthalten.", "features.gov.annex.title": "Annex-IV-Verifier", "features.gov.annex.body": "forgelm verify-annex-iv validiert die Annex-IV-§1-9-Schemavollständigkeit und den manifest_hash des Bundles zur Manipulationserkennung. (Audit-Ketten-Integrität ist der separate Befehl forgelm verify-audit.)", "features.gov.approve.title": "Artikel-14-Staging-Gate", @@ -4498,9 +4498,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.fr, { "compliance.gdpr15.title": "Droit d'accès", - "compliance.gdpr15.body": "forgelm reverse-pii scanne les corpus JSONL masqués à la recherche de résidus en clair ou de digests SHA-256 salés d'un identifiant de personne concernée et signale chaque ligne de chaque split où cette personne apparaît. Il ne trouve que ce qui figure littéralement dans les fichiers scannés : les sources brutes et le modèle entraîné ne sont pas parcourus, et le traitement d'une demande d'accès reste votre processus.", + "compliance.gdpr15.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche d'occurrences en clair, ou de digests SHA-256 salés écrits par un pipeline de masquage externe, d'un identifiant de personne concernée et signale chaque ligne des fichiers analysés qui le contient. C'est un élément de votre réponse à l'article 15, qui doit aussi fournir une copie des données et les informations de l'article 15(1)(a) à (h) ; les sources brutes et le modèle entraîné ne sont pas parcourus.", "compliance.gdpr17.title": "Droit à l'effacement", - "compliance.gdpr17.body": "forgelm purge supprime les lignes d'une personne concernée des corpus d'entraînement et estampille les événements data.erasure_requested + data.erasure_completed dans le log d'audit append-only, de sorte que les exécutions post-suppression soient liées de manière forensique à l'événement d'effacement.", + "compliance.gdpr17.body": "forgelm purge supprime des lignes par id d'un corpus d'entraînement et consigne les événements data.erasure_requested + data.erasure_completed dans le log d'audit append-only. C'est une étape d'un effacement au titre de l'article 17 : les poids entraînés, les documents sources et les copies ailleurs ne changent pas.", "compliance.iso.title": "Aligné, non certifié", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "Le logiciel ne peut pas être certifié ISO 27001 ou SOC 2 — seules les organisations le peuvent. ForgeLM est aligné avec les contrôles qu'un auditeur demande au déployeur : il produit les preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain à chaque exécution, prêtes à être déposées dans votre classeur d'audit.", @@ -4516,7 +4516,7 @@ "features.gov.heading": "Outillage Compliance & RGPD", "features.gov.body": "Articles 9-17 de l'EU AI Act, RGPD Articles 15 + 17, preuves d'alignement ISO 27001 / SOC 2 Type II.", "features.gov.access.title": "Accès RGPD Article 15", - "features.gov.access.body": "forgelm reverse-pii scanne les corpus JSONL masqués à la recherche de correspondances de sous-chaînes forensiques ou de digests salés qui font remonter chaque ligne où apparaît un identifiant de personne concernée.", + "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de correspondances en clair ou de digests salés d'un identifiant de personne concernée et liste les lignes des fichiers analysés qui le contiennent.", "features.gov.annex.title": "Vérificateur Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valide la complétude du schéma Annex IV §1-9 et le manifest_hash du bundle pour la détection de falsification. (L'intégrité de la chaîne d'audit est la commande séparée forgelm verify-audit.)", "features.gov.approve.title": "Gate de staging Article 14", @@ -4544,9 +4544,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.es, { "compliance.gdpr15.title": "Derecho de acceso", - "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL enmascarados en busca de residuos en texto claro o digests SHA-256 salados de un identificador de interesado y reporta cada fila de cada split donde aparece ese sujeto. Solo encuentra lo que figura literalmente en los archivos que escanea: no busca en las fuentes sin procesar ni en el modelo entrenado, y la gestión de una solicitud de acceso sigue siendo su proceso.", + "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL en busca de apariciones en texto claro, o de digests SHA-256 salados escritos por un pipeline de enmascaramiento externo, de un identificador de interesado y reporta cada línea de los archivos escaneados que lo contiene. Es un elemento de su respuesta del Artículo 15, que además debe incluir una copia de los datos y la información del Artículo 15(1)(a) a (h); no se buscan las fuentes sin procesar ni el modelo entrenado.", "compliance.gdpr17.title": "Derecho de supresión", - "compliance.gdpr17.body": "forgelm purge elimina las filas de un interesado de los corpus de entrenamiento y estampa eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only, de modo que las ejecuciones posteriores a la supresión queden vinculadas forensemente al evento de borrado.", + "compliance.gdpr17.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra los eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only. Es un paso de una supresión del Artículo 17: los pesos entrenados, los documentos de origen y las copias en otros lugares no cambian.", "compliance.iso.title": "Alineado, no certificado", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "El software no puede ser certificado ISO 27001 o SOC 2 — solo las organizaciones pueden. ForgeLM está alineado con los controles que un auditor le pregunta al deployer: produce las pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro en cada ejecución, listas para incorporarse a su carpeta de auditoría.", @@ -4562,7 +4562,7 @@ "features.gov.heading": "Tooling Compliance & RGPD", "features.gov.body": "Artículos 9-17 del EU AI Act, RGPD Artículos 15 + 17, pruebas de alineamiento ISO 27001 / SOC 2 Type II.", "features.gov.access.title": "Acceso RGPD Artículo 15", - "features.gov.access.body": "forgelm reverse-pii escanea corpus JSONL enmascarados en busca de coincidencias de subcadenas forenses o digests salados que sacan a la luz cada fila en la que aparece un identificador de interesado.", + "features.gov.access.body": "forgelm reverse-pii escanea corpus JSONL en busca de coincidencias en texto claro o de digests salados de un identificador de interesado y lista las líneas de los archivos escaneados que lo contienen.", "features.gov.annex.title": "Verificador Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valida la completitud del esquema Annex IV §1-9 y el manifest_hash del bundle para la detección de manipulación. (La integridad de la cadena de auditoría es el comando separado forgelm verify-audit.)", "features.gov.approve.title": "Gate de staging del Artículo 14", @@ -4590,9 +4590,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.zh, { "compliance.gdpr15.title": "访问权", - "compliance.gdpr15.body": "forgelm reverse-pii 扫描已掩码的 JSONL 语料库,查找数据主体标识的明文残留或加盐 SHA-256 摘要,并报告该主体出现的每个 split 的每一行。它只能找到所扫描文件中字面存在的内容:不会搜索原始来源或已训练的模型,处理访问请求仍是你的流程。", + "compliance.gdpr15.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识的明文出现,或由外部掩码流水线写入的加盐 SHA-256 摘要,并报告所扫描文件中包含它的每一行。它只是第 15 条答复的输入之一,该答复还必须提供数据副本和第 15 条第 1 款(a)至(h)项信息;不会搜索原始来源或已训练的模型。", "compliance.gdpr17.title": "删除权", - "compliance.gdpr17.body": "forgelm purge 从训练语料中删除某个数据主体的行,并在 append-only 审计日志中盖上 data.erasure_requested + data.erasure_completed 事件,使删除后的运行在取证上与该删除事件相连。", + "compliance.gdpr17.body": "forgelm purge 按 id 从训练语料中删除行,并将 data.erasure_requested + data.erasure_completed 事件记录到 append-only 审计日志。它只是第 17 条删除的其中一步:已训练的权重、源文档和其他位置的副本不会改变。", "compliance.iso.title": "对齐,而非认证", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "软件无法获得 ISO 27001 或 SOC 2 认证 — 只有组织可以。ForgeLM 与审计员向 deployer 询问的控制项保持对齐:它在每次运行时产出审计追踪、变更管理、数据血缘和供应链证据,可直接放入你的审计文件夹。", @@ -4608,7 +4608,7 @@ "features.gov.heading": "合规 & GDPR 工具", "features.gov.body": "EU AI Act 第 9-17 条、GDPR 第 15 + 17 条、ISO 27001 / SOC 2 Type II 对齐证据。", "features.gov.access.title": "GDPR 第 15 条访问", - "features.gov.access.body": "forgelm reverse-pii 扫描已掩码的 JSONL 语料库,查找取证子串或加盐摘要匹配,使每一条数据主体标识出现的行浮现出来。", + "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识的明文或加盐摘要匹配,并列出所扫描文件中包含它的行。", "features.gov.annex.title": "Annex IV 验证器", "features.gov.annex.body": "forgelm verify-annex-iv 验证 Annex IV §1-9 模式完整性以及 bundle 的 manifest_hash 防篡改。(审计链完整性由独立的 forgelm verify-audit 命令负责。)", "features.gov.approve.title": "第 14 条 staging gate", diff --git a/tests/site/site_translations.test.cjs b/tests/site/site_translations.test.cjs index b1b6df55..b016355a 100644 --- a/tests/site/site_translations.test.cjs +++ b/tests/site/site_translations.test.cjs @@ -38,11 +38,13 @@ test('claims the code does not back stay out of the site copy', () => { en: [/dual-control/i, /\bauto-deletes\b/, /run fails \(exit 3\)/, /satisfying the access-request/, /Slack, Teams, Discord, or any HTTP/, /Slack \/ Teams \/ Discord/, /enforced upstream/, /GDPR-compliant TTLs/, /SHA-256 \+ size manifest/, /6 quant levels and SHA-256 manifest/, /ZeRO-2 \/ ZeRO-3 presets/, - /PII redacted at ingest/, /GGUF export at q4_k_m/, /both wired into the audit log/, /matches EU AI Act guidance/, - /rejected models in final_model\.staging/], + /PII redacted at ingest/, /GGUF export at q4_k_m/, /wired into the audit log/, /Article 17 erasure with/, /matches EU AI Act guidance/, + /rejected models in final_model\.staging/, /forensically linked/, /every row of every split/, + /surface every row/], tr: [/ikili[- ]kontrol/i, /otomatik siler/, /GDPR uyumlu TTL/, /erişim-talebi tarafını karşılar/, /Slack, Teams, Discord veya/, /Slack \/ Teams \/ Discord/, /upstream'de uygulanır/, /her ikisi de audit log'a bağlı/, - /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/], + /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/, /adli olarak bağlanır/, + /geçtiği her split/, /yüzeye çıkarır/], }; const found = []; for (const [locale, patterns] of Object.entries(retired)) { @@ -60,6 +62,7 @@ test('claims the code does not back stay out of the site copy', () => { for (const key of ['wizard.retention.audit_log.hint', 'wizard.retention.staging.hint']) { assert.match(table[key], /--check-policy/, `${locale} ${key}: a horizon is only reported, not enforced`); } + assert.doesNotMatch(table['wizard.retention.audit_log.hint'], /EU AI Act/, `${locale}: the 5-year default is not a legal period`); } }); @@ -73,6 +76,12 @@ test('the home GDPR card keeps the trained-weights caveat in every locale', () = for (const [locale, pattern] of Object.entries(caveat)) { assert.match(translations[locale]['home.cap.gdpr.body'], pattern, `${locale}: home.cap.gdpr.body`); } + const weights = { en: /weights/, tr: /ağırlıklar/, de: /Gewichte/, fr: /poids/, es: /pesos/, zh: /权重/ }; + for (const [locale, pattern] of Object.entries(weights)) { + assert.match(translations[locale]['compliance.gdpr17.body'], pattern, `${locale}: purge does not change trained weights`); + } const html = fs.readFileSync(path.join(root, 'site', 'index.html'), 'utf8'); - assert.match(html, /trained weights are not changed/, 'index.html static fallback'); + const fallback = html.match(/data-i18n-html="home\.cap\.gdpr\.body">([^<]*(?:[^<]*<\/code>[^<]*)*)<\/p>/); + assert.ok(fallback, 'index.html keeps a static fallback for the card'); + assert.equal(fallback[1], translations.en['home.cap.gdpr.body'], 'the static fallback equals the English string'); }); From 3c8ec65df557f9b2bdbb5c1461eed953c6742b29 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:00:47 +0300 Subject: [PATCH 10/25] =?UTF-8?q?fix(docs-guard):=20review=20follow-up=20?= =?UTF-8?q?=E2=80=94=20multi-document=20YAML,=20language=20handling,=20sta?= =?UTF-8?q?le-row=20messages;=20correct=20the=20chat=20and=20cost=20pages?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Guard (CR-A17a-007): every document of a multi-document snippet is checked; a file directly under the manual root is not a language directory; an empty tree, or one with no YAML blocks, can no longer pass vacuously; a stale row says which languages diverged (fix the page, not the row) instead of always saying "delete the row"; the vLLM row's reason no longer claims `forgelm deploy` emits keys it does not; the docstring states the exit codes (argparse exits 2) and the known limits (unparseable snippets and YAML-1.1 non-string keys such as `on:`). New tests pin the behaviours a mutation review found unpinned: known blocks inside mixed or exempted snippets are still walked, stale rows alone fail the run, rows match one by one rather than per page, key matching is case-sensitive. Pages: `cost_source` has three values (`user_config`, `auto_detected` for an exact name, `fuzzy_match`) and is absent when no rate resolves; without a CUDA GPU there is no detected GPU, so set `training.gpu_cost_per_hour`; quantised chat loading needs CUDA and bitsandbytes and 4-bit wins when both flags are given; top-p, top-k and the repetition penalty are fixed at 0.9 / 50 / 1.1. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/usermanuals/en/deployment/chat.md | 4 +- docs/usermanuals/en/operations/gpu-cost.md | 2 +- docs/usermanuals/tr/deployment/chat.md | 4 +- docs/usermanuals/tr/operations/gpu-cost.md | 4 +- tests/test_check_usermanual_schema_drift.py | 91 +++++++++++++++- tools/check_usermanual_schema_drift.py | 113 +++++++++++++------- 6 files changed, 169 insertions(+), 49 deletions(-) diff --git a/docs/usermanuals/en/deployment/chat.md b/docs/usermanuals/en/deployment/chat.md index 132b7246..1935ddcc 100644 --- a/docs/usermanuals/en/deployment/chat.md +++ b/docs/usermanuals/en/deployment/chat.md @@ -45,9 +45,9 @@ yolunu izleyebilirsiniz... | Sampling temperature | `--temperature` (change it later with `/temperature`) | `0.7` | | Maximum tokens per reply | `--max-new-tokens` | `512` | | Streaming output | `--no-stream` turns it off | on | -| Quantised loading | `--load-in-4bit` or `--load-in-8bit` | off | +| Quantised loading (CUDA with bitsandbytes only; ignored otherwise) | `--load-in-4bit` (NF4) or `--load-in-8bit`; 4-bit wins if both are given | off | -There is no top-p setting and no history file that persists across sessions; write a conversation out with `/save`. +Top-p, top-k and the repetition penalty are fixed (0.9 / 50 / 1.1) and cannot be set, and there is no history file that persists across sessions; write a conversation out with `/save`. ## Loading a checkpoint diff --git a/docs/usermanuals/en/operations/gpu-cost.md b/docs/usermanuals/en/operations/gpu-cost.md index 4695a74b..82c3aca5 100644 --- a/docs/usermanuals/en/operations/gpu-cost.md +++ b/docs/usermanuals/en/operations/gpu-cost.md @@ -40,7 +40,7 @@ training: gpu_cost_per_hour: 1.10 # USD per hour for the training GPU ``` -Set `training.gpu_cost_per_hour` to your real rate. When it is unset, ForgeLM falls back to its built-in price table for the detected GPU; the run's `resource_usage` records which one it used (`cost_source`: `user_config` or `auto_detected`). There is one rate per run: per-GPU-model rate tables are not supported. +Set `training.gpu_cost_per_hour` to your real rate. When it is unset, ForgeLM falls back to its built-in price table for the detected GPU; the run's `resource_usage` records which one it used (`cost_source`: `user_config`, `auto_detected` for an exact GPU-name match, or `fuzzy_match` for a partial-name match; the key is absent when no rate could be resolved). Without a CUDA GPU there is no detected GPU, so set `training.gpu_cost_per_hour` explicitly. There is one rate per run: per-GPU-model rate tables are not supported. ## Output diff --git a/docs/usermanuals/tr/deployment/chat.md b/docs/usermanuals/tr/deployment/chat.md index 4054ae7d..9d74a5d4 100644 --- a/docs/usermanuals/tr/deployment/chat.md +++ b/docs/usermanuals/tr/deployment/chat.md @@ -45,9 +45,9 @@ Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... | Sampling temperature | `--temperature` (sonra `/temperature` ile değiştirin) | `0.7` | | Yanıt başına maksimum token | `--max-new-tokens` | `512` | | Streaming çıktı | `--no-stream` kapatır | açık | -| Quantize yükleme | `--load-in-4bit` veya `--load-in-8bit` | kapalı | +| Quantize yükleme (yalnızca CUDA + bitsandbytes; aksi halde yok sayılır) | `--load-in-4bit` (NF4) veya `--load-in-8bit`; ikisi de verilirse 4-bit kazanır | kapalı | -top-p ayarı ve oturumlar arası kalıcı bir history dosyası yoktur; bir konuşmayı `/save` ile yazın. +top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz; oturumlar arası kalıcı bir history dosyası da yoktur; bir konuşmayı `/save` ile yazın. ## Checkpoint yükleme diff --git a/docs/usermanuals/tr/operations/gpu-cost.md b/docs/usermanuals/tr/operations/gpu-cost.md index a1b33482..9574c44c 100644 --- a/docs/usermanuals/tr/operations/gpu-cost.md +++ b/docs/usermanuals/tr/operations/gpu-cost.md @@ -40,7 +40,7 @@ training: gpu_cost_per_hour: 1.10 # eğitim GPU'su için saatlik USD ``` -`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config` veya `auto_detected`). Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. +`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config`; tam GPU adı eşleşmesi için `auto_detected`; kısmi ad eşleşmesi için `fuzzy_match`; tarife çözülemediyse anahtar yoktur). CUDA GPU yoksa tespit edilen bir GPU da yoktur, bu yüzden `training.gpu_cost_per_hour`'u açıkça ayarlayın. Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. ## Çıktı @@ -92,7 +92,7 @@ Kalibrasyon yaklaşımı genelde gerçeğin %20 içindedir. Kontrolden çıkabilecek koşular için planlanan `cost_tracking` bloğu eşik tabanlı alarmı (alarm eşiği aşılınca webhook tetiklenir) ve durdurmayı -(durdurma eşiğinde eğitim durur) destekleyecekti. Böyle bir ayar bugün +(durdurma eşiğinde eğitim durur) destekleyecek. Böyle bir ayar bugün `forgelm/config.py`'de yoktur. İmplemente edildiğinde, alarm konfigüre webhook'u fırlatır (bkz. [Webhook'lar](#/operations/webhooks)) — yanlış konfigüre koşunun gece bütçeyi haftalık bitirmesini CI'da yakalamak için faydalı. O zamana kadar maliyeti audit log + zamanlayıcınızdaki bütçe-tarafı koruyucu ile manuel izleyin. diff --git a/tests/test_check_usermanual_schema_drift.py b/tests/test_check_usermanual_schema_drift.py index bc54a7f8..5de0620a 100644 --- a/tests/test_check_usermanual_schema_drift.py +++ b/tests/test_check_usermanual_schema_drift.py @@ -302,16 +302,17 @@ def test_a_row_must_match_the_exact_key_set(self, tool, synthetic_schema, tmp_pa self._write(tmp_path, lang, "operations/cicd.md", "jobs: {}\nchat: {}\n") result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) assert [sorted(u.keys) for u in result.unknown_blocks] == [["chat", "jobs"], ["chat", "jobs"]] - assert [row for _, row, _ in result.stale_rows] == [("operations/cicd.md", frozenset({"jobs"}))] + assert [(row.page, row.keys) for row in result.stale_rows] == [("operations/cicd.md", frozenset({"jobs"}))] def test_a_row_that_matches_no_snippet_is_stale(self, tool, synthetic_schema, tmp_path): rows = {("operations/gpu-cost.md", frozenset({"gpu_profiles"})): "tracked elsewhere"} for lang in ("en", "tr"): self._write(tmp_path, lang, "operations/gpu-cost.md", "training:\n learning_rate: 0.1\n") result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, rows) - assert [(name, row) for name, row, _ in result.stale_rows] == [ - ("_KNOWN_OPEN", ("operations/gpu-cost.md", frozenset({"gpu_profiles"}))) + assert [(row.table, row.page, row.keys) for row in result.stale_rows] == [ + ("_KNOWN_OPEN", "operations/gpu-cost.md", frozenset({"gpu_profiles"})) ] + assert not result.clean def test_a_row_matched_in_only_one_language_is_stale(self, tool, synthetic_schema, tmp_path): rows = {("operations/gpu-cost.md", frozenset({"gpu_profiles"})): "tracked elsewhere"} @@ -319,6 +320,90 @@ def test_a_row_matched_in_only_one_language_is_stale(self, tool, synthetic_schem self._write(tmp_path, "tr", "operations/gpu-cost.md", "training:\n learning_rate: 0.1\n") result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, rows) assert len(result.stale_rows) == 1 + stale = result.stale_rows[0] + assert (stale.matched_in, stale.missing_in) == (("en",), ("tr",)) + + def test_known_blocks_inside_mixed_or_exempted_snippets_are_still_walked(self, tool, synthetic_schema, tmp_path): + rows = {("operations/cicd.md", frozenset({"chat"})): "tracked elsewhere"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "operations/cicd.md", "training:\n bogus: 1\nchat: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert sorted({d.key_path for d in result.drifts}) == ["training.bogus"] + assert result.unknown_blocks == [] and result.stale_rows == [] + # Without the row, the same snippet reports BOTH the nested drift and the unknown block. + bare = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, {}) + assert len(bare.drifts) == 2 and len(bare.unknown_blocks) == 2 + + def test_each_row_is_matched_on_its_own_not_per_page(self, tool, synthetic_schema, tmp_path): + rows = { + ("ops/a.md", frozenset({"jobs"})): "GitHub Actions", + ("ops/a.md", frozenset({"concurrency"})): "GitHub Actions concurrency", + } + for lang in ("en", "tr"): + self._write(tmp_path, lang, "ops/a.md", "jobs: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert [(row.page, row.keys) for row in result.stale_rows] == [("ops/a.md", frozenset({"concurrency"}))] + + def test_key_matching_is_case_sensitive(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + assert tool.unknown_top_level_keys(self._snippet(tool, "Training:\n bogus: 1\n"), top) == frozenset( + {"Training"} + ) + + def test_every_document_of_a_multi_document_snippet_is_checked(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + body = "chat: {}\n---\noutput: {}\n---\ntraining:\n bogus: 1\n" + assert tool.unknown_top_level_keys(self._snippet(tool, body), top) == frozenset({"chat", "output"}) + assert [d.key_path for d in tool.check_snippet(self._snippet(tool, body), synthetic_schema, top)] == [ + "training.bogus" + ] + + def test_an_empty_tree_cannot_pass_vacuously(self, tool, synthetic_schema, tmp_path, monkeypatch): + rows = {("ops/a.md", frozenset({"jobs"})): "GitHub Actions"} + empty = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert not empty.clean and empty.snippet_count == 0 + config_path = tmp_path / "config.py" + config_path.write_text(_SYNTHETIC_CONFIG, encoding="utf-8") + root = tmp_path / "usermanuals" + root.mkdir() + monkeypatch.setattr(tool, "CONFIG_PATH", config_path) + monkeypatch.setattr(tool, "USERMANUAL_ROOT", root) + assert tool.main(["--strict", "--quiet"]) == 1 + + def test_a_stray_top_level_file_is_not_a_language_directory(self, tool, synthetic_schema, tmp_path): + rows = {("ops/a.md", frozenset({"jobs"})): "GitHub Actions"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "ops/a.md", "jobs: {}\n") + (tmp_path / "README.md").write_text("```yaml\njobs: {}\n```\n", encoding="utf-8") + assert tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}).clean + + def test_a_partial_language_directory_names_the_missing_language(self, tool, synthetic_schema, tmp_path, capsys): + rows = {("ops/a.md", frozenset({"jobs"})): "GitHub Actions"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "ops/a.md", "jobs: {}\n") + self._write(tmp_path, "de", "ops/b.md", "training:\n learning_rate: 0.1\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert [(row.missing_in, row.matched_in) for row in result.stale_rows] == [(("de",), ("en", "tr"))] + tool._report(result) + assert "unmatched in de but matched in en, tr" in capsys.readouterr().out + + def test_main_fails_strict_when_only_a_stale_row_remains( + self, tool, synthetic_schema, tmp_path, monkeypatch, capsys + ): + config_path = tmp_path / "config.py" + config_path.write_text(_SYNTHETIC_CONFIG, encoding="utf-8") + root = tmp_path / "usermanuals" + for lang in ("en", "tr"): + self._write(root, lang, "ops/a.md", "training:\n learning_rate: 0.1\n") + monkeypatch.setattr(tool, "CONFIG_PATH", config_path) + monkeypatch.setattr(tool, "USERMANUAL_ROOT", root) + monkeypatch.setattr(tool, "REPO_ROOT", tmp_path) + monkeypatch.setattr(tool, "_NON_FORGELM_YAML", {}) + monkeypatch.setattr(tool, "_KNOWN_OPEN", {("ops/a.md", frozenset({"gone"})): "was fixed"}) + assert tool.main(["--strict"]) == 1 + out = capsys.readouterr().out + assert "_KNOWN_OPEN: ops/a.md gone" in out and "delete the row" in out + assert tool.main([]) == 0 def test_main_strict_exits_one_on_an_unlisted_block_and_zero_in_advisory_mode( self, tool, synthetic_schema, tmp_path, monkeypatch, capsys diff --git a/tools/check_usermanual_schema_drift.py b/tools/check_usermanual_schema_drift.py index bdac85bd..eeded2fc 100644 --- a/tools/check_usermanual_schema_drift.py +++ b/tools/check_usermanual_schema_drift.py @@ -77,7 +77,13 @@ - ``0`` — every checked key path resolves against the schema. - ``1`` — at least one fabricated key path, unrecognised top-level block - or stale exemption row found (strict mode), or invalid arguments. + or stale exemption row found (strict mode), or no YAML block found at all. + (Invalid command-line arguments exit ``2`` through ``argparse``.) + +Known limits: a snippet ``yaml.safe_load_all`` cannot parse is skipped +(``check_yaml_snippets.py`` reports parse errors on full configs), and a +top-level key YAML 1.1 reads as a non-string (``on:``, ``yes:``, ``1:``) is +not checked — GitHub Actions' ``on:`` is exactly such a key. CI wiring: this guard runs in ``.github/workflows/ci.yml`` with ``--strict`` — any fabricated key path fails the build. The @@ -152,7 +158,7 @@ "trust_remote_code", } ), - ): "vLLM server config that `forgelm deploy --target vllm` generates", + ): "vLLM engine config sample (tokenizer_mode / served_model_name / chat_template are not emitted by `forgelm deploy`: #435)", ("operations/cicd.md", frozenset({"jobs", "name"})): "GitHub Actions workflow", ("operations/cicd.md", frozenset({"audit", "deploy", "stages", "train"})): "GitLab CI pipeline", ("operations/cicd.md", frozenset({"concurrency"})): "GitHub Actions concurrency block", @@ -201,13 +207,25 @@ class UnknownBlock: keys: FrozenSet[str] +@dataclass(frozen=True) +class StaleRow: + """An exemption row that does not match a snippet in every language directory.""" + + table: str + page: str + keys: FrozenSet[str] + reason: str + matched_in: Tuple[str, ...] + missing_in: Tuple[str, ...] + + @dataclass(frozen=True) class ScanResult: """Everything one pass over a usermanual tree found.""" drifts: List[KeyDrift] unknown_blocks: List[UnknownBlock] - stale_rows: List[Tuple[str, _PageKeys, str]] # (table name, row key, reason) + stale_rows: List[StaleRow] snippet_count: int @property @@ -367,28 +385,26 @@ def _walk_value( return drift -def _parse_fragment(snippet: "Snippet") -> Optional[dict]: - """Return the parsed mapping of a fragment this guard checks, else ``None``. +def _parse_fragments(snippet: "Snippet") -> List[dict]: + """Return the fragment mappings of ``snippet`` that this guard checks. - ``None`` covers unparseable YAML (``check_yaml_snippets.py`` reports parse - errors on full configs), non-mapping documents, and full configs that - ``check_yaml_snippets.py`` already validates with a real ``ForgeConfig(**data)``. + A snippet may hold several YAML documents (``---``); each mapping is + returned. Unparseable YAML, non-mapping documents and full configs that + ``check_yaml_snippets.py`` already validates with a real + ``ForgeConfig(**data)`` are left out. """ try: - parsed = yaml.safe_load(snippet.body) + documents = list(yaml.safe_load_all(snippet.body)) except yaml.YAMLError: - return None - if not isinstance(parsed, dict) or _FULL_CONFIG_KEYS.issubset(parsed.keys()): - return None - return parsed + return [] + return [doc for doc in documents if isinstance(doc, dict) and not _FULL_CONFIG_KEYS.issubset(doc.keys())] def unknown_top_level_keys(snippet: "Snippet", top_level: Dict[str, FieldType]) -> FrozenSet[str]: """Return the string top-level keys of ``snippet`` that are not ForgeConfig blocks.""" - parsed = _parse_fragment(snippet) - if parsed is None: - return frozenset() - return frozenset(key for key in parsed if isinstance(key, str) and key not in top_level) + return frozenset( + key for doc in _parse_fragments(snippet) for key in doc if isinstance(key, str) and key not in top_level + ) def check_snippet( @@ -399,18 +415,16 @@ def check_snippet( Only the keys *inside* recognised ForgeConfig blocks are walked here; an unrecognised top-level block is :func:`unknown_top_level_keys`'s concern. """ - parsed = _parse_fragment(snippet) - if parsed is None: - return [] drifts: List[KeyDrift] = [] - for key, sub_value in parsed.items(): - if not isinstance(key, str): - continue - field_type = top_level.get(key) - if field_type is None or field_type.nested_class is None: - continue # unrecognised or opaque top-level block — see unknown_top_level_keys - for key_path in _walk_value(sub_value, field_type.nested_class, schema, key): - drifts.append(KeyDrift(path=snippet.path, line=snippet.line_start, key_path=key_path)) + for doc in _parse_fragments(snippet): + for key, sub_value in doc.items(): + if not isinstance(key, str): + continue + field_type = top_level.get(key) + if field_type is None or field_type.nested_class is None: + continue # unrecognised or opaque top-level block — see unknown_top_level_keys + for key_path in _walk_value(sub_value, field_type.nested_class, schema, key): + drifts.append(KeyDrift(path=snippet.path, line=snippet.line_start, key_path=key_path)) return drifts @@ -419,9 +433,14 @@ def walk_usermanuals(root: Path) -> List[Path]: return [p for p in sorted(root.rglob("*.md")) if p.is_file()] -def _language_and_page(root: Path, md_path: Path) -> Tuple[str, str]: - """Split ``//
/.md`` into ``(lang, "
/.md")``.""" +def _language_and_page(root: Path, md_path: Path) -> Optional[Tuple[str, str]]: + """Split ``//
/.md`` into ``(lang, "
/.md")``. + + A file directly under ``root`` belongs to no language directory and returns ``None``. + """ parts = md_path.relative_to(root).parts + if len(parts) < 2: + return None return parts[0], "/".join(parts[1:]) @@ -439,7 +458,10 @@ def scan( languages: Set[str] = set() snippet_count = 0 for md_path in walk_usermanuals(root): - language, page = _language_and_page(root, md_path) + located = _language_and_page(root, md_path) + if located is None: + continue + language, page = located languages.add(language) for snippet in extract_yaml_snippets(md_path): snippet_count += 1 @@ -451,11 +473,13 @@ def scan( matched.add((language, page, keys)) else: unknown.append(UnknownBlock(path=snippet.path, line=snippet.line_start, keys=keys)) - stale: List[Tuple[str, _PageKeys, str]] = [] + stale: List[StaleRow] = [] for table_name, table in (("_NON_FORGELM_YAML", non_forgelm), ("_KNOWN_OPEN", known_open)): for (page, keys), reason in table.items(): - if any((language, page, keys) not in matched for language in sorted(languages)): - stale.append((table_name, (page, keys), reason)) + present = tuple(lang for lang in sorted(languages) if (lang, page, keys) in matched) + missing = tuple(lang for lang in sorted(languages) if (lang, page, keys) not in matched) + if missing or not languages: + stale.append(StaleRow(table_name, page, keys, reason, present, missing)) return ScanResult(drifts=drifts, unknown_blocks=unknown, stale_rows=stale, snippet_count=snippet_count) @@ -497,6 +521,9 @@ def main(argv: Optional[List[str]] = None) -> int: result = scan(USERMANUAL_ROOT, schema, top_level, _NON_FORGELM_YAML, _KNOWN_OPEN) + if result.snippet_count == 0: + print(f"check_usermanual_schema_drift: no yaml blocks found under {USERMANUAL_ROOT}.", file=sys.stderr) + return 1 if not result.clean: _report(result) return 1 if args.strict else 0 @@ -505,7 +532,7 @@ def main(argv: Optional[List[str]] = None) -> int: print( f"OK: {result.snippet_count} yaml block(s) under docs/usermanuals/ checked; " f"every recognised ForgeConfig key path resolves against the schema and every " - f"other top-level block is another tool's YAML." + f"other top-level block is another tool's YAML or a listed known-open block." ) return 0 @@ -525,12 +552,20 @@ def _report(result: ScanResult) -> None: rel = u.path.relative_to(REPO_ROOT) if u.path.is_absolute() else u.path print( f" {rel}:{u.line} {', '.join(sorted(u.keys))} (not a ForgeConfig top-level block: correct the " - f"example, or list it in _NON_FORGELM_YAML with a reason if it is another tool's YAML)" + f"example, or list it in _NON_FORGELM_YAML with a reason if it is another tool's YAML, or in " + f"_KNOWN_OPEN if a tracked issue owns the fix)" ) if result.stale_rows: - print(f"FAIL: {len(result.stale_rows)} exemption row(s) match no YAML block in every language.") - for table_name, (page, keys), reason in result.stale_rows: - print(f" {table_name}: {page} {', '.join(sorted(keys))} ({reason}) — delete the row") + print(f"FAIL: {len(result.stale_rows)} exemption row(s) do not match a YAML block in every language.") + for row in result.stale_rows: + where = f"{row.table}: {row.page} {', '.join(sorted(row.keys))} ({row.reason})" + if row.matched_in: + print( + f" {where} — unmatched in {', '.join(row.missing_in)} but matched in " + f"{', '.join(row.matched_in)}: the language copies diverged; fix the page, not the row" + ) + else: + print(f" {where} — matches no snippet in any language: delete the row") if __name__ == "__main__": From 2f33b0506585a0ae9b8c9116c209cf94e0e672a2 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:07:38 +0300 Subject: [PATCH 11/25] =?UTF-8?q?docs(site):=20language=20review=20round?= =?UTF-8?q?=202=20=E2=80=94=20the=20erasure=20card,=20"overdue",=20article?= =?UTF-8?q?=20references=20and=20Turkish=20terms?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review round over the six-language site strings and the Turkish pages: - `features.gov.purge.body` (the sibling of the access card, still saying purge "drops a data-subject's rows") now says purge deletes rows by id and is one step of an Article 17 erasure that leaves trained weights unchanged, in all six languages plus the HTML fallback; the home card says reverse-pii finds the lines that contain an identifier you supply. - French, Spanish and Chinese no longer "respond to an article": they respond to an access request (Article 15); German names the DSGVO and the identifier's owner; Chinese uses 已超期 (overdue) instead of 已过期 (expired); Turkish uses "süresi geçmiş" instead of "geciken" (delayed) in the wizard hints and the reference/guide rows, "düz metin olarak geçtiği yerler" for occurrences, and spells out "Madde 17 kapsamındaki silme". - Art. 87 reads "set the specific conditions for processing" (its wording) on every page, the recital-14 reference is attached to the company IBAN and switchboard number only, and the Turkish sentence names what is not personal data. - The cost page no longer links "the full cost_tracking block" (there is none) and the dangling colon in the multi-GPU sentence is gone; the chat page says "Quantized" and the Turkish top-p sentence is two sentences. - Tests: the old purge-card wording is retired in English and Turkish, and every locale keeps the trained-weights caveat on both the compliance and the features erasure cards. German, French, Spanish and Chinese still have no native review. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/data_audit-tr.md | 4 +- docs/guides/data_audit.md | 3 +- docs/guides/gdpr_erasure-tr.md | 2 +- docs/reference/configuration-tr.md | 8 +-- docs/usermanuals/en/data/pii-ml.md | 2 +- docs/usermanuals/en/deployment/chat.md | 2 +- docs/usermanuals/en/operations/gpu-cost.md | 4 +- docs/usermanuals/tr/deployment/chat.md | 2 +- docs/usermanuals/tr/operations/gpu-cost.md | 6 +- .../usermanuals/tr/reference/configuration.md | 2 +- site/features.html | 2 +- site/index.html | 2 +- site/js/translations.js | 72 +++++++++---------- tests/site/site_translations.test.cjs | 9 ++- 14 files changed, 61 insertions(+), 59 deletions(-) diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 4095d689..230beeff 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -416,8 +416,8 @@ install ile uyumlu kalır. Varsayılan regex detector, yaygın yapılandırılmış identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. Her biri, kimliği belirli veya belirlenebilir bir gerçek kişiye ilişkin olduğunda GDPR Madde 4(1) -kapsamında kişisel veridir; bir şirketin IBAN'ı, santral numarası veya test -kart numarası değildir (Gerekçe 14). +kapsamında kişisel veridir; bir şirketin IBAN'ı veya santral numarası (Gerekçe 14) ile bir test +kart numarası ise kişisel veri değildir. Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier kategorilerini ekleyen opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index 8ea429d4..b8916dfb 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -409,8 +409,7 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. The default regex detector covers the common structured identifiers (email, phone, IBAN, credit card, national IDs). Each is personal data under GDPR Art. 4(1) when it relates to an identified or identifiable natural -person; a company's IBAN or switchboard number, or a test card number, is not -(recital 14). +person; a company's IBAN or switchboard number (recital 14), or a test card number, is not. Phase 12.5 adds an opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adapter that layers ML-NER on top — adding the unstructured-identifier categories regex inherently misses: `person`, `organization`, `location`. diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 5c9560cb..bad0021a 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -75,7 +75,7 @@ Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical ```yaml retention: audit_log_retention_days: 1825 # 5 yıl (ForgeLM varsayılanı, yasal süre değildir) - staging_ttl_days: 7 # purge --check-policy'nin staging dizinini geciken saydığı yaş + staging_ttl_days: 7 # purge --check-policy'nin staging dizinini süresi geçmiş saydığı yaş ephemeral_artefact_retention_days: 90 # üç-aylık review cadence'i raw_documents_retention_days: 90 # data audit yeniden çalıştırmadan önce ingestion penceresi enforce: log_only # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess) diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index d27b4e80..635a8e51 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -282,10 +282,10 @@ ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "geciken" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (uygulanabildiği ölçüde bkz. AI Act Madde 18(1): dokümantasyon 10 yıl; Madde 19(1) ve 26(6): sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | -| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve legacy `final_model.staging/`'i) "geciken" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | -| `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "geciken" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | -| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "geciken" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | +| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "süresi geçmiş" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (uygulanabildiği ölçüde bkz. AI Act Madde 18(1): dokümantasyon 10 yıl; Madde 19(1) ve 26(6): sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | +| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve eski `final_model.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | +| `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | +| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Hiçbir komut onu kullanmaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa eğitim ve `--dry-run` uyarır. | > **Kaldırıldı:** `evaluation.staging_ttl_days` (v0.5.5 itibarıyla kullanımdan diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index 99d18c43..0a954fcd 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -5,7 +5,7 @@ description: Optional Microsoft Presidio adapter that adds person / organization # ML-NER PII Detection — `--pii-ml` -The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the common *structured* identifiers (email, phone, IBAN, credit card, national IDs). They are personal data under GDPR Article 4(1) whenever they relate to an identifiable natural person; Article 87 additionally lets Member States restrict how national identification numbers are processed. Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. +The default [PII Masking](#/data/pii-masking) layer is regex-anchored and covers the common *structured* identifiers (email, phone, IBAN, credit card, national IDs). They are personal data under GDPR Article 4(1) whenever they relate to an identified or identifiable natural person; Article 87 additionally lets Member States set the specific conditions for processing national identification numbers. Presidio NER adds the *unstructured* identifiers regex inherently misses: person names, organization names, geographic locations. This is an **opt-in** layer. The default audit behaviour is unchanged when `--pii-ml` is not passed (`forgelm ingest` has no `--pii-ml` flag). diff --git a/docs/usermanuals/en/deployment/chat.md b/docs/usermanuals/en/deployment/chat.md index 1935ddcc..b223be2e 100644 --- a/docs/usermanuals/en/deployment/chat.md +++ b/docs/usermanuals/en/deployment/chat.md @@ -45,7 +45,7 @@ yolunu izleyebilirsiniz... | Sampling temperature | `--temperature` (change it later with `/temperature`) | `0.7` | | Maximum tokens per reply | `--max-new-tokens` | `512` | | Streaming output | `--no-stream` turns it off | on | -| Quantised loading (CUDA with bitsandbytes only; ignored otherwise) | `--load-in-4bit` (NF4) or `--load-in-8bit`; 4-bit wins if both are given | off | +| Quantized loading (CUDA with bitsandbytes only; ignored otherwise) | `--load-in-4bit` (NF4) or `--load-in-8bit`; 4-bit wins if both are given | off | Top-p, top-k and the repetition penalty are fixed (0.9 / 50 / 1.1) and cannot be set, and there is no history file that persists across sessions; write a conversation out with `/save`. diff --git a/docs/usermanuals/en/operations/gpu-cost.md b/docs/usermanuals/en/operations/gpu-cost.md index 82c3aca5..2e06488a 100644 --- a/docs/usermanuals/en/operations/gpu-cost.md +++ b/docs/usermanuals/en/operations/gpu-cost.md @@ -84,7 +84,7 @@ The calibration approach is typically within 20% of actual. ## Multi-GPU and distributed -For multi-GPU training, ForgeLM multiplies the single `training.gpu_cost_per_hour` rate by the GPU count (`gpu_hours` is the training time times the GPU count): +For multi-GPU training, ForgeLM multiplies the single `training.gpu_cost_per_hour` rate by the GPU count (`gpu_hours` is the training time times the GPU count). A 4×A100 run for 2 hours = 4 × 2 × $1.10 = $8.80, regardless of whether you use ZeRO or FSDP. @@ -131,4 +131,4 @@ Drop this in your project root as `gpu_profiles.local.yaml`; ForgeLM auto-merges - [VRAM Fit-Check](#/operations/vram-fit-check) — runs alongside cost estimation. - [CI/CD Pipelines](#/operations/cicd) — cost alerts in CI. -- [Configuration Reference](#/reference/configuration) — full `cost_tracking` block. +- [Configuration Reference](#/reference/configuration) — the `training.gpu_cost_per_hour` field. diff --git a/docs/usermanuals/tr/deployment/chat.md b/docs/usermanuals/tr/deployment/chat.md index 9d74a5d4..f2d0941d 100644 --- a/docs/usermanuals/tr/deployment/chat.md +++ b/docs/usermanuals/tr/deployment/chat.md @@ -47,7 +47,7 @@ Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... | Streaming çıktı | `--no-stream` kapatır | açık | | Quantize yükleme (yalnızca CUDA + bitsandbytes; aksi halde yok sayılır) | `--load-in-4bit` (NF4) veya `--load-in-8bit`; ikisi de verilirse 4-bit kazanır | kapalı | -top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz; oturumlar arası kalıcı bir history dosyası da yoktur; bir konuşmayı `/save` ile yazın. +Top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz. Oturumlar arası kalıcı bir history dosyası da yoktur; bir konuşmayı `/save` ile kaydedin. ## Checkpoint yükleme diff --git a/docs/usermanuals/tr/operations/gpu-cost.md b/docs/usermanuals/tr/operations/gpu-cost.md index 9574c44c..90042fa4 100644 --- a/docs/usermanuals/tr/operations/gpu-cost.md +++ b/docs/usermanuals/tr/operations/gpu-cost.md @@ -40,7 +40,7 @@ training: gpu_cost_per_hour: 1.10 # eğitim GPU'su için saatlik USD ``` -`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config`; tam GPU adı eşleşmesi için `auto_detected`; kısmi ad eşleşmesi için `fuzzy_match`; tarife çözülemediyse anahtar yoktur). CUDA GPU yoksa tespit edilen bir GPU da yoktur, bu yüzden `training.gpu_cost_per_hour`'u açıkça ayarlayın. Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. +`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config`; tam GPU adı eşleşmesi için `auto_detected`; kısmi ad eşleşmesi için `fuzzy_match`; tarife belirlenemediyse anahtar bulunmaz). CUDA GPU yoksa tespit edilen bir GPU da yoktur, bu yüzden `training.gpu_cost_per_hour`'u açıkça ayarlayın. Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. ## Çıktı @@ -84,7 +84,7 @@ Kalibrasyon yaklaşımı genelde gerçeğin %20 içindedir. ## Çoklu-GPU ve dağıtık -Çoklu-GPU eğitiminde ForgeLM tek `training.gpu_cost_per_hour` tarifesini GPU sayısıyla çarpar (`gpu_hours`, eğitim süresi çarpı GPU sayısıdır): +Çoklu-GPU eğitiminde ForgeLM tek `training.gpu_cost_per_hour` tarifesini GPU sayısıyla çarpar (`gpu_hours`, eğitim süresi çarpı GPU sayısıdır). 4×A100 koşusu 2 saat = 4 × 2 × $1.10 = $8.80; ZeRO veya FSDP kullanmak fark etmez. @@ -131,4 +131,4 @@ Bunu proje kökünüze `gpu_profiles.local.yaml` olarak bırakın; ForgeLM otoma - [VRAM Fit-Check](#/operations/vram-fit-check) — maliyet tahmini ile birlikte koşar. - [CI/CD Hatları](#/operations/cicd) — CI'da maliyet alarmları. -- [Konfigürasyon Referansı](#/reference/configuration) — tam `cost_tracking` bloğu. +- [Konfigürasyon Referansı](#/reference/configuration) — `training.gpu_cost_per_hour` alanı. diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 794e8cdd..5fad453f 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -368,7 +368,7 @@ kaynağı kaydedilmemiştir. ```yaml retention: audit_log_retention_days: 1825 # 5 yıl: ForgeLM varsayılanı (proje tercihi; GDPR Madde 5(1)(e) sabit süre koymaz) - staging_ttl_days: 7 # staging dizininin purge --check-policy tarafından geciken sayıldığı yaş + staging_ttl_days: 7 # staging dizininin purge --check-policy tarafından süresi geçmiş sayıldığı yaş ephemeral_artefact_retention_days: 90 # uyumluluk paketleri, denetim raporları raw_documents_retention_days: 90 # ingest edilen PDF/DOCX/EPUB/TXT/Markdown enforce: "log_only" # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only | warn_on_excess | block_on_excess) diff --git a/site/features.html b/site/features.html index 4e26d200..6a42630a 100644 --- a/site/features.html +++ b/site/features.html @@ -415,7 +415,7 @@

Article 14 staging gate

GDPR Article 17 erasure

-

forgelm purge drops a data-subject's rows from training corpora and stamps the deletion in the append-only audit log.

+

forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log; it is one step of an Article 17 erasure (trained weights are not changed).

diff --git a/site/index.html b/site/index.html index 7bb24214..157705b2 100644 --- a/site/index.html +++ b/site/index.html @@ -536,7 +536,7 @@

ISO 27001 / SOC 2 alignment

GDPR rights tooling

-

Tooling for GDPR data-subject requests: forgelm reverse-pii finds the rows that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

+

Tooling for GDPR data-subject requests: forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

forgelm purge forgelm reverse-pii diff --git a/site/js/translations.js b/site/js/translations.js index c507181f..dc29d416 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -390,7 +390,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 alignment", "home.cap.iso.body": "Audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor asks for. Software is aligned, not certified.", "home.cap.gdpr.title": "GDPR rights tooling", - "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii finds the rows that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", + "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", "home.cap.see_all": "See the full feature matrix", "home.yaml.eyebrow": "A real config — not a screenshot", "home.yaml.title": "One YAML, end-to-end run", @@ -496,7 +496,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 hizalaması", "home.cap.iso.body": "Deployer denetçisinin sorduğu audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtları. Yazılım sertifikalı değil, hizalanmıştır.", "home.cap.gdpr.title": "GDPR hakları araçları", - "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 yanıtının bir girdisi); forgelm purge satırları eğitim corpus'larından siler (Madde 17 silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", + "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 yanıtının bir girdisi); forgelm purge satırları eğitim corpus'larından siler (Madde 17 kapsamındaki silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", "home.cap.see_all": "Tüm özellik matrisini gör", "home.yaml.eyebrow": "Gerçek bir konfigürasyon — ekran görüntüsü değil", "home.yaml.title": "Bir YAML, uçtan uca koşu", @@ -708,7 +708,7 @@ "home.cap.iso.title": "Alignement ISO 27001 / SOC 2", "home.cap.iso.body": "Piste d'audit, gestion du changement, lignée des données et preuves de chaîne logistique demandées par l'auditeur du déployeur. Le logiciel est aligné, pas certifié.", "home.cap.gdpr.title": "Outils de droits RGPD", - "home.cap.gdpr.body": "Outils pour les demandes de personnes concernées au titre du RGPD : forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, Article 15) ; forgelm purge supprime des lignes des corpus d'entraînement (une étape de l'effacement, Article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", + "home.cap.gdpr.body": "Outils pour les demandes des personnes concernées au titre du RGPD : forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, article 15) ; forgelm purge supprime des lignes des corpus d'entraînement (une étape de l'effacement, article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", "home.cap.see_all": "Voir la matrice complète des fonctionnalités", "home.yaml.eyebrow": "Une vraie configuration — pas une capture d'écran", "home.yaml.title": "Un YAML, exécution de bout en bout", @@ -920,7 +920,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 对齐", "home.cap.iso.body": "部署方审计员要求的审计轨迹、变更管理、数据沿袭和供应链证据。软件是对齐的,不是认证的。", "home.cap.gdpr.title": "GDPR 权利工具", - "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 查找包含你所提供标识符的行(第 15 条答复的输入之一),forgelm purge 从训练语料中删除行(第 17 条删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", + "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 查找包含你所提供标识符的行(对第 15 条访问请求答复的输入之一),forgelm purge 从训练语料中删除行(第 17 条所涉删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", "home.cap.see_all": "查看完整功能矩阵", "home.yaml.eyebrow": "真实配置 — 不是截图", "home.yaml.title": "一个 YAML,端到端运行", @@ -1033,7 +1033,7 @@ "features.gov.approve.title": "Article 14 staging gate", "features.gov.approve.body": "forgelm approve / forgelm reject + forgelm approvals implement the human-oversight staging directory with audit-stamped sign-off.", "features.gov.purge.title": "GDPR Article 17 erasure", - "features.gov.purge.body": "forgelm purge drops a data-subject's rows from training corpora and stamps the deletion in the append-only audit log.", + "features.gov.purge.body": "forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log; it is one step of an Article 17 erasure (trained weights are not changed).", "features.gov.access.title": "GDPR Article 15 access", "features.gov.access.body": "forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.", "features.gov.annex.title": "Annex IV verifier", @@ -1138,9 +1138,9 @@ "features.gov.approve.title": "Madde 14 staging kapısı", "features.gov.approve.body": "forgelm approve / forgelm reject + forgelm approvals, audit-damgalı onayla insan-gözetimli staging dizinini uygular.", "features.gov.purge.title": "GDPR Madde 17 silme", - "features.gov.purge.body": "forgelm purge bir veri sahibinin satırlarını eğitim corpus'undan düşürür ve silmeyi append-only audit log'a damgalar.", + "features.gov.purge.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve silmeyi append-only audit log'a kaydeder; Madde 17 kapsamındaki silmenin bir adımıdır (eğitilmiş ağırlıklar değişmez).", "features.gov.access.title": "GDPR Madde 15 erişim", - "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz-metin ya da tuzlanmış-digest eşleşmelerine bakar ve taranan dosyalarda onu içeren satırları listeler.", + "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da tuzlanmış digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler.", "features.gov.annex.title": "Annex IV doğrulayıcı", "features.gov.annex.body": "forgelm verify-annex-iv Annex IV §1-9 şema tamlığını ve bundle'ın manifest_hash'ini tampering tespiti için doğrular. (Audit-chain bütünlüğü ayrı bir forgelm verify-audit komutudur.)", "features.gov.gguf.title": "GGUF bütünlük doğrulayıcı", @@ -2209,15 +2209,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API key env değişkeni", "wizard.synthetic.api_key_env.hint": "Teacher API anahtarını taşıyan ortam değişkeni adı. Boş = yerel teacher modeli.", "wizard.retention.title": "Saklama ufukları (GDPR Madde 5+17)", - "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları geciken olarak raporladığı yaşlar; hiçbir şey silinmez.", + "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları süresi geçmiş olarak raporladığı yaşlar; hiçbir şey silinmez.", "wizard.retention.audit_log.label": "Audit log saklama (gün)", - "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra geciken olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", + "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra süresi geçmiş olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", "wizard.retention.staging.label": "Staging ufku (gün)", - "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging.*/ (veya legacy final_model.staging/) dizinini hangi yaşta geciken olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", + "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging.*/ (veya eski final_model.staging/) dizinini hangi yaşta süresi geçmiş olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", "wizard.retention.ephemeral.label": "Geçici artefact'lar (gün)", - "wizard.retention.ephemeral.hint": "forgelm purge --check-policy compliance/ ve data_audit_report.json'u hangi yaşta geciken olarak raporlasın. Varsayılan 90.", + "wizard.retention.ephemeral.hint": "forgelm purge --check-policy compliance/ ve data_audit_report.json'u hangi yaşta süresi geçmiş olarak raporlasın. Varsayılan 90.", "wizard.retention.raw_documents.label": "Ham dokümanlar (gün)", - "wizard.retention.raw_documents.hint": "forgelm purge --check-policy raw_documents/ veya ingestion_output/ dizinini hangi yaşta geciken olarak raporlasın. Varsayılan 90.", + "wizard.retention.raw_documents.hint": "forgelm purge --check-policy raw_documents/ veya ingestion_output/ dizinini hangi yaşta süresi geçmiş olarak raporlasın. Varsayılan 90.", "wizard.retention.enforce.label": "Politika modu (ayrılmış)", "wizard.retention.enforce.hint": "Ayrılmış: hiçbir ForgeLM komutu bunu henüz kullanmaz, bu yüzden üç mod da aynı davranır. Saklama ufukları yalnızca forgelm purge --check-policy ile raporlanır; hiçbir şey otomatik silinmez.", "wizard.retention.enforce.log_only": "log_only — varsayılan (hiçbir komut kullanmaz)", @@ -2997,7 +2997,7 @@ "wizard.synthetic.api_key_env.label": "Variable env clé API teacher", "wizard.synthetic.api_key_env.hint": "Nom variable env portant clé API teacher. Vide = modèle teacher local.", "wizard.retention.title": "Horizons de rétention (RGPD Art. 5+17)", - "wizard.retention.hint": "Âges à partir desquels forgelm purge --check-policy signale des artefacts comme dépassés ; rien n'est supprimé.", + "wizard.retention.hint": "Seuils d'âge à partir desquels forgelm purge --check-policy signale des artefacts comme dépassés ; rien n'est supprimé.", "wizard.retention.audit_log.label": "Rétention audit log (jours)", "wizard.retention.audit_log.hint": "Après combien de jours forgelm purge --check-policy signale audit_log.jsonl comme dépassé. Default 1825 (5 ans) est un choix de ForgeLM, pas un délai légal.", "wizard.retention.staging.label": "Horizon staging (jours)", @@ -3391,11 +3391,11 @@ "wizard.synthetic.api_key_env.label": "Variable env clave API teacher", "wizard.synthetic.api_key_env.hint": "Nombre variable env con clave API teacher. Vacío = modelo teacher local.", "wizard.retention.title": "Horizontes de retención (RGPD Art. 5+17)", - "wizard.retention.hint": "Antigüedades a partir de las cuales forgelm purge --check-policy reporta artefactos como vencidos; no se borra nada.", + "wizard.retention.hint": "Antigüedad a partir de la cual forgelm purge --check-policy reporta los artefactos como vencidos; no se elimina nada.", "wizard.retention.audit_log.label": "Retención audit log (días)", "wizard.retention.audit_log.hint": "Tras cuántos días forgelm purge --check-policy reporta audit_log.jsonl como vencido. Default 1825 (5 años) es una elección de ForgeLM, no un plazo legal.", "wizard.retention.staging.label": "Horizonte staging (días)", - "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta un directorio final_model.staging.*/ (o el antiguo final_model.staging/) como vencido; no se borra nada automáticamente. Default 7.", + "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta un directorio final_model.staging.*/ (o el antiguo final_model.staging/) como vencido; no se elimina nada automáticamente. Default 7.", "wizard.retention.ephemeral.label": "Artefactos efímeros (días)", "wizard.retention.ephemeral.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta compliance/ y data_audit_report.json como vencidos. Default 90.", "wizard.retention.raw_documents.label": "Documentos crudos (días)", @@ -3785,15 +3785,15 @@ "wizard.synthetic.api_key_env.label": "Teacher API 密钥环境变量", "wizard.synthetic.api_key_env.hint": "承载 teacher API 密钥的环境变量名。空 = 本地 teacher 模型。", "wizard.retention.title": "保留期限(GDPR 第 5+17 条)", - "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已过期的时长;不会删除任何内容。", + "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已超期的期限(天数);不会删除任何内容。", "wizard.retention.audit_log.label": "Audit log 保留(天)", - "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已过期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", + "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已超期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", "wizard.retention.staging.label": "Staging 期限(天)", - "wizard.retention.staging.hint": "final_model.staging.*/(或旧的 final_model.staging/)目录存在多少天后,forgelm purge --check-policy 会将其报告为已过期;不会自动删除任何内容。默认 7。", + "wizard.retention.staging.hint": "final_model.staging.*/(或旧的 final_model.staging/)目录存在多少天后,forgelm purge --check-policy 会将其报告为已超期;不会自动删除任何内容。默认 7。", "wizard.retention.ephemeral.label": "临时 artefact(天)", - "wizard.retention.ephemeral.hint": "compliance/ 和 data_audit_report.json 存在多少天后,forgelm purge --check-policy 会将其报告为已过期。默认 90。", + "wizard.retention.ephemeral.hint": "compliance/ 和 data_audit_report.json 存在多少天后,forgelm purge --check-policy 会将其报告为已超期。默认 90。", "wizard.retention.raw_documents.label": "原始文档(天)", - "wizard.retention.raw_documents.hint": "raw_documents/ 或 ingestion_output/ 存在多少天后,forgelm purge --check-policy 会将其报告为已过期。默认 90。", + "wizard.retention.raw_documents.hint": "raw_documents/ 或 ingestion_output/ 存在多少天后,forgelm purge --check-policy 会将其报告为已超期。默认 90。", "wizard.retention.enforce.label": "策略模式(保留)", "wizard.retention.enforce.hint": "保留项:目前没有任何 ForgeLM 命令使用它,因此三种模式行为相同。保留期限仅由 forgelm purge --check-policy 报告;不会自动删除任何内容。", "wizard.retention.enforce.log_only": "log_only — 默认(没有命令使用它)", @@ -3929,9 +3929,9 @@ "compliance.gdpr.title": "Ingest'te PII redaksiyonu", "compliance.gdpr.body": "--pii-mask chunk'lar JSONL'a düşmeden e-posta, telefon, kredi kartı (Luhn), IBAN, ulusal kimlik (TR/DE/FR/US-SSN) maskeler. Eğitime ulaşanı azaltır ve veri minimizasyonunu (Madde 5(1)(c)) destekler; isimler ve serbest metin tespit edilmez, bu yüzden bir yardımcıdır, garanti değildir.", "compliance.gdpr15.title": "Erişim hakkı", - "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz-metin geçişlerini veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", + "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz metin olarak geçtiği yerleri veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", "compliance.gdpr17.title": "Silme hakkı", - "compliance.gdpr17.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını kaydeder. Madde 17 silmenin bir adımıdır: eğitilmiş ağırlıklar, kaynak dokümanlar ve başka yerlerdeki kopyalar değişmez.", + "compliance.gdpr17.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını kaydeder. Madde 17 kapsamındaki silmenin bir adımıdır: eğitilmiş ağırlıklar, kaynak dokümanlar ve başka yerlerdeki kopyalar değişmez.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.title": "Hizalanmış, sertifikalı değil", "compliance.iso.lead": "Yazılım ISO 27001 veya SOC 2 sertifikalı olamaz — sadece organizasyonlar olabilir. ForgeLM, denetçinin deployer'a sorduğu kontrollerle hizalıdır: her çalışmada audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtlarını üretir; doğrudan audit klasörünüze düşer.", @@ -4452,7 +4452,7 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.de, { "compliance.gdpr15.title": "Auskunftsrecht", - "compliance.gdpr15.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext-Vorkommen oder, bei einer externen Maskierungs-Pipeline, nach gesalzenen SHA-256-Digests einer Betroffenenkennung und meldet jede Zeile der gescannten Dateien, die sie enthält. Das ist ein Baustein Ihrer Auskunft nach Artikel 15, die außerdem eine Datenkopie und die Informationen nach Artikel 15 Abs. 1 Buchst. a bis h enthalten muss; Rohquellen und das trainierte Modell werden nicht durchsucht.", + "compliance.gdpr15.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext-Vorkommen oder nach von einer externen Maskierungs-Pipeline geschriebenen gesalzenen SHA-256-Digests der Kennung einer betroffenen Person und meldet jede Zeile der gescannten Dateien, die sie enthält. Das ist ein Baustein Ihrer Auskunft nach Artikel 15 DSGVO, die außerdem eine Datenkopie und die Informationen nach Artikel 15 Abs. 1 Buchst. a bis h enthalten muss; Rohquellen und das trainierte Modell werden nicht durchsucht.", "compliance.gdpr17.title": "Recht auf Löschung", "compliance.gdpr17.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert data.erasure_requested + data.erasure_completed im Append-only-Audit-Log. Es ist ein Schritt einer Löschung nach Artikel 17: trainierte Gewichte, Quelldokumente und Kopien an anderen Orten bleiben unverändert.", "compliance.iso.title": "Aligned, nicht zertifiziert", @@ -4470,7 +4470,7 @@ "features.gov.heading": "Compliance- & DSGVO-Tooling", "features.gov.body": "EU-AI-Act-Artikel 9-17, DSGVO-Artikel 15 + 17, ISO-27001- / SOC-2-Type-II-Alignment-Nachweise.", "features.gov.access.title": "DSGVO-Artikel-15-Auskunft", - "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern einer Betroffenenkennung und listet die Zeilen der gescannten Dateien auf, die sie enthalten.", + "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern der Kennung einer betroffenen Person und listet die Zeilen der gescannten Dateien auf, die sie enthalten.", "features.gov.annex.title": "Annex-IV-Verifier", "features.gov.annex.body": "forgelm verify-annex-iv validiert die Annex-IV-§1-9-Schemavollständigkeit und den manifest_hash des Bundles zur Manipulationserkennung. (Audit-Ketten-Integrität ist der separate Befehl forgelm verify-audit.)", "features.gov.approve.title": "Artikel-14-Staging-Gate", @@ -4480,7 +4480,7 @@ "features.gov.iso.title": "ISO 27001 / SOC 2 Alignment", "features.gov.iso.body": "Audit-Trail-, Change-Management-, Daten-Lineage- und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Die Software ist aligned, nicht zertifiziert.", "features.gov.purge.title": "DSGVO-Artikel-17-Löschung", - "features.gov.purge.body": "forgelm purge entfernt die Zeilen einer betroffenen Person aus Trainingskorpora und stempelt die Löschung in das Append-only-Audit-Log.", + "features.gov.purge.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log; es ist ein Schritt einer Löschung nach Artikel 17 (trainierte Gewichte bleiben unverändert).", "features.eval.safetyeval.title": "Standalone-Sicherheits-Eval", "features.eval.safetyeval.body": "forgelm safety-eval führt Llama Guard gegen eine --probes-JSONL-Prompt-Datei aus (oder --default-probes für das gebündelte 51-Prompt-Set über 18 Schadkategorien) und gibt eine kategorienspezifische Sicherheitszusammenfassung aus (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "Bibliotheks-API", @@ -4498,9 +4498,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.fr, { "compliance.gdpr15.title": "Droit d'accès", - "compliance.gdpr15.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche d'occurrences en clair, ou de digests SHA-256 salés écrits par un pipeline de masquage externe, d'un identifiant de personne concernée et signale chaque ligne des fichiers analysés qui le contient. C'est un élément de votre réponse à l'article 15, qui doit aussi fournir une copie des données et les informations de l'article 15(1)(a) à (h) ; les sources brutes et le modèle entraîné ne sont pas parcourus.", + "compliance.gdpr15.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche d'occurrences en clair, ou de digests SHA-256 salés écrits par un pipeline de masquage externe, de l'identifiant d'une personne concernée et signale chaque ligne des fichiers analysés qui le contient. C'est un élément de votre réponse à une demande d'accès (article 15), qui doit aussi fournir une copie des données et les informations de l'article 15, paragraphe 1, points a) à h) ; les sources brutes et le modèle entraîné ne sont pas parcourus.", "compliance.gdpr17.title": "Droit à l'effacement", - "compliance.gdpr17.body": "forgelm purge supprime des lignes par id d'un corpus d'entraînement et consigne les événements data.erasure_requested + data.erasure_completed dans le log d'audit append-only. C'est une étape d'un effacement au titre de l'article 17 : les poids entraînés, les documents sources et les copies ailleurs ne changent pas.", + "compliance.gdpr17.body": "forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne les événements data.erasure_requested + data.erasure_completed dans le log d'audit append-only. C'est une étape d'un effacement au titre de l'article 17 : les poids entraînés, les documents sources et les copies conservées ailleurs ne sont pas modifiés.", "compliance.iso.title": "Aligné, non certifié", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "Le logiciel ne peut pas être certifié ISO 27001 ou SOC 2 — seules les organisations le peuvent. ForgeLM est aligné avec les contrôles qu'un auditeur demande au déployeur : il produit les preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain à chaque exécution, prêtes à être déposées dans votre classeur d'audit.", @@ -4516,7 +4516,7 @@ "features.gov.heading": "Outillage Compliance & RGPD", "features.gov.body": "Articles 9-17 de l'EU AI Act, RGPD Articles 15 + 17, preuves d'alignement ISO 27001 / SOC 2 Type II.", "features.gov.access.title": "Accès RGPD Article 15", - "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de correspondances en clair ou de digests salés d'un identifiant de personne concernée et liste les lignes des fichiers analysés qui le contiennent.", + "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de correspondances en clair ou de digests salés de l'identifiant d'une personne concernée et liste les lignes des fichiers analysés qui le contiennent.", "features.gov.annex.title": "Vérificateur Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valide la complétude du schéma Annex IV §1-9 et le manifest_hash du bundle pour la détection de falsification. (L'intégrité de la chaîne d'audit est la commande séparée forgelm verify-audit.)", "features.gov.approve.title": "Gate de staging Article 14", @@ -4526,7 +4526,7 @@ "features.gov.iso.title": "Alignement ISO 27001 / SOC 2", "features.gov.iso.body": "Preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain que l'auditeur du déployeur demande. Le logiciel est aligné, non certifié.", "features.gov.purge.title": "Effacement RGPD Article 17", - "features.gov.purge.body": "forgelm purge supprime les lignes d'une personne concernée des corpus d'entraînement et estampille la suppression dans le log d'audit append-only.", + "features.gov.purge.body": "forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne l'effacement dans le log d'audit append-only ; c'est une étape d'un effacement au titre de l'article 17 (les poids entraînés ne changent pas).", "features.eval.safetyeval.title": "Évaluation sécurité autonome", "features.eval.safetyeval.body": "forgelm safety-eval exécute Llama Guard contre un fichier de prompts JSONL --probes (ou --default-probes pour le set bundlé de 51 prompts couvrant 18 catégories de préjudice) et émet un résumé de sécurité par catégorie (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "API bibliothèque", @@ -4544,9 +4544,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.es, { "compliance.gdpr15.title": "Derecho de acceso", - "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL en busca de apariciones en texto claro, o de digests SHA-256 salados escritos por un pipeline de enmascaramiento externo, de un identificador de interesado y reporta cada línea de los archivos escaneados que lo contiene. Es un elemento de su respuesta del Artículo 15, que además debe incluir una copia de los datos y la información del Artículo 15(1)(a) a (h); no se buscan las fuentes sin procesar ni el modelo entrenado.", + "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL en busca de apariciones en texto claro, o de digests SHA-256 salados escritos por un pipeline de enmascaramiento externo, de un identificador de interesado y reporta cada línea de los archivos escaneados que lo contiene. Es un elemento de su respuesta a una solicitud de acceso (Artículo 15), que además debe incluir una copia de los datos y la información del Artículo 15, apartado 1, letras a) a h); no se buscan las fuentes sin procesar ni el modelo entrenado.", "compliance.gdpr17.title": "Derecho de supresión", - "compliance.gdpr17.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra los eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only. Es un paso de una supresión del Artículo 17: los pesos entrenados, los documentos de origen y las copias en otros lugares no cambian.", + "compliance.gdpr17.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra los eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only. Es un paso de una supresión conforme al Artículo 17: los pesos entrenados, los documentos de origen y las copias que existan en otros lugares no cambian.", "compliance.iso.title": "Alineado, no certificado", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "El software no puede ser certificado ISO 27001 o SOC 2 — solo las organizaciones pueden. ForgeLM está alineado con los controles que un auditor le pregunta al deployer: produce las pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro en cada ejecución, listas para incorporarse a su carpeta de auditoría.", @@ -4572,7 +4572,7 @@ "features.gov.iso.title": "Alineamiento ISO 27001 / SOC 2", "features.gov.iso.body": "Pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro que pide el auditor del deployer. El software está alineado, no certificado.", "features.gov.purge.title": "Supresión RGPD Artículo 17", - "features.gov.purge.body": "forgelm purge elimina las filas de un interesado de los corpus de entrenamiento y estampa la supresión en el log de auditoría append-only.", + "features.gov.purge.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra la supresión en el log de auditoría append-only; es un paso de una supresión conforme al Artículo 17 (los pesos entrenados no cambian).", "features.eval.safetyeval.title": "Evaluación de seguridad autónoma", "features.eval.safetyeval.body": "forgelm safety-eval ejecuta Llama Guard contra un archivo de prompts JSONL --probes (o --default-probes para el set bundleado de 51 prompts que cubre 18 categorías de daño) y emite un resumen de seguridad por categoría (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "API de biblioteca", @@ -4590,9 +4590,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.zh, { "compliance.gdpr15.title": "访问权", - "compliance.gdpr15.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识的明文出现,或由外部掩码流水线写入的加盐 SHA-256 摘要,并报告所扫描文件中包含它的每一行。它只是第 15 条答复的输入之一,该答复还必须提供数据副本和第 15 条第 1 款(a)至(h)项信息;不会搜索原始来源或已训练的模型。", + "compliance.gdpr15.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文,或由外部掩码流水线写入的该标识符的加盐 SHA-256 摘要,并报告所扫描文件中包含它的每一行。它只是对第 15 条访问请求答复的输入之一,该答复还必须提供数据副本和第 15 条第 1 款(a)至(h)项信息;不会搜索原始来源或已训练的模型。", "compliance.gdpr17.title": "删除权", - "compliance.gdpr17.body": "forgelm purge 按 id 从训练语料中删除行,并将 data.erasure_requested + data.erasure_completed 事件记录到 append-only 审计日志。它只是第 17 条删除的其中一步:已训练的权重、源文档和其他位置的副本不会改变。", + "compliance.gdpr17.body": "forgelm purge 按 id 从训练语料中删除行,并将 data.erasure_requested + data.erasure_completed 事件记录到 append-only 审计日志。它只是第 17 条所涉删除的其中一步:已训练的权重、源文档和其他位置的副本不会改变。", "compliance.iso.title": "对齐,而非认证", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "软件无法获得 ISO 27001 或 SOC 2 认证 — 只有组织可以。ForgeLM 与审计员向 deployer 询问的控制项保持对齐:它在每次运行时产出审计追踪、变更管理、数据血缘和供应链证据,可直接放入你的审计文件夹。", @@ -4608,7 +4608,7 @@ "features.gov.heading": "合规 & GDPR 工具", "features.gov.body": "EU AI Act 第 9-17 条、GDPR 第 15 + 17 条、ISO 27001 / SOC 2 Type II 对齐证据。", "features.gov.access.title": "GDPR 第 15 条访问", - "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识的明文或加盐摘要匹配,并列出所扫描文件中包含它的行。", + "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文或加盐摘要匹配,并列出所扫描文件中包含它的行。", "features.gov.annex.title": "Annex IV 验证器", "features.gov.annex.body": "forgelm verify-annex-iv 验证 Annex IV §1-9 模式完整性以及 bundle 的 manifest_hash 防篡改。(审计链完整性由独立的 forgelm verify-audit 命令负责。)", "features.gov.approve.title": "第 14 条 staging gate", @@ -4618,7 +4618,7 @@ "features.gov.iso.title": "ISO 27001 / SOC 2 对齐", "features.gov.iso.body": "deployer 的审计员需要的审计追踪、变更管理、数据血缘和供应链证据。软件是对齐,而不是被认证。", "features.gov.purge.title": "GDPR 第 17 条删除", - "features.gov.purge.body": "forgelm purge 从训练语料中移除某个数据主体的行,并将该删除盖入 append-only 审计日志。", + "features.gov.purge.body": "forgelm purge 按 id 从训练语料中删除行,并将该删除记录到 append-only 审计日志;它只是第 17 条所涉删除的其中一步(已训练的权重不会改变)。", "features.eval.safetyeval.title": "独立安全评估", "features.eval.safetyeval.body": "forgelm safety-eval 对 --probes JSONL 提示文件 (或用 --default-probes 调用打包的、覆盖 18 个危害类别的 51-提示集) 运行 Llama Guard,并产出按类别的安全摘要 (safety_results.json + safety_trend.jsonl)。", "features.ent.api.title": "库 API", diff --git a/tests/site/site_translations.test.cjs b/tests/site/site_translations.test.cjs index b016355a..ffc7b06c 100644 --- a/tests/site/site_translations.test.cjs +++ b/tests/site/site_translations.test.cjs @@ -40,11 +40,12 @@ test('claims the code does not back stay out of the site copy', () => { /SHA-256 \+ size manifest/, /6 quant levels and SHA-256 manifest/, /ZeRO-2 \/ ZeRO-3 presets/, /PII redacted at ingest/, /GGUF export at q4_k_m/, /wired into the audit log/, /Article 17 erasure with/, /matches EU AI Act guidance/, /rejected models in final_model\.staging/, /forensically linked/, /every row of every split/, - /surface every row/], + /surface every row/, /drops a data-subject's rows/], tr: [/ikili[- ]kontrol/i, /otomatik siler/, /GDPR uyumlu TTL/, /erişim-talebi tarafını karşılar/, /Slack, Teams, Discord veya/, /Slack \/ Teams \/ Discord/, /upstream'de uygulanır/, /her ikisi de audit log'a bağlı/, /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/, /adli olarak bağlanır/, - /geçtiği her split/, /yüzeye çıkarır/], + /geçtiği her split/, /yüzeye çıkarır/, + /satırlarını eğitim corpus'undan düşürür/], }; const found = []; for (const [locale, patterns] of Object.entries(retired)) { @@ -78,7 +79,9 @@ test('the home GDPR card keeps the trained-weights caveat in every locale', () = } const weights = { en: /weights/, tr: /ağırlıklar/, de: /Gewichte/, fr: /poids/, es: /pesos/, zh: /权重/ }; for (const [locale, pattern] of Object.entries(weights)) { - assert.match(translations[locale]['compliance.gdpr17.body'], pattern, `${locale}: purge does not change trained weights`); + for (const key of ['compliance.gdpr17.body', 'features.gov.purge.body']) { + assert.match(translations[locale][key], pattern, `${locale} ${key}: purge does not change trained weights`); + } } const html = fs.readFileSync(path.join(root, 'site', 'index.html'), 'utf8'); const fallback = html.match(/data-i18n-html="home\.cap\.gdpr\.body">([^<]*(?:[^<]*<\/code>[^<]*)*)<\/p>/); From c2c3c79dfda60f79e84e8189b1433a6b2a89eaf3 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:16:26 +0300 Subject: [PATCH 12/25] =?UTF-8?q?fix:=20review=20round=202=20=E2=80=94=20m?= =?UTF-8?q?odule-size=20budgets,=20enforce=20wording=20test,=20retention?= =?UTF-8?q?=20hints,=20reverse-pii=20limits,=20sturdier=20guard=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by the second review round and the first full test run of the branch: - The two docstring/help rewrites had grown `forgelm/cli/_parser.py` by 1 line and `forgelm/cli/subcommands/_purge.py` by 4, over their module-size ratchets; the wording is kept and the line counts are back to the base (the ratchet test and the gauntlet step fail otherwise). `tests/test_config.py` still asserted the old "no command reads it" wording of the reserved `retention.enforce` description. - GDPR page: `--check-policy` needs `--output-dir ` (it scans the current directory otherwise); reverse-pii matching is case-sensitive and compares raw file text, so spelling/case variants and ASCII-escaped non-ASCII names need `--type custom --query '(?i)...'`; names and locations are what the English `--pii-ml` model counts (Presidio ignores the organization label by default); a model that cannot be shown to be anonymous (EDPB Opinion 28/2024) can bring access and erasure requests to the model itself; KVKK's principles, access and erasure rights and security duty are placed in Art. 4, 11/7 and 12. - pii-ml page: the example no longer lists an IBAN under a "medium" worst tier, and says `organization` stays 0 with `--pii-ml-language en`. - Wizard hints for the ephemeral and raw-document horizons say the age is counted from the audit log's first event (file age when there is no log), in all six languages; the Chinese staging hint no longer says "has existed for N days". - Recital 14 is cited for legal persons only, with the sole-trader exception. - Tests: the guard is now pinned against the mutations the review found alive (a nested drift alone fails strict mode and names file:line:key, a row with extra keys does not exempt a trimmed snippet, a flat language page is scanned, an empty tree fails with a message, an opaque top-level field is not an unknown block); the retired-phrase lists tolerate hard-wrapping, `Art. 9` needs a word boundary, the quadratic pattern is gone, and the legacy-marker reader may only call `is_file`, `read_text` and `strip`. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/data_audit-tr.md | 5 +- docs/guides/data_audit.md | 2 +- docs/usermanuals/en/compliance/gdpr.md | 10 ++-- docs/usermanuals/en/data/pii-ml.md | 4 +- docs/usermanuals/tr/compliance/gdpr.md | 10 ++-- docs/usermanuals/tr/data/pii-ml.md | 4 +- forgelm/cli/_parser.py | 5 +- forgelm/cli/subcommands/_purge.py | 16 +++--- site/js/translations.js | 26 ++++----- tests/site/site_translations.test.cjs | 2 +- tests/test_check_usermanual_schema_drift.py | 54 ++++++++++++++++++- tests/test_config.py | 2 +- tests/test_gdpr_erasure.py | 6 ++- .../test_usermanual_reference_consistency.py | 14 ++--- 14 files changed, 107 insertions(+), 53 deletions(-) diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 230beeff..330914af 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -416,8 +416,9 @@ install ile uyumlu kalır. Varsayılan regex detector, yaygın yapılandırılmış identifier'ları (email, telefon, IBAN, kredi kartı, ulusal kimlik) kapsar. Her biri, kimliği belirli veya belirlenebilir bir gerçek kişiye ilişkin olduğunda GDPR Madde 4(1) -kapsamında kişisel veridir; bir şirketin IBAN'ı veya santral numarası (Gerekçe 14) ile bir test -kart numarası ise kişisel veri değildir. +kapsamında kişisel veridir; bir tüzel kişinin IBAN'ı veya santral numarası, şahıs tacir gibi bir gerçek +kişiyi tanımlamadıkça kişisel veri değildir (Gerekçe 14); bir test kart +numarası ise hiç kimseye ilişkin değildir. Faz 12.5, regex'in doğal olarak kaçırdığı yapılandırılmamış identifier kategorilerini ekleyen opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adaptörünü ML-NER olarak üstüne ekler: `person`, `organization`, diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index b8916dfb..bca4e9c3 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -409,7 +409,7 @@ the audit reproducible (Annex IV requirement) and bare-install-friendly. The default regex detector covers the common structured identifiers (email, phone, IBAN, credit card, national IDs). Each is personal data under GDPR Art. 4(1) when it relates to an identified or identifiable natural -person; a company's IBAN or switchboard number (recital 14), or a test card number, is not. +person; a legal person's IBAN or switchboard number is not (recital 14) unless it identifies a natural person such as a sole trader, and a test card number relates to no one. Phase 12.5 adds an opt-in **Presidio** ([data-privacy-stack/presidio](https://github.com/data-privacy-stack/presidio)) adapter that layers ML-NER on top — adding the unstructured-identifier categories regex inherently misses: `person`, `organization`, `location`. diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index bc83b452..7a654347 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -19,7 +19,7 @@ GDPR (Europe) and KVKK (Türkiye) impose data protection requirements on persona | **Integrity & confidentiality** | GDPR Art. 5(1)(f) | Protect against unauthorised access. | | **Accountability** | GDPR Art. 5(2) | You must be able to demonstrate compliance. | -KVKK (Türkiye) mirrors these principles closely. +KVKK (Türkiye, Law 6698) sets similar principles in its Article 4 and gives data subjects access and erasure rights in Articles 11 and 7; it has no separate accountability principle, and its security duties sit in Article 12. ## How ForgeLM addresses each @@ -31,7 +31,7 @@ PII masking at ingest replaces the identifier formats ForgeLM detects with a pla $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. To measure how many names, organisations and locations remain, run `forgelm audit --pii-ml`: it adds Presidio NER **detection**, which reports per-category counts for each split (not which rows or which names) and masks nothing. It needs the `ingestion-pii-ml` extra and a spaCy model (for a language without a dedicated model, such as Turkish, `--pii-ml-language xx` uses `xx_ent_wiki_sm`). To find a known name, search for it with `forgelm reverse-pii`. See [PII Masking](#/data/pii-masking) and [ML-NER PII](#/data/pii-ml). +The detectors cover email, phone, credit card, IBAN, and the Turkish, German, French and US national-ID formats; every match becomes `[REDACTED]`. **Names, addresses, dates of birth, health data and other free text are not detected**, and the patterns are tuned to miss a borderline match rather than redact ordinary prose. Masking reduces the personal data in the JSONL; it does not remove identifiable subjects from it. To measure how many names and locations (and, for some language models, organisations) remain, run `forgelm audit --pii-ml`: it adds Presidio NER **detection**, which reports per-category counts for each split (not which rows or which names) and masks nothing. It needs the `ingestion-pii-ml` extra and a spaCy model (for a language without a dedicated model, such as Turkish, `--pii-ml-language xx` uses `xx_ent_wiki_sm`). To find a known name, search for it with `forgelm reverse-pii`. See [PII Masking](#/data/pii-masking) and [ML-NER PII](#/data/pii-ml). ### Accountability (Art. 5(2)) @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml` lists artefacts that are past their horizon and always exits 0. Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) +`forgelm purge --check-policy --config config.yaml --output-dir ` lists the artefacts in that directory that are past their horizon and always exits 0 (without `--output-dir` it scans the current directory, not `training.output_dir`). Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) ## Data subject requests @@ -70,7 +70,7 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed value is the first 16 hex digits of the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning; with a `data/*.jsonl` glob that puts `audit_log.jsonl` among your corpus files, so always pass `--output-dir`. It finds what is literally in the files you point it at. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. +The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed value is the first 16 hex digits of the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning; with a `data/*.jsonl` glob that puts `audit_log.jsonl` among your corpus files, so always pass `--output-dir`. It finds what is literally in the files you point it at: the match is case-sensitive and compares raw file text, so search spelling and case variants as well (`--type custom --query '(?i)…'`), and note that a corpus written with ASCII escapes such as `\u00e7` hides non-ASCII names from a literal query. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. ### Right to erasure (Art. 17) @@ -111,7 +111,7 @@ Train from the file with ids. `forgelm reverse-pii` reports the file and line wh ### Model-level erasure -LLMs can memorise rare strings from training. Even with PII masking, removing all traces from a deployed model is hard. ForgeLM's defences: +LLMs can memorise rare strings from training. Even with PII masking, removing all traces from a deployed model is hard, and under EDPB Opinion 28/2024 a model trained on personal data cannot always be considered anonymous: if you cannot show that yours is, an access or erasure request may reach the model itself, not only the corpus. ForgeLM's defences: - **Reduce what goes in:** PII masking and deduplication (memorised data is usually duplicated data). - **Memorisation is not measured:** ForgeLM does not test whether a trained model reproduces a given string. diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index 0a954fcd..03fa2483 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -39,7 +39,7 @@ Without the spaCy model, `forgelm audit --pii-ml` raises a typed `ImportError` * ```shell $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml ✓ format: instructions (8,400 rows) -⚠ PII: 12 emails, 3 phone, 1 IBAN (regex layer) +⚠ PII: 12 emails, 3 phone (regex layer) ⚠ PII (ML): 47 person, 18 organization, 9 location (Presidio) worst tier: medium ``` @@ -62,6 +62,8 @@ The Presidio findings merge into the same `pii_summary` and `pii_severity` block } ``` +Presidio's default English configuration ignores the `organization` label (it produces many false positives), so with `--pii-ml-language en` the `organization` count stays 0; the figures above assume a language model that reports it, for example `xx`. + ## Severity tiers The new categories sit under a dedicated table in `forgelm.data_audit.PII_ML_SEVERITY`: diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index ad20c135..4835b203 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -19,7 +19,7 @@ GDPR (Avrupa) ve KVKK (Türkiye), eğitimde kullanılan kişisel verilere koruma | **Bütünlük & gizlilik** | GDPR Md. 5(1)(f) | Yetkisiz erişime karşı koruyun. | | **Hesap verebilirlik** | GDPR Md. 5(2) | Uyumu gösterebilmelisiniz. | -KVKK (Türkiye) bu ilkeleri yakından yansıtır. +KVKK (Türkiye, 6698 sayılı Kanun) benzer ilkeleri 4. maddesinde sayar, ilgili kişilere erişim ve silme haklarını 11. ve 7. maddelerde tanır; ayrı bir hesap verebilirlik ilkesi yoktur ve güvenlik yükümlülükleri 12. maddededir. ## ForgeLM her birini nasıl ele alıyor @@ -31,7 +31,7 @@ Ingest'te PII maskeleme, ForgeLM'in tespit ettiği tanımlayıcı biçimlerini b $ forgelm ingest ./policies/ --recursive --strategy markdown --pii-mask --output data/policies.jsonl ``` -Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. Geriye kaç isim, kurum ve yer kaldığını ölçmek için `forgelm audit --pii-ml` çalıştırın: Presidio NER **tespiti** ekler; bu tespit her split için kategori başına sayıları raporlar (hangi satırların ya da hangi isimlerin olduğunu değil) ve hiçbir şeyi maskelemez. `ingestion-pii-ml` extra'sını ve bir spaCy modelini gerektirir (Türkçe gibi özel modeli olmayan diller için `--pii-ml-language xx`, `xx_ent_wiki_sm` kullanır). Bilinen bir ismi bulmak için onu `forgelm reverse-pii` ile arayın. Bkz. [PII Maskeleme](#/data/pii-masking) ve [ML-NER PII](#/data/pii-ml). +Dedektörler e-posta, telefon, kredi kartı, IBAN ile Türk, Alman, Fransız ve ABD ulusal kimlik biçimlerini kapsar; her eşleşme `[REDACTED]` olur. **İsimler, adresler, doğum tarihleri, sağlık verisi ve diğer serbest metin tespit edilmez**; desenler sıradan düzyazıyı maskelemek yerine sınırda bir eşleşmeyi kaçırmaya ayarlıdır. Maskeleme JSONL'daki kişisel veriyi azaltır; tanımlanabilir özneleri ondan kaldırmaz. Geriye kaç isim ve yerin (bazı dil modellerinde kurumların da) kaldığını ölçmek için `forgelm audit --pii-ml` çalıştırın: Presidio NER **tespiti** ekler; bu tespit her split için kategori başına sayıları raporlar (hangi satırların ya da hangi isimlerin olduğunu değil) ve hiçbir şeyi maskelemez. `ingestion-pii-ml` extra'sını ve bir spaCy modelini gerektirir (Türkçe gibi özel modeli olmayan diller için `--pii-ml-language xx`, `xx_ent_wiki_sm` kullanır). Bilinen bir ismi bulmak için onu `forgelm reverse-pii` ile arayın. Bkz. [PII Maskeleme](#/data/pii-masking) ve [ML-NER PII](#/data/pii-ml). ### Hesap verebilirlik (Md. 5(2)) @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml` ufku aşmış artefaktları listeler ve her zaman 0 ile çıkar. Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) +`forgelm purge --check-policy --config config.yaml --output-dir ` o dizinde ufku aşmış artefaktları listeler ve her zaman 0 ile çıkar (`--output-dir` verilmezse `training.output_dir`'i değil, bulunulan dizini tarar). Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) ## Veri sahibi talepleri @@ -70,7 +70,7 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan değer, audit event'inin kaydettiği tuzlu tanımlayıcı hash'inin ilk 16 hex hanesidir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar; `data/*.jsonl` gibi bir glob ile bu, `audit_log.jsonl`'i corpus dosyalarınızın arasına koyar, bu yüzden her zaman `--output-dir` verin. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. +Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan değer, audit event'inin kaydettiği tuzlu tanımlayıcı hash'inin ilk 16 hex hanesidir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar; `data/*.jsonl` gibi bir glob ile bu, `audit_log.jsonl`'i corpus dosyalarınızın arasına koyar, bu yüzden her zaman `--output-dir` verin. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur: eşleşme büyük/küçük harfe duyarlıdır ve ham dosya metnini karşılaştırır, bu yüzden yazım ve harf varyantlarını da arayın (`--type custom --query '(?i)…'`); `\u00e7` gibi ASCII kaçışlarıyla yazılmış bir corpus ASCII dışı isimleri literal sorgudan gizler. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. ### Silme hakkı (Md. 17) @@ -111,7 +111,7 @@ Eğitimi id'li dosyadan yapın. `forgelm reverse-pii` tanımlayıcının geçti ### Model-seviyesi silme -LLM'ler eğitimden nadir string'leri ezberleyebilir. PII maskelemeyle bile deploy edilmiş bir modelden tüm izleri kaldırmak zordur. ForgeLM'in savunmaları: +LLM'ler eğitimden nadir string'leri ezberleyebilir. PII maskelemeyle bile deploy edilmiş bir modelden tüm izleri kaldırmak zordur; EDPB Görüş 28/2024'e göre kişisel veriyle eğitilmiş bir model her durumda anonim sayılamaz: modelinizin anonim olduğunu gösteremiyorsanız, bir erişim veya silme talebi yalnızca corpus'a değil modelin kendisine de uzanabilir. ForgeLM'in savunmaları: - **Girenleri azalt:** PII maskeleme ve deduplikasyon (ezberlenen veri genelde tekrarlanan veridir). - **Ezber ölçülmez:** ForgeLM, eğitilmiş bir modelin belirli bir string'i üretip üretmediğini test etmez. diff --git a/docs/usermanuals/tr/data/pii-ml.md b/docs/usermanuals/tr/data/pii-ml.md index f7df352f..415bac42 100644 --- a/docs/usermanuals/tr/data/pii-ml.md +++ b/docs/usermanuals/tr/data/pii-ml.md @@ -39,7 +39,7 @@ spaCy modeli yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce** ```shell $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml ✓ format: instructions (8.400 satır) -⚠ PII: 12 email, 3 telefon, 1 IBAN (regex katmanı) +⚠ PII: 12 email, 3 telefon (regex katmanı) ⚠ PII (ML): 47 person, 18 organization, 9 location (Presidio) en kötü kat: medium ``` @@ -62,6 +62,8 @@ Presidio bulguları aynı `pii_summary` ve `pii_severity` bloklarına disjoint k } ``` +Presidio'nun varsayılan İngilizce yapılandırması `organization` etiketini yok sayar (çok yanlış pozitif üretir); bu yüzden `--pii-ml-language en` ile `organization` sayısı 0 kalır. Yukarıdaki rakamlar bu etiketi raporlayan bir dil modelini (örneğin `xx`) varsayar. + ## Şiddet katmanları Yeni kategoriler `forgelm.data_audit.PII_ML_SEVERITY` adında özel bir tabloda yaşar: diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 56804e2c..ced63004 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -628,9 +628,8 @@ def _add_audit_subcommand(subparsers) -> None: "Phase 12.5: language code passed to Presidio's NLP engine when --pii-ml is on " "(default: 'en'). Presidio raises a typed exception if no engine is registered for " "the requested language — surface it to the operator instead of silently " - "running an English NER on a non-English corpus. Set to e.g. 'de' on a German " - "corpus, or 'xx' (multilingual fallback) for a language without a dedicated spaCy " - "model such as Turkish, AND make sure the matching spaCy model is installed." + "running an English NER on a non-English corpus. Set to e.g. 'de' on a German corpus, or 'xx' (multilingual fallback) for a language " + "without a spaCy model such as Turkish, AND install the matching spaCy model." ), ) p.add_argument( diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index e66dccd4..39414fd9 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -43,8 +43,7 @@ - 0 — success or a successful ``--check-policy`` report (report-not-gate semantic per design §10 Q5). - 1 — config error (unknown ``row-id``, no matching run-scoped target, - missing corpus, mutually exclusive flag combination, a ``--config`` - that cannot be loaded). + missing corpus, conflicting flags, an unloadable ``--config``). - 2 — runtime error (I/O, permission denied, atomic rename failure). """ @@ -1011,14 +1010,11 @@ def _legacy_staging_owned_by(legacy_dir: Path, run_id: str) -> bool: def _artefact_targets_for_run(base: Path, run_id: str) -> List[Path]: """Return ``compliance/`` files whose name embeds ``run_id`` as a token. - Any regular file directly under ``compliance/`` whose name carries - the run-id as a token is matched, whatever its extension. The - standard bundle ForgeLM writes uses fixed basenames - (``compliance_report.json`` ...) with no run-id, so it is never matched - (pinned by ``test_artefacts_kind_finds_nothing_in_the_standard_fixed_name_bundle``). - The token-boundary check (`_filename_contains_run_id`) defends against - short-run-id false positives where a substring match would - accidentally pick up files belonging to a different run. + Any regular file directly under ``compliance/`` whose name carries the + run-id as a token matches, whatever its extension. The standard bundle + (fixed basenames such as ``compliance_report.json``) has no run-id and is + never matched. The token-boundary check (`_filename_contains_run_id`) + defends against short-run-id false positives on other runs' files. """ compliance_dir = base / "compliance" if not compliance_dir.is_dir(): diff --git a/site/js/translations.js b/site/js/translations.js index dc29d416..3222aaec 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -1817,9 +1817,9 @@ "wizard.retention.staging.label": "Staging horizon (days)", "wizard.retention.staging.hint": "Age at which forgelm purge --check-policy reports a final_model.staging.*/ (or legacy final_model.staging/) directory as overdue; nothing is deleted automatically. Default 7.", "wizard.retention.ephemeral.label": "Ephemeral artefacts (days)", - "wizard.retention.ephemeral.hint": "Age at which forgelm purge --check-policy reports compliance/ and data_audit_report.json as overdue. Default 90.", + "wizard.retention.ephemeral.hint": "Days after the audit log's first event (file age if there is no log) at which forgelm purge --check-policy reports compliance/ and data_audit_report.json as overdue. Default 90.", "wizard.retention.raw_documents.label": "Raw documents (days)", - "wizard.retention.raw_documents.hint": "Age at which forgelm purge --check-policy reports raw_documents/ or ingestion_output/ as overdue. Default 90.", + "wizard.retention.raw_documents.hint": "Days after the audit log's first event (file age if there is no log) at which forgelm purge --check-policy reports raw_documents/ or ingestion_output/ as overdue. Default 90.", "wizard.retention.enforce.label": "Policy mode (reserved)", "wizard.retention.enforce.hint": "Reserved: no ForgeLM command acts on this yet, so all three modes behave the same. Retention horizons are only reported by forgelm purge --check-policy; nothing is deleted automatically.", "wizard.retention.enforce.log_only": "log_only — default (no command acts on it)", @@ -2215,9 +2215,9 @@ "wizard.retention.staging.label": "Staging ufku (gün)", "wizard.retention.staging.hint": "forgelm purge --check-policy bir final_model.staging.*/ (veya eski final_model.staging/) dizinini hangi yaşta süresi geçmiş olarak raporlasın; hiçbir şey otomatik silinmez. Varsayılan 7.", "wizard.retention.ephemeral.label": "Geçici artefact'lar (gün)", - "wizard.retention.ephemeral.hint": "forgelm purge --check-policy compliance/ ve data_audit_report.json'u hangi yaşta süresi geçmiş olarak raporlasın. Varsayılan 90.", + "wizard.retention.ephemeral.hint": "forgelm purge --check-policy, audit log'un ilk olayından itibaren (audit log yoksa dosya yaşı) kaç gün sonra compliance/ ve data_audit_report.json'u süresi geçmiş olarak raporlasın. Varsayılan 90.", "wizard.retention.raw_documents.label": "Ham dokümanlar (gün)", - "wizard.retention.raw_documents.hint": "forgelm purge --check-policy raw_documents/ veya ingestion_output/ dizinini hangi yaşta süresi geçmiş olarak raporlasın. Varsayılan 90.", + "wizard.retention.raw_documents.hint": "forgelm purge --check-policy, audit log'un ilk olayından itibaren (audit log yoksa dosya yaşı) kaç gün sonra raw_documents/ veya ingestion_output/ dizinini süresi geçmiş olarak raporlasın. Varsayılan 90.", "wizard.retention.enforce.label": "Politika modu (ayrılmış)", "wizard.retention.enforce.hint": "Ayrılmış: hiçbir ForgeLM komutu bunu henüz kullanmaz, bu yüzden üç mod da aynı davranır. Saklama ufukları yalnızca forgelm purge --check-policy ile raporlanır; hiçbir şey otomatik silinmez.", "wizard.retention.enforce.log_only": "log_only — varsayılan (hiçbir komut kullanmaz)", @@ -2609,9 +2609,9 @@ "wizard.retention.staging.label": "Staging-Horizont (Tage)", "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging.*/-Verzeichnis (oder das ältere final_model.staging/) als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", "wizard.retention.ephemeral.label": "Ephemere Artefakte (Tage)", - "wizard.retention.ephemeral.hint": "Ab welchem Alter forgelm purge --check-policy compliance/ und data_audit_report.json als überfällig meldet. Default 90.", + "wizard.retention.ephemeral.hint": "Nach wie vielen Tagen seit dem ersten Ereignis im Audit-Log (ohne Audit-Log: Dateialter) forgelm purge --check-policy compliance/ und data_audit_report.json als überfällig meldet. Default 90.", "wizard.retention.raw_documents.label": "Rohdokumente (Tage)", - "wizard.retention.raw_documents.hint": "Ab welchem Alter forgelm purge --check-policy raw_documents/ oder ingestion_output/ als überfällig meldet. Default 90.", + "wizard.retention.raw_documents.hint": "Nach wie vielen Tagen seit dem ersten Ereignis im Audit-Log (ohne Audit-Log: Dateialter) forgelm purge --check-policy raw_documents/ oder ingestion_output/ als überfällig meldet. Default 90.", "wizard.retention.enforce.label": "Richtlinienmodus (reserviert)", "wizard.retention.enforce.hint": "Reserviert: Noch wertet kein ForgeLM-Befehl dies aus, daher verhalten sich alle drei Modi gleich. Aufbewahrungshorizonte meldet nur forgelm purge --check-policy; nichts wird automatisch gelöscht.", "wizard.retention.enforce.log_only": "log_only — Default (kein Befehl wertet es aus)", @@ -3003,9 +3003,9 @@ "wizard.retention.staging.label": "Horizon staging (jours)", "wizard.retention.staging.hint": "À partir de quel âge forgelm purge --check-policy signale un répertoire final_model.staging.*/ (ou l'ancien final_model.staging/) comme dépassé ; rien n'est supprimé automatiquement. Default 7.", "wizard.retention.ephemeral.label": "Artefacts éphémères (jours)", - "wizard.retention.ephemeral.hint": "À partir de quel âge forgelm purge --check-policy signale compliance/ et data_audit_report.json comme dépassés. Default 90.", + "wizard.retention.ephemeral.hint": "Après combien de jours depuis le premier événement du log d'audit (à défaut de log : l'âge du fichier) forgelm purge --check-policy signale compliance/ et data_audit_report.json comme dépassés. Default 90.", "wizard.retention.raw_documents.label": "Documents bruts (jours)", - "wizard.retention.raw_documents.hint": "À partir de quel âge forgelm purge --check-policy signale raw_documents/ ou ingestion_output/ comme dépassés. Default 90.", + "wizard.retention.raw_documents.hint": "Après combien de jours depuis le premier événement du log d'audit (à défaut de log : l'âge du fichier) forgelm purge --check-policy signale raw_documents/ ou ingestion_output/ comme dépassés. Default 90.", "wizard.retention.enforce.label": "Mode de politique (réservé)", "wizard.retention.enforce.hint": "Réservé : aucune commande ForgeLM n'agit encore dessus, donc les trois modes se comportent de la même façon. Les horizons de rétention ne sont signalés que par forgelm purge --check-policy ; rien n'est supprimé automatiquement.", "wizard.retention.enforce.log_only": "log_only — défaut (aucune commande n'en tient compte)", @@ -3397,9 +3397,9 @@ "wizard.retention.staging.label": "Horizonte staging (días)", "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta un directorio final_model.staging.*/ (o el antiguo final_model.staging/) como vencido; no se elimina nada automáticamente. Default 7.", "wizard.retention.ephemeral.label": "Artefactos efímeros (días)", - "wizard.retention.ephemeral.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta compliance/ y data_audit_report.json como vencidos. Default 90.", + "wizard.retention.ephemeral.hint": "Tras cuántos días desde el primer evento del log de auditoría (sin log: la antigüedad del archivo) forgelm purge --check-policy reporta compliance/ y data_audit_report.json como vencidos. Default 90.", "wizard.retention.raw_documents.label": "Documentos crudos (días)", - "wizard.retention.raw_documents.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta raw_documents/ o ingestion_output/ como vencidos. Default 90.", + "wizard.retention.raw_documents.hint": "Tras cuántos días desde el primer evento del log de auditoría (sin log: la antigüedad del archivo) forgelm purge --check-policy reporta raw_documents/ o ingestion_output/ como vencidos. Default 90.", "wizard.retention.enforce.label": "Modo de política (reservado)", "wizard.retention.enforce.hint": "Reservado: ningún comando de ForgeLM actúa aún sobre esto, así que los tres modos se comportan igual. Los horizontes de retención solo los informa forgelm purge --check-policy; nada se elimina automáticamente.", "wizard.retention.enforce.log_only": "log_only — default (ningún comando lo usa)", @@ -3789,11 +3789,11 @@ "wizard.retention.audit_log.label": "Audit log 保留(天)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已超期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", "wizard.retention.staging.label": "Staging 期限(天)", - "wizard.retention.staging.hint": "final_model.staging.*/(或旧的 final_model.staging/)目录存在多少天后,forgelm purge --check-policy 会将其报告为已超期;不会自动删除任何内容。默认 7。", + "wizard.retention.staging.hint": "自该运行的第一条审计事件起,final_model.staging.*/(或旧的 final_model.staging/)目录满多少天后,forgelm purge --check-policy 会将其报告为已超期;不会自动删除任何内容。默认 7。", "wizard.retention.ephemeral.label": "临时 artefact(天)", - "wizard.retention.ephemeral.hint": "compliance/ 和 data_audit_report.json 存在多少天后,forgelm purge --check-policy 会将其报告为已超期。默认 90。", + "wizard.retention.ephemeral.hint": "自审计日志第一条事件起(无审计日志时按文件修改时间)满多少天后,forgelm purge --check-policy 会将 compliance/ 和 data_audit_report.json 报告为已超期。默认 90。", "wizard.retention.raw_documents.label": "原始文档(天)", - "wizard.retention.raw_documents.hint": "raw_documents/ 或 ingestion_output/ 存在多少天后,forgelm purge --check-policy 会将其报告为已超期。默认 90。", + "wizard.retention.raw_documents.hint": "自审计日志第一条事件起(无审计日志时按文件修改时间)满多少天后,forgelm purge --check-policy 会将 raw_documents/ 或 ingestion_output/ 报告为已超期。默认 90。", "wizard.retention.enforce.label": "策略模式(保留)", "wizard.retention.enforce.hint": "保留项:目前没有任何 ForgeLM 命令使用它,因此三种模式行为相同。保留期限仅由 forgelm purge --check-policy 报告;不会自动删除任何内容。", "wizard.retention.enforce.log_only": "log_only — 默认(没有命令使用它)", diff --git a/tests/site/site_translations.test.cjs b/tests/site/site_translations.test.cjs index ffc7b06c..bc806415 100644 --- a/tests/site/site_translations.test.cjs +++ b/tests/site/site_translations.test.cjs @@ -43,7 +43,7 @@ test('claims the code does not back stay out of the site copy', () => { /surface every row/, /drops a data-subject's rows/], tr: [/ikili[- ]kontrol/i, /otomatik siler/, /GDPR uyumlu TTL/, /erişim-talebi tarafını karşılar/, /Slack, Teams, Discord veya/, /Slack \/ Teams \/ Discord/, /upstream'de uygulanır/, /her ikisi de audit log'a bağlı/, - /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/, /adli olarak bağlanır/, + /EU AI Act rehberiyle uyumlu/, /reddedilen modeller ne kadar tutulsun/, /adli olarak .{0,30}bağlanır/, /geçtiği her split/, /yüzeye çıkarır/, /satırlarını eğitim corpus'undan düşürür/], }; diff --git a/tests/test_check_usermanual_schema_drift.py b/tests/test_check_usermanual_schema_drift.py index 5de0620a..5614ec71 100644 --- a/tests/test_check_usermanual_schema_drift.py +++ b/tests/test_check_usermanual_schema_drift.py @@ -98,6 +98,7 @@ class ForgeConfig(BaseModel): data: DataConfig = Field(description="data block") leaf: Optional[LeafConfig] = Field(default=None, description="top-level leaf block") pipeline: Optional[PipelineConfig] = Field(default=None, description="pipeline block") + metadata: Dict[str, Any] = Field(default={}, description="opaque top-level map") """ @@ -158,7 +159,7 @@ def test_forward_ref_string_resolves(self, tool, synthetic_schema): class TestBuildSchemaMap: def test_forge_config_top_level_matches_declared_blocks(self, synthetic_schema): top = synthetic_schema["ForgeConfig"] - assert set(top.keys()) == {"model", "training", "data", "leaf", "pipeline"} + assert set(top.keys()) == {"model", "training", "data", "leaf", "pipeline", "metadata"} def test_alias_indexed_alongside_field_name(self, synthetic_schema): training = synthetic_schema["TrainingConfig"] @@ -405,6 +406,57 @@ def test_main_fails_strict_when_only_a_stale_row_remains( assert "_KNOWN_OPEN: ops/a.md gone" in out and "delete the row" in out assert tool.main([]) == 0 + def test_an_opaque_top_level_field_is_not_an_unknown_block(self, tool, synthetic_schema): + top = synthetic_schema["ForgeConfig"] + snippet = self._snippet(tool, "metadata:\n anything: 1\n") + assert tool.unknown_top_level_keys(snippet, top) == frozenset() + assert tool.check_snippet(snippet, synthetic_schema, top) == [] + + def test_a_row_with_extra_keys_does_not_exempt_a_trimmed_snippet(self, tool, synthetic_schema, tmp_path): + rows = {("ops/a.md", frozenset({"jobs", "name"})): "GitHub Actions"} + for lang in ("en", "tr"): + self._write(tmp_path, lang, "ops/a.md", "jobs: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], rows, {}) + assert [sorted(u.keys) for u in result.unknown_blocks] == [["jobs"], ["jobs"]] + assert [(row.page, row.keys) for row in result.stale_rows] == [("ops/a.md", frozenset({"jobs", "name"}))] + + def test_a_page_directly_under_a_language_directory_is_scanned(self, tool, synthetic_schema, tmp_path): + for lang in ("en", "tr"): + self._write(tmp_path, lang, "flat.md", "chat: {}\n") + result = tool.scan(tmp_path, synthetic_schema, synthetic_schema["ForgeConfig"], {}, {}) + assert [u.path.relative_to(tmp_path).as_posix() for u in result.unknown_blocks] == ["en/flat.md", "tr/flat.md"] + + def test_main_strict_fails_on_a_nested_drift_alone_and_names_file_line_and_key( + self, tool, tmp_path, monkeypatch, capsys + ): + config_path = tmp_path / "config.py" + config_path.write_text(_SYNTHETIC_CONFIG, encoding="utf-8") + root = tmp_path / "usermanuals" + for lang in ("en", "tr"): + self._write(root, lang, "ops/a.md", "training:\n bogus: 1\n") + monkeypatch.setattr(tool, "CONFIG_PATH", config_path) + monkeypatch.setattr(tool, "USERMANUAL_ROOT", root) + monkeypatch.setattr(tool, "REPO_ROOT", tmp_path) + monkeypatch.setattr(tool, "_NON_FORGELM_YAML", {}) + monkeypatch.setattr(tool, "_KNOWN_OPEN", {}) + assert tool.main(["--strict"]) == 1 + out = capsys.readouterr().out + assert "usermanuals/en/ops/a.md:3 training.bogus (no such field on the ForgeConfig schema)" in out + assert tool.main([]) == 0 + + def test_an_empty_tree_fails_in_both_modes_even_with_empty_tables(self, tool, tmp_path, monkeypatch, capsys): + config_path = tmp_path / "config.py" + config_path.write_text(_SYNTHETIC_CONFIG, encoding="utf-8") + root = tmp_path / "usermanuals" + root.mkdir() + monkeypatch.setattr(tool, "CONFIG_PATH", config_path) + monkeypatch.setattr(tool, "USERMANUAL_ROOT", root) + monkeypatch.setattr(tool, "_NON_FORGELM_YAML", {}) + monkeypatch.setattr(tool, "_KNOWN_OPEN", {}) + assert tool.main(["--strict"]) == 1 + assert "no yaml blocks found" in capsys.readouterr().err + assert tool.main([]) == 1 + def test_main_strict_exits_one_on_an_unlisted_block_and_zero_in_advisory_mode( self, tool, synthetic_schema, tmp_path, monkeypatch, capsys ): diff --git a/tests/test_config.py b/tests/test_config.py index 70296912..32fb8b74 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -943,7 +943,7 @@ def test_the_description_does_not_promise_a_gate(self): from forgelm.config import RetentionConfig description = RetentionConfig.model_fields["enforce"].description or "" - assert "no command reads it" in description + assert "no command acts on it" in description assert "EXIT_EVAL_FAILURE" not in description and "aborts" not in description def test_canonical_retention_staging_ttl_days_still_works(self): diff --git a/tests/test_gdpr_erasure.py b/tests/test_gdpr_erasure.py index 3e34ff42..d66ef402 100644 --- a/tests/test_gdpr_erasure.py +++ b/tests/test_gdpr_erasure.py @@ -348,10 +348,12 @@ def test_only_the_purge_reader_names_the_legacy_staging_marker(self) -> None: 1 for n in ast.walk(fn) if isinstance(n, ast.Constant) and n.value == "staging_run_id" ) reader_calls |= { - n.func.attr for n in ast.walk(fn) if isinstance(n, ast.Call) and isinstance(n.func, ast.Attribute) + n.func.attr if isinstance(n.func, ast.Attribute) else getattr(n.func, "id", "?") + for n in ast.walk(fn) + if isinstance(n, ast.Call) } assert total == inside_reader == 1, "the marker name must appear only in _legacy_staging_owned_by" - assert reader_calls <= {"is_file", "exists", "read_text", "strip"}, ( + assert reader_calls <= {"is_file", "read_text", "strip"}, ( f"the marker reader must stay read-only: {reader_calls}" ) diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 308241c3..6deab160 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -93,12 +93,12 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: r"17\(3\)\(b\)\s+preserves", r"17\(3\)\(b\)\s+(?:exemption|defence)", r"17\(3\)\(b\)\s+records", - r"17\(3\)\(b\) savunması", + r"17\(3\)\(b\)\s+savunması", r"17\(3\)\(b\)\s+audit\s*/\s*muhasebe", - r"17\(3\)\(b\) kayıt", - r"append-only (?:Article|Madde) 17\(3\)\(b\)", + r"17\(3\)\(b\)\s+kayıt", + r"append-only\s+(?:Article|Madde)\s+17\(3\)\(b\)", r"5\s+years\s+default\s+\(Article\s+5\(1\)\(e\)\)", - r"varsayılan 5 yıl \(Madde 5\(1\)\(e\)\)", + r"varsayılan\s+5\s+yıl\s+\(Madde\s+5\(1\)\(e\)\)", r"legal-obligation\s+record\s+under", r"compliant\s+out\s+of\s+the\s+box", r"years\s+\(Article\s+12\s+record-keeping\)|Article\s+12\s+record-keeping\s+baseline", @@ -159,9 +159,9 @@ def test_reject_does_not_claim_discard_staging(self, lang: str) -> None: ) _RETIRED_PII_CITATIONS = ( - r"(?:Art(?:icle|\.)?|Madde|Md\.)\s*9\b", + r"\b(?:Art(?:icle|\.)?|Madde|Md\.)\s*9\b", r"GDPR[\s\u2010-\u2015-]*(?:mandat|requir)", - r"regulat\w*[\s\u2010-\u2015-]*mandated", + r"regulation[\s\u2010-\u2015-]+mandated", r"(?:Article|Madde)\s+10\b[^.\n]{0,60}(?:cares|concerned|önemsediği)", r"consensus\s+regulatory", r"konsensüs\s+regülatif", @@ -201,7 +201,7 @@ def test_every_article_17_3_b_mention_is_conditional(rel: str) -> None: text = (_REPO_ROOT / rel).read_text(encoding="utf-8") unqualified = [ text[max(0, m.start() - 60) : m.end() + 60] - for m in re.finditer(r"17\(3\)\(b\)", text) + for m in re.finditer(r"17\s*\(3\)\s*\(b\)", text) if not _CONDITIONAL_17_3_B.search(text[max(0, m.start() - 250) : m.end() + 250]) ] assert unqualified == [], f"{rel}: Article 17(3)(b) mentioned without a limiting qualifier: {unqualified}" From 8282a51119c304fb01e39a4a0d5a4b9458204492 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:16:46 +0300 Subject: [PATCH 13/25] docs: release note, agent guides and Phase 16 roadmap for the GDPR-claims fix - CHANGELOG [Unreleased]: the contract this fix restores (a deployer can copy a documented command or YAML block, and quote a documented limit in a DPIA or an access reply, without it being untrue) with the per-topic corrections, and the schema-drift guard extension under Changed. - CLAUDE.md / AGENTS.md and docs/standards/documentation.md: the user-manual schema-drift guard also fails on a top-level block ForgeConfig rejects unless the page and key set are listed with a reason, and on a listed row that no longer matches. - docs/roadmap/phase-16-trust-surface-hardening.md (S15): records that the fenced-YAML half of the schema-prose guard landed and what is still open. Fixes #101 Co-Authored-By: Claude Sonnet 5.5 --- AGENTS.md | 6 +++- CHANGELOG.md | 36 +++++++++++++++++++ CLAUDE.md | 6 +++- .../phase-16-trust-surface-hardening.md | 6 ++++ docs/standards/documentation.md | 2 +- 5 files changed, 53 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ffe04563..d9d99a8c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -231,7 +231,11 @@ Default workflow for a non-trivial change: `check_usermanual_schema_drift.py --strict` — validates that every fenced YAML key under `docs/usermanuals/` resolves against the real `ForgeConfig` schema, catching fabricated-field examples that would - fail `--dry-run`). The four after the import-origin guard are the + fail `--dry-run`; it also fails on a top-level block `ForgeConfig` does + not have — a `chat:` or `output:` example — unless the page and exact + key set are listed with a reason as another tool's YAML or as a + known-open block, and on a listed row that no longer matches). The four + after the import-origin guard are the historical gauntlet; the three doc guards (Wave 3 / Wave 4 / Wave 5 additions) catch bilingual structural drift, broken markdown anchors, and CLI ↔ docs diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c1aa61b..78e45cad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,11 @@ All notable changes to ForgeLM are documented here. ### Changed +- `tools/check_usermanual_schema_drift.py` now also fails on a top-level block in a user-manual YAML example that `ForgeConfig` + does not have (a `chat:` or `output:` block that Pydantic rejects with `extra_forbidden`), instead of skipping it as "someone + else's YAML". Another tool's YAML (a CI workflow, a compose file, a vLLM config) is listed in the guard with a reason, a + fabricated block that a tracked issue still owns goes in a second table, and a listed row that no longer matches a snippet in + every language fails, so the tables cannot become a blanket allowlist (CR-A17a-007, #101). - Admit newer majors of five dependencies: `huggingface_hub` 2, `pypdf` 6 and `xxhash` 4 (the `ingestion` extra), `datasketch` 2 (the `ingestion-scale` extra) and `pytest-randomly` 4 and 5 (the `dev` extra). The ingestion, data-audit and doctor test modules pass on pypdf 6.21.1, xxhash 4.0.1 and datasketch 2.0.0 with the same results as on the previous majors; CI installs datasketch 2.0.0 and @@ -112,6 +117,37 @@ All notable changes to ForgeLM are documented here. ### Fixed +- Make the GDPR, erasure, retention and PII documentation say what ForgeLM does, so a deployer can copy a documented command or + YAML block, and quote a documented limit in a DPIA or an access reply, without it being untrue (#101). The GDPR / KVKK manual + page was rewritten earlier in this cycle; this completes it: + - Article 17(3)(b) is no longer presented as preserving the audit log as a class. It disapplies erasure only to the extent a + specific Union or Member State obligation requires the processing, and keeping entries needs a documented basis (the guides, + reference, user manual, design note, `forgelm purge --help`, `RetentionConfig` and `config_template.yaml`). + - The retention horizons are described as operator-set limits that `forgelm purge --check-policy` reports and never enforces; the + defaults (5 years for the audit log, 7 / 90 / 90 days) are ForgeLM choices, not legal periods. `staging_ttl_days` is the age + after which a staging directory is reported (from the run's first audit event, else the log's first event, else mtime), not a + scheduled cleanup after `forgelm reject`; the ephemeral and raw-document rows name the directories the scan covers; the + `--check-policy` JSON `age_source` is `audit` or `mtime`. + - `forgelm purge --kind artefacts` is documented as what it does: it matches files under `compliance/` whose names contain the + run id, so it finds nothing in the standard bundle (fixed basenames such as `compliance_report.json`), emits + `data.erasure_failed` (`NoMatchingArtefacts`) and exits 1 without deleting anything. The legacy `final_model.staging/` is + purged only with a `staging_run_id` marker that ForgeLM never writes. The code half of the artefacts limitation stays open. + - With `FORGELM_AUDIT_SECRET` set, `forgelm purge` XORs it into the identifier salt, so rotating it changes `target_id` + digests (the user manual said they were stable); the external-copies warning is row mode only; run-scoped deletion is not + atomic; `--output-dir` defaults to the current directory in run mode; erasure notices to recipients are Article 19. + - The PII severity tiers cite GDPR Art. 87 and Art. 32 with ENISA (not Art. 9), and the regex detector set is described as + structured identifiers that are personal data under Art. 4(1), not "what Article 10 cares about" or "GDPR-mandated". + - The GDPR page's access-request example passes `--output-dir` and shows the real output line, and points to `forgelm audit + --pii-ml` (detection only) for names. The site's home GDPR card says `forgelm reverse-pii` locates rows (an input to an + Article 15 response) and `forgelm purge` removes them from training corpora (one step of Article 17 erasure; trained weights + are not changed), in all six languages; the wizard's retention hints no longer say the 5-year default "matches EU AI Act + guidance". The German, French, Spanish and Chinese strings have no native review. + - The access-request page says what `forgelm reverse-pii` cannot see: matching is case-sensitive and compares raw file text, + `--check-policy` needs `--output-dir `, and a model that cannot be shown to be anonymous (EDPB Opinion + 28/2024) can bring access and erasure requests to the model itself. The wizard's retention hints and the site's access and + erasure cards (six languages) say what `forgelm purge --check-policy`, `forgelm reverse-pii` and `forgelm purge` do. + - The `forgelm chat` page shows the real flags instead of a `chat:` YAML block, and the GPU-cost page the real + `training.gpu_cost_per_hour` field instead of an `output.cost_tracking` block that does not exist. - Fail closed when judge replies contain multiple score objects or duplicate JSON keys, and reject non-object replies, including a score object that only appears inside an array or after an undecodable `{`, with the existing judge-failure sentinel. The rejection reason is logged without echoing the reply diff --git a/CLAUDE.md b/CLAUDE.md index a41fd170..1855daf8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -231,7 +231,11 @@ Default workflow for a non-trivial change: `check_usermanual_schema_drift.py --strict` — validates that every fenced YAML key under `docs/usermanuals/` resolves against the real `ForgeConfig` schema, catching fabricated-field examples that would - fail `--dry-run`). The four after the import-origin guard are the + fail `--dry-run`; it also fails on a top-level block `ForgeConfig` does + not have — a `chat:` or `output:` example — unless the page and exact + key set are listed with a reason as another tool's YAML or as a + known-open block, and on a listed row that no longer matches). The four + after the import-origin guard are the historical gauntlet; the three doc guards (Wave 3 / Wave 4 / Wave 5 additions) catch bilingual structural drift, broken markdown anchors, and CLI ↔ docs diff --git a/docs/roadmap/phase-16-trust-surface-hardening.md b/docs/roadmap/phase-16-trust-surface-hardening.md index c3b719f2..6b1d5441 100644 --- a/docs/roadmap/phase-16-trust-surface-hardening.md +++ b/docs/roadmap/phase-16-trust-surface-hardening.md @@ -760,6 +760,12 @@ committed, then the same with Sonnet, then the next step. that would have caught it**, and they share one hard constraint: EN, TR and all six site locales change in the same commit. + *Landed with the GDPR-claims fix:* `check_usermanual_schema_drift.py` now + also fails on a fenced-YAML top-level block `ForgeConfig` does not have + (the `chat:` / `output:` / `gdpr.md` blocks), with a reasoned exemption + table and stale-row detection. Still open here: the inline-code + config-key-in-prose half, and the two known-open blocks the guard lists. + Regulation (EU) 2026/1744 moves Annex III high-risk obligations to **2 December 2027** and Annex I to **2 August 2028**; five current surfaces still tie "full enforcement" to 2 August 2026, and the original wording diff --git a/docs/standards/documentation.md b/docs/standards/documentation.md index a75f267d..e4b593d5 100644 --- a/docs/standards/documentation.md +++ b/docs/standards/documentation.md @@ -230,7 +230,7 @@ See [localization.md](localization.md) for the full rule. Summary: - User-facing docs in `docs/` root, `docs/reference/`, `docs/roadmap.md`, `docs/qms/`, and `docs/usermanuals/{en,tr}/` have `-tr.md` (or `tr/`-tree) mirrors. - Internal docs (`standards/`, `marketing/`, `analysis/`, `design/`) are English only. -- Mirrors must stay in **H2 + H3 + H4 structural sync** (Wave 3 / Faz 24 formalisation): every heading at depth 2-4 in the EN file must have a matching same-position heading in the TR file. Cosmetic wording differences inside a section are fine; structural divergence (sections added in one, missing in other; reordered headings) is a bug. The `tools/check_bilingual_parity.py --strict` gate fails CI on any mismatch — its scope covers every hand-registered EN/TR pair (across `docs/guides/`, `docs/qms/`, `docs/reference/`) plus the auto-discovered `docs/usermanuals/` pairs; the pair count self-updates as mirrors are added, so it is not pinned here. The companion `tools/check_bilingual_code_blocks.py --strict` gate additionally enforces matching fenced-block counts and per-block YAML-key sets across the same pairs, and `tools/check_usermanual_schema_drift.py --strict` validates that every fenced YAML key under `docs/usermanuals/` resolves against the real `ForgeConfig` schema. +- Mirrors must stay in **H2 + H3 + H4 structural sync** (Wave 3 / Faz 24 formalisation): every heading at depth 2-4 in the EN file must have a matching same-position heading in the TR file. Cosmetic wording differences inside a section are fine; structural divergence (sections added in one, missing in other; reordered headings) is a bug. The `tools/check_bilingual_parity.py --strict` gate fails CI on any mismatch — its scope covers every hand-registered EN/TR pair (across `docs/guides/`, `docs/qms/`, `docs/reference/`) plus the auto-discovered `docs/usermanuals/` pairs; the pair count self-updates as mirrors are added, so it is not pinned here. The companion `tools/check_bilingual_code_blocks.py --strict` gate additionally enforces matching fenced-block counts and per-block YAML-key sets across the same pairs, and `tools/check_usermanual_schema_drift.py --strict` validates that every fenced YAML key under `docs/usermanuals/` resolves against the real `ForgeConfig` schema and that no snippet uses a top-level block `ForgeConfig` rejects (another tool's YAML, such as a CI workflow, is listed in the guard with a reason). ## Cross-linking from bilingual content to EN-only docs From 14eb0447a9dad8b86afd98ba531c98fcc2087e03 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 02:34:38 +0300 Subject: [PATCH 14/25] =?UTF-8?q?docs:=20CodeRabbit=20follow-ups=20?= =?UTF-8?q?=E2=80=94=20a=20CI=20gate=20that=20fails,=20digest/salt=20condi?= =?UTF-8?q?tions,=20exit=20codes,=20a=20reproducible=20example?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The "pipe to `jq '.violations | length'`" CI-gate advice printed a count and exited 0 whatever it was; it is now `jq -e '.success and (.violations | length == 0)'` (guides, purge reference, design note, `_purge.py` docstring). - The purge/reverse-pii digests match only for the same value, the same `--output-dir` and the same salt source; with `FORGELM_AUDIT_SECRET` set at purge time, query with `--salt-source env_var` and the same secret. - An unwritable `--audit-dir` is a runtime error (exit 2), not a config error. - The pii-ml quick example runs the default English model, which never reports `organization`; the example no longer shows one and the note says how to get it (`--pii-ml-language xx`). - The Article 17(3)(b) test now ties the qualifier to the claim's own sentence (or to a next sentence that continues it); an unrelated "only to staff" nearby no longer satisfies it, and near-miss cases are pinned both ways. Refs #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 4 +- docs/guides/gdpr_erasure-tr.md | 6 ++- docs/guides/gdpr_erasure.md | 8 +-- docs/reference/purge_subcommand-tr.md | 2 +- docs/reference/purge_subcommand.md | 2 +- .../usermanuals/en/compliance/gdpr-erasure.md | 4 +- docs/usermanuals/en/data/pii-ml.md | 7 ++- .../usermanuals/tr/compliance/gdpr-erasure.md | 4 +- docs/usermanuals/tr/data/pii-ml.md | 7 ++- forgelm/cli/subcommands/_purge.py | 2 +- .../test_usermanual_reference_consistency.py | 51 +++++++++++++++++-- 11 files changed, 70 insertions(+), 27 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 59e346d3..1dd57989 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -210,7 +210,7 @@ Standard ForgeLM 0/1/2/3/4 contract — all codes from `forgelm.cli._exit_codes` | 3 | `EXIT_EVAL_FAILURE` | Reserved for the trainer pre-flight gate when `enforce: block_on_excess` is configured (see §3.4 in the spirit of the master `error-handling.md` exit-code contract). **Not** used by `forgelm purge --check-policy` — that path is a report, not a gate; see §10 Q5 + the paragraph below. | | 4 | `EXIT_AWAITING_APPROVAL` | Reserved for its trainer-pipeline meaning per `docs/standards/error-handling.md` ("Training + evals passed, but `require_human_approval: true` — staged, awaiting human sign-off"). **Not** reused by `forgelm purge`. Since the shipped `forgelm purge` is non-interactive (no prompt, no `--yes` flag), there is no "operator declined the prompt" path; consent is recorded in the operator-supplied `--justification` and the audit chain. | -**`--check-policy` always exits 0** (per §10 Q5 — the resolved decision: report-not-gate semantic). Operators who want a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves; CI then branches on that count rather than on the exit code. This keeps the public contract `docs/standards/error-handling.md` (0/1/2/3/4 trainer + 5 wizard-cancel) consistent across every ForgeLM subcommand: a code-3 from `forgelm purge` always means "trainer pre-flight gate failed", never "report found something". +**`--check-policy` always exits 0** (per §10 Q5 — the resolved decision: report-not-gate semantic). Operators who want a CI gate use `--output-format json` and pipe to `jq -e '.violations | length == 0'` themselves; CI then branches on that count rather than on the exit code. This keeps the public contract `docs/standards/error-handling.md` (0/1/2/3/4 trainer + 5 wizard-cancel) consistent across every ForgeLM subcommand: a code-3 from `forgelm purge` always means "trainer pre-flight gate failed", never "report found something". ### 4.4 Atomicity @@ -444,7 +444,7 @@ The closure plan §15.5 lists three open items that intersect Phase 21: ### Q5. Should `--check-policy` return non-zero when violations are found? -**Decision:** no. `--check-policy` is a *report*, not a *gate*. The CI gate is the trainer pre-flight (when `enforce: block_on_excess` is configured). Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves. +**Decision:** no. `--check-policy` is a *report*, not a *gate*. The CI gate is the trainer pre-flight (when `enforce: block_on_excess` is configured). Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and pipe to `jq -e '.violations | length == 0'` themselves. ### Q6. Should we delete the synthetic-data snapshots on row erasure? diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index bad0021a..3f64c6bc 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -68,7 +68,7 @@ $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. -**Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve `jq '.violations | length'`'e pipe'lar; bu, public exit-code kontratını `0/1/2/3/4` her ForgeLM subcommand'ında tutarlı tutar. +**Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist)'e pipe'lar; bu, public exit-code kontratını `0/1/2/3/4` her ForgeLM subcommand'ında tutarlı tutar. ## `retention:` config block'u @@ -146,7 +146,9 @@ Madde 15 erişim talepleri kendisi de subject'in verisini audit log'a sızdırmamalıdır. Salt'lı form, compliance reviewer'ın aynı subject için Madde 17 (purge) ve Madde 15 (reverse-pii) olaylarını cleartext görmeden ilişkilendirmesini sağlar (`--row-id` ve `--query`'ye aynı değeri -verdiğinizde digest'ler eşleşir). +verdiğinizde, aynı `--output-dir`'i ve aynı salt kaynağını kullandığınızda +digest'ler eşleşir: purge için `FORGELM_AUDIT_SECRET` ayarlıysa aynı secret ile +`--salt-source env_var` kullanarak sorgulayın). **Identifier tipleri** (`--type`): `literal` (varsayılan), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 481f9447..7fecd93f 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -68,7 +68,7 @@ $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs Read-only scan. Walks the output directory, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. -**Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves; this keeps the public exit-code contract `0/1/2/3/4` consistent across every ForgeLM subcommand. +**Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist) themselves; this keeps the public exit-code contract `0/1/2/3/4` consistent across every ForgeLM subcommand. ## The `retention:` config block @@ -144,8 +144,10 @@ access requests must not themselves leak the subject's data into the audit log. The salted form is a stable per-identifier fingerprint that lets a compliance reviewer correlate Article 17 (purge) and Article 15 (reverse-pii) events for the same subject (the digests match -when you pass the same value to `--row-id` and `--query`) without seeing -the cleartext. +when you pass the same value to `--row-id` and `--query`, use the same +`--output-dir`, and use the same salt source: if `FORGELM_AUDIT_SECRET` was set +for the purge, query with `--salt-source env_var` and the same secret) without +seeing the cleartext. **Identifier types** (`--type`): `literal` (default), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. All diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index 6b13d95b..9b9266f0 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -58,7 +58,7 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. -**Başarılı policy raporu her zaman 0 ile çıkar** (report-not-gate semantik, design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (1) ile çıkar — eksik, okunamaz veya Pydantic doğrulamasından geçemeyen explicit `--config` non-zero olarak yüzeyselleşir; böylece operatör "loader başarısız" durumunu "ihlal yok" sanmaz. CI gate kuran operatörler `--output-format json` kullanıp `jq '.violations | length'`'e pipe eder. +**Başarılı policy raporu her zaman 0 ile çıkar** (report-not-gate semantik, design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (1) ile çıkar — eksik, okunamaz veya Pydantic doğrulamasından geçemeyen explicit `--config` non-zero olarak yüzeyselleşir; böylece operatör "loader başarısız" durumunu "ihlal yok" sanmaz. CI gate kuran operatörler `--output-format json` kullanıp `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist)'e pipe eder. ## Salt çözümlemesi diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index 48d6ff06..dc9f64e6 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -58,7 +58,7 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai Read-only scan. Walks ``, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. -**Successful policy reports always exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit `EXIT_CONFIG_ERROR` (1) — an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as a non-zero so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq '.violations | length'` themselves. +**Successful policy reports always exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit `EXIT_CONFIG_ERROR` (1) — an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as a non-zero so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist) themselves. ## Salt resolution diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 138b5389..057b6a84 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -100,8 +100,8 @@ Both subcommands follow the project's `0/1/2/3/4` contract: | Code | Meaning | |---|---| | 0 | Success. `forgelm purge --check-policy` exits 0 even when violations are found (report-not-gate semantic). `forgelm reverse-pii` exits 0 even when no matches are found (Article 15 explicitly accepts "no records" as a valid answer). | -| 1 | Config error: bad flags, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded, unwritable `--audit-dir`. | -| 2 | Runtime error: I/O failure, permission denied, ReDoS SIGALRM timeout, atomic-rename failure. | +| 1 | Config error: bad flags, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded. | +| 2 | Runtime error: I/O failure, permission denied, unwritable `--audit-dir`, ReDoS SIGALRM timeout, atomic-rename failure. | Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of either subcommand's surface. diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index 03fa2483..e3c0a297 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -40,7 +40,7 @@ Without the spaCy model, `forgelm audit --pii-ml` raises a typed `ImportError` * $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml ✓ format: instructions (8,400 rows) ⚠ PII: 12 emails, 3 phone (regex layer) -⚠ PII (ML): 47 person, 18 organization, 9 location (Presidio) +⚠ PII (ML): 47 person, 9 location (Presidio) worst tier: medium ``` @@ -52,17 +52,16 @@ The Presidio findings merge into the same `pii_summary` and `pii_severity` block "email": 12, "phone": 3, "person": 47, - "organization": 18, "location": 9 }, "pii_severity": { - "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 12}, "worst_tier": "medium" } } ``` -Presidio's default English configuration ignores the `organization` label (it produces many false positives), so with `--pii-ml-language en` the `organization` count stays 0; the figures above assume a language model that reports it, for example `xx`. +Presidio's default English configuration ignores the `organization` label (it produces many false positives), so with the default `--pii-ml-language en` the `organization` count stays 0 and the example above shows none; a language model that reports it, for example `--pii-ml-language xx`, adds an `organization` count. ## Severity tiers diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 6e6d1d7a..6be876dc 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -100,8 +100,8 @@ Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kend | Kod | Anlamı | |---|---| | 0 | Başarı. İhlal bulunduğunda bile `forgelm purge --check-policy` 0 ile çıkar (report-not-gate semantik). Eşleşme bulunmadığında bile `forgelm reverse-pii` 0 ile çıkar (Madde 15 "kayıt yok"u geçerli yanıt olarak açıkça kabul eder). | -| 1 | Config hatası: kötü flag'ler, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`, yazılamaz `--audit-dir`. | -| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | +| 1 | Config hatası: kötü flag'ler, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`. | +| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, yazılamaz `--audit-dir`, ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın da yüzeyinin parçası değildir. diff --git a/docs/usermanuals/tr/data/pii-ml.md b/docs/usermanuals/tr/data/pii-ml.md index 415bac42..12edbf1c 100644 --- a/docs/usermanuals/tr/data/pii-ml.md +++ b/docs/usermanuals/tr/data/pii-ml.md @@ -40,7 +40,7 @@ spaCy modeli yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce** $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml ✓ format: instructions (8.400 satır) ⚠ PII: 12 email, 3 telefon (regex katmanı) -⚠ PII (ML): 47 person, 18 organization, 9 location (Presidio) +⚠ PII (ML): 47 person, 9 location (Presidio) en kötü kat: medium ``` @@ -52,17 +52,16 @@ Presidio bulguları aynı `pii_summary` ve `pii_severity` bloklarına disjoint k "email": 12, "phone": 3, "person": 47, - "organization": 18, "location": 9 }, "pii_severity": { - "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 12}, "worst_tier": "medium" } } ``` -Presidio'nun varsayılan İngilizce yapılandırması `organization` etiketini yok sayar (çok yanlış pozitif üretir); bu yüzden `--pii-ml-language en` ile `organization` sayısı 0 kalır. Yukarıdaki rakamlar bu etiketi raporlayan bir dil modelini (örneğin `xx`) varsayar. +Presidio'nun varsayılan İngilizce yapılandırması `organization` etiketini yok sayar (çok yanlış pozitif üretir); bu yüzden varsayılan `--pii-ml-language en` ile `organization` sayısı 0 kalır ve yukarıdaki örnek onu göstermez; bu etiketi raporlayan bir dil modeli (örneğin `--pii-ml-language xx`) bir `organization` sayısı ekler. ## Şiddet katmanları diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 39414fd9..4baf5fa0 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -1122,7 +1122,7 @@ def _run_purge_check_policy(args, output_format: str) -> None: Always exits 0 (per design §10 Q5: report-not-gate semantic). Operators wiring a CI gate use ``--output-format json`` and pipe - to ``jq '.violations | length'`` themselves. + to ``jq -e '.violations | length == 0'`` themselves. Strict config loading: ``--check-policy`` is the one purge mode where the operator explicitly asked for a retention report. A diff --git a/tests/test_usermanual_reference_consistency.py b/tests/test_usermanual_reference_consistency.py index 6deab160..ddbfa62f 100644 --- a/tests/test_usermanual_reference_consistency.py +++ b/tests/test_usermanual_reference_consistency.py @@ -196,17 +196,58 @@ def test_retired_pii_citations_stay_out(rel: str) -> None: assert found == [], f"{rel} still attributes the PII detector set or tiers to the wrong GDPR provision: {found}" +_ARTICLE_17_3_B = re.compile(r"17\s*\(3\)\s*\(b\)") +# A block ends at a blank line, a list item or a table row; a hard-wrapped newline inside a sentence does not end it. +_BLOCK_BREAK = re.compile(r"\n\s*\n|\n(?=\s*(?:[-*]\s|\d+\.\s|\|))") +# A sentence ends at ". " before a capital, backtick, asterisk, pipe or dash ("Art. 17" does not end one). +_SENTENCE_BREAK = re.compile(r"(?<=[.!?])\s+(?=[A-ZÇĞİÖŞÜ`*|\-])") +# The sentence after a mention only belongs to its claim when it picks the claim up ("It applies only to the extent ..."). +_CONTINUES_THE_CLAIM = re.compile(r"(?:It|Each|Both|They|This|These|That|Bu|Her|Her biri|Bunlar|Her ikisi)\b") + + +def unqualified_article_17_3_b_mentions(text: str) -> list: + """Return the Article 17(3)(b) mentions with no limiting qualifier in their own sentence. + + A qualifier in the next sentence counts only when that sentence continues the claim; one in an earlier or an + unrelated sentence, or in another list item, row or paragraph, does not. + """ + bad = [] + for block in _BLOCK_BREAK.split(text): + sentences = _SENTENCE_BREAK.split(block) + for index, sentence in enumerate(sentences): + if not _ARTICLE_17_3_B.search(sentence) or _CONDITIONAL_17_3_B.search(sentence): + continue + following = sentences[index + 1] if index + 1 < len(sentences) else "" + if _CONTINUES_THE_CLAIM.match(following.lstrip()) and _CONDITIONAL_17_3_B.search(following): + continue + bad.append(sentence.strip()[:120]) + return bad + + @pytest.mark.parametrize("rel", _GDPR_CLAIM_FILES) def test_every_article_17_3_b_mention_is_conditional(rel: str) -> None: text = (_REPO_ROOT / rel).read_text(encoding="utf-8") - unqualified = [ - text[max(0, m.start() - 60) : m.end() + 60] - for m in re.finditer(r"17\s*\(3\)\s*\(b\)", text) - if not _CONDITIONAL_17_3_B.search(text[max(0, m.start() - 250) : m.end() + 250]) - ] + unqualified = unqualified_article_17_3_b_mentions(text) assert unqualified == [], f"{rel}: Article 17(3)(b) mentioned without a limiting qualifier: {unqualified}" +@pytest.mark.parametrize( + ("text", "expected_bad"), + [ + ("Article 17(3)(b) applies only to the extent a specific obligation requires it.", 0), + ("Article 17(3)(b) can justify keeping entries. It applies only to the extent an obligation requires it.", 0), + ("Access is limited only to staff. Article 17(3)(b) exempts every audit log.", 1), + ("Article 17(3)(b) exempts every audit log.\n\nAccess is limited only to staff.", 1), + ("- Art. 17(3)(b) preserves audit records.\n- Retention applies only to the extent the user sets.", 1), + ("Madde 17(3)(b) audit kayıtlarını korur.", 1), + ("Madde 17(3)(b) yalnızca belirli bir yükümlülüğün gerektirdiği ölçüde uygulanır.", 0), + ("Article 17 (3) (b) is a blanket defence.", 1), + ], +) +def test_the_article_17_3_b_invariant_ties_the_qualifier_to_its_own_claim(text: str, expected_bad: int) -> None: + assert len(unqualified_article_17_3_b_mentions(text)) == expected_bad + + @pytest.mark.parametrize("rel", _PURGE_ARTEFACT_WARNING_FILES) def test_purge_docs_warn_that_artefacts_misses_the_standard_bundle(rel: str) -> None: # CR-A10-002 / DOC-D1B05-005: the exporter writes fixed basenames and the From 65b33054e0fac30523065509e05045373d61798e Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 03:26:24 +0300 Subject: [PATCH 15/25] fix(docs): digest-correlation rule, CI gate, exit codes, JSON envelope, pseudonymisation and the Article 15/17 framing A second, independent review of the GDPR / erasure documentation checked it against the running commands and found statements that were still wrong or that the first pass had introduced: - `reverse-pii --salt-source` is not a way to match purge digests: it switches to a hash-mask scan that looks for digests inside the corpus and finds nothing (exit 0) in a corpus that holds plaintext. The correlation rule is now stated once and identically everywhere (guide, both references, manual, deployer guide, incident SOP): same value, same `--output-dir`, same `FORGELM_AUDIT_SECRET` state, default plaintext mode; link requests to erasures by the ticket in `--justification` when ids differ. An end-to-end test runs both commands with and without a secret. - The CI-gate recipe passed on an unchecked directory (no `retention:` block, `{"success": false}`). It is now `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`, with the cases where `count` is absent explained. - Exit-code tables: an argument-parser error (two modes together, `--kind logs`) exits 2, other config errors 1; "always exits 0" now says what it means (a successful report, however many it finds). - The purge JSON envelope section is regenerated from real output: every envelope carries `success`, the row envelope has `row_id_hash`, `artefact_kind` takes the real values, paths follow `--output-dir`, and the failure example no longer echoes the identifier. - A salted digest is pseudonymisation, not anonymisation (Art. 4(5), recital 26) and the salt file sits next to the log: the guide, manual, deployer guide, reverse-pii reference and `--help` say so. - Art. 15 / 17 are described as inputs to / steps of a response everywhere (CLI help and epilog, cli.md, compliance summary, both references, manual capability table, site privacy page and feature titles in six languages), with a "complete response" paragraph (identity, one month, copy of the data, 15(4), 17(1)/17(3), Art. 19, Art. 12(4) reasons) and the Art. 19 qualifiers. - Art. 17(3)(b) text now carries the public-interest limb and "to which the controller is subject"; an entry may also stay because no Art. 17(1) ground applies; the AI Act examples are gone (a circular example, not yet applicable, and system logs are not the training audit log). - Retention: `0` stops the report (nothing is ever retained or deleted by it), compliance bundles and raw documents are aged from the audit log's first event, so a shared output directory can report a fresh bundle as overdue (documented); the wizard prompts say "0 = never report". - External-copies warning conditions (row mode, `--config`, an exported `url_env`), the `--kind artefacts` no-match message and `--help`, the `--dry-run` help, `--pii-ml` examples for the English default (no `organization`), `--pii-ml-language` advice (`xx`, not `tr`), the shared-output-directory caveat and the notebook's "redact names" claim. - Site Art. 10 card no longer claims bias detection; Turkish terminology and line references fixed. Fixes #101 Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 14 +++--- docs/guides/data_audit-tr.md | 8 ++-- docs/guides/data_audit.md | 8 ++-- docs/guides/gdpr_erasure-tr.md | 40 +++++++++------- docs/guides/gdpr_erasure.md | 38 ++++++++------- docs/guides/iso_soc2_deployer_guide-tr.md | 19 ++++---- docs/guides/iso_soc2_deployer_guide.md | 19 ++++---- docs/qms/sop_incident_response-tr.md | 25 ++++++---- docs/qms/sop_incident_response.md | 25 ++++++---- docs/reference/audit_event_catalog-tr.md | 2 +- docs/reference/audit_event_catalog.md | 2 +- docs/reference/compliance_summary-tr.md | 8 ++-- docs/reference/compliance_summary.md | 8 ++-- docs/reference/configuration-tr.md | 12 +++-- docs/reference/configuration.md | 12 +++-- docs/reference/purge_subcommand-tr.md | 48 ++++++++++++------- docs/reference/purge_subcommand.md | 48 ++++++++++++------- docs/reference/reverse_pii_subcommand-tr.md | 10 ++-- docs/reference/reverse_pii_subcommand.md | 10 ++-- .../usermanuals/en/compliance/gdpr-erasure.md | 29 +++++------ docs/usermanuals/en/compliance/gdpr.md | 4 +- docs/usermanuals/en/data/audit.md | 2 +- docs/usermanuals/en/data/pii-ml.md | 15 ++++-- docs/usermanuals/en/reference/cli.md | 4 +- .../usermanuals/tr/compliance/gdpr-erasure.md | 29 +++++------ docs/usermanuals/tr/compliance/gdpr.md | 4 +- docs/usermanuals/tr/data/audit.md | 2 +- docs/usermanuals/tr/data/pii-ml.md | 15 ++++-- docs/usermanuals/tr/reference/cli.md | 4 +- forgelm/cli/_parser.py | 20 ++++---- forgelm/cli/subcommands/_purge.py | 20 ++++---- forgelm/cli/subcommands/_reverse_pii.py | 9 ++-- forgelm/config.py | 12 ++--- forgelm/wizard/_collectors.py | 8 ++-- notebooks/ingestion_playground.ipynb | 4 +- site/compliance.html | 4 +- site/features.html | 4 +- site/js/translations.js | 48 +++++++++---------- tests/test_reverse_pii.py | 47 ++++++++++++++++++ 39 files changed, 379 insertions(+), 261 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 1dd57989..40e9dfdb 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -45,7 +45,7 @@ Phase 21 closes the gap with a real implementation. This Phase 20 document spec **Article 19** requires the controller to tell each recipient the data was disclosed to about the erasure, and **Article 17(2)** adds reasonable steps to inform other controllers where the controller made the data public. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. -**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to name it: no provision of EU law protects a training audit log merely because ForgeLM produces it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis. +**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to be able to name it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis; once the data subject withdraws consent or a successful objection removes that basis, it no longer carries them (Art. 17(1)(b)-(c)), and a refusal still needs reasons and a complaint route within the Art. 12 deadline (Art. 12(4)). The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability), and ForgeLM never edits or deletes an entry. Whether the entries that reference the now-erased subject may stay is the controller's decision, and the controller documents why: they are still necessary for their purpose, or an Art. 17(3) exception applies (for (3)(b) and (3)(e), only to the extent a specific obligation or claim requires it). @@ -72,7 +72,7 @@ The four artefact kinds the operator can act on in-tool: `corpus_rows`, `staging Article 5(1)(e) requires personal data to be kept "no longer than is necessary". ForgeLM ships a config block so the operator can declare their retention horizon and the tool can warn / refuse runs that violate it. -> **As shipped:** the horizons are only *reported* by `forgelm purge --check-policy` (read-only, always exit 0). `retention.enforce` is reserved: no command acts on it (the training CLI only logs a warning when it is set), so nothing warns, refuses or deletes on age. Ages come from the audit log first (the run's first event, else the log's first event) and fall back to `mtime`. Read the warn / refuse and `mtime` passages below as the original design; §12 lists the deviations. +> **As shipped:** the horizons are only *reported* by `forgelm purge --check-policy` (read-only; exits 0 however many it finds). `retention.enforce` is reserved: no command acts on it (the training CLI only logs a warning when it is set), so nothing warns, refuses or deletes on age. Ages come from the audit log first (the run's first event, else the log's first event) and fall back to `mtime`. Read the warn / refuse and `mtime` passages below as the original design; §12 lists the deviations. ### 3.1 Prior-state acknowledgement (audited 2026-05-02) @@ -106,9 +106,9 @@ class RetentionConfig(BaseModel): # --check-policy` reports `audit_log.jsonl` as overdue. 1825 days # (5 years) is a ForgeLM choice, not a legal period: GDPR Art. 5(1)(e) # sets no fixed period. Align it with the obligations that apply to you - # (for example AI Act Art. 18(1), documentation kept 10 years, and - # Art. 19(1) / 26(6), system logs at least six months) and with your - # own necessity assessment. Set to 0 to retain indefinitely. + # (for example a sector retention rule, or AI Act Art. 18(1) for + # provider documentation once it applies to you) and with your own + # necessity assessment. Set to 0 to stop reporting it. audit_log_retention_days: int = Field(default=1825, ge=0) # Staging directory time-to-live. Article 14 staging is meant to hold a @@ -210,7 +210,7 @@ Standard ForgeLM 0/1/2/3/4 contract — all codes from `forgelm.cli._exit_codes` | 3 | `EXIT_EVAL_FAILURE` | Reserved for the trainer pre-flight gate when `enforce: block_on_excess` is configured (see §3.4 in the spirit of the master `error-handling.md` exit-code contract). **Not** used by `forgelm purge --check-policy` — that path is a report, not a gate; see §10 Q5 + the paragraph below. | | 4 | `EXIT_AWAITING_APPROVAL` | Reserved for its trainer-pipeline meaning per `docs/standards/error-handling.md` ("Training + evals passed, but `require_human_approval: true` — staged, awaiting human sign-off"). **Not** reused by `forgelm purge`. Since the shipped `forgelm purge` is non-interactive (no prompt, no `--yes` flag), there is no "operator declined the prompt" path; consent is recorded in the operator-supplied `--justification` and the audit chain. | -**`--check-policy` always exits 0** (per §10 Q5 — the resolved decision: report-not-gate semantic). Operators who want a CI gate use `--output-format json` and pipe to `jq -e '.violations | length == 0'` themselves; CI then branches on that count rather than on the exit code. This keeps the public contract `docs/standards/error-handling.md` (0/1/2/3/4 trainer + 5 wizard-cancel) consistent across every ForgeLM subcommand: a code-3 from `forgelm purge` always means "trainer pre-flight gate failed", never "report found something". +**`--check-policy` exits 0 on a successful report** (per §10 Q5 — the resolved decision: report-not-gate semantic). Operators who want a CI gate use `--output-format json` and test the report themselves (for example `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`; `count` is absent when no `retention:` block was loaded); CI then branches on that rather than on the exit code. This keeps the public contract `docs/standards/error-handling.md` (the exit-code table in that standard) consistent across every ForgeLM subcommand: a code-3 from `forgelm purge` always means "trainer pre-flight gate failed", never "report found something". ### 4.4 Atomicity @@ -444,7 +444,7 @@ The closure plan §15.5 lists three open items that intersect Phase 21: ### Q5. Should `--check-policy` return non-zero when violations are found? -**Decision:** no. `--check-policy` is a *report*, not a *gate*. The CI gate is the trainer pre-flight (when `enforce: block_on_excess` is configured). Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and pipe to `jq -e '.violations | length == 0'` themselves. +**Decision:** no. `--check-policy` is a *report*, not a *gate*. The CI gate is the trainer pre-flight (when `enforce: block_on_excess` is configured). Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and test the report themselves (see §4.3). ### Q6. Should we delete the synthetic-data snapshots on row erasure? diff --git a/docs/guides/data_audit-tr.md b/docs/guides/data_audit-tr.md index 330914af..20e0e5e3 100644 --- a/docs/guides/data_audit-tr.md +++ b/docs/guides/data_audit-tr.md @@ -431,7 +431,8 @@ forgelm audit data/ --output ./audit/ --pii-ml Yeni kategoriler mevcut `pii_summary` ve `pii_severity` bloklarına disjoint isimlerle birleşir, bu yüzden regex baseline ML sinyaliyle -yan yana görünür kalır: +yan yana görünür kalır (örnek varsayılan İngilizce model içindir; o model +`organization` raporlamaz, diğer dil modelleri eder): ```json { @@ -439,15 +440,14 @@ yan yana görünür kalır: "email": 12, "phone": 3, "person": 47, // ← Presidio - "organization": 18, // ← Presidio "location": 9 // ← Presidio }, "pii_severity": { - "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 12}, "by_type": { "email": {"count": 12, "tier": "medium"}, "person": {"count": 47, "tier": "medium"}, - "organization": {"count": 18, "tier": "low"} + "location": {"count": 9, "tier": "low"} }, "worst_tier": "medium" } diff --git a/docs/guides/data_audit.md b/docs/guides/data_audit.md index bca4e9c3..00628319 100644 --- a/docs/guides/data_audit.md +++ b/docs/guides/data_audit.md @@ -421,7 +421,8 @@ forgelm audit data/ --output ./audit/ --pii-ml The new categories merge into the existing `pii_summary` and `pii_severity` blocks under disjoint names so the regex baseline stays -visible alongside the ML signal: +visible alongside the ML signal (the example is for the default English +model, which never reports `organization`; other language models do): ```json { @@ -429,15 +430,14 @@ visible alongside the ML signal: "email": 12, "phone": 3, "person": 47, // ← Presidio - "organization": 18, // ← Presidio "location": 9 // ← Presidio }, "pii_severity": { - "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 30}, + "by_tier": {"critical": 0, "high": 0, "medium": 59, "low": 12}, "by_type": { "email": {"count": 12, "tier": "medium"}, "person": {"count": 47, "tier": "medium"}, - "organization": {"count": 18, "tier": "low"} + "location": {"count": 9, "tier": "low"} }, "worst_tier": "medium" } diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 3f64c6bc..05c08757 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -3,9 +3,12 @@ > **AB Genel Veri Koruma Tüzüğü Madde 17** ("unutulma hakkı") veri > sahibine, sorumlunun kendisi hakkında tuttuğu kişisel verilerin > silinmesini isteme hakkı verir. ForgeLM'in `forgelm purge` -> subcommand'ı, bir ForgeLM dağıtımının ürettiği veya tükettiği eğitim -> corpus'larına ve koşum-kapsamlı artefact'lara karşı bu talepleri -> karşılamak için operator-facing araçtır. +> subcommand'ı böyle bir talebi yanıtlamanın bir adımıdır: bir ForgeLM +> dağıtımının ürettiği veya tükettiği eğitim corpus'larından ve +> koşum-kapsamlı artefact'lardan satır kaldırır. Kişiyi tanımlamak, bir +> Madde 17(1) gerekçesinin geçerli olup olmadığına karar vermek, Madde 12 +> süresi içinde yanıtlamak ve eğitilmiş ağırlıklarla ilgili her şey sizin +> sürecinizde kalır. ## `forgelm purge` neyi yapar (ve neyi yapmaz) @@ -15,14 +18,15 @@ - Bir koşumun staging dizinini ve adı run id'yi embed eden compliance dosyalarını siler (`--run-id --kind {staging,artefacts}`; `artefacts` standart paketi bulamaz, aşağıya bakın). - Yüklü config'in `retention:` block'una karşı retention-policy ihlallerini raporlar (`--check-policy`). - Her aksiyonu tamper-evident bir audit-event zinciriyle kaydeder — `request → completed` (veya `request → failed`) — böylece adli inceleyici tam olarak ne olduğunu yeniden inşa edebilir. -- Operatör tarafından sağlanan satır identifier'larını audit log'a girmeden önce SHA-256 ile hash'ler; böylece zincirin kendisi kişisel-veri sızıntısına dönüşmez (Madde 5(1)(c) veri minimizasyonu). +- Operatör tarafından sağlanan satır identifier'larını audit log'a girmeden önce per-output-dir salt ile hash'ler; böylece log identifier'ı açık halde tutmaz (Madde 5(1)(c) veri minimizasyonu). Bu pseudonymisation'dır, anonimleştirme değildir (Madde 4(5), gerekçe 26): log'un yanındaki `.forgelm_audit_salt` dosyasını okuyabilen herkes tahmin ettiği bir identifier'dan digest'i yeniden hesaplayabilir; bu yüzden log'u ve salt'ı aynı erişim kontrolleriyle koruyun ve digest'leri kişisel veri sayın. **Yapmaz:** - Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. -- Audit log'un kendisinden entry silmez. Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibine atıf yapan girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır (log teknik dokümantasyonunuzun parçasıysa AI Act Madde 18(1)'in 10 yıllık dokümantasyon saklaması ya da kullanıma alınmış yüksek riskli bir sistemin ürettiği loglar için Madde 19(1) ile 26(6)'nın log saklama yükümlülükleri gibi) ve bu yükümlülüğü adlandırabilmeniz gerekir. -- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (Madde 19 verinin ifşa edildiği her alıcıya bildirim yapılmasını ister; Madde 17(2) ise kontrolör veriyi kamuya açtıysa makul adımlar ekler). Tool yüklü config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. +- Audit log'un kendisinden entry silmez. Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Bir Madde 17(1) gerekçesi geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki talepleri kapsar. Hiçbir Madde 17(1) gerekçesi geçerli değilse (örneğin girdiler toplandıkları amaç için hâlâ gereklidir) silme gerekmez; ancak talebe yine de Madde 12 süresi içinde gerekçeli bir yanıt verilir ve şikâyet hakkı bildirilir (Madde 12(4)). +- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve ilgili kişi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). Tool, row modunda `--config` ile verdiğiniz config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. +- Yanıtın tamamını vermez. Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); ilgili kişiye verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). ## Üç mod @@ -38,11 +42,11 @@ $ forgelm purge \ Tool şunları yapar: -1. Per-output-dir salt'ı `/.forgelm_audit_salt`'ta çözer (ilk kullanımda oluşturur; mode `0600`; `FORGELM_AUDIT_SECRET[:16]` set'liyse XOR'lar). **Not:** bu XOR yalnızca **identifier hashing**'i besler — audit-chain HMAC anahtarı bağımsız olarak `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` şeklinde türetilir (bkz. [`docs/qms/access_control.md`](../qms/access_control-tr.md) §3.4 / `forgelm/compliance.py:104-114`). İki primitif kasıtlı olarak ayrıdır. +1. Per-output-dir salt'ı `/.forgelm_audit_salt`'ta çözer (ilk kullanımda oluşturur; mode `0600`; `FORGELM_AUDIT_SECRET[:16]` set'liyse XOR'lar). **Not:** bu XOR yalnızca **identifier hashing**'i besler — audit-chain HMAC anahtarı bağımsız olarak `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` şeklinde türetilir (bkz. [`docs/qms/access_control.md`](../qms/access_control-tr.md) §3.4 / `forgelm/compliance.py` içindeki `AuditLogger.__init__`). İki primitif kasıtlı olarak ayrıdır. 2. Audit log'a `data.erasure_requested` yazar — **hash'lenmiş** target_id (`SHA-256(salt + value)`) ve operatör-sağlanan justification ile. 3. Eşleşen JSONL satırını `id` (veya `row_id`) field'ı ile bulur — re-order edilmiş bir dosyada sessiz yanlış-satır silmesini engellemek için satır-numarası fallback'i **reddedilir**. ForgeLM şu an bir id-doldurma yardımcısı sunmuyor (Phase 28 backlog'unda `forgelm audit --add-row-ids` flag'i bekliyor); id'siz corpus'lara sahip operatörler purge'den önce id'leri operatör-tarafı bir script ile (örn. `jq -c 'to_entries | with_entries(...)'` ya da tek-seferlik bir Python döngüsü) pre-populate etmeli. 4. Corpus'u atomic olarak yeniden yazar (kardeş bir temp dosyaya yazar + `os.replace`); operatörler ya tam silme-öncesi dosyayı ya da tam silme-sonrası dosyayı görür, asla kısmi state'i değil. -5. Uyarı koşullarını tespit eder ve eşleşen event'leri yayar (memorization, synthetic-data varlığı, dış kopyalar). +5. Uyarı koşullarını tespit eder ve eşleşen event'leri yayar (memorization, synthetic-data varlığı, dış kopyalar). Dış-kopya kontrolü `--config` ile verdiğiniz config'in `webhook` bloğunu okur (literal bir `url` ya da purge ortamında export edilmiş bir `url_env` değişkeni); `--config` olmadan ya da `url_env` ayarsızsa tetiklenemez. 6. Disk operasyonu raise ederse `data.erasure_failed`, başarılıysa `data.erasure_completed` yazar; zincir nihai state'i yansıtır. Çoklu-satır eşleşmeleri varsayılan olarak reddedilir (`--row-matches one`); id'yi paylaşan her satırı silmek için `--row-matches all` geç (operatör niyeti onaylar). @@ -68,7 +72,7 @@ $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: filesystem `mtime` + `age_source=mtime` flag'i ile) yüklü config'in `retention:` horizon'larına karşı kıyaslar ve structured violation list yayar. -**Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist)'e pipe'lar; bu, public exit-code kontratını `0/1/2/3/4` her ForgeLM subcommand'ında tutarlı tutar. +**Başarılı policy raporları 0 ile çıkar** (rapor-değil-gate semantiği — design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar: eksik / okunamayan / Pydantic validation'ı geçemeyen explicit `--config`, "loader failed" durumunu "ihlal yok" sanmamak için `EXIT_CONFIG_ERROR` olarak yüzeye çıkar. CI gate isteyen operatörler `--output-format json` kullanır ve raporu kendileri sınar, örneğin `set -o pipefail` altında `jq -e '.success == true and has("count") and .count == 0'`: `count`, hiçbir `retention:` bloğu yüklenmediğinde yoktur; böylece gate hiç kontrol edilmemiş bir dizinde geçmek yerine başarısız olur, ve `--output-dir` gerçek eğitim çıktı dizini olmalıdır (olmayan bir dizin `count: 0` raporlar). Bu, public exit-code kontratını her ForgeLM subcommand'ında tutarlı tutar. ## `retention:` config block'u @@ -81,7 +85,7 @@ retention: enforce: log_only # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess) ``` -Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi devre dışı bırakır (sınırsız tut). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur, her zaman 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. `enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz, bu yüzden üç değer de `log_only` gibi davranır. +Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapatır (her iki durumda da hiçbir şey silinmez). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur; kaç tane bulursa bulsun 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. `enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz, bu yüzden üç değer de `log_only` gibi davranır. ## Kaldırıldı: `evaluation.staging_ttl_days` @@ -109,8 +113,8 @@ Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için policy'yi d | Kod | Anlamı | |---|---| | 0 | Başarı veya başarılı `--check-policy` raporu (rapor-değil-gate semantiği). | -| 1 | Config hatası: bilinmeyen `--row-id`, eksik `--corpus`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), mutually-exclusive flag kombinasyonu veya eksik / okunamayan / Pydantic validation'ı geçemeyen `--check-policy --config `. | -| 2 | Runtime hatası: I/O failure, permission denied, atomic-rename failure. | +| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus` ya da `--corpus`'suz `--row-id`, eksik `--kind`, seçilmemiş mod veya eksik / okunamayan / Pydantic validation'ı geçemeyen `--check-policy --config `. | +| 2 | Runtime hatası: I/O failure, permission denied, atomic-rename failure; ayrıca argüman-parser kullanım hatası (`--row-id` ile `--run-id` gibi birlikte verilen iki mod, `--kind logs`), JSON zarfı yerine usage basar. | `--check-policy` asla 3 kodu döndürmez. Başarılı bir rapor 0 ile çıkar; verilen `--config`'i yükleme başarısızlığı ise yanıltıcı bir "ihlal yok" raporuna düşmek yerine `EXIT_CONFIG_ERROR` (sıfır olmayan) ile çıkar. CI gate isteyen operatörler ihlal sayısını JSON çıktıdan kendileri hesaplar (design §10 Q5). @@ -145,10 +149,14 @@ için kullandığı aynı per-output-dir salt'ı yeniden kullanılır. Madde 15 erişim talepleri kendisi de subject'in verisini audit log'a sızdırmamalıdır. Salt'lı form, compliance reviewer'ın aynı subject için Madde 17 (purge) ve Madde 15 (reverse-pii) olaylarını cleartext -görmeden ilişkilendirmesini sağlar (`--row-id` ve `--query`'ye aynı değeri -verdiğinizde, aynı `--output-dir`'i ve aynı salt kaynağını kullandığınızda -digest'ler eşleşir: purge için `FORGELM_AUDIT_SECRET` ayarlıysa aynı secret ile -`--salt-source env_var` kullanarak sorgulayın). +görmeden ilişkilendirmesini sağlar. Digest'ler yalnızca `--row-id` ve +`--query` değerleri aynı metinse, iki komut da aynı `--output-dir`'i +kullanıyorsa ve `FORGELM_AUDIT_SECRET` ikisinde de aynı durumdaysa (ayarsız ya +da aynı değer) eşleşir; erişim talebini varsayılan düz-metin modunda çalıştırın, +çünkü `--salt-source` yukarıdaki hash-mask taramasına geçirir ve purge +digest'lerini eşleştirmenin bir yolu değildir. Id'ler farklıysa bir talebi +silmesine, `forgelm purge --justification`'a verdiğiniz ticket referansı +üzerinden bağlayın. **Identifier tipleri** (`--type`): `literal` (varsayılan), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 7fecd93f..c5a028a1 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -3,9 +3,11 @@ > **Article 17 of the EU General Data Protection Regulation** (the > "right to be forgotten") gives data subjects the right to request > erasure of personal data a controller holds about them. ForgeLM's -> `forgelm purge` subcommand is the operator-facing tool for honouring -> those requests against training corpora and run-scoped artefacts a -> ForgeLM deployment produced or consumed. +> `forgelm purge` subcommand is one step of answering such a request: it +> removes rows from the training corpora and run-scoped artefacts a ForgeLM +> deployment produced or consumed. Identifying the person, deciding whether +> an Article 17(1) ground applies, answering within the Article 12 deadline +> and everything about the trained weights remain your process. ## What `forgelm purge` does (and does not) @@ -15,14 +17,15 @@ - Deletes a run's staging directory, and compliance files whose names embed the run id (`--run-id --kind {staging,artefacts}`; `artefacts` does not find the standard bundle, see below). - Reports retention-policy violations against the loaded config's `retention:` block (`--check-policy`). - Records every action with a tamper-evident audit-event chain — `request → completed` (or `request → failed`) — so a forensic reviewer can reconstruct exactly what happened. -- Hashes operator-supplied row identifiers with SHA-256 before they enter the audit log so the chain itself does not become a personal-data leak (Article 5(1)(c) data minimisation). +- Hashes operator-supplied row identifiers with a per-output-dir salt before they enter the audit log, so the log does not hold the identifier in clear (Article 5(1)(c) data minimisation). This is pseudonymisation, not anonymisation (Article 4(5), recital 26): anyone who can read the `.forgelm_audit_salt` file kept next to the log can recompute a digest from a guessed identifier, so protect the log and the salt with the same access controls and treat the digests as personal data. **Does not:** - Re-train models. Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` when a `final_model/` directory exists directly under `--output-dir` (which defaults to the corpus's directory, so pass your training output directory), so the operator sees the gap. It cannot tell which runs consumed the corpus. -- Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history. Whether entries that reference the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing (such as the 10-year documentation keeping of AI Act Art. 18(1) where the log forms part of your technical documentation, or the log-retention duties of Art. 19(1) and Art. 26(6) for logs a deployed high-risk system generates), and you have to be able to name that obligation. -- Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (Article 19 requires telling each recipient the data was disclosed to; Article 17(2) adds reasonable steps where the controller made the data public). The tool emits `data.erasure_warning_external_copies` when the loaded config has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. +- Delete entries from the audit log itself. The tool records the erasure as a new event rather than rewriting history, and never deletes an audit-log entry. Whether entries that name the data subject may be kept is the controller's decision, and it has to be documented. If an Article 17(1) ground applies, the entries stay only where an Article 17(3) exception covers them: 17(3)(b) applies only to the extent processing is necessary for compliance with a legal obligation under Union or Member State law to which the controller is subject (or for a task in the public interest or in the exercise of official authority), and 17(3)(e) covers legal claims. If no Article 17(1) ground applies (for example the entries are still necessary for the purpose they were collected for), erasure is not owed, but the request still gets a reasoned answer within the Article 12 deadline, with the right to complain (Article 12(4)). +- Push erasure notices to downstream consumers. Webhook receivers, dataset mirrors, deployed model endpoints, and third-party fine-tunes derived from a published checkpoint are the operator's runtime-layer responsibility (GDPR Article 19 requires telling each recipient the data was disclosed to about the erasure, unless that proves impossible or involves disproportionate effort, and telling the data subject about those recipients on request; Article 17(2) adds reasonable steps where the controller made the data public). The tool emits `data.erasure_warning_external_copies` in row mode when the config you pass with `--config` has a webhook block, so a downstream consumer querying the audit log sees the explicit reminder. - Erase backups. Replicas, snapshots, and storage backups outside the operator's `` are infrastructure-side concerns. +- Make the whole response. A complete response also needs steps ForgeLM does not perform: verifying the requester's identity (Article 12(6)); answering within one month of receipt (Article 12(3); extendable by two further months for complex or numerous requests, with notice in the first month); giving the data subject a copy of the data and the Article 15(1)(a)-(h) information (Article 15(3)) without adversely affecting other people's rights (Article 15(4)); deciding whether an Article 17(1) ground applies and which Article 17(3) exception, if any, covers what you keep; telling recipients (Article 19); re-running the access search after a purge to confirm the rows are gone; and, if you refuse a request, giving reasons and the right to complain within the same deadline (Article 12(4)). Whether the trained model counts as personal data is a separate question (EDPB Opinion 28/2024). ## Three modes @@ -42,7 +45,7 @@ The tool: 2. Writes `data.erasure_requested` to the audit log with the **hashed** target_id (`SHA-256(salt + value)`) and the operator-supplied justification. 3. Locates the matching JSONL row by its `id` (or `row_id`) field — line-number fallback is **rejected** to defend against silent wrong-row deletion on a re-ordered file. ForgeLM does not currently provide an id-population helper (a `forgelm audit --add-row-ids` flag is on the Phase 28 backlog); operators with id-less corpora must pre-populate ids with an operator-side script (e.g. `jq -c 'to_entries | with_entries(...)'` or a one-shot Python loop) before invoking purge. 4. Atomically rewrites the corpus (writes a sibling temp file + `os.replace`); operators get either the full pre-erasure file or the full post-erasure file, never a partial state. -5. Detects warning conditions and emits the matching events (memorisation, synthetic-data presence, external copies). +5. Detects warning conditions and emits the matching events (memorisation, synthetic-data presence, external copies). The external-copies check reads the `webhook` block of the config you pass with `--config` (a literal `url`, or a `url_env` variable that is exported in the purge environment); without `--config`, or with an unset `url_env`, it cannot fire. 6. Writes `data.erasure_completed` (or `data.erasure_failed` if the disk operation raised) so the chain reflects the final state. Multi-row matches default to refusal (`--row-matches one`); pass `--row-matches all` to delete every row sharing the id (operator confirms intent). @@ -68,7 +71,7 @@ $ forgelm purge --check-policy --config configs/run.yaml --output-dir ./outputs Read-only scan. Walks the output directory, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime` flag) against the loaded config's `retention:` horizons, and emits a structured violation list. -**Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist) themselves; this keeps the public exit-code contract `0/1/2/3/4` consistent across every ForgeLM subcommand. +**Successful policy reports exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit with `EXIT_CONFIG_ERROR` (non-zero): an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as `EXIT_CONFIG_ERROR` so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and test the report themselves, for example `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`: `count` is absent when no `retention:` block was loaded, so the gate fails instead of passing on a directory that was never checked, and `--output-dir` must be the real training output directory (a missing one reports `count: 0`). This keeps the public exit-code contract consistent across every ForgeLM subcommand. ## The `retention:` config block @@ -81,7 +84,7 @@ retention: enforce: log_only # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess) ``` -Setting any horizon to `0` disables the policy for that artefact kind (retain indefinitely). The horizons are reported by `forgelm purge --check-policy` (read-only, always exits 0); nothing is deleted or blocked automatically. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. +Setting any horizon to `0` turns the report off for that artefact kind (nothing is deleted either way). The horizons are reported by `forgelm purge --check-policy` (read-only; exits 0 however many it finds); nothing is deleted or blocked automatically. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. ## Removed: `evaluation.staging_ttl_days` @@ -109,8 +112,8 @@ Six new events ship with `forgelm purge` (catalogued in `docs/reference/audit_ev | Code | Meaning | |---|---| | 0 | Success, or a successful `--check-policy` report (report-not-gate semantic). | -| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | -| 2 | Runtime error: I/O failure, permission denied, atomic-rename failure. | +| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), a missing `--corpus` or `--row-id` without `--corpus`, a missing `--kind`, no mode selected, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | +| 2 | Runtime error: I/O failure, permission denied, atomic-rename failure; also an argument-parser usage error (two modes together such as `--row-id` with `--run-id`, `--kind logs`), which prints usage instead of a JSON envelope. | `--check-policy` never returns code 3. A successful report exits 0; a failure to load the supplied `--config` exits with `EXIT_CONFIG_ERROR` (non-zero) rather than degrading to a misleading "no violations" report. Operators wiring a CI gate compute the violation count from JSON output themselves (per design §10 Q5). @@ -143,11 +146,14 @@ that `forgelm purge` uses for its `target_id` field — Article 15 access requests must not themselves leak the subject's data into the audit log. The salted form is a stable per-identifier fingerprint that lets a compliance reviewer correlate Article 17 (purge) and -Article 15 (reverse-pii) events for the same subject (the digests match -when you pass the same value to `--row-id` and `--query`, use the same -`--output-dir`, and use the same salt source: if `FORGELM_AUDIT_SECRET` was set -for the purge, query with `--salt-source env_var` and the same secret) without -seeing the cleartext. +Article 15 (reverse-pii) events for the same subject without seeing the +cleartext. The digests match only when the `--row-id` and `--query` values +are the same text, both commands use the same `--output-dir`, and +`FORGELM_AUDIT_SECRET` is in the same state for both (unset, or the same +value); run the access request in its default plaintext mode, because +`--salt-source` switches to the hash-mask scan above and is not a way to +match purge digests. When the ids differ, link a request to its erasure +through the ticket reference you pass to `forgelm purge --justification`. **Identifier types** (`--type`): `literal` (default), `email`, `phone`, `tr_id`, `us_ssn`, `iban`, `credit_card`, `custom`. All diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index 196d0689..62d425d8 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -281,8 +281,9 @@ forgelm reverse-pii --query "alice@example.com" --type email \ --output-dir ./outputs data/*.jsonl # JSON envelope { matches: [...], match_count: N } döner # Audit chain QUERY HASH'LENMİŞ ŞEKİLDE bir -# `data.access_request_query` event damgalar (asla raw — salt audit -# log'un kendisine karşı wordlist saldırılarına karşı koruma sağlar). +# `data.access_request_query` event damgalar (asla raw). Salt'lı digest +# pseudonymised'dır, anonim değildir: `.forgelm_audit_salt`'ı log ile aynı +# erişim kontrolleri altında tutun, yoksa tahmin edilen bir identifier ona karşı sınanabilir. ``` Madde 17 (silme hakkı): @@ -292,15 +293,17 @@ forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ --output-dir ./outputs # Audit chain `target_id` HASH'LENMİŞ ŞEKİLDE # `data.erasure_requested` → `data.erasure_completed` (veya -# `data.erasure_failed`) damgalar. Eğer satırı eğiten bir model -# `final_model/`e sahipse `data.erasure_warning_memorisation` DA -# yangın eder ve memorisation residual riskini özneye bildirmek -# zorundasınız. +# `data.erasure_failed`) damgalar. Eğer `--output-dir` altında doğrudan bir +# `final_model/` dizini varsa `data.erasure_warning_memorisation` DA +# tetiklenir; satırı hangi modelin gördüğünü söylemez, bu yüzden +# corpus ile eğitilmiş her modeli kapsam içinde sayın ve memorisation +# residual riskini özneye bildirin. ``` Cross-tool digest correlation (purge `target_id` == reverse-pii -`query_hash` aynı `output_dir`'deki aynı identifier için) Wave 3 -follow-up testleriyle pin'lenmiştir. +`query_hash`: aynı `output_dir`, aynı değer ve aynı +`FORGELM_AUDIT_SECRET` durumu, `--salt-source` olmadan) testlerle +pin'lenmiştir. ## Yaygın tuzaklar diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md index 9d020375..3dfbec16 100644 --- a/docs/guides/iso_soc2_deployer_guide.md +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -282,8 +282,9 @@ forgelm reverse-pii --query "alice@example.com" --type email \ --output-dir ./outputs data/*.jsonl # JSON envelope returns { matches: [...], match_count: N } # The audit chain stamps a `data.access_request_query` event with -# the QUERY HASHED (never raw — the salt protects against wordlist -# attacks against the audit log itself). +# the QUERY HASHED (never raw). The salted digest is pseudonymised, +# not anonymous: keep `.forgelm_audit_salt` under the same access +# controls as the log, or a guessed identifier can be tested against it. ``` Article 17 (right to erasure): @@ -293,15 +294,17 @@ forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ --output-dir ./outputs # The audit chain stamps `data.erasure_requested` → # `data.erasure_completed` (or `data.erasure_failed`) with -# `target_id` HASHED. If a model trained on the row has -# `final_model/`, `data.erasure_warning_memorisation` ALSO fires -# and you must communicate the memorisation residual risk to the -# subject. +# `target_id` HASHED. If a `final_model/` directory exists directly +# under `--output-dir`, `data.erasure_warning_memorisation` ALSO fires; +# it does not say which model saw the row, so treat every model +# trained on the corpus as in scope and communicate the memorisation +# residual risk to the subject. ``` The cross-tool digest correlation (purge `target_id` == -reverse-pii `query_hash` for the same identifier in the same -`output_dir`) is pinned by Wave 3 follow-up tests. +reverse-pii `query_hash` for the same value in the same `output_dir` +under the same `FORGELM_AUDIT_SECRET` state, with no `--salt-source`) +is pinned by tests. ## Common pitfalls diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md index b1aabf3d..32517287 100644 --- a/docs/qms/sop_incident_response-tr.md +++ b/docs/qms/sop_incident_response-tr.md @@ -244,17 +244,21 @@ olabilir. **Tetik:** Veri sahibi operatörün DSAR portalı veya iletişim formu üzerinden erişim talebi gönderir. -**Önem:** Medium (regülasyon deadline 30 gün). +**Önem:** Medium (yanıt, talebin alınmasından itibaren bir ay içinde, GDPR Madde 12(3); bildirimle iki ay uzatılabilir; KVKK Madde 13(2)'de 30 gün). **Runbook:** 1. [ ] **Sahip kimliğini doğrula** operatörün DSAR prosedürü başına. 2. [ ] **`forgelm reverse-pii --query --type data/*.jsonl --output-dir ` çalıştır**. -3. [ ] **Hash-mask scan** ForgeLM'in maskelediği herhangi bir - korpus için (`forgelm reverse-pii --query - --salt-source per_dir --output-dir `). -4. [ ] **Yanıt mektubu oluştur** DSAR şablonu başına; koşumun +3. [ ] **Diğer kopyaları da ara.** Eşleşme büyük/küçük harfe duyarlıdır + ve ham dosya metnini karşılaştırır; yazım ve harf varyantlarını da + ara; `forgelm ingest --pii-mask` digest değil `[REDACTED]` yazar, + bu yüzden hash-mask taraması (`--salt-source`) yalnızca ForgeLM + dışında hash'lenmiş korpuslar içindir; ham kaynak dokümanları ayrıca ara. +4. [ ] **Yanıt mektubu oluştur** DSAR şablonu başına: verinin bir + kopyası ve Madde 15(1)(a)-(h) bilgileri; başkalarının verisini + açığa çıkarmamaya dikkat et (Madde 15(4)); koşumun `data.access_request_query` audit-event id'sine atıf yap. 5. [ ] **Yanıt mektubunu sakla** audit chain yanında. @@ -262,8 +266,9 @@ olabilir. **Tetik:** Sahip silme talebi gönderir. -**Önem:** Medium (regülasyon deadline 30 gün; bazı AB üye -ülkeleri daha kısa yanıt gerektirir). +**Önem:** Medium (yanıt, talebin alınmasından itibaren bir ay içinde, +GDPR Madde 12(3); bildirimle iki ay uzatılabilir; KVKK Madde 13(2)'de +30 gün). **Runbook:** @@ -271,8 +276,10 @@ olabilir. 2. [ ] **`forgelm purge --row-id --corpus data/.jsonl --output-dir ` çalıştır**. 3. [ ] **`data.erasure_warning_memorisation` olayını kontrol et** — - satırı eğiten bir model `final_model/` artefaktına sahipse, - memorisation residual riski uygulanır. + yalnızca `--output-dir` altında doğrudan bir `final_model/` + dizini varsa tetiklenir ve satırı hangi modelin gördüğünü + söylemez; uyarı tetiklenmese bile corpus ile eğitilmiş her + modeli kapsam içinde say. 4. [ ] **Memorisation caveat'ı sahibe bildir** yanıt mektubunda (şablon: "satırı sildik ama önceden eğitilmiş bir model bunu memorise etmiş olabilir; uygun şekilde yeniden eğiteceğiz veya diff --git a/docs/qms/sop_incident_response.md b/docs/qms/sop_incident_response.md index bf322545..bacebb3d 100644 --- a/docs/qms/sop_incident_response.md +++ b/docs/qms/sop_incident_response.md @@ -246,17 +246,21 @@ that an attacker may have observed. **Trigger:** Data subject submits an access request via deployer's DSAR portal or contact form. -**Severity:** Medium (regulatory deadline 30 days). +**Severity:** Medium (response due within one month of receipt, GDPR Art. 12(3), extendable by two months with notice; 30 days under KVKK Art. 13(2)). **Runbook:** 1. [ ] **Verify subject identity** per deployer's DSAR procedure. 2. [ ] **Run** `forgelm reverse-pii --query --type data/*.jsonl --output-dir ` -3. [ ] **Hash-mask scan** for any corpus that ForgeLM masked - (`forgelm reverse-pii --query --salt-source per_dir - --output-dir `). -4. [ ] **Compose response letter** per DSAR template; cite the run's +3. [ ] **Search the other copies too.** The match is case-sensitive + and compares raw file text, so search spelling and case variants; + `forgelm ingest --pii-mask` writes `[REDACTED]`, not digests, so a + hash-mask scan (`--salt-source`) is only for corpora hashed outside + ForgeLM; search the raw source documents separately. +4. [ ] **Compose response letter** per DSAR template: a copy of the + data and the Art. 15(1)(a)-(h) information, taking care not to + expose other people's data (Art. 15(4)); cite the run's `data.access_request_query` audit-event id. 5. [ ] **Retain** the response letter alongside the audit chain. @@ -264,8 +268,9 @@ DSAR portal or contact form. **Trigger:** Subject submits an erasure request. -**Severity:** Medium (regulatory deadline 30 days; some EU member -states require shorter response). +**Severity:** Medium (response due within one month of receipt, GDPR +Art. 12(3), extendable by two months with notice; 30 days under KVKK +Art. 13(2)). **Runbook:** @@ -273,8 +278,10 @@ states require shorter response). 2. [ ] **Run** `forgelm purge --row-id --corpus data/.jsonl --output-dir `. 3. [ ] **Check** for `data.erasure_warning_memorisation` event — - if a model trained on the row has a `final_model/` artefact, - memorisation residual risk applies. + it fires only when a `final_model/` directory exists directly + under `--output-dir` and does not say which model saw the row, so + treat every model trained on the corpus as in scope even when no + warning fires. 4. [ ] **Communicate the memorisation caveat** to the subject in the response letter (template: "we deleted the row but a previously-trained model may have memorised it; we will diff --git a/docs/reference/audit_event_catalog-tr.md b/docs/reference/audit_event_catalog-tr.md index 452d4e73..d2a8b755 100644 --- a/docs/reference/audit_event_catalog-tr.md +++ b/docs/reference/audit_event_catalog-tr.md @@ -80,7 +80,7 @@ Hash zinciri, satır diske düştükten (`flush` + `fsync`) sonra ilerler; kirli | `data.erasure_failed` | Disk operasyonu raise etti VEYA eşleşen satır/koşum bulunamadı VEYA çoklu-satır policy belirsizliği reddetti. | Tüm `requested` field'ları + `error_class`, `error_message` | 17 | | `data.erasure_warning_memorisation` | Row erasure × `--output-dir` altında doğrudan bir `final_model/` dizini mevcut (birebir ad; `--output-dir` varsayılan olarak corpus'un dizinidir, bu yüzden eğitim çıktı dizininizi verin). ForgeLM corpus'u hangi koşumların tükettiğini belirlemez. | Tüm `completed` field'ları + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `output_dir`'de `synthetic_data*.jsonl` mevcut. | Tüm `completed` field'ları + `synthetic_files` | 17 | -| `data.erasure_warning_external_copies` | Yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | 17 | +| `data.erasure_warning_external_copies` | Yalnızca `--config` ile row-modu purge: config'in `webhook` bloğu boş değil (literal `url` ya da export edilmiş `url_env`); downstream tüketiciler bildirim almış olabilir. | Tüm `completed` field'ları + `webhook_targets` (redact'li URL'ler) | 17 | ### Madde 15 — GDPR Erişim Hakkı (Phase 38 — `forgelm reverse-pii`) diff --git a/docs/reference/audit_event_catalog.md b/docs/reference/audit_event_catalog.md index d5b93ebd..2eed9532 100644 --- a/docs/reference/audit_event_catalog.md +++ b/docs/reference/audit_event_catalog.md @@ -80,7 +80,7 @@ The hash chain advances after the line lands on disk (`flush` + `fsync`), so an | `data.erasure_failed` | Disk operation raised, OR no matching row/run found, OR multi-row policy refused on ambiguity. | All `requested` fields + `error_class`, `error_message` | 17 | | `data.erasure_warning_memorisation` | Row erasure × a `final_model/` directory exists directly under `--output-dir` (literal name; `--output-dir` defaults to the corpus's directory, so pass your training output directory). ForgeLM does not determine which runs consumed the corpus. | All `completed` fields + `affected_run_ids` | 17 | | `data.erasure_warning_synthetic_data_present` | Row erasure × `synthetic_data*.jsonl` exists in `output_dir`. | All `completed` fields + `synthetic_files` | 17 | -| `data.erasure_warning_external_copies` | Loaded config has a non-empty `webhook` block; downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | 17 | +| `data.erasure_warning_external_copies` | Row-mode purge with `--config` only: the config's `webhook` block is non-empty (literal `url`, or `url_env` exported); downstream consumers may have received notices. | All `completed` fields + `webhook_targets` (redacted URLs) | 17 | ### Article 15 — GDPR Right-of-Access (Phase 38 — `forgelm reverse-pii`) diff --git a/docs/reference/compliance_summary-tr.md b/docs/reference/compliance_summary-tr.md index 1d9b3def..03d306bb 100644 --- a/docs/reference/compliance_summary-tr.md +++ b/docs/reference/compliance_summary-tr.md @@ -36,8 +36,8 @@ ForgeLM kutudan çıktığı gibi şunları ship eder: `forgelm verify-integrity`. - **EU AI Act Madde 17** QMS şablonları: `docs/qms/` (Wave 0 baseline + Wave 4 ISO eklemeleri). -- **GDPR Madde 15** erişim hakkı: `forgelm reverse-pii`. -- **GDPR Madde 17** silinme hakkı: `forgelm purge`. +- **GDPR Madde 15**: `forgelm reverse-pii`, bir identifier'ı içeren satırları bulur (erişim yanıtının bir girdisi). +- **GDPR Madde 17**: `forgelm purge`, bir corpus'tan satır kaldırır (silmenin bir adımı). - **ISO 27001 / SOC 2 uyumluluğu** — bkz. Wave 4 design doc + deployer rehberi. @@ -259,9 +259,9 @@ Tam alan semantiği ve config yüzeyi: ### GDPR Madde 15 + 17 (Wave 2b + Wave 3) -- Madde 17 silme: `forgelm purge --row-id --corpus +- Madde 17 (silmenin bir adımı): `forgelm purge --row-id --corpus data/file.jsonl`, salted-hash audit ile (`data.erasure_*` event'leri). -- Madde 15 erişim: `forgelm reverse-pii --query --type +- Madde 15 (erişim yanıtının bir girdisi): `forgelm reverse-pii --query --type email|phone|... data/*.jsonl`, salted-hash audit ile (`data.access_request_query` event'i). diff --git a/docs/reference/compliance_summary.md b/docs/reference/compliance_summary.md index 1bd714c3..dd78419f 100644 --- a/docs/reference/compliance_summary.md +++ b/docs/reference/compliance_summary.md @@ -37,8 +37,8 @@ ForgeLM ships out-of-the-box: post-deployment verification via `forgelm verify-integrity`. - **EU AI Act Article 17** QMS templates in `docs/qms/` (Wave 0 baseline + Wave 4 ISO additions). -- **GDPR Article 15** right-of-access via `forgelm reverse-pii`. -- **GDPR Article 17** right-to-erasure via `forgelm purge`. +- **GDPR Article 15**: `forgelm reverse-pii` finds the rows that contain an identifier (an input to an access response). +- **GDPR Article 17**: `forgelm purge` removes rows from a corpus (one step of an erasure). - **ISO 27001 / SOC 2 alignment** — see Wave 4 design doc + deployer guide. @@ -258,9 +258,9 @@ Full field semantics and the config surface: ### GDPR Article 15 + 17 (Wave 2b + Wave 3) -- Article 17 erasure: `forgelm purge --row-id --corpus +- Article 17 (one step of an erasure): `forgelm purge --row-id --corpus data/file.jsonl` with salted-hash audit (`data.erasure_*` events). -- Article 15 access: `forgelm reverse-pii --query --type +- Article 15 (an input to an access response): `forgelm reverse-pii --query --type email|phone|... data/*.jsonl` with salted-hash audit (`data.access_request_query` event). diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index 635a8e51..a073513e 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -276,18 +276,20 @@ sınırlarıdır (talepler gereksiz gecikmeden kaçınılarak, en geç Madde 12( aylık süre içinde yanıtlanır; bu süre iki ay daha uzatılabilir). Değerleri kendi gereklilik (necessity) değerlendirmenize göre belirleyin; varsayılanlar ForgeLM tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: -`forgelm purge --check-policy` ufku aşanları listeler ve her zaman 0 ile çıkar; +`forgelm purge --check-policy` ufku aşanları listeler ve kaç tane bulursa bulsun 0 ile çıkar (yüklenemeyen `--config` 1 ile çıkar); hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı ayrılmıştır: henüz hiçbir komut onu okumaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| -| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "süresi geçmiş" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir (uygulanabildiği ölçüde bkz. AI Act Madde 18(1): dokümantasyon 10 yıl; Madde 19(1) ve 26(6): sistem logları en az altı ay). `0` süresiz saklamayı belirtir; girdilerin neden gerekli kaldığını veya dayandığınız yasal yükümlülüğü belgeleyin. | -| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve eski `final_model.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` süresiz saklama anlamına gelir. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | -| `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | -| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` süresiz saklama. | +| `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "süresi geçmiş" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir. `0` bu raporu kapatır (hiçbir şey silinmez); log'u yalnızca girdiler gerekli kaldığı sürece ya da adlandırabileceğiniz belirli bir yasal saklama yükümlülüğü gerektirdiği sürece tutun. | +| `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve eski `final_model.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` bu raporu kapatır. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | +| `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` bu raporu kapatır. | +| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` bu raporu kapatır. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Hiçbir komut onu kullanmaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa eğitim ve `--dry-run` uyarır. | +**Paylaşılan çıktı dizinleri.** `compliance/`, `data_audit_report.json` ve ham-belge dizinlerinin yaşı, dosyanın en son yazıldığı andan değil, audit log'un *ilk* event'inden sayılır. Birçok koşum için yeniden kullanılan bir dizinde bugün yazılmış bir paket de ufku aşmış olarak raporlanabilir: raporun listelediği bir şeyi silmeden önce dosyanın kendi tarihine bakın. + > **Kaldırıldı:** `evaluation.staging_ttl_days` (v0.5.5 itibarıyla kullanımdan > kaldırılmıştı) v0.8.0'da kaldırılmıştır. Artık geçerli tek form > `retention.staging_ttl_days`'dir. Eski anahtarı hâlâ set eden YAML dosyaları diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index e01cdc7f..cd1aee6a 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -275,18 +275,20 @@ long as necessary) and support erasure handling under Article 17 (requests are answered without undue delay, within the Article 12(3) one-month period, extendable by two further months). Choose values from your own necessity assessment; the defaults are ForgeLM choices, not legal periods. The horizons are **reported**, not enforced: `forgelm purge ---check-policy` lists what is past them and always exits 0, and nothing is +--check-policy` lists what is past them and exits 0 however many it finds (an unloadable `--config` exits 1), and nothing is deleted, blocked or scheduled automatically. The `enforce` knob is reserved: no command reads it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period (see AI Act Art. 18(1), documentation kept 10 years, and Art. 19(1) / 26(6), system logs at least six months, as applicable). Set to `0` to retain indefinitely; document why the entries stay necessary or the legal obligation you rely on. | -| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to retain indefinitely. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | -| `ephemeral_artefact_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/compliance/` and `/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to retain indefinitely. | -| `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to retain indefinitely. | +| `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period. Set to `0` to stop reporting it (nothing is ever deleted); keep it only for as long as the entries stay necessary or a specific legal retention duty you can name requires it. | +| `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to stop reporting it. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | +| `ephemeral_artefact_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/compliance/` and `/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to stop reporting it. | +| `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to stop reporting it. | | `enforce` | string | `"log_only"` | Reserved policy mode. No command acts on it: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and training and `--dry-run` warn when a config sets either of the last two. | +**Shared output directories.** `compliance/`, `data_audit_report.json` and the raw-document directories are aged from the audit log's *first* event, not from when the file was last written. In a directory reused for many runs, a bundle written today can be reported as past its horizon: check the file's own date before deleting anything the report lists. + > **Removed:** `evaluation.staging_ttl_days` (deprecated as of v0.5.5) was > removed in v0.8.0. `retention.staging_ttl_days` is now the only accepted form. > YAML files that still set the legacy key will fail config-load with `EXIT_CONFIG_ERROR`. diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index 9b9266f0..4214a26b 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -1,9 +1,9 @@ # `forgelm purge` — Subcommand Referansı -> **Hedef kitle:** Eğitim corpus'larına ve koşum-kapsamlı artefact'lara karşı GDPR Madde 17 silme taleplerini karşılayan ForgeLM operatörleri ve ortaya çıkan `data.erasure_*` audit zincirini doğrulayan denetçiler. +> **Hedef kitle:** Eğitim corpus'larına ve koşum-kapsamlı artefact'lara karşı GDPR Madde 17 silme taleplerinin corpus adımını yürüten ForgeLM operatörleri ve ortaya çıkan `data.erasure_*` audit zincirini doğrulayan denetçiler. > **Ayna:** [purge_subcommand.md](purge_subcommand.md) -`forgelm purge`, ForgeLM eğitim corpus'ları ve koşum artefact'ları için **GDPR Madde 17 silinme hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 21). Bir corpus satırını atomik olarak siler; `--run-id` ile bir koşumun staging dizinini (`--kind staging`) veya adı run id'yi embed eden compliance dosyalarını (`--kind artefacts`, aşağıya bakın) siler; her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. +`forgelm purge`, bir **GDPR Madde 17 silmesinin** corpus ve koşum-artefact adımı için operatör-yüzlü araçtır (Phase 21); yanıtın geri kalanı ve eğitilmiş ağırlıklar sizin sürecinizde kalır. Bir corpus satırını atomik olarak siler; `--run-id` ile bir koşumun staging dizinini (`--kind staging`) veya adı run id'yi embed eden compliance dosyalarını (`--kind artefacts`, aşağıya bakın) siler; her adımı tamper-evident `audit_log.jsonl` zincirine kaydeder. Deployer akışı (DSAR ticket → CLI çağrısı → doğrulama) için bkz. [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure-tr.md). Bu sayfa flag-başına, event-başına referanstır. @@ -52,13 +52,15 @@ JSONL eğitim corpus'undan `id` (veya `row_id`) field'ı ile tanımlanan tek bir > **`--kind artefacts` standart compliance paketini bulamaz.** Trainer ve exporter sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`; yapılandırıldığında `risk_assessment.json` / `annex_iv_metadata.json`) ve hiçbiri run id içermez. Normal bir çıktı dizininde komut bu yüzden hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar ve 1 ile çıkar; hiçbir şey silmez. Yalnızca adına run id'yi kendiniz yazdığınız dosyalara etki eder. Resolver düzeltilene kadar paketi depolama katmanınız üzerinden kaldırın. -`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +`--kind logs` **kasıtlı olarak yok**: audit log append-only'dir ve tool onu asla silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun belgelenmiş kararıdır: bir Madde 17(1) gerekçesi geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır (17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülük ya da kamu yararına bir görev için gerekli olduğu ölçüde; 17(3)(e) hukuki talepleri kapsar); hiçbir gerekçe geçerli değilse silme gerekmez, ancak talebe yine de Madde 12 süresi içinde gerekçeli bir yanıt verilir (Madde 12(4)). ### Retention-policy raporu (`--check-policy`) Salt-okunur tarama. `` üzerinde dolaşır, her artefact'ın yaşını (kanonik: `final_model.staging./` dizini için koşumun ilk audit event'i, yoksa audit-log genesis `timestamp`'i; fallback: dosya sistemi `mtime`'ı, `age_source=mtime` flag'iyle) yüklü config'in `retention:` horizon'larıyla karşılaştırır ve yapısal bir ihlal listesi yayar. -**Başarılı policy raporu her zaman 0 ile çıkar** (report-not-gate semantik, design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (1) ile çıkar — eksik, okunamaz veya Pydantic doğrulamasından geçemeyen explicit `--config` non-zero olarak yüzeyselleşir; böylece operatör "loader başarısız" durumunu "ihlal yok" sanmaz. CI gate kuran operatörler `--output-format json` kullanıp `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist)'e pipe eder. +**Paylaşılan çıktı dizinleri.** `compliance/`, `data_audit_report.json` ve ham-belge dizinlerinin yaşı, dosyanın en son yazıldığı andan değil, audit log'un *ilk* event'inden sayılır. Birçok koşum için yeniden kullanılan bir dizinde bugün yazılmış bir paket de ufku aşmış olarak raporlanabilir: raporun listelediği bir şeyi silmeden önce dosyanın kendi tarihine bakın. + +**Başarılı policy raporu her zaman 0 ile çıkar** (report-not-gate semantik, design §10 Q5). Config-load başarısızlıkları `EXIT_CONFIG_ERROR` (1) ile çıkar — eksik, okunamaz veya Pydantic doğrulamasından geçemeyen explicit `--config` non-zero olarak yüzeyselleşir; böylece operatör "loader başarısız" durumunu "ihlal yok" sanmaz. CI gate kuran operatörler `--output-format json` kullanır ve raporu kendileri sınar, örneğin `set -o pipefail` altında `jq -e '.success == true and has("count") and .count == 0'`: `count`, hiçbir `retention:` bloğu yüklenmediğinde yoktur; böylece gate hiç kontrol edilmemiş bir dizinde geçmek yerine başarısız olur, ve `--output-dir` gerçek eğitim çıktı dizini olmalıdır (olmayan bir dizin `count: 0` raporlar). ## Salt çözümlemesi @@ -71,7 +73,7 @@ Per-output-dir salt `/.forgelm_audit_salt`'ta ilk kullanımda oluşt ## `forgelm reverse-pii` ile cross-tool digest korelasyonu -`forgelm purge --row-id ` ve `forgelm reverse-pii --query --salt-source per_dir` **aynı** ``'e (yani aynı `.forgelm_audit_salt`'ı tüketerek) çalıştırıldığında ve purge için `FORGELM_AUDIT_SECRET` ayarlı değilse (`salt_source=per_dir`; ayarlıysa purge onu salt'a XOR'lar ve `salt_source=env_var` kaydeder, bu durumda bunun yerine aynı secret ile `--salt-source env_var` kullanarak sorgulayın), `data.erasure_*` event'lerindeki `target_id` ile `data.access_request_query` event'indeki `query_hash` byte-byte aynıdır. Compliance reviewer böylece aynı veri sahibi için Madde 17 silme ve Madde 15 erişim taleplerini cleartext identifier'ı asla görmeden korele edebilir. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. +`forgelm purge` `target_id` kaydeder (salt ile `--row-id` değerinin SHA-256'sı); `forgelm reverse-pii` ise `--query` değerinin aynı fonksiyonunu `query_hash` olarak kaydeder. İkisi yalnızca şu üç koşulun hepsi sağlandığında byte-byte aynıdır: değer aynı metindir (satırın `id` / `row_id` alanı ile sorgu aynı string olmalıdır; opak UUID satır id'leri bir e-posta sorgusuna asla eşit olmaz), iki komut da aynı `--output-dir`'i (aynı `.forgelm_audit_salt`'ı) kullanır ve `FORGELM_AUDIT_SECRET` ikisinde de aynı durumdadır (ayarsız ya da aynı değerle ayarlı; ayarlıysa purge `salt_source=env_var` kaydeder). Erişim talebini varsayılan düz-metin modunda çalıştırın: `--salt-source` gerekmez ve purge digest'lerini eşleştirmenin bir yolu değildir; çünkü `reverse-pii`'yi corpus *içinde* digest arayan bir hash-mask taramasına geçirir ve düz metin içeren bir corpus'ta hiçbir şey bulmaz. Id'ler farklıysa bir talebi silmesine, `forgelm purge --justification`'a verdiğiniz ticket referansı üzerinden bağlayın. Compliance reviewer böylece aynı veri sahibi için Madde 17 silme ve Madde 15 erişim taleplerini cleartext identifier'ı asla görmeden korele edebilir. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. ## Yayılan audit event'leri @@ -93,21 +95,23 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak | Kod | Anlamı | |---|---| | 0 | Başarı veya başarılı `--check-policy` raporu (report-not-gate semantik). | -| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus`, karşılıklı dışlayıcı flag kombinasyonu veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | -| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, atomic-rename başarısızlığı. | +| 1 | Config hatası: bilinmeyen `--row-id`, `--run-id` ile eşleşen staging dizini / compliance dosyası yok (standart sabit-adlı pakete karşı `--kind artefacts` dahil), eksik `--corpus` ya da `--corpus`'suz `--row-id`, eksik `--kind`, seçilmemiş mod veya eksik / okunamaz / Pydantic doğrulamasından geçemeyen `--check-policy --config `. | +| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, atomic-rename başarısızlığı; ayrıca argüman-parser kullanım hatası (`--row-id` ile `--run-id` gibi birlikte verilen iki mod, `--kind logs`), JSON zarfı yerine usage basar. | `--check-policy` asla 3 veya 4 dönmez. Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) eğitim pipeline'ı için ayrılmıştır ve bu subcommand'ın yüzeyinin parçası değildir. ## JSON çıktı zarfı -`--output-format json` ile her çağrı stdout'a tam olarak bir JSON nesnesi yazdırır. Şekil moda göre değişir — toplam üç zarf. +`--output-format json` ile argüman ayrıştırmayı geçen her çağrı stdout'a tam olarak bir JSON nesnesi yazdırır (argüman-parser hatası bunun yerine usage basar ve 2 ile çıkar). Başarılı ya da başarısız her zarf boolean bir `success` taşır; ona göre dallanın. Şekil moda göre değişir. **Row modu (başarı)** — `_perform_row_erasure_and_audit`'den yayılır: ```json { + "success": true, "mode": "row", "dry_run": false, + "row_id_hash": "01aeebf6566c0f6ec4a95a4678a2eb3ef45022ffb7bd9d5662b706d6d6506efc", "salt_source": "per_dir", "corpus_path": "/abs/path/data/train.jsonl", "matches": 1, @@ -117,22 +121,23 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak } ``` -`--dry-run` çağrılarında aynı zarf `dry_run: true` ile ve `bytes_freed` olmadan görünür (rewrite hiç olmaz). +`row_id_hash`, salt ile `--row-id` değerinin tam SHA-256 hex digest'idir; `data.erasure_*` event'leri onun ilk 16 hex hanesini `target_id` olarak taşır. `--dry-run` çağrılarında aynı zarf `dry_run: true` ile ve `bytes_freed` olmadan görünür (rewrite hiç olmaz). **Run modu (başarı)** — `_run_purge_run_id`'den yayılır: ```json { + "success": true, "mode": "run", "kind": "staging", "dry_run": false, "run_id": "fg-abc123def456", - "deleted": ["/abs/path/outputs/run42/final_model.staging.fg-abc123def456"], - "bytes_freed": 102400000 + "deleted": ["/final_model.staging.fg-abc123def456"], + "bytes_freed": 2048 } ``` -`deleted` gerçekten silinen mutlak yolların **listesidir** (ya da `--dry-run`'da `would_delete` anahtarı aynı şekli taşır ve `deleted` yoktur). +`deleted` gerçekten silinen yolların **listesidir**; yollar `--output-dir`'e verdiğiniz biçimde yazılır (ya da `--dry-run`'da `would_delete` anahtarı aynı şekli taşır ve `deleted` yoktur). **`--check-policy` modu** — `_run_purge_check_policy`'ten yayılır: @@ -141,29 +146,36 @@ Altı event de [`audit_event_catalog.md`](audit_event_catalog-tr.md)'deki ortak "success": true, "violations": [ { - "artefact_kind": "ephemeral_artefact", - "path": "/abs/path/outputs/run42/compliance/data_audit_report.json", + "artefact_kind": "compliance_bundle", + "path": "/compliance", "age_days": 121.4, "horizon_days": 90, "age_source": "audit" + }, + { + "artefact_kind": "staging_dir[fg-old]", + "path": "/final_model.staging.fg-old", + "age_days": 121.4, + "horizon_days": 7, + "age_source": "audit" } ], - "count": 1 + "count": 2 } ``` -`age_source` ∈ `{audit, mtime}` — `audit` kanonik yaş (koşumun ilk audit event timestamp'i); `mtime` operatörün farkındalığı için işaretlenmiş filesystem fallback'i. +`artefact_kind` şunlardan biridir: `audit_log`, `staging_dir`, `staging_dir[]`, `compliance_bundle`, `data_audit_report` veya `raw_documents[]`. `age_source` ∈ `{audit, mtime}` — `audit` yaşın audit log'dan sayıldığı anlamına gelir (staging dizini için koşumun ilk event'i, yoksa log'un ilk event'i); `mtime` operatörün farkındalığı için işaretlenmiş filesystem fallback'idir. Hiçbir `retention:` bloğu yüklenmediğinde (`--config` yok ya da bloğu olmayan bir config) zarf `count`'suz `{"success": true, "violations": [], "note": "..."}` olur: hiçbir şey kontrol edilmemiştir. **Hata zarfı (her mod)** — `_output_error_and_exit`: ```json { "success": false, - "error": "Row id 'ali@example.com' not found in 'data/train.jsonl'." + "error": "No row with id= in 'data/train.jsonl'. Refusing to delete." } ``` -Asimetri notu: row/run başarı zarfları açık `success` field'ı **taşımaz** (`mode` ayraç başarı sinyalidir); `--check-policy` ve hata zarfı taşır. CI gate'leri `mode` anahtarına (başarı yolu) ya da `success: false`'a (hata yolu) dallanmalı; `success: true`'nin varlığı/yokluğuna değil. +Tanımlayıcı asla geri yazılmaz: mesaj `--row-id` değeri yerine bir hash öneki taşır. ## Bkz. diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index dc9f64e6..dbe74ec6 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -1,9 +1,9 @@ # `forgelm purge` — Subcommand Reference -> **Audience:** ForgeLM operators honouring GDPR Article 17 erasure requests against training corpora and run-scoped artefacts, plus auditors verifying the resulting `data.erasure_*` audit chain. +> **Audience:** ForgeLM operators carrying out the corpus step of GDPR Article 17 erasure requests against training corpora and run-scoped artefacts, plus auditors verifying the resulting `data.erasure_*` audit chain. > **Mirror:** [purge_subcommand-tr.md](purge_subcommand-tr.md) -`forgelm purge` is the operator-facing implementation of the **GDPR Article 17 right-to-erasure** for ForgeLM training corpora and run artefacts (Phase 21). It atomically erases a corpus row; with `--run-id` it deletes a run's staging directory (`--kind staging`) or compliance files whose names embed the run id (`--kind artefacts`, see below); and it records every step in the tamper-evident `audit_log.jsonl` chain. +`forgelm purge` is the operator-facing tool for the corpus and run-artefact step of a **GDPR Article 17 erasure** (Phase 21); the rest of the response, and the trained weights, stay your process. It atomically erases a corpus row; with `--run-id` it deletes a run's staging directory (`--kind staging`) or compliance files whose names embed the run id (`--kind artefacts`, see below); and it records every step in the tamper-evident `audit_log.jsonl` chain. For the deployer-flow walkthrough (DSAR ticket → CLI invocation → verification), see [`../guides/gdpr_erasure.md`](../guides/gdpr_erasure.md). This page is the per-flag, per-event reference. @@ -52,13 +52,15 @@ Erases a single row identified by its `id` (or `row_id`) field from a JSONL trai > **`--kind artefacts` does not find the standard compliance bundle.** The trainer and exporter write fixed basenames (`compliance_report.json`, `training_manifest.yaml`, `data_provenance.json`, `data_governance_report.json`, and `risk_assessment.json` / `annex_iv_metadata.json` when configured), none of which contains a run id. On a normal output directory the command therefore finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`) and exits 1; it deletes nothing. It acts only on files you have named with the run id yourself. Until the resolver is fixed, remove the bundle through your storage layer. -`--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it. Whether entries that name the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. +`--kind logs` is **intentionally absent**: the audit log is append-only and the tool never deletes it. Whether entries that name a data subject may be kept is the controller's documented decision: if an Article 17(1) ground applies they stay only where an Article 17(3) exception covers them (17(3)(b) only to the extent processing is necessary for a legal obligation under Union or Member State law to which the controller is subject, or for a task in the public interest; 17(3)(e) covers legal claims); if no ground applies, erasure is not owed, but the request still gets a reasoned answer within the Article 12 deadline (Article 12(4)). ### Retention-policy report (`--check-policy`) Read-only scan. Walks ``, compares each artefact's age (canonical: the run's first audit event for a `final_model.staging./` directory, else the audit log's genesis `timestamp`; fallback: filesystem `mtime` with `age_source=mtime`) against the loaded config's `retention:` horizons, and emits a structured violation list. -**Successful policy reports always exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit `EXIT_CONFIG_ERROR` (1) — an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as a non-zero so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and pipe to `jq -e '.success and (.violations | length == 0)'` (exit 1 while violations exist) themselves. +**Shared output directories.** `compliance/`, `data_audit_report.json` and the raw-document directories are aged from the audit log's *first* event, not from when the file was last written. In a directory reused for many runs, a bundle written today can be reported as past its horizon: check the file's own date before deleting anything the report lists. + +**Successful policy reports always exit 0** (report-not-gate semantic per design §10 Q5). Config-load failures exit `EXIT_CONFIG_ERROR` (1) — an explicit `--config` that is missing, unreadable, or fails Pydantic validation surfaces as a non-zero so the operator does not mistake "loader failed" for "no violations". Operators wiring a CI gate use `--output-format json` and test the report themselves, for example `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`: `count` is absent when no `retention:` block was loaded, so the gate fails instead of passing on a directory that was never checked, and `--output-dir` must be the real training output directory (a missing one reports `count: 0`). ## Salt resolution @@ -71,7 +73,7 @@ The per-output-dir salt at `/.forgelm_audit_salt` is created on firs ## Cross-tool digest correlation with `forgelm reverse-pii` -When `forgelm purge --row-id ` and `forgelm reverse-pii --query --salt-source per_dir` run against the **same** `` (i.e. consume the same `.forgelm_audit_salt`) and `FORGELM_AUDIT_SECRET` was unset for the purge (`salt_source=per_dir`; when it was set, the purge XORs it into the salt and records `salt_source=env_var`, so query with `--salt-source env_var` and the same secret instead), the `target_id` field on `data.erasure_*` events and the `query_hash` field on `data.access_request_query` events are byte-identical. A compliance reviewer can therefore correlate Article 17 erasures and Article 15 access requests for the same data subject without ever seeing the cleartext identifier. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. +`forgelm purge` records `target_id` (SHA-256 of the salt plus the `--row-id` value) and `forgelm reverse-pii` records `query_hash` (the same function of the `--query` value). They are byte-identical only when all three hold: the value is the same text (a row's `id` / `row_id` and the query must be the same string; opaque UUID row ids never equal an e-mail query), both commands use the same `--output-dir` (the same `.forgelm_audit_salt`), and `FORGELM_AUDIT_SECRET` is in the same state for both (unset, or set to the same value; purge records `salt_source=env_var` when it is set). Run the access request in its default plaintext mode: `--salt-source` is not needed and is not a way to match purge digests, because it switches `reverse-pii` to a hash-mask scan that looks for digests *inside the corpus* and finds nothing in a corpus that holds plaintext. When the ids differ, link a request to its erasure through the ticket reference you pass to `forgelm purge --justification`. A compliance reviewer can therefore correlate Article 17 erasures and Article 15 access requests for the same data subject without ever seeing the cleartext identifier. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. ## Audit events emitted @@ -93,21 +95,23 @@ All six events ride the common envelope from [`audit_event_catalog.md`](audit_ev | Code | Meaning | |---|---| | 0 | Success, or a successful `--check-policy` report (report-not-gate semantic). | -| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), missing `--corpus`, mutually-exclusive flag combination, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | -| 2 | Runtime error: I/O failure, permission denied, atomic-rename failure. | +| 1 | Config error: unknown `--row-id`, no staging directory / compliance file matching `--run-id` (including `--kind artefacts` against the standard fixed-name bundle), a missing `--corpus` or `--row-id` without `--corpus`, a missing `--kind`, no mode selected, or `--check-policy --config ` that is missing / unreadable / fails Pydantic validation. | +| 2 | Runtime error: I/O failure, permission denied, atomic-rename failure; also an argument-parser usage error (two modes together such as `--row-id` with `--run-id`, `--kind logs`), which prints usage instead of a JSON envelope. | `--check-policy` never returns code 3 or 4. Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are reserved for the training pipeline and are not part of this subcommand's surface. ## JSON output envelope -With `--output-format json` every invocation prints exactly one JSON object on stdout. The shape varies by mode — three envelopes in total. +With `--output-format json` every invocation that gets past argument parsing prints exactly one JSON object on stdout (an argument-parser error prints usage and exits 2 instead). Every envelope, success or failure, carries a boolean `success`; branch on it. The shape varies by mode. **Row mode (success)** — emitted by `_perform_row_erasure_and_audit`: ```json { + "success": true, "mode": "row", "dry_run": false, + "row_id_hash": "01aeebf6566c0f6ec4a95a4678a2eb3ef45022ffb7bd9d5662b706d6d6506efc", "salt_source": "per_dir", "corpus_path": "/abs/path/data/train.jsonl", "matches": 1, @@ -117,22 +121,23 @@ With `--output-format json` every invocation prints exactly one JSON object on s } ``` -For `--dry-run` invocations the same envelope appears with `dry_run: true` and without `bytes_freed` (the rewrite never happens). +`row_id_hash` is the full SHA-256 hex digest of the salt plus the `--row-id` value; the `data.erasure_*` events carry its first 16 hex digits as `target_id`. For `--dry-run` invocations the same envelope appears with `dry_run: true` and without `bytes_freed` (the rewrite never happens). **Run mode (success)** — emitted by `_run_purge_run_id`: ```json { + "success": true, "mode": "run", "kind": "staging", "dry_run": false, "run_id": "fg-abc123def456", - "deleted": ["/abs/path/outputs/run42/final_model.staging.fg-abc123def456"], - "bytes_freed": 102400000 + "deleted": ["/final_model.staging.fg-abc123def456"], + "bytes_freed": 2048 } ``` -`deleted` is a **list** of absolute paths actually removed (or, on `--dry-run`, the `would_delete` key carries the same shape and `deleted` is absent). +`deleted` is a **list** of the paths actually removed, spelled relative to `--output-dir` as you passed it (or, on `--dry-run`, the `would_delete` key carries the same shape and `deleted` is absent). **`--check-policy` mode** — emitted by `_run_purge_check_policy`: @@ -141,29 +146,36 @@ For `--dry-run` invocations the same envelope appears with `dry_run: true` and w "success": true, "violations": [ { - "artefact_kind": "ephemeral_artefact", - "path": "/abs/path/outputs/run42/compliance/data_audit_report.json", + "artefact_kind": "compliance_bundle", + "path": "/compliance", "age_days": 121.4, "horizon_days": 90, "age_source": "audit" + }, + { + "artefact_kind": "staging_dir[fg-old]", + "path": "/final_model.staging.fg-old", + "age_days": 121.4, + "horizon_days": 7, + "age_source": "audit" } ], - "count": 1 + "count": 2 } ``` -`age_source` ∈ `{audit, mtime}` — `audit` is the canonical age (run's first audit event timestamp); `mtime` is the filesystem fallback flagged for the operator's awareness. +`artefact_kind` is one of `audit_log`, `staging_dir`, `staging_dir[]`, `compliance_bundle`, `data_audit_report` or `raw_documents[]`. `age_source` ∈ `{audit, mtime}` — `audit` is the age counted from the audit log (the run's first event for a staging directory, else the log's first event); `mtime` is the filesystem fallback flagged for the operator's awareness. When no `retention:` block was loaded (no `--config`, or a config without one) the envelope is `{"success": true, "violations": [], "note": "..."}` with no `count`: nothing was checked. **Failure envelope (any mode)** — `_output_error_and_exit`: ```json { "success": false, - "error": "Row id 'ali@example.com' not found in 'data/train.jsonl'." + "error": "No row with id= in 'data/train.jsonl'. Refusing to delete." } ``` -Note the asymmetry: row/run success envelopes do **not** carry an explicit `success` field (the `mode` discriminator is the success signal); `--check-policy` and the failure envelope do. Operators wiring CI gates should branch on the `mode` key (success path) or `success: false` (failure path), not on the presence/absence of `success: true`. +The identifier is never echoed: the message carries a hash prefix instead of the `--row-id` value. ## See also diff --git a/docs/reference/reverse_pii_subcommand-tr.md b/docs/reference/reverse_pii_subcommand-tr.md index 55cc2c5e..ad5be001 100644 --- a/docs/reference/reverse_pii_subcommand-tr.md +++ b/docs/reference/reverse_pii_subcommand-tr.md @@ -1,9 +1,9 @@ # `forgelm reverse-pii` — Subcommand Referansı -> **Hedef kitle:** Eğitim corpus'larına karşı GDPR Madde 15 erişim hakkı taleplerini yanıtlayan ForgeLM operatörleri ve ortaya çıkan `data.access_request_query` audit satırını doğrulayan denetçiler. +> **Hedef kitle:** Eğitim corpus'larına karşı GDPR Madde 15 erişim yanıtları hazırlayan ForgeLM operatörleri ve ortaya çıkan `data.access_request_query` audit satırını doğrulayan denetçiler. > **Ayna:** [reverse_pii_subcommand.md](reverse_pii_subcommand.md) -`forgelm reverse-pii`, ForgeLM eğitim corpus'ları için **GDPR Madde 15 erişim hakkı**'nın operatör-yüzlü uygulamasıdır (Phase 38). [`forgelm purge`](purge_subcommand-tr.md) "satırımı sil" derken, `reverse-pii` "identifier'ımın geçtiği her satırı bul" der. +`forgelm reverse-pii`, ForgeLM eğitim corpus'ları için bir **GDPR Madde 15 erişim yanıtının** girdisidir (Phase 38): bir identifier'ın nerede geçtiğini bulur; sıfır eşleşme yalnızca literal metnin taradığınız dosyalarda olmadığını söyler. [`forgelm purge`](purge_subcommand-tr.md) "satırımı sil" derken, `reverse-pii` "identifier'ımın geçtiği her satırı bul" der. ## Synopsis @@ -47,7 +47,7 @@ $ forgelm reverse-pii --query "alice@example.com" --type email \ --salt-source per_dir --output-dir ./outputs data/*.jsonl ``` -`--salt-source env_var` `FORGELM_AUDIT_SECRET`'in set olmasını gerektirir; `per_dir` `/.forgelm_audit_salt`'taki salt dosyasını okur. +`--salt-source env_var` `FORGELM_AUDIT_SECRET`'in set olmasını gerektirir; `per_dir` `/.forgelm_audit_salt`'taki salt dosyasını okur. `forgelm ingest --pii-mask` digest değil `[REDACTED]` yazar, bu yüzden bu mod ForgeLM'in kendi maskelediği bir corpus'ta hiçbir şey bulmaz; onlar için varsayılan düz-metin taramasını kullanın. ## Identifier tipleri @@ -59,7 +59,7 @@ $ forgelm reverse-pii --query "alice@example.com" --type email \ ## `forgelm purge` ile cross-tool digest korelasyonu -`forgelm reverse-pii --query --salt-source per_dir` ve `forgelm purge --row-id ` **aynı** ``'e (yani aynı `.forgelm_audit_salt`'ı tüketerek) çalıştırıldığında, `data.access_request_query` üzerindeki `query_hash` ile `data.erasure_*` event'leri üzerindeki `target_id` byte-byte aynıdır. Bu, compliance reviewer'ın Madde 15 erişim talebini ve aynı subject için Madde 17 silmesini tek bir audit zincirinde, cleartext identifier operatör terminalinden hiç çıkmadan korele etmesini sağlar. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. +`forgelm purge` `target_id` kaydeder (salt ile `--row-id` değerinin SHA-256'sı); `forgelm reverse-pii` ise `--query` değerinin aynı fonksiyonunu `query_hash` olarak kaydeder. İkisi yalnızca şu üç koşulun hepsi sağlandığında byte-byte aynıdır: değer aynı metindir (satırın `id` / `row_id` alanı ile sorgu aynı string olmalıdır; opak UUID satır id'leri bir e-posta sorgusuna asla eşit olmaz), iki komut da aynı `--output-dir`'i (aynı `.forgelm_audit_salt`'ı) kullanır ve `FORGELM_AUDIT_SECRET` ikisinde de aynı durumdadır (ayarsız ya da aynı değerle ayarlı; ayarlıysa purge `salt_source=env_var` kaydeder). Erişim talebini varsayılan düz-metin modunda çalıştırın: `--salt-source` gerekmez ve purge digest'lerini eşleştirmenin bir yolu değildir; çünkü `reverse-pii`'yi corpus *içinde* digest arayan bir hash-mask taramasına geçirir ve düz metin içeren bir corpus'ta hiçbir şey bulmaz. Id'ler farklıysa bir talebi silmesine, `forgelm purge --justification`'a verdiğiniz ticket referansı üzerinden bağlayın. Bu, compliance reviewer'ın Madde 15 erişim talebini ve aynı subject için Madde 17 silmesini tek bir audit zincirinde, cleartext identifier operatör terminalinden hiç çıkmadan korele etmesini sağlar. `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir` ile pin'lendi. `/.forgelm_audit_salt`'taki salt **yalnızca identifier-hash salt'ıdır** — audit-chain HMAC anahtarına katılmaz; o anahtar bağımsız olarak `forgelm.compliance.AuditLogger.__init__` içinde `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` şeklinde türetilir. @@ -71,7 +71,7 @@ Her çağrı tam olarak bir event yayar ([katalog satırı](audit_event_catalog- |---|---|---| | `data.access_request_query` | Scan tamamlandıktan sonra (veya mid-scan I/O hatası sonrası — `error_class` / `error_message` ile). | `query_hash` (raw identifier'ın salt'lı SHA-256'sı — asla raw; purge'ün per-output-dir salt'ını yeniden kullanır), `identifier_type` ∈ `{literal, email, phone, tr_id, us_ssn, iban, credit_card, custom}`, `scan_mode` ∈ `{plaintext, hash}`, `salt_source` ∈ `{plaintext, per_dir, env_var}`, `files_scanned` (path'ler), `match_count`, opsiyonel `error_class` / `error_message` | -Chain satırı **hash'li** identifier'ı kaydeder; böylece zincirin kendisi subject verisini sızdırmaz. +Chain satırı **hash'li** identifier'ı kaydeder; böylece zincir onu açık halde tutmaz. Digest pseudonymised'dır, anonim değildir: salt dosyası log'un yanında durur, bu yüzden ikisini de aynı erişim kontrolleriyle koruyun ve digest'leri kişisel veri sayın. ## Exit kodları diff --git a/docs/reference/reverse_pii_subcommand.md b/docs/reference/reverse_pii_subcommand.md index 97c9b638..89012d3a 100644 --- a/docs/reference/reverse_pii_subcommand.md +++ b/docs/reference/reverse_pii_subcommand.md @@ -1,9 +1,9 @@ # `forgelm reverse-pii` — Subcommand Reference -> **Audience:** ForgeLM operators answering GDPR Article 15 right-of-access requests against training corpora, plus auditors verifying the resulting `data.access_request_query` audit row. +> **Audience:** ForgeLM operators preparing GDPR Article 15 access responses against training corpora, plus auditors verifying the resulting `data.access_request_query` audit row. > **Mirror:** [reverse_pii_subcommand-tr.md](reverse_pii_subcommand-tr.md) -`forgelm reverse-pii` is the operator-facing implementation of the **GDPR Article 15 right of access** for ForgeLM training corpora (Phase 38). Where [`forgelm purge`](purge_subcommand.md) answers "delete my row", `reverse-pii` answers "find every line where my identifier appears". +`forgelm reverse-pii` is an input to a **GDPR Article 15 access response** for ForgeLM training corpora (Phase 38): it finds where an identifier appears, and a zero-match scan only says the literal text is absent from the files you scanned. Where [`forgelm purge`](purge_subcommand.md) answers "delete my row", `reverse-pii` answers "find every line where my identifier appears". ## Synopsis @@ -47,7 +47,7 @@ $ forgelm reverse-pii --query "alice@example.com" --type email \ --salt-source per_dir --output-dir ./outputs data/*.jsonl ``` -`--salt-source env_var` requires `FORGELM_AUDIT_SECRET` to be set; `per_dir` reads the salt file at `/.forgelm_audit_salt`. +`--salt-source env_var` requires `FORGELM_AUDIT_SECRET` to be set; `per_dir` reads the salt file at `/.forgelm_audit_salt`. `forgelm ingest --pii-mask` writes `[REDACTED]`, not digests, so this mode finds nothing in a corpus ForgeLM masked itself; use the default plaintext scan for those. ## Identifier types @@ -59,7 +59,7 @@ $ forgelm reverse-pii --query "alice@example.com" --type email \ ## Cross-tool digest correlation with `forgelm purge` -When `forgelm reverse-pii --query --salt-source per_dir` and `forgelm purge --row-id ` run against the **same** `` (i.e. consume the same `.forgelm_audit_salt`), the `query_hash` field on `data.access_request_query` and the `target_id` field on `data.erasure_*` events are byte-identical. This lets a compliance reviewer correlate the Article 15 access request and the Article 17 erasure for the same data subject in a single audit chain, without the cleartext identifier ever leaving the operator's terminal. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. +`forgelm purge` records `target_id` (SHA-256 of the salt plus the `--row-id` value) and `forgelm reverse-pii` records `query_hash` (the same function of the `--query` value). They are byte-identical only when all three hold: the value is the same text (a row's `id` / `row_id` and the query must be the same string; opaque UUID row ids never equal an e-mail query), both commands use the same `--output-dir` (the same `.forgelm_audit_salt`), and `FORGELM_AUDIT_SECRET` is in the same state for both (unset, or set to the same value; purge records `salt_source=env_var` when it is set). Run the access request in its default plaintext mode: `--salt-source` is not needed and is not a way to match purge digests, because it switches `reverse-pii` to a hash-mask scan that looks for digests *inside the corpus* and finds nothing in a corpus that holds plaintext. When the ids differ, link a request to its erasure through the ticket reference you pass to `forgelm purge --justification`. This lets a compliance reviewer correlate the Article 15 access request and the Article 17 erasure for the same data subject in a single audit chain, without the cleartext identifier ever leaving the operator's terminal. Pinned by `tests/test_reverse_pii.py::test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`. The salt at `/.forgelm_audit_salt` is the **identifier-hash salt only** — it does NOT participate in the audit-chain HMAC key, which is derived independently as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` inside `forgelm.compliance.AuditLogger.__init__`. @@ -71,7 +71,7 @@ Every invocation emits exactly one event ([catalog row](audit_event_catalog.md#a |---|---|---| | `data.access_request_query` | After the scan completes (or after a mid-scan I/O failure — with `error_class` / `error_message`). | `query_hash` (salted SHA-256 of raw identifier — never raw; reuses purge's per-output-dir salt), `identifier_type` ∈ `{literal, email, phone, tr_id, us_ssn, iban, credit_card, custom}`, `scan_mode` ∈ `{plaintext, hash}`, `salt_source` ∈ `{plaintext, per_dir, env_var}`, `files_scanned` (paths), `match_count`, optional `error_class` / `error_message` | -The chain row records the **hashed** identifier so the chain itself does not leak the subject's data. +The chain row records the **hashed** identifier so the chain does not store it in clear. The digest is pseudonymised, not anonymous: the salt file sits next to the log, so protect both with the same access controls and treat the digests as personal data. ## Exit codes diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index 057b6a84..aa015fe1 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -1,22 +1,22 @@ --- title: GDPR Erasure & Access -description: Honour Article 15 (right of access) and Article 17 (right to erasure) requests with the `forgelm purge` and `forgelm reverse-pii` subcommands. +description: Support Article 15 (access) and Article 17 (erasure) requests with the `forgelm purge` and `forgelm reverse-pii` subcommands. --- # GDPR Erasure & Access -ForgeLM ships sister subcommands for the two most-common GDPR data-subject rights: `forgelm purge` (Article 17 — right to erasure) and `forgelm reverse-pii` (Article 15 — right of access). They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain — without the cleartext identifier ever leaving the operator's terminal. +ForgeLM ships sister subcommands that support two common GDPR data-subject requests: `forgelm purge` removes rows (one step of an Article 17 erasure) and `forgelm reverse-pii` finds rows (an input to an Article 15 response). Neither is a complete response on its own. They share a per-output-dir salt so a compliance reviewer can correlate erasures and access requests for the same data subject in a single tamper-evident audit chain — without the cleartext identifier ever leaving the operator's terminal. This page is the operator quick-reference. The deeper deployer-flow walkthrough lives in the [GDPR erasure deployer guide on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure.md); per-flag references are in the in-manual [CLI Reference](#/reference/cli) and the dedicated subcommand pages [`purge_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand.md) + [`reverse_pii_subcommand.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand.md) on GitHub. ## Two rights, one chain -| Right | Subcommand | What it answers | +| Request | Subcommand | What it does | |---|---|---| -| **Article 15** — right of access | `forgelm reverse-pii` | "Do you hold any record of me?" — scans JSONL corpora for the subject's identifier. | -| **Article 17** — right to erasure | `forgelm purge` | "Delete every record of me." — atomically removes a corpus row; with `--run-id` it removes a staging directory or compliance files whose names embed the run id. | +| **Article 15** — access | `forgelm reverse-pii` | Finds the lines that contain the subject's identifier in the JSONL corpora you scan: an input to the response, not the response. | +| **Article 17** — erasure | `forgelm purge` | Atomically removes a corpus row by id (one step of an erasure, not all of it); with `--run-id` it removes a staging directory or compliance files whose names embed the run id. | -Both subcommands hash the subject's identifier with the **same** per-output-dir salt (`/.forgelm_audit_salt`) before recording the audit row. The `query_hash` field on `data.access_request_query` and the `target_id` field on `data.erasure_*` are byte-identical for the same subject, which lets a reviewer correlate the two without cleartext. +Both subcommands hash the subject's identifier with the **same** per-output-dir salt (`/.forgelm_audit_salt`) before recording the audit row. The `query_hash` field on `data.access_request_query` and the `target_id` field on `data.erasure_*` are byte-identical only when the `--row-id` value and the `--query` value are the same text, both commands use the same `--output-dir`, and `FORGELM_AUDIT_SECRET` is in the same state for both; otherwise link the two with the ticket reference you pass to `forgelm purge --justification`. ## Article 17 — `forgelm purge` @@ -44,15 +44,16 @@ Six events ride the audit chain (see the [Audit Event Catalog on GitHub](https:/ - `data.erasure_failed` — emitted instead when the disk operation raised, or no matching row/run was found. - `data.erasure_warning_memorisation` — corpus-row erasure × a `final_model/` exists directly under `--output-dir` (the row is gone from disk but may still be memorised in the trained weights). - `data.erasure_warning_synthetic_data_present` — corpus-row erasure × `synthetic_data*.jsonl` exists in `output_dir`. -- `data.erasure_warning_external_copies` — loaded config has a non-empty `webhook` block; downstream consumers may have received notices. +- `data.erasure_warning_external_copies` — corpus-row erasure only, and only when you pass `--config`: the config's `webhook` block is non-empty (a literal `url`, or a `url_env` variable exported in the purge environment); downstream consumers may have received notices. ### What it does NOT do - **Re-train models.** Removing a row from the corpus does not unmemorise it from already-trained weights — full retraining without the erased row is the only proper mitigation. The tool emits `data.erasure_warning_memorisation` so the gap is visible. -- **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history. Whether entries that name the data subject may be kept is the controller's decision, and the controller documents why: the entries are still necessary for their purpose, or an Article 17(3) exception applies. Article 17(3)(b), for example, applies only to the extent a specific Union or Member State legal obligation requires the processing, and you have to be able to name that obligation. +- **Delete the audit log.** The tool records the erasure as a new event rather than rewriting history, and never deletes an audit-log entry. Whether entries that name the data subject may be kept is the controller's decision, and it has to be documented. If an Article 17(1) ground applies, the entries stay only where an Article 17(3) exception covers them: 17(3)(b) applies only to the extent processing is necessary for compliance with a legal obligation under Union or Member State law to which the controller is subject (or for a task in the public interest or in the exercise of official authority), and 17(3)(e) covers legal claims. If no Article 17(1) ground applies (for example the entries are still necessary for the purpose they were collected for), erasure is not owed, but the request still gets a reasoned answer within the Article 12 deadline, with the right to complain (Article 12(4)). - **Remove an unmarked legacy `final_model.staging/`.** `--kind staging` deletes `final_model.staging./`, and the legacy `final_model.staging/` only when it holds a `staging_run_id` file equal to the run id. ForgeLM does not write that marker, so remove an unmarked legacy directory yourself. - **Find the standard compliance bundle with `--kind artefacts`.** The trainer writes fixed basenames (`compliance_report.json`, `training_manifest.yaml`, ...) that contain no run id, and the resolver matches only filenames that embed it. On a normal output directory the command finds nothing, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing; remove the bundle through your storage layer. -- **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (Article 19 requires telling each recipient the data was disclosed to; Article 17(2) adds reasonable steps where the controller made the data public). The `data.erasure_warning_external_copies` event is the explicit reminder. +- **Push notices to downstream consumers.** Webhook receivers, dataset mirrors, and deployed model endpoints are the operator's runtime-layer responsibility (GDPR Article 19 requires telling each recipient the data was disclosed to about the erasure, unless that proves impossible or involves disproportionate effort, and telling the data subject about those recipients on request; Article 17(2) adds reasonable steps where the controller made the data public). The `data.erasure_warning_external_copies` event is the explicit reminder. +- **Make the whole response.** A complete response also needs steps ForgeLM does not perform: verifying the requester's identity (Article 12(6)); answering within one month of receipt (Article 12(3); extendable by two further months for complex or numerous requests, with notice in the first month); giving the data subject a copy of the data and the Article 15(1)(a)-(h) information (Article 15(3)) without adversely affecting other people's rights (Article 15(4)); deciding whether an Article 17(1) ground applies and which Article 17(3) exception, if any, covers what you keep; telling recipients (Article 19); re-running the access search after a purge to confirm the rows are gone; and, if you refuse a request, giving reasons and the right to complain within the same deadline (Article 12(4)). Whether the trained model counts as personal data is a separate question (EDPB Opinion 28/2024). ## Article 15 — `forgelm reverse-pii` @@ -77,7 +78,7 @@ One event per invocation (see catalog): - `data.access_request_query` — emitted after the scan (or after a mid-scan I/O failure, with `error_*` fields). Carries the **hashed** `query_hash`, the `identifier_type`, the `scan_mode`, the list of `files_scanned`, the `match_count`, and the `salt_source`. -The chain row never carries the cleartext identifier — the salt protects against wordlist attacks against the audit log itself. +The chain row never carries the cleartext identifier, only a salted digest. The digest is pseudonymised, not anonymous: the salt file sits next to the log, so whoever can read both can test a guessed identifier against it. Protect the log and the salt with the same access controls and treat the digests as personal data. ### `--type custom` regex caveat @@ -89,7 +90,7 @@ The per-output-dir salt at `/.forgelm_audit_salt` (mode `0600`, atom The two primitives are intentionally separate: -- Rotating `FORGELM_AUDIT_SECRET` rotates the chain HMAC key **and**, whenever the variable is set, the identifier-hash salt: `forgelm purge` XORs the first 16 bytes of the secret into the per-output-dir salt and records `salt_source=env_var`, so its `target_id` digests change with the secret. A `forgelm reverse-pii` query correlates with earlier purge events only when it uses the same salt source (`--salt-source env_var` with the same secret, or `--salt-source per_dir` for purges run without the variable). +- Rotating `FORGELM_AUDIT_SECRET` rotates the chain HMAC key **and**, whenever the variable is set, the identifier-hash salt: `forgelm purge` XORs the first 16 bytes of the secret into the per-output-dir salt and records `salt_source=env_var`, so its `target_id` digests change with the secret. A plaintext `forgelm reverse-pii` query (no `--salt-source`) resolves the salt the same way, so it correlates with a purge run under the same secret state. `--salt-source` is a different feature: a hash-mask scan for corpora whose identifiers were hashed outside ForgeLM, which finds nothing in a plaintext corpus. - Inspecting either primitive does not reveal the other. - The per-output-dir salt is always an input to identifier hashing and persists for the lifetime of the output dir; the secret, when set, is a second input. @@ -99,9 +100,9 @@ Both subcommands follow the project's `0/1/2/3/4` contract: | Code | Meaning | |---|---| -| 0 | Success. `forgelm purge --check-policy` exits 0 even when violations are found (report-not-gate semantic). `forgelm reverse-pii` exits 0 even when no matches are found (Article 15 explicitly accepts "no records" as a valid answer). | -| 1 | Config error: bad flags, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded. | -| 2 | Runtime error: I/O failure, permission denied, unwritable `--audit-dir`, ReDoS SIGALRM timeout, atomic-rename failure. | +| 0 | Success. `forgelm purge --check-policy` exits 0 even when violations are found (report-not-gate semantic). `forgelm reverse-pii` exits 0 even when no matches are found (a zero-match scan only says the literal text is absent from the files you scanned; it does not by itself confirm that you hold no data about the person). | +| 1 | Config error: a missing `--kind`, `--row-id` without `--corpus`, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded. | +| 2 | Runtime error: I/O failure, permission denied, unwritable `--audit-dir`, an argument-parser usage error (two modes together, `--kind logs`), ReDoS SIGALRM timeout, atomic-rename failure. | Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of either subcommand's surface. diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index 7a654347..b3ad4323 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml --output-dir ` lists the artefacts in that directory that are past their horizon and always exits 0 (without `--output-dir` it scans the current directory, not `training.output_dir`). Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) +`forgelm purge --check-policy --config config.yaml --output-dir ` lists the artefacts in that directory that are past their horizon and exits 0 however many it finds (without `--output-dir` it scans the current directory, not `training.output_dir`). Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) ## Data subject requests @@ -111,7 +111,7 @@ Train from the file with ids. `forgelm reverse-pii` reports the file and line wh ### Model-level erasure -LLMs can memorise rare strings from training. Even with PII masking, removing all traces from a deployed model is hard, and under EDPB Opinion 28/2024 a model trained on personal data cannot always be considered anonymous: if you cannot show that yours is, an access or erasure request may reach the model itself, not only the corpus. ForgeLM's defences: +LLMs can memorise rare strings from training. Even with PII masking, removing all traces from a deployed model is hard. EDPB Opinion 28/2024 says a model trained on personal data is not always anonymous, so whether yours is has to be assessed case by case; until you can show that it is, plan your answers to access and erasure requests as if the trained weights might still contain personal data. ForgeLM's defences: - **Reduce what goes in:** PII masking and deduplication (memorised data is usually duplicated data). - **Memorisation is not measured:** ForgeLM does not test whether a trained model reproduces a given string. diff --git a/docs/usermanuals/en/data/audit.md b/docs/usermanuals/en/data/audit.md index 8c720938..972c27b6 100644 --- a/docs/usermanuals/en/data/audit.md +++ b/docs/usermanuals/en/data/audit.md @@ -137,7 +137,7 @@ Authoritative source: `forgelm/cli/_parser.py::_add_audit_subcommand`. | `--quality-filter` / `--no-quality-filter` | Heuristic quality checks (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). **Default ON from v0.6.0** (Phase 15 Task 5); pre-v0.6.0 the flag was opt-in. Pass `--no-quality-filter` to skip. Populates `quality_summary` in the report. See [Quality Filter](#/data/quality-filter). | | `--croissant` | Emit a [Google Croissant 1.0](https://mlcommons.org/croissant/) dataset card under the report's `croissant` key. See [Croissant 1.0 Dataset Card](#/data/croissant-card). | | `--pii-ml` | Layer Presidio's ML-NER PII detection on top of the regex detector. Requires the optional `forgelm[ingestion-pii-ml]` extra **and** a spaCy NER model (e.g. `python -m spacy download en_core_web_lg`). See [ML-NER PII (Presidio)](#/data/pii-ml). | -| `--pii-ml-language LANG` | spaCy NLP language code for `--pii-ml` (default `en`). Set to e.g. `tr` on a Turkish corpus AND make sure the matching spaCy model is installed. | +| `--pii-ml-language LANG` | spaCy NLP language code for `--pii-ml` (default `en`). Set to e.g. `de` for German, or `xx` (the multilingual `xx_ent_wiki_sm` fallback) for a language without a dedicated spaCy model such as Turkish; the matching spaCy model must be installed. | | `--workers N` | Worker processes for the split-level pipeline (default 1, sequential). Set to 2-4 on multi-split corpora for near-linear speedup. The audit JSON is byte-identical across worker counts (determinism contract). | | `--output-format {text,json}` | Stdout renderer. The `json` mode prints a machine-readable summary; `text` is the default human-readable form. | | `--allow-secrets` | Record credential findings without failing the run (exit `0` with a `SUPPRESSED` warning instead of `3`). The scan itself cannot be disabled. | diff --git a/docs/usermanuals/en/data/pii-ml.md b/docs/usermanuals/en/data/pii-ml.md index e3c0a297..3b5834b3 100644 --- a/docs/usermanuals/en/data/pii-ml.md +++ b/docs/usermanuals/en/data/pii-ml.md @@ -38,10 +38,15 @@ Without the spaCy model, `forgelm audit --pii-ml` raises a typed `ImportError` * ```shell $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml -✓ format: instructions (8,400 rows) -⚠ PII: 12 emails, 3 phone (regex layer) -⚠ PII (ML): 47 person, 9 location (Presidio) - worst tier: medium +Data audit summary + Source : data/customer-letters.jsonl + Total samples : 8400 + Splits : train + └─ train: n=8400 + languages (top-3): en=8400 + PII : email=12, location=9, person=47, phone=3 + PII severity : worst tier = medium + by tier : medium=59, low=12 ``` The Presidio findings merge into the same `pii_summary` and `pii_severity` blocks under disjoint category names, so the regex baseline stays visible alongside the ML signal: @@ -116,7 +121,7 @@ _require_presidio() text = "Alice Williams visits Acme Corp's Berlin office on Monday." counts = detect_pii_ml(text, language="en") print(counts) -# {'person': 1, 'organization': 1, 'location': 1} +# {'person': 1, 'location': 1} (Presidio's default English configuration ignores organizations) ``` The function returns an empty dict for non-string input, missing extras, or transient analyzer errors (a single bad row never blocks the audit). Hard failures (missing spaCy model, language not registered) are caught up front by `_require_presidio(language=...)` and raised before any rows are scanned — call it explicitly when you bypass `audit_dataset` and want the same pre-flight guarantees. diff --git a/docs/usermanuals/en/reference/cli.md b/docs/usermanuals/en/reference/cli.md index 5ee639a8..beb37503 100644 --- a/docs/usermanuals/en/reference/cli.md +++ b/docs/usermanuals/en/reference/cli.md @@ -26,8 +26,8 @@ ForgeLM ships a single `forgelm` binary with subcommands. This page is the canon | `forgelm approve` | Sign a human approval request and promote `final_model.staging/`. | | `forgelm reject` | Reject a human approval request; the staging directory is preserved for forensics. | | `forgelm approvals` | List pending approvals (`--pending`) or inspect one (`--show RUN_ID`). | -| `forgelm purge` | GDPR Article 17 erasure: row-id, run-id, or `--check-policy` retention report. | -| `forgelm reverse-pii` | GDPR Article 15 right-of-access: search masked corpora for a subject's identifier (plaintext or hash-mask scan). | +| `forgelm purge` | One step of a GDPR Article 17 erasure: row-id, run-id, or `--check-policy` retention report. | +| `forgelm reverse-pii` | An input to a GDPR Article 15 response: search corpora for a subject's identifier (plaintext or hash-mask scan). | | `forgelm cache-models` | Air-gap workflow: pre-populate the HuggingFace Hub cache for one or more models. | | `forgelm cache-tasks` | Air-gap workflow: pre-populate the lm-eval task dataset cache (requires `[eval]` extra). | | `forgelm safety-eval` | Standalone safety evaluation against a model checkpoint (Llama Guard by default). | diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 6be876dc..1c837d3a 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -1,22 +1,22 @@ --- title: GDPR Silme & Erişim -description: `forgelm purge` ve `forgelm reverse-pii` subcommand'ları ile Madde 15 (erişim hakkı) ve Madde 17 (silinme hakkı) taleplerini karşılayın. +description: `forgelm purge` ve `forgelm reverse-pii` subcommand'ları ile Madde 15 (erişim) ve Madde 17 (silme) taleplerini destekleyin. --- # GDPR Silme & Erişim -ForgeLM, en yaygın iki GDPR veri-sahibi hakkı için kardeş subcommand'lar ship eder: `forgelm purge` (Madde 17 — silinme hakkı) ve `forgelm reverse-pii` (Madde 15 — erişim hakkı). Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde — cleartext identifier operatör terminalinden hiç çıkmadan — korele edebilir. +ForgeLM, iki yaygın GDPR veri-sahibi talebini destekleyen kardeş subcommand'lar ship eder: `forgelm purge` satır kaldırır (Madde 17 silmenin bir adımı) ve `forgelm reverse-pii` satır bulur (Madde 15 yanıtının bir girdisi). Hiçbiri tek başına eksiksiz bir yanıt değildir. Per-output-dir salt'ı paylaşırlar; böylece compliance reviewer aynı veri sahibi için silme ve erişim taleplerini tek bir tamper-evident audit zincirinde — cleartext identifier operatör terminalinden hiç çıkmadan — korele edebilir. Bu sayfa operatör hızlı-referansıdır. Daha derin deployer akışı [GitHub'daki GDPR silme deployer kılavuzunda](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/gdpr_erasure-tr.md) bulunur; flag-başına referanslar manuel içindeki [CLI Referansı](#/reference/cli) ve GitHub'daki özel subcommand sayfaları [`purge_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/purge_subcommand-tr.md) + [`reverse_pii_subcommand-tr.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/reference/reverse_pii_subcommand-tr.md) sayfalarındadır. ## İki hak, tek zincir -| Hak | Subcommand | Ne yanıtlar | +| Talep | Subcommand | Ne yapar | |---|---|---| -| **Madde 15** — erişim hakkı | `forgelm reverse-pii` | "Hakkımda kayıt tutuyor musunuz?" — JSONL corpus'ları subject identifier'ı için tarar. | -| **Madde 17** — silinme hakkı | `forgelm purge` | "Hakkımdaki her kaydı silin." — bir corpus satırını atomik olarak siler; `--run-id` ile bir staging dizinini veya adı run id'yi embed eden compliance dosyalarını siler. | +| **Madde 15** — erişim | `forgelm reverse-pii` | Taradığınız JSONL corpus'larında subject identifier'ını içeren satırları bulur: yanıtın bir girdisidir, yanıtın kendisi değil. | +| **Madde 17** — silme | `forgelm purge` | Bir corpus satırını id ile atomik olarak siler (silmenin bir adımı, tamamı değil); `--run-id` ile bir staging dizinini veya adı run id'yi embed eden compliance dosyalarını siler. | -İki subcommand da audit satırını kaydetmeden önce subject identifier'ını **aynı** per-output-dir salt (`/.forgelm_audit_salt`) ile hash'ler. `data.access_request_query` üzerindeki `query_hash` ve `data.erasure_*` üzerindeki `target_id` aynı subject için byte-byte aynıdır; reviewer bu sayede ikisini cleartext olmadan korele edebilir. +İki subcommand da audit satırını kaydetmeden önce subject identifier'ını **aynı** per-output-dir salt (`/.forgelm_audit_salt`) ile hash'ler. `data.access_request_query` üzerindeki `query_hash` ve `data.erasure_*` üzerindeki `target_id` yalnızca `--row-id` değeri ile `--query` değeri aynı metinse, iki komut da aynı `--output-dir`'i kullanıyorsa ve `FORGELM_AUDIT_SECRET` ikisinde de aynı durumdaysa byte-byte aynıdır; aksi halde ikisini `forgelm purge --justification`'a verdiğiniz ticket referansıyla bağlayın. ## Madde 17 — `forgelm purge` @@ -44,15 +44,16 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - `data.erasure_failed` — disk operasyonu raise ettiğinde veya eşleşen satır/koşum bulunmadığında bunun yerine yayılır. - `data.erasure_warning_memorisation` — corpus-satır silme × `--output-dir` altında doğrudan bir `final_model/` mevcut (satır diskten gitti ama eğitilmiş ağırlıklarda hâlâ memorize edilmiş olabilir). - `data.erasure_warning_synthetic_data_present` — corpus-satır silme × `output_dir`'de `synthetic_data*.jsonl` mevcut. -- `data.erasure_warning_external_copies` — yüklü config boş-olmayan `webhook` block'u içeriyor; downstream tüketiciler bildirim almış olabilir. +- `data.erasure_warning_external_copies` — yalnızca corpus-satır silmede ve yalnızca `--config` verdiğinizde: config'in `webhook` bloğu boş değil (literal bir `url` ya da purge ortamında export edilmiş bir `url_env` değişkeni); downstream tüketiciler bildirim almış olabilir. ### Neyi YAPMAZ - **Modelleri yeniden eğitmez.** Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözümdür. Tool `data.erasure_warning_memorisation` yayar; böylece boşluk görünür olur. -- **Audit log'unu silmez.** Tool silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve veri sorumlusu nedenini belgeler: girdiler amaçları için hâlâ gereklidir ya da Madde 17(3) istisnalarından biri uygulanır. Örneğin Madde 17(3)(b) yalnızca AB hukukundan ya da üye devlet hukukundan doğan belirli bir yükümlülüğün işlemeyi gerektirdiği ölçüde uygulanır ve bu yükümlülüğü adlandırabilmeniz gerekir. +- **Audit log'unu silmez.** Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Bir Madde 17(1) gerekçesi geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki talepleri kapsar. Hiçbir Madde 17(1) gerekçesi geçerli değilse (örneğin girdiler toplandıkları amaç için hâlâ gereklidir) silme gerekmez; ancak talebe yine de Madde 12 süresi içinde gerekçeli bir yanıt verilir ve şikâyet hakkı bildirilir (Madde 12(4)). - **İşaretsiz legacy `final_model.staging/` dizinini silmez.** `--kind staging`, `final_model.staging./` dizinini siler; legacy `final_model.staging/` dizinini yalnızca içinde run id'ye eşit bir `staging_run_id` dosyası varsa siler. ForgeLM bu işaretçiyi yazmaz, bu yüzden işaretsiz bir legacy dizini kendiniz silin. - **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca run id'yi embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. -- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (Madde 19 verinin ifşa edildiği her alıcıya bildirim yapılmasını ister; Madde 17(2) ise kontrolör veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. +- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve ilgili kişi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. +- **Yanıtın tamamını vermez.** Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); ilgili kişiye verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). ## Madde 15 — `forgelm reverse-pii` @@ -77,7 +78,7 @@ forgelm reverse-pii --query "alice@example.com" --type email \ - `data.access_request_query` — scan'den sonra (veya mid-scan I/O hatası sonrası, `error_*` field'larıyla) yayılır. **Hash'lenmiş** `query_hash`'i, `identifier_type`'ı, `scan_mode`'u, `files_scanned` listesini, `match_count`'u ve `salt_source`'ü taşır. -Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kendisine karşı wordlist saldırılarına karşı korur. +Chain satırı cleartext identifier'ı asla taşımaz, yalnızca salt'lı bir digest taşır. Digest pseudonymised'dır, anonim değildir: salt dosyası log'un yanında durur; ikisini de okuyabilen biri tahmin ettiği bir identifier'ı onunla sınayabilir. Log'u ve salt'ı aynı erişim kontrolleriyle koruyun ve digest'leri kişisel veri sayın. ### `--type custom` regex caveat @@ -89,7 +90,7 @@ Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kend İki primitif kasıtlı olarak ayrıdır: -- `FORGELM_AUDIT_SECRET` rotasyonu chain HMAC anahtarını **ve**, değişken ayarlıysa, identifier-hash salt'ını rotate eder: `forgelm purge` secret'ın ilk 16 byte'ını per-output-dir salt'a XOR'lar ve `salt_source=env_var` kaydeder; bu yüzden `target_id` digest'leri secret ile değişir. Bir `forgelm reverse-pii` sorgusu, önceki purge event'leriyle yalnızca aynı salt kaynağını kullandığında korele eder (aynı secret ile `--salt-source env_var` ya da değişken olmadan koşturulmuş purge'lar için `--salt-source per_dir`). +- `FORGELM_AUDIT_SECRET` rotasyonu chain HMAC anahtarını **ve**, değişken ayarlıysa, identifier-hash salt'ını rotate eder: `forgelm purge` secret'ın ilk 16 byte'ını per-output-dir salt'a XOR'lar ve `salt_source=env_var` kaydeder; bu yüzden `target_id` digest'leri secret ile değişir. Düz-metin bir `forgelm reverse-pii` sorgusu (`--salt-source` yok) salt'ı aynı şekilde çözer; bu yüzden aynı secret durumunda koşturulmuş bir purge ile korele eder. `--salt-source` farklı bir özelliktir: identifier'ları ForgeLM dışında hash'lenmiş corpus'lar için hash-mask taramasıdır ve düz metin içeren bir corpus'ta hiçbir şey bulmaz. - Birini incelemek diğerini ifşa etmez. - Per-output-dir salt identifier hash'lemenin her zaman bir girdisidir ve output dizininin yaşam süresi boyunca kalıcıdır; secret, ayarlıysa, ikinci girdidir. @@ -99,9 +100,9 @@ Chain satırı cleartext identifier'ı asla taşımaz — salt audit log'un kend | Kod | Anlamı | |---|---| -| 0 | Başarı. İhlal bulunduğunda bile `forgelm purge --check-policy` 0 ile çıkar (report-not-gate semantik). Eşleşme bulunmadığında bile `forgelm reverse-pii` 0 ile çıkar (Madde 15 "kayıt yok"u geçerli yanıt olarak açıkça kabul eder). | -| 1 | Config hatası: kötü flag'ler, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`. | -| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, yazılamaz `--audit-dir`, ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | +| 0 | Başarı. İhlal bulunduğunda bile `forgelm purge --check-policy` 0 ile çıkar (report-not-gate semantik). Eşleşme bulunmadığında bile `forgelm reverse-pii` 0 ile çıkar (sıfır eşleşme yalnızca literal metnin taradığınız dosyalarda olmadığını söyler; kişi hakkında veri tutmadığınızı tek başına doğrulamaz). | +| 1 | Config hatası: eksik `--kind`, `--corpus`'suz `--row-id`, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`. | +| 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, yazılamaz `--audit-dir`, argüman-parser kullanım hatası (birlikte verilen iki mod, `--kind logs`), ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın da yüzeyinin parçası değildir. diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index 4835b203..e804fb4a 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml --output-dir ` o dizinde ufku aşmış artefaktları listeler ve her zaman 0 ile çıkar (`--output-dir` verilmezse `training.output_dir`'i değil, bulunulan dizini tarar). Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) +`forgelm purge --check-policy --config config.yaml --output-dir ` o dizinde ufku aşmış artefaktları listeler ve kaç tane bulursa bulsun 0 ile çıkar (`--output-dir` verilmezse `training.output_dir`'i değil, bulunulan dizini tarar). Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) ## Veri sahibi talepleri @@ -111,7 +111,7 @@ Eğitimi id'li dosyadan yapın. `forgelm reverse-pii` tanımlayıcının geçti ### Model-seviyesi silme -LLM'ler eğitimden nadir string'leri ezberleyebilir. PII maskelemeyle bile deploy edilmiş bir modelden tüm izleri kaldırmak zordur; EDPB Görüş 28/2024'e göre kişisel veriyle eğitilmiş bir model her durumda anonim sayılamaz: modelinizin anonim olduğunu gösteremiyorsanız, bir erişim veya silme talebi yalnızca corpus'a değil modelin kendisine de uzanabilir. ForgeLM'in savunmaları: +LLM'ler eğitimden nadir string'leri ezberleyebilir. PII maskelemeyle bile deploy edilmiş bir modelden tüm izleri kaldırmak zordur. EDPB Görüş 28/2024'e göre kişisel veriyle eğitilmiş bir model her zaman anonim değildir; modelinizin anonim olup olmadığı olay bazında değerlendirilmelidir. Anonim olduğunu gösterene kadar erişim ve silme taleplerine vereceğiniz yanıtları, eğitilmiş ağırlıklar hâlâ kişisel veri içerebilirmiş gibi planlayın. ForgeLM'in savunmaları: - **Girenleri azalt:** PII maskeleme ve deduplikasyon (ezberlenen veri genelde tekrarlanan veridir). - **Ezber ölçülmez:** ForgeLM, eğitilmiş bir modelin belirli bir string'i üretip üretmediğini test etmez. diff --git a/docs/usermanuals/tr/data/audit.md b/docs/usermanuals/tr/data/audit.md index 68bc1c3d..e86d10a4 100644 --- a/docs/usermanuals/tr/data/audit.md +++ b/docs/usermanuals/tr/data/audit.md @@ -137,7 +137,7 @@ Yetkili kaynak: `forgelm/cli/_parser.py::_add_audit_subcommand`. | `--quality-filter` / `--no-quality-filter` | Heuristik kalite kontrolleri (mean word length, alphabetic-character ratio, end-of-line punctuation ratio, repeated-line ratio, short-paragraph ratio). **v0.6.0'dan itibaren default-AÇIK** (Faz 15 Görev 5); öncesinde opt-in idi. Atlamak için `--no-quality-filter` geçirin. Rapora `quality_summary` ekler. Bkz. [Kalite Filtresi](#/data/quality-filter). | | `--croissant` | Raporun `croissant` anahtarı altına bir [Google Croissant 1.0](https://mlcommons.org/croissant/) dataset card emit eder. Bkz. [Croissant 1.0 Dataset Kartı](#/data/croissant-card). | | `--pii-ml` | Regex detector'ın üzerine Presidio'nun ML-NER PII tespitini katmanlar. Opsiyonel `forgelm[ingestion-pii-ml]` extra'sını **ve** bir spaCy NER modelini (örn. `python -m spacy download en_core_web_lg`) gerektirir. Bkz. [ML-NER PII (Presidio)](#/data/pii-ml). | -| `--pii-ml-language LANG` | `--pii-ml` için spaCy NLP dili kodu (varsayılan `en`). Türkçe corpus için örn. `tr` set edin VE eşleşen spaCy modelinin kurulu olduğundan emin olun. | +| `--pii-ml-language LANG` | `--pii-ml` için spaCy NLP dili kodu (varsayılan `en`). Örn. Almanca için `de`; Türkçe gibi özel spaCy modeli olmayan bir dil için `xx` (çok dilli `xx_ent_wiki_sm` fallback'i) set edin; eşleşen spaCy modeli kurulu olmalıdır. | | `--workers N` | Split-düzeyi pipeline için worker process sayısı (varsayılan 1, sıralı). Multi-split corpus'larda 2-4'e set ederek near-linear hızlanma sağlanır. Audit JSON worker sayısından bağımsız byte-identical (determinism contract). | | `--output-format {text,json}` | Stdout renderer. `json` modu makine-okunabilir bir özet, `text` varsayılan insan-okunabilir formdur. | | `--allow-secrets` | Credential bulgularını koşuyu başarısız kılmadan kaydeder (`3` yerine bir `SUPPRESSED` uyarısıyla exit `0`). Taramanın kendisi devre dışı bırakılamaz. | diff --git a/docs/usermanuals/tr/data/pii-ml.md b/docs/usermanuals/tr/data/pii-ml.md index 12edbf1c..1697b73f 100644 --- a/docs/usermanuals/tr/data/pii-ml.md +++ b/docs/usermanuals/tr/data/pii-ml.md @@ -38,10 +38,15 @@ spaCy modeli yoksa `forgelm audit --pii-ml` hiçbir satır taranmadan **önce** ```shell $ forgelm audit data/customer-letters.jsonl --output ./audit/ --pii-ml -✓ format: instructions (8.400 satır) -⚠ PII: 12 email, 3 telefon (regex katmanı) -⚠ PII (ML): 47 person, 9 location (Presidio) - en kötü kat: medium +Data audit summary + Source : data/customer-letters.jsonl + Total samples : 8400 + Splits : train + └─ train: n=8400 + languages (top-3): en=8400 + PII : email=12, location=9, person=47, phone=3 + PII severity : worst tier = medium + by tier : medium=59, low=12 ``` Presidio bulguları aynı `pii_summary` ve `pii_severity` bloklarına disjoint kategori isimleriyle birleşir, böylece regex baseline ML sinyaliyle yan yana görünür kalır: @@ -116,7 +121,7 @@ _require_presidio() text = "Alice Williams Pazartesi Acme Corp'un Berlin ofisini ziyaret ediyor." counts = detect_pii_ml(text, language="en") print(counts) -# {'person': 1, 'organization': 1, 'location': 1} +# {'person': 1, 'location': 1} (Presidio'nun varsayılan İngilizce yapılandırması organization'ı yok sayar) ``` Fonksiyon non-string input, eksik extra'lar veya geçici analyzer hataları için boş dict döndürür (tek bir kötü satır audit'i bloke etmez). Sert arızalar (eksik spaCy modeli, kayıtlı olmayan dil) `_require_presidio(language=...)` tarafından peşinen yakalanır ve hiçbir satır taranmadan önce raise edilir — `audit_dataset`'i bypass ettiğinizde aynı pre-flight garantilerini istiyorsanız bunu açıkça çağırın. diff --git a/docs/usermanuals/tr/reference/cli.md b/docs/usermanuals/tr/reference/cli.md index c552e54b..deca1490 100644 --- a/docs/usermanuals/tr/reference/cli.md +++ b/docs/usermanuals/tr/reference/cli.md @@ -26,8 +26,8 @@ ForgeLM, subcommand'larla tek bir `forgelm` binary'si yayınlar. Bu sayfa kanoni | `forgelm approve` | İnsan onay isteğini imzala ve `final_model.staging/`'i promote et. | | `forgelm reject` | İnsan onay isteğini reddet; staging dizini adli inceleme için korunur. | | `forgelm approvals` | Bekleyen onayları listele (`--pending`) veya tek birini incele (`--show RUN_ID`). | -| `forgelm purge` | GDPR Madde 17 silme: row-id, run-id veya `--check-policy` retention raporu. | -| `forgelm reverse-pii` | GDPR Madde 15 erişim hakkı: maskelenmiş corpora'da subject kimlik bilgisini ara (plaintext veya hash-mask scan). | +| `forgelm purge` | GDPR Madde 17 silmenin bir adımı: row-id, run-id veya `--check-policy` retention raporu. | +| `forgelm reverse-pii` | GDPR Madde 15 yanıtının bir girdisi: corpora'da subject kimlik bilgisini ara (plaintext veya hash-mask scan). | | `forgelm cache-models` | Air-gap workflow: bir veya birden fazla model için HuggingFace Hub cache'ini önceden doldur. | | `forgelm cache-tasks` | Air-gap workflow: lm-eval task dataset cache'ini önceden doldur (`[eval]` extra'sı gerekir). | | `forgelm safety-eval` | Bir model checkpoint'ine karşı standalone safety evaluation (varsayılan Llama Guard). | diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index ced63004..5d7780b6 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -629,7 +629,7 @@ def _add_audit_subcommand(subparsers) -> None: "(default: 'en'). Presidio raises a typed exception if no engine is registered for " "the requested language — surface it to the operator instead of silently " "running an English NER on a non-English corpus. Set to e.g. 'de' on a German corpus, or 'xx' (multilingual fallback) for a language " - "without a spaCy model such as Turkish, AND install the matching spaCy model." + "with no dedicated spaCy model such as Turkish; the matching spaCy model must be installed." ), ) p.add_argument( @@ -1089,9 +1089,9 @@ def _add_purge_subcommand(subparsers) -> None: """ p = subparsers.add_parser( "purge", - help="GDPR Article 17 erasure: remove corpus rows or run-scoped artefacts.", + help="Remove corpus rows or run-scoped artefacts (one step of a GDPR Article 17 erasure).", description=( - "Article 17 right-to-erasure: deletes a corpus row, a staging " + "One step of a GDPR Article 17 erasure: deletes a corpus row, a staging " "directory, or a run's compliance files whose names embed the run id, and records a tamper-evident " "audit-event chain (request → completed / failed). Use --check-policy " "for a read-only retention-violation report against the loaded config." @@ -1129,7 +1129,7 @@ def _add_purge_subcommand(subparsers) -> None: type=str, default=None, choices=["staging", "artefacts"], - help="Run-scoped erasure target: staging directory OR compliance files whose names embed the run id. `logs` is intentionally absent (the audit log is append-only and never deleted by this command).", + help="Run-scoped erasure target: staging directory OR compliance files whose names embed the run id (the standard compliance bundle has fixed filenames without a run id and is never matched). `logs` is intentionally absent (the audit log is append-only and never deleted by this command).", ) p.add_argument( "--row-matches", @@ -1163,7 +1163,7 @@ def _add_purge_subcommand(subparsers) -> None: "--dry-run", action="store_true", default=argparse.SUPPRESS, - help="Show what would be deleted; do not modify disk. Audit events still emit (with dry_run=True) so the chain reflects intent.", + help="Show what would be deleted without touching the corpus or the staging files. The salt file is still created and audit events still emit (with dry_run=True) so the chain reflects intent.", ) _add_common_subparser_flags(p, include_output_format=True) @@ -1190,13 +1190,13 @@ def _add_reverse_pii_subcommand(subparsers) -> None: """ p = subparsers.add_parser( "reverse-pii", - help="GDPR Article 15 right-of-access: search masked corpora for a subject's identifier.", + help="Search corpora for a subject's identifier (an input to a GDPR Article 15 response).", description=( - "Article 15 right-of-access: walks a glob of JSONL corpora and reports " + "An input to a GDPR Article 15 response: walks a glob of JSONL corpora and reports " "every line where the supplied identifier appears (plaintext residual " "scan by default; hash-mask scan when --salt-source is set). Audit " "event `data.access_request_query` records the hashed identifier so " - "the chain itself does not leak the subject's data." + "the chain does not store it in clear (a pseudonymised digest, not an anonymous one)." ), ) p.add_argument( @@ -1294,8 +1294,8 @@ def parse_args() -> argparse.Namespace: " forgelm approve RUN_ID Promote a staged model after human review (Art. 14)\n" " forgelm reject RUN_ID Reject a staged model (preserves staging dir for forensics)\n" " forgelm approvals --pending List runs awaiting human approval (or --show RUN_ID)\n" - " forgelm purge --row-id ID GDPR Article 17 erasure (corpus row / run artefacts / --check-policy)\n" - " forgelm reverse-pii --query VAL GDPR Article 15 right-of-access (find subject identifier in masked corpora)\n" + " forgelm purge --row-id ID One step of a GDPR Art. 17 erasure (corpus row / run artefacts / --check-policy)\n" + " forgelm reverse-pii --query VAL Input to a GDPR Art. 15 response (find a subject identifier in corpora)\n" " forgelm cache-models --model M Pre-populate HF Hub cache (air-gap workflow)\n" " forgelm cache-tasks --tasks CSV Pre-populate lm-eval task datasets (requires [eval] extra)\n" " forgelm verify-annex-iv PATH Verify EU AI Act Annex IV artifact (field set + manifest hash)\n" diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 4baf5fa0..bf665181 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -865,17 +865,13 @@ def _run_purge_run_id(args, output_format: str) -> None: target_paths = _resolve_run_kind_targets(output_dir, args.run_id, args.kind) if not target_paths: - audit.log_event( - _EVT_ERASURE_FAILED, - **request_fields, - error_class="NoMatchingArtefacts", - error_message=f"No {args.kind!r} artefacts found for run_id={args.run_id!r}.", - ) - _output_error_and_exit( - output_format, - f"No {args.kind!r} artefacts found for run_id={args.run_id!r} under {output_dir!r}.", - EXIT_CONFIG_ERROR, + # The standard compliance bundle has fixed filenames with no run id, so `--kind artefacts` never matches it. + hint = " The standard compliance bundle has fixed filenames without a run id; remove it through your storage layer." + reason = f"No {args.kind!r} artefacts found for run_id={args.run_id!r}." + ( + hint if args.kind == "artefacts" else "" ) + audit.log_event(_EVT_ERASURE_FAILED, **request_fields, error_class="NoMatchingArtefacts", error_message=reason) + _output_error_and_exit(output_format, f"{reason} (searched under {output_dir!r})", EXIT_CONFIG_ERROR) # Boundary check (F-P5-OPUS-02): refuse any resolved target that escapes # ``output_dir``. ``--run-id`` is free-form, so a ``..``-bearing value @@ -1120,9 +1116,9 @@ def _delete_path(path: Path) -> int: def _run_purge_check_policy(args, output_format: str) -> None: """Handle ``forgelm purge --check-policy [--config ]``. - Always exits 0 (per design §10 Q5: report-not-gate semantic). + A successful report exits 0 (design §10 Q5: report-not-gate); an unloadable config exits 1. Operators wiring a CI gate use ``--output-format json`` and pipe - to ``jq -e '.violations | length == 0'`` themselves. + to ``jq -e '.success == true and has("count") and .count == 0'``. Strict config loading: ``--check-policy`` is the one purge mode where the operator explicitly asked for a retention report. A diff --git a/forgelm/cli/subcommands/_reverse_pii.py b/forgelm/cli/subcommands/_reverse_pii.py index 84eb82bc..e07ac27a 100644 --- a/forgelm/cli/subcommands/_reverse_pii.py +++ b/forgelm/cli/subcommands/_reverse_pii.py @@ -484,10 +484,11 @@ def _hash_for_audit(query: str, salt: bytes) -> str: is set, the env secret too). Salt-free SHA-256 of low-entropy identifiers (e-mails, phone numbers) is brute-forcible from a commodity wordlist; the salt closes that gap. - 2. A purge → reverse-pii cycle for the same subject in the same - ``output_dir`` produces matching digests, so a compliance - reviewer correlating Article 17 + Article 15 events for one - subject sees a connected timeline. + 2. A purge → reverse-pii cycle for the same value (the ``--row-id`` + and the ``--query`` text) in the same ``output_dir`` and under the + same ``FORGELM_AUDIT_SECRET`` state produces matching digests, so a + compliance reviewer correlating Article 17 + Article 15 events sees a + connected timeline. F-W3FU-08 (priv) fix: ``salt`` is required (was ``Optional[bytes]`` with an unsalted-fallback branch). The legacy ``None`` branch was diff --git a/forgelm/config.py b/forgelm/config.py index a2eec696..52b030ec 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1598,7 +1598,7 @@ class RetentionConfig(BaseModel): Choose values from your own necessity assessment. Setting any horizon to ``0`` disables the policy for that artefact - kind (retain indefinitely). The horizons are *reported*, not enforced: + kind (it is never reported). The horizons are *reported*, not enforced: ``forgelm purge --check-policy`` lists artefacts past them and always exits 0; nothing is deleted, blocked or scheduled automatically. ``enforce`` is reserved: no command acts on it yet, so @@ -1613,7 +1613,7 @@ class RetentionConfig(BaseModel): ge=0, description=( "Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). " - "Default 1825 = 5 years, a ForgeLM choice, not a legal period. Set to 0 to retain indefinitely; document why the entries stay necessary or the legal obligation you rely on." + "Default 1825 = 5 years, a ForgeLM choice, not a legal period. Set to 0 to stop reporting it; keep the log only while it stays necessary or a nameable legal duty requires it." ), ) staging_ttl_days: int = Field( @@ -1621,7 +1621,7 @@ class RetentionConfig(BaseModel): ge=0, description=( "Days after which `forgelm purge --check-policy` reports a `final_model.staging./` (or legacy `final_model.staging/`) directory as overdue (aged from the run's first audit event, else the log's first event, else mtime); nothing is deleted. " - "Set to 0 to retain indefinitely. This is the canonical staging-retention field; " + "Set to 0 to stop reporting it. This is the canonical staging-retention field; " "the legacy `evaluation.staging_ttl_days` alias was removed in v0.8.0." ), ) @@ -1630,7 +1630,7 @@ class RetentionConfig(BaseModel): ge=0, description=( "Age in days after which `forgelm purge --check-policy` reports `/compliance/` and " - "`/data_audit_report.json` as overdue; nothing is deleted. Set to 0 to retain indefinitely." + "`/data_audit_report.json` as overdue; nothing is deleted. Set to 0 to stop reporting it." ), ) raw_documents_retention_days: int = Field( @@ -1638,7 +1638,7 @@ class RetentionConfig(BaseModel): ge=0, description=( "Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or " - "`/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue; nothing is deleted. Set to 0 to retain indefinitely. " + "`/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue; nothing is deleted. Set to 0 to stop reporting it. " "Closes ghost-features GH-023 (was nested as `ingestion.retention.raw_documents.ttl_days`; now top-level)." ), ) @@ -1646,7 +1646,7 @@ class RetentionConfig(BaseModel): default="log_only", description=( "Reserved: no command acts on it. Retention horizons are only reported, read-only, by `forgelm purge " - "--check-policy` (always exit 0); nothing is deleted, warned about or blocked automatically. " + "--check-policy` (exit 0 whatever it finds); nothing is deleted, warned about or blocked automatically. " "`warn_on_excess` and `block_on_excess` are accepted but behave like `log_only`; training and `--dry-run` warn when set." ), ) diff --git a/forgelm/wizard/_collectors.py b/forgelm/wizard/_collectors.py index d7574192..a28f76d3 100644 --- a/forgelm/wizard/_collectors.py +++ b/forgelm/wizard/_collectors.py @@ -841,19 +841,19 @@ def _collect_retention() -> Optional[Dict[str, Any]]: ): return None audit = _prompt_int( - "audit_log_retention_days (0 = retain indefinitely)", + "audit_log_retention_days (0 = never report)", 1825, min_val=0, max_val=18250, ) - staging = _prompt_int("staging_ttl_days (0 = retain indefinitely)", 7, min_val=0, max_val=365) + staging = _prompt_int("staging_ttl_days (0 = never report)", 7, min_val=0, max_val=365) ephemeral = _prompt_int( - "ephemeral_artefact_retention_days (0 = retain indefinitely)", + "ephemeral_artefact_retention_days (0 = never report)", 90, min_val=0, max_val=3650, ) - raw_docs = _prompt_int("raw_documents_retention_days (0 = retain indefinitely)", 90, min_val=0, max_val=3650) + raw_docs = _prompt_int("raw_documents_retention_days (0 = never report)", 90, min_val=0, max_val=3650) enforce = _prompt_choice( "enforcement mode (reserved: no command acts on it yet; `forgelm purge --check-policy` only reports):", [ diff --git a/notebooks/ingestion_playground.ipynb b/notebooks/ingestion_playground.ipynb index 9ae8f151..7159c660 100644 --- a/notebooks/ingestion_playground.ipynb +++ b/notebooks/ingestion_playground.ipynb @@ -103,7 +103,7 @@ "|---|---|\n", "| `STRATEGY` | `\"paragraph\"` (default) / `\"sliding\"` / `\"markdown\"`. Markdown is heading-aware and best for `.md` corpora. Paragraph is the safe default for mixed corpora. |\n", "| `CHUNK_SIZE` | Soft per-chunk character cap (default 2048). 600–1024 is good for tight LMs (≤4k context); 4096 for bigger LMs. |\n", - "| `MASK_PII` | Redact names / emails / phone numbers / IDs before writing. |\n", + "| `MASK_PII` | Redact the formats the regex layer detects (emails, phone numbers, card / IBAN / national-ID formats; **not names**) before writing. |\n", "| `MASK_SECRETS` | Redact API keys / tokens / SSH keys / connection strings. |\n", "| `RECURSIVE` | Descend into subdirectories of `./raw_docs/`. |\n", "\n", @@ -132,7 +132,7 @@ "# corpora, \"en\" for English, etc. Leave as None to disable the check.\n", "LANGUAGE_HINT = None # e.g. \"tr\" for a Turkish corpus\n", "\n", - "MASK_PII = True # redact PII (names, emails, phones, IDs)\n", + "MASK_PII = True # redact detected emails, phones, cards, IBANs, IDs (not names)\n", "MASK_SECRETS = True # redact API keys, tokens, SSH/PGP keys, JWTs\n", "RECURSIVE = True # descend into subdirectories\n", "OUTPUT_PATH = \"./curated/dataset.jsonl\" # where the curated JSONL lands\n", diff --git a/site/compliance.html b/site/compliance.html index fc14b304..333d87ed 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -146,7 +146,7 @@

Risk management

Article 10

Data & data governance

-

Quality, relevance, representativeness criteria. Bias detection, PII redaction. Optional Presidio ML-NER for unstructured PII; optional Croissant 1.0 dataset card emitted alongside the audit.

+

Quality, relevance, representativeness criteria you declare. PII counts and masking, cross-split leakage gate. Optional Presidio ML-NER PII layer and Croissant 1.0 dataset card emitted alongside the audit. Every audit report can be filed as part of your Article 10 evidence; collection, provenance and bias-examination documentation still come from you.

forgelm/data_audit/ · forgelm/ingestion.py
@@ -277,7 +277,7 @@

A run's compliance footprint

checkpoints/policy-bot/             # training.output_dir from your YAML
 ├── audit_log.jsonl                 # Article 12 append-only event log
 ├── audit_log.jsonl.manifest.json   # SHA-256 sidecar over the audit log
-├── data_audit_report.json          # Article 10 quality/PII/bias scan — you place it here (see note)
+├── data_audit_report.json          # Article 10 quality/PII/leakage audit — you place it here (see note)
 ├── compliance/                     # Annex IV bundle (Articles 9-17)
 │   ├── annex_iv_metadata.json      # Article 11 technical documentation (flat JSON)
 │   ├── compliance_report.json      # Full Article 9-17 manifest — superset of the per-artefact files
diff --git a/site/features.html b/site/features.html
index 6a42630a..c1e690aa 100644
--- a/site/features.html
+++ b/site/features.html
@@ -414,12 +414,12 @@ 

Article 14 staging gate

-

GDPR Article 17 erasure

+

Row erasure (Article 17 step)

forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log; it is one step of an Article 17 erasure (trained weights are not changed).

-

GDPR Article 15 access

+

Identifier search (Article 15 input)

forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.

diff --git a/site/js/translations.js b/site/js/translations.js index 3222aaec..68ad29f4 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -1032,9 +1032,9 @@ "features.gov.body": "EU AI Act Articles 9-17, GDPR Article 15 + 17, ISO 27001 / SOC 2 Type II alignment evidence.", "features.gov.approve.title": "Article 14 staging gate", "features.gov.approve.body": "forgelm approve / forgelm reject + forgelm approvals implement the human-oversight staging directory with audit-stamped sign-off.", - "features.gov.purge.title": "GDPR Article 17 erasure", + "features.gov.purge.title": "Row erasure (Article 17 step)", "features.gov.purge.body": "forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log; it is one step of an Article 17 erasure (trained weights are not changed).", - "features.gov.access.title": "GDPR Article 15 access", + "features.gov.access.title": "Identifier search (Article 15 input)", "features.gov.access.body": "forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.", "features.gov.annex.title": "Annex IV verifier", "features.gov.annex.body": "forgelm verify-annex-iv validates the Annex IV §1-9 schema completeness and the bundle's manifest_hash for tampering detection. (Audit-chain integrity is the separate forgelm verify-audit command.)", @@ -1137,9 +1137,9 @@ "features.gov.body": "EU AI Act Madde 9-17, GDPR Madde 15 + 17, ISO 27001 / SOC 2 Type II hizalama kanıtı.", "features.gov.approve.title": "Madde 14 staging kapısı", "features.gov.approve.body": "forgelm approve / forgelm reject + forgelm approvals, audit-damgalı onayla insan-gözetimli staging dizinini uygular.", - "features.gov.purge.title": "GDPR Madde 17 silme", + "features.gov.purge.title": "Satır silme (Madde 17 adımı)", "features.gov.purge.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve silmeyi append-only audit log'a kaydeder; Madde 17 kapsamındaki silmenin bir adımıdır (eğitilmiş ağırlıklar değişmez).", - "features.gov.access.title": "GDPR Madde 15 erişim", + "features.gov.access.title": "Tanımlayıcı arama (Madde 15 girdisi)", "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da tuzlanmış digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler.", "features.gov.annex.title": "Annex IV doğrulayıcı", "features.gov.annex.body": "forgelm verify-annex-iv Annex IV §1-9 şema tamlığını ve bundle'ın manifest_hash'ini tampering tespiti için doğrular. (Audit-chain bütünlüğü ayrı bir forgelm verify-audit komutudur.)", @@ -3854,7 +3854,7 @@ "compliance.art9.title": "Risk management", "compliance.art9.body": "A continuous, iterative process: identify foreseeable risks, evaluate, mitigate. Auto-revert on threshold breach is the technical mitigation; trend tracking is the monitoring loop.", "compliance.art10.title": "Data & data governance", - "compliance.art10.body": "Quality, relevance, representativeness criteria. Bias detection, PII redaction, cross-split leakage gate. Optional Presidio ML-NER PII layer and Croissant 1.0 dataset card emitted alongside the audit. Every audit report becomes an Article 10 evidence document.", + "compliance.art10.body": "Quality, relevance, representativeness criteria you declare. PII counts and masking, cross-split leakage gate. Optional Presidio ML-NER PII layer and Croissant 1.0 dataset card emitted alongside the audit. Every audit report can be filed as part of your Article 10 evidence; collection, provenance and bias-examination documentation still come from you.", "compliance.art11.title": "Technical documentation", "compliance.art11.body": "Annex IV's nine required §1-9 sections — auto-populated from the run: system identification, intended purpose, components, computational resources, data governance, technical documentation, monitoring & logging, performance metrics, risk management.", "compliance.art12.title": "Record-keeping (logs)", @@ -3913,7 +3913,7 @@ "compliance.art9.title": "Risk yönetimi", "compliance.art9.body": "Sürekli, iteratif süreç: öngörülebilir riskleri tanımla, değerlendir, azalt. Eşik aşımında otomatik artefakt silme teknik mitigasyondur; izleme döngüsünü trend izleme sağlar.", "compliance.art10.title": "Veri ve veri yönetimi", - "compliance.art10.body": "Kalite, uygunluk, temsiliyet kriterleri. Bias tespiti, PII redaksiyonu, split-arası sızıntı kapısı. Opsiyonel Presidio ML-NER PII katmanı ve audit ile birlikte yayınlanan Croissant 1.0 dataset card. Her audit raporu Madde 10 kanıt belgesi olur.", + "compliance.art10.body": "Beyan ettiğiniz kalite, uygunluk, temsiliyet kriterleri. PII sayıları ve maskeleme, split-arası sızıntı kapısı. Opsiyonel Presidio ML-NER PII katmanı ve audit ile birlikte yayınlanan Croissant 1.0 dataset card. Her audit raporu Madde 10 kanıtınızın bir parçası olarak sunulabilir; toplama, köken ve önyargı-inceleme dokümantasyonu yine sizden gelir.", "compliance.art11.title": "Teknik dokümantasyon", "compliance.art11.body": "Annex IV'ün §1-9 dokuz zorunlu bölümü — koşudan otomatik doldurulur: sistem kimliği, amaçlanan kullanım, bileşenler, hesaplama kaynakları, veri yönetişimi, teknik dokümantasyon, izleme ve loglama, performans metrikleri, risk yönetimi.", "compliance.art12.title": "Kayıt tutma (loglar)", @@ -3972,7 +3972,7 @@ "compliance.art9.title": "Risikomanagement", "compliance.art9.body": "Ein kontinuierlicher, iterativer Prozess: vorhersehbare Risiken identifizieren, bewerten, mindern. Das automatische Verwerfen der Artefakte bei Schwellenüberschreitung ist die technische Maßnahme; Trend-Tracking ist die Monitoring-Schleife.", "compliance.art10.title": "Daten & Datengovernance", - "compliance.art10.body": "Kriterien für Qualität, Relevanz, Repräsentativität. Bias-Erkennung, PII-Schwärzung, Split-übergreifender Leak-Check. Optionale Presidio-ML-NER-PII-Schicht und Croissant-1.0-Datensatzkarte, die zusammen mit dem Audit ausgegeben wird. Jeder Audit-Bericht wird zu einem Artikel-10-Beweisdokument.", + "compliance.art10.body": "Von Ihnen deklarierte Kriterien für Qualität, Relevanz, Repräsentativität. PII-Zählungen und -Schwärzung, Split-übergreifender Leak-Check. Optionale Presidio-ML-NER-PII-Schicht und Croissant-1.0-Datensatzkarte, die zusammen mit dem Audit ausgegeben wird. Jeder Audit-Bericht kann als Teil Ihrer Artikel-10-Nachweise abgelegt werden; Erhebungs-, Herkunfts- und Bias-Prüfungsdokumentation liefern weiterhin Sie.", "compliance.art11.title": "Technische Dokumentation", "compliance.art11.body": "Die neun Pflichtabschnitte §1-9 von Annex IV — automatisch aus dem Lauf befüllt: Systemidentifikation, Zweckbestimmung, Komponenten, Rechenressourcen, Daten-Governance, technische Dokumentation, Monitoring & Logging, Leistungskennzahlen, Risikomanagement.", "compliance.art12.title": "Aufzeichnungen (Logs)", @@ -4015,7 +4015,7 @@ "compliance.art9.title": "Gestion des risques", "compliance.art9.body": "Un processus continu et itératif : identifier les risques prévisibles, évaluer, atténuer. La suppression automatique des artefacts en cas de dépassement de seuil est la mitigation technique ; le suivi de tendance est la boucle de surveillance.", "compliance.art10.title": "Données & gouvernance des données", - "compliance.art10.body": "Critères de qualité, pertinence, représentativité. Détection de biais, masquage des PII, gate de fuite inter-split. Couche PII Presidio ML-NER optionnelle et carte de jeu de données Croissant 1.0 émise aux côtés de l'audit. Chaque rapport d'audit devient un document de preuve pour l'Article 10.", + "compliance.art10.body": "Critères de qualité, pertinence, représentativité que vous déclarez. Comptages de PII et masquage, gate de fuite inter-split. Couche PII Presidio ML-NER optionnelle et carte de jeu de données Croissant 1.0 émise aux côtés de l'audit. Chaque rapport d'audit peut être versé à vos preuves de l'Article 10 ; la documentation sur la collecte, la provenance et l'examen des biais reste de votre côté.", "compliance.art11.title": "Documentation technique", "compliance.art11.body": "Les neuf sections obligatoires §1-9 d'Annex IV — auto-remplies à partir de l'exécution : identification du système, finalité prévue, composants, ressources de calcul, gouvernance des données, documentation technique, surveillance & journalisation, métriques de performance, gestion des risques.", "compliance.art12.title": "Tenue de registres (logs)", @@ -4058,7 +4058,7 @@ "compliance.art9.title": "Gestión de riesgos", "compliance.art9.body": "Un proceso continuo e iterativo: identifique riesgos previsibles, evalúe, mitigue. El descarte automático de artefactos ante violación de umbral es la mitigación técnica; el seguimiento de tendencias es el bucle de monitorización.", "compliance.art10.title": "Datos y gobernanza de datos", - "compliance.art10.body": "Criterios de calidad, relevancia, representatividad. Detección de sesgos, redacción de PII, puerta de fuga entre splits. Capa PII de Presidio ML-NER opcional y tarjeta de dataset Croissant 1.0 emitida junto con la auditoría. Cada informe de auditoría se convierte en un documento probatorio del Artículo 10.", + "compliance.art10.body": "Criterios de calidad, relevancia y representatividad que usted declara. Recuentos de PII y enmascaramiento, puerta de fuga entre splits. Capa PII de Presidio ML-NER opcional y tarjeta de dataset Croissant 1.0 emitida junto con la auditoría. Cada informe de auditoría puede archivarse como parte de sus pruebas del Artículo 10; la documentación de recopilación, procedencia y examen de sesgos sigue siendo suya.", "compliance.art11.title": "Documentación técnica", "compliance.art11.body": "Las nueve secciones obligatorias §1-9 del Annex IV — auto-rellenadas desde la ejecución: identificación del sistema, propósito previsto, componentes, recursos computacionales, gobernanza de datos, documentación técnica, monitorización y registro, métricas de rendimiento, gestión de riesgos.", "compliance.art12.title": "Registro (logs)", @@ -4101,7 +4101,7 @@ "compliance.art9.title": "风险管理", "compliance.art9.body": "一个持续的、迭代的过程:识别可预见的风险、评估、缓解。阈值越界时自动丢弃训练产物是技术缓解;趋势跟踪是监控回路。", "compliance.art10.title": "数据 & 数据治理", - "compliance.art10.body": "质量、相关性、代表性标准。偏见检测、PII 屏蔽、split 间泄漏 gate。可选的 Presidio ML-NER PII 层,以及与审计一同输出的 Croissant 1.0 数据集卡片。每份审计报告都成为 Article 10 的证据文档。", + "compliance.art10.body": "你声明的质量、相关性、代表性标准。PII 计数与屏蔽、split 间泄漏 gate。可选的 Presidio ML-NER PII 层,以及与审计一同输出的 Croissant 1.0 数据集卡片。每份审计报告都可以作为你的 Article 10 证据的一部分归档;数据收集、来源和偏见审查文档仍由你提供。", "compliance.art11.title": "技术文档", "compliance.art11.body": "Annex IV §1-9 的九个必需章节 — 由运行自动填充:系统标识、预期目的、组件、计算资源、数据治理、技术文档、监控与日志、性能指标、风险管理。", "compliance.art12.title": "记录保存 (日志)", @@ -4344,7 +4344,7 @@ "privacy.header.eyebrow": "Privacy", "privacy.header.title": "Privacy policy", "privacy.header.lead": "ForgeLM is open-source software you run on your own infrastructure. This page covers what this website does — not the toolkit itself.", - "privacy.body": "

Short version

This site is static HTML. We do not run analytics, tracking pixels, advertising trackers, A/B tests, fingerprinting, or any third-party telemetry. Three third-party requests can happen here: Google Fonts is fetched on every page load (the Inter and JetBrains Mono typefaces); Mermaid is loaded from jsDelivr on the Guide page only (to render diagrams); and Formspree only when you submit the contact form. All three are detailed below; nothing else.

What we collect

From normal browsing

Nothing. No cookies are set. Two values are stored in localStorage on your device:

  • forgelm-lang — your chosen interface language (en, tr, de, fr, es, zh).
  • forgelm-theme — your chosen colour theme (dark or light).

Both stay on your device and are never transmitted. Clearing your browser storage removes them.

From the contact form

When you submit the form on the Contact page, the values you typed (name, email, organisation, topic, subject, message) are sent to Formspree, which forwards them to the project maintainer. Formspree's privacy practices are governed by their own policy at formspree.io/legal/privacy-policy.

Your email is used only to reply to your message. We do not add it to a mailing list, do not share it with third parties, and do not retain it after the conversation is resolved.

Hosting

This site is served as static files. Your browser will exchange standard request headers (User-Agent, Accept-Language, IP address) with whichever CDN / web server hosts the files; that exchange is governed by the hosting provider's privacy practices and is not under ForgeLM's control. We do not log this traffic ourselves.

Third-party assets

The site loads:

  • Google Fonts — the Inter and JetBrains Mono typefaces. Google may log requests as described in their privacy policy.
  • jsDelivr — Mermaid 10.9.4 is loaded on the Guide page only to render the architecture / flow diagrams. jsDelivr may log requests as described in their privacy policy.
  • Formspree — only when you submit the contact form, as described above.

Nothing else is loaded from a third party. There is no Google Analytics, no Facebook pixel, no Hotjar, no Mixpanel, no whatever-the-current-tracker is.

Children

This site is aimed at developers, ML engineers, and compliance professionals. It is not directed at children under 16.

Your rights (GDPR / KVKK)

If you believe we hold personal data about you and you want it accessed, corrected, or deleted, write to us via the contact form (or GitHub Issues) and we will respond within 30 days. Because the only personal data we hold is what you sent in the contact form, deletion is straightforward: we delete the email thread.

The toolkit itself

The ForgeLM Python package — installed via pip install forgelm — does not phone home. It does not collect telemetry, does not send usage data anywhere, and works offline (air-gap-capable). Models, datasets, and tokenizers you reference in your YAML config are downloaded only at your explicit request and only from sources you specify.

Data-subject rights you can ship in your own pipeline

If your organisation deploys ForgeLM, the toolkit gives you concrete primitives for the data-subject rights side of GDPR: forgelm reverse-pii scans the corpus for plaintext residuals or salted-digest matches of a subject identifier and reports every row where they appear (Article 15 access), and forgelm purge deletes those rows from your training corpora and stamps the deletion in the append-only audit log (Article 17 erasure). On top of that, ForgeLM is aligned with ISO 27001 / SOC 2 Type II controls — see the Compliance page for the full evidence map.

Changes

If we update this page, the date at the top changes. Material changes are noted in the project's CHANGELOG.md.

Questions

Open the contact form or a GitHub Discussion.

" + "privacy.body": "

Short version

This site is static HTML. We do not run analytics, tracking pixels, advertising trackers, A/B tests, fingerprinting, or any third-party telemetry. Three third-party requests can happen here: Google Fonts is fetched on every page load (the Inter and JetBrains Mono typefaces); Mermaid is loaded from jsDelivr on the Guide page only (to render diagrams); and Formspree only when you submit the contact form. All three are detailed below; nothing else.

What we collect

From normal browsing

Nothing. No cookies are set. Two values are stored in localStorage on your device:

  • forgelm-lang — your chosen interface language (en, tr, de, fr, es, zh).
  • forgelm-theme — your chosen colour theme (dark or light).

Both stay on your device and are never transmitted. Clearing your browser storage removes them.

From the contact form

When you submit the form on the Contact page, the values you typed (name, email, organisation, topic, subject, message) are sent to Formspree, which forwards them to the project maintainer. Formspree's privacy practices are governed by their own policy at formspree.io/legal/privacy-policy.

Your email is used only to reply to your message. We do not add it to a mailing list, do not share it with third parties, and do not retain it after the conversation is resolved.

Hosting

This site is served as static files. Your browser will exchange standard request headers (User-Agent, Accept-Language, IP address) with whichever CDN / web server hosts the files; that exchange is governed by the hosting provider's privacy practices and is not under ForgeLM's control. We do not log this traffic ourselves.

Third-party assets

The site loads:

  • Google Fonts — the Inter and JetBrains Mono typefaces. Google may log requests as described in their privacy policy.
  • jsDelivr — Mermaid 10.9.4 is loaded on the Guide page only to render the architecture / flow diagrams. jsDelivr may log requests as described in their privacy policy.
  • Formspree — only when you submit the contact form, as described above.

Nothing else is loaded from a third party. There is no Google Analytics, no Facebook pixel, no Hotjar, no Mixpanel, no whatever-the-current-tracker is.

Children

This site is aimed at developers, ML engineers, and compliance professionals. It is not directed at children under 16.

Your rights (GDPR / KVKK)

If you believe we hold personal data about you and you want it accessed, corrected, or deleted, write to us via the contact form (or GitHub Issues) and we will respond within 30 days. Because the only personal data we hold is what you sent in the contact form, deletion is straightforward: we delete the email thread.

The toolkit itself

The ForgeLM Python package — installed via pip install forgelm — does not phone home. It does not collect telemetry, does not send usage data anywhere, and works offline (air-gap-capable). Models, datasets, and tokenizers you reference in your YAML config are downloaded only at your explicit request and only from sources you specify.

Data-subject rights you can ship in your own pipeline

If your organisation deploys ForgeLM, the toolkit gives you concrete primitives for the data-subject rights side of GDPR: forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 access response), and forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log (one step of an Article 17 erasure; trained weights are not changed). On top of that, ForgeLM is aligned with ISO 27001 / SOC 2 Type II controls — see the Compliance page for the full evidence map.

Changes

If we update this page, the date at the top changes. Material changes are noted in the project's CHANGELOG.md.

Questions

Open the contact form or a GitHub Discussion.

" }); Object.assign(T.tr, { @@ -4353,7 +4353,7 @@ "privacy.header.eyebrow": "Gizlilik", "privacy.header.title": "Gizlilik politikası", "privacy.header.lead": "ForgeLM, kendi altyapınızda çalıştırdığınız açık kaynaklı bir yazılımdır. Bu sayfa web sitesinin ne yaptığını açıklar — toolkit'i değil.", - "privacy.body": "

Kısa versiyon

Bu site statik HTML'dir. Analitik, izleme pixel'i, reklam takipçisi, A/B test, fingerprinting veya üçüncü-taraf telemetri çalıştırmıyoruz. Burada üç üçüncü-taraf isteği gerçekleşebilir: Google Fonts her sayfa yüklemesinde alınır (Inter ve JetBrains Mono yazıtipleri); Mermaid yalnızca Rehber sayfasında jsDelivr'dan yüklenir (diyagramları render etmek için); ve Formspree yalnızca iletişim formunu gönderdiğinizde. Üçü de aşağıda detaylıdır; başka hiçbir şey yok.

Neler topluyoruz

Normal gezinmeden

Hiçbir şey. Hiçbir cookie set edilmez. Cihazınızda yalnızca iki değer localStorage'a yazılır:

  • forgelm-lang — seçtiğiniz arayüz dili (en, tr, de, fr, es, zh).
  • forgelm-theme — seçtiğiniz renk teması (dark veya light).

İkisi de cihazınızda kalır ve hiçbir yere iletilmez. Tarayıcı depolamasını temizlemek bunları siler.

İletişim formundan

İletişim sayfasındaki formu gönderdiğinizde girdiğiniz değerler (isim, e-posta, organizasyon, konu başlığı, konu, mesaj) Formspree'ye gönderilir; Formspree de bunları proje yürütücüsüne iletir. Formspree'nin gizlilik uygulamaları kendi politikasıyla yönetilir: formspree.io/legal/privacy-policy.

E-postanız yalnızca mesajınıza cevap vermek için kullanılır. Mailing listesine eklenmez, üçüncü taraflarla paylaşılmaz, görüşme sona erdikten sonra saklanmaz.

Hosting

Site statik dosya olarak servis edilir. Tarayıcınız, dosyaları barındıran CDN / web sunucusuyla standart istek header'larını (User-Agent, Accept-Language, IP) takas eder; bu takas hosting sağlayıcısının gizlilik uygulamalarıyla yönetilir ve ForgeLM'in kontrolünde değildir. Bu trafiği biz kendimiz loglamıyoruz.

Üçüncü-taraf varlıklar

Site şunları yükler:

  • Google Fonts — Inter ve JetBrains Mono yazıtipleri. Google istekleri kendi gizlilik politikasında açıklandığı gibi loglayabilir.
  • jsDelivr — Mermaid 10.9.4 yalnızca Rehber sayfasında mimari / akış diyagramlarını render etmek için yüklenir. jsDelivr istekleri kendi gizlilik politikasında açıklandığı gibi loglayabilir.
  • Formspree — yalnızca iletişim formunu gönderdiğinizde, yukarıda açıklandığı gibi.

Üçüncü taraftan başka hiçbir şey yüklenmez. Google Analytics yok, Facebook pixel yok, Hotjar yok, Mixpanel yok, güncel-takipçi-her-neyse yok.

Çocuklar

Bu site geliştiricilere, ML mühendislerine ve compliance profesyonellerine yöneliktir. 16 yaş altındaki çocukları hedeflemez.

Haklarınız (KVKK / GDPR)

Hakkınızda kişisel veri tuttuğumuzu düşünüyor ve erişim, düzeltme veya silme istiyorsanız iletişim formundan (veya GitHub Issues üzerinden) bize yazın; 30 gün içinde döneriz. Tuttuğumuz tek kişisel veri iletişim formundan gönderdikleriniz olduğu için silme kolay: e-posta thread'ini sileriz.

Toolkit'in kendisi

pip install forgelm ile kurulan ForgeLM Python paketi eve telefon etmez. Telemetri toplamaz, kullanım verisi göndermez, offline çalışır (air-gap uyumlu). YAML konfigürasyonunuzda referans verdiğiniz modeller, veri setleri ve tokenizer'lar yalnızca açık talebinizle ve yalnızca sizin belirttiğiniz kaynaklardan indirilir.

Kendi pipeline'ınızda sevk edebileceğiniz veri sahibi hakları

Organizasyonunuz ForgeLM dağıtıyorsa, toolkit GDPR'ın veri sahibi hakları tarafına somut primitifler sunar: forgelm reverse-pii bir özne tanımlayıcısının düz-metin kalıntılarını veya tuzlanmış-digest eşleşmelerini corpus'ta tarar ve geçtiği her satırı raporlar (Madde 15 erişim) ve forgelm purge bu satırları eğitim corpus'unuzdan siler ve silmeyi append-only audit log'a damgalar (Madde 17 silme). Üstüne, ForgeLM ISO 27001 / SOC 2 Type II kontrolleriyle hizalıdır — tam kanıt haritası için Uyumluluk sayfasına bakın.

Değişiklikler

Bu sayfayı güncellersek yukarıdaki tarih değişir. Maddi değişiklikler projenin CHANGELOG.md'sine düşer.

Sorular

İletişim formunu veya bir GitHub Discussion'ı açın.

" + "privacy.body": "

Kısa versiyon

Bu site statik HTML'dir. Analitik, izleme pixel'i, reklam takipçisi, A/B test, fingerprinting veya üçüncü-taraf telemetri çalıştırmıyoruz. Burada üç üçüncü-taraf isteği gerçekleşebilir: Google Fonts her sayfa yüklemesinde alınır (Inter ve JetBrains Mono yazıtipleri); Mermaid yalnızca Rehber sayfasında jsDelivr'dan yüklenir (diyagramları render etmek için); ve Formspree yalnızca iletişim formunu gönderdiğinizde. Üçü de aşağıda detaylıdır; başka hiçbir şey yok.

Neler topluyoruz

Normal gezinmeden

Hiçbir şey. Hiçbir cookie set edilmez. Cihazınızda yalnızca iki değer localStorage'a yazılır:

  • forgelm-lang — seçtiğiniz arayüz dili (en, tr, de, fr, es, zh).
  • forgelm-theme — seçtiğiniz renk teması (dark veya light).

İkisi de cihazınızda kalır ve hiçbir yere iletilmez. Tarayıcı depolamasını temizlemek bunları siler.

İletişim formundan

İletişim sayfasındaki formu gönderdiğinizde girdiğiniz değerler (isim, e-posta, organizasyon, konu başlığı, konu, mesaj) Formspree'ye gönderilir; Formspree de bunları proje yürütücüsüne iletir. Formspree'nin gizlilik uygulamaları kendi politikasıyla yönetilir: formspree.io/legal/privacy-policy.

E-postanız yalnızca mesajınıza cevap vermek için kullanılır. Mailing listesine eklenmez, üçüncü taraflarla paylaşılmaz, görüşme sona erdikten sonra saklanmaz.

Hosting

Site statik dosya olarak servis edilir. Tarayıcınız, dosyaları barındıran CDN / web sunucusuyla standart istek header'larını (User-Agent, Accept-Language, IP) takas eder; bu takas hosting sağlayıcısının gizlilik uygulamalarıyla yönetilir ve ForgeLM'in kontrolünde değildir. Bu trafiği biz kendimiz loglamıyoruz.

Üçüncü-taraf varlıklar

Site şunları yükler:

  • Google Fonts — Inter ve JetBrains Mono yazıtipleri. Google istekleri kendi gizlilik politikasında açıklandığı gibi loglayabilir.
  • jsDelivr — Mermaid 10.9.4 yalnızca Rehber sayfasında mimari / akış diyagramlarını render etmek için yüklenir. jsDelivr istekleri kendi gizlilik politikasında açıklandığı gibi loglayabilir.
  • Formspree — yalnızca iletişim formunu gönderdiğinizde, yukarıda açıklandığı gibi.

Üçüncü taraftan başka hiçbir şey yüklenmez. Google Analytics yok, Facebook pixel yok, Hotjar yok, Mixpanel yok, güncel-takipçi-her-neyse yok.

Çocuklar

Bu site geliştiricilere, ML mühendislerine ve compliance profesyonellerine yöneliktir. 16 yaş altındaki çocukları hedeflemez.

Haklarınız (KVKK / GDPR)

Hakkınızda kişisel veri tuttuğumuzu düşünüyor ve erişim, düzeltme veya silme istiyorsanız iletişim formundan (veya GitHub Issues üzerinden) bize yazın; 30 gün içinde döneriz. Tuttuğumuz tek kişisel veri iletişim formundan gönderdikleriniz olduğu için silme kolay: e-posta thread'ini sileriz.

Toolkit'in kendisi

pip install forgelm ile kurulan ForgeLM Python paketi eve telefon etmez. Telemetri toplamaz, kullanım verisi göndermez, offline çalışır (air-gap uyumlu). YAML konfigürasyonunuzda referans verdiğiniz modeller, veri setleri ve tokenizer'lar yalnızca açık talebinizle ve yalnızca sizin belirttiğiniz kaynaklardan indirilir.

Kendi pipeline'ınızda sevk edebileceğiniz veri sahibi hakları

Organizasyonunuz ForgeLM dağıtıyorsa, toolkit GDPR'ın veri sahibi hakları tarafına somut primitifler sunar: forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 erişim yanıtının bir girdisi) ve forgelm purge bir eğitim corpus'undan satırları id ile siler ve silmeyi append-only audit log'a kaydeder (Madde 17 kapsamındaki silmenin bir adımı; eğitilmiş ağırlıklar değişmez). Üstüne, ForgeLM ISO 27001 / SOC 2 Type II kontrolleriyle hizalıdır — tam kanıt haritası için Uyumluluk sayfasına bakın.

Değişiklikler

Bu sayfayı güncellersek yukarıdaki tarih değişir. Maddi değişiklikler projenin CHANGELOG.md'sine düşer.

Sorular

İletişim formunu veya bir GitHub Discussion'ı açın.

" }); Object.assign(T.de, { @@ -4362,7 +4362,7 @@ "privacy.header.eyebrow": "Datenschutz", "privacy.header.title": "Datenschutzerklärung", "privacy.header.lead": "ForgeLM ist Open-Source-Software, die Sie auf Ihrer eigenen Infrastruktur betreiben. Diese Seite behandelt, was diese Website tut — nicht das Toolkit selbst.", - "privacy.body": "

Kurzversion

Diese Seite ist statisches HTML. Wir betreiben keine Analytik, keine Tracking-Pixel, keine Werbe-Tracker, keine A/B-Tests, kein Fingerprinting und keine Drittanbieter-Telemetrie. Hier können drei Drittanbieter-Anfragen erfolgen: Google Fonts wird bei jedem Seitenaufruf abgerufen (die Schriften Inter und JetBrains Mono); Mermaid wird ausschließlich auf der Guide-Seite von jsDelivr geladen (zum Rendern der Diagramme); und Formspree nur, wenn Sie das Kontaktformular absenden. Alle drei werden unten detailliert beschrieben; sonst nichts.

Was wir erfassen

Beim normalen Surfen

Nichts. Es werden keine Cookies gesetzt. Auf Ihrem Gerät werden zwei Werte in localStorage gespeichert:

  • forgelm-lang — Ihre gewählte Oberflächensprache (en, tr, de, fr, es, zh).
  • forgelm-theme — Ihr gewähltes Farbschema (dark oder light).

Beide bleiben auf Ihrem Gerät und werden nie übertragen. Das Leeren Ihres Browser-Speichers entfernt sie.

Vom Kontaktformular

Wenn Sie das Formular auf der Kontakt-Seite absenden, werden die eingegebenen Werte (Name, E-Mail, Organisation, Thema, Betreff, Nachricht) an Formspree gesendet, das sie an den Projekt-Maintainer weiterleitet. Die Datenschutzpraktiken von Formspree unterliegen ihrer eigenen Richtlinie unter formspree.io/legal/privacy-policy.

Ihre E-Mail wird ausschließlich zur Beantwortung Ihrer Nachricht verwendet. Wir nehmen sie nicht in eine Mailingliste auf, geben sie nicht an Dritte weiter und behalten sie nach Abschluss der Konversation nicht zurück.

Hosting

Die Seite wird als statische Dateien ausgeliefert. Ihr Browser tauscht Standard-Request-Header (User-Agent, Accept-Language, IP-Adresse) mit dem CDN / Webserver aus, der die Dateien hostet; dieser Austausch unterliegt den Datenschutzpraktiken des Hosting-Anbieters und liegt nicht in der Kontrolle von ForgeLM. Wir loggen diesen Verkehr selbst nicht.

Drittanbieter-Assets

Die Seite lädt:

  • Google Fonts — die Schriften Inter und JetBrains Mono. Google kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • jsDelivr — Mermaid 10.9.4 wird ausschließlich auf der Guide-Seite geladen, um die Architektur- / Flussdiagramme zu rendern. jsDelivr kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • Formspree — nur wenn Sie das Kontaktformular absenden, wie oben beschrieben.

Sonst wird nichts von Drittanbietern geladen. Kein Google Analytics, kein Facebook-Pixel, kein Hotjar, kein Mixpanel, kein wie-auch-immer-aktueller-Tracker.

Kinder

Diese Seite richtet sich an Entwickler, ML-Ingenieure und Compliance-Fachleute. Sie richtet sich nicht an Kinder unter 16.

Ihre Rechte (DSGVO / KVKK)

Wenn Sie der Meinung sind, dass wir personenbezogene Daten über Sie speichern, und Sie Auskunft, Berichtigung oder Löschung wünschen, schreiben Sie uns über das Kontaktformular (oder GitHub Issues); wir antworten innerhalb von 30 Tagen. Da die einzigen personenbezogenen Daten, die wir halten, das sind, was Sie im Kontaktformular gesendet haben, ist die Löschung einfach: Wir löschen den E-Mail-Thread.

Das Toolkit selbst

Das ForgeLM-Python-Paket — installiert über pip install forgelm — telefoniert nicht nach Hause. Es sammelt keine Telemetrie, sendet keine Nutzungsdaten irgendwohin und arbeitet offline (Air-Gap-fähig). Modelle, Datensätze und Tokenizer, die Sie in Ihrer YAML-Konfiguration referenzieren, werden nur auf Ihre ausdrückliche Anforderung und nur aus von Ihnen angegebenen Quellen heruntergeladen.

Datensubjekt-Rechte, die Sie in Ihrer eigenen Pipeline ausliefern können

Wenn Ihre Organisation ForgeLM einsetzt, liefert das Toolkit konkrete Primitive für die Datensubjekt-Rechte-Seite der DSGVO: forgelm reverse-pii durchsucht den Korpus nach Klartextresten oder gesalzenen Digest-Treffern einer Subjekt-Kennung und meldet jede Zeile, in der diese Person erscheint (Artikel 15 Auskunft), und forgelm purge löscht diese Zeilen aus Ihren Trainingskorpora und stempelt die Löschung in das Append-only-Audit-Log (Artikel 17 Löschung). Darüber hinaus ist ForgeLM mit ISO 27001 / SOC 2 Type II Kontrollen aligned — siehe die Compliance-Seite für die vollständige Nachweis-Karte.

Änderungen

Wenn wir diese Seite aktualisieren, ändert sich das Datum oben. Wesentliche Änderungen werden im CHANGELOG.md des Projekts vermerkt.

Fragen

Öffnen Sie das Kontaktformular oder eine GitHub Discussion.

" + "privacy.body": "

Kurzversion

Diese Seite ist statisches HTML. Wir betreiben keine Analytik, keine Tracking-Pixel, keine Werbe-Tracker, keine A/B-Tests, kein Fingerprinting und keine Drittanbieter-Telemetrie. Hier können drei Drittanbieter-Anfragen erfolgen: Google Fonts wird bei jedem Seitenaufruf abgerufen (die Schriften Inter und JetBrains Mono); Mermaid wird ausschließlich auf der Guide-Seite von jsDelivr geladen (zum Rendern der Diagramme); und Formspree nur, wenn Sie das Kontaktformular absenden. Alle drei werden unten detailliert beschrieben; sonst nichts.

Was wir erfassen

Beim normalen Surfen

Nichts. Es werden keine Cookies gesetzt. Auf Ihrem Gerät werden zwei Werte in localStorage gespeichert:

  • forgelm-lang — Ihre gewählte Oberflächensprache (en, tr, de, fr, es, zh).
  • forgelm-theme — Ihr gewähltes Farbschema (dark oder light).

Beide bleiben auf Ihrem Gerät und werden nie übertragen. Das Leeren Ihres Browser-Speichers entfernt sie.

Vom Kontaktformular

Wenn Sie das Formular auf der Kontakt-Seite absenden, werden die eingegebenen Werte (Name, E-Mail, Organisation, Thema, Betreff, Nachricht) an Formspree gesendet, das sie an den Projekt-Maintainer weiterleitet. Die Datenschutzpraktiken von Formspree unterliegen ihrer eigenen Richtlinie unter formspree.io/legal/privacy-policy.

Ihre E-Mail wird ausschließlich zur Beantwortung Ihrer Nachricht verwendet. Wir nehmen sie nicht in eine Mailingliste auf, geben sie nicht an Dritte weiter und behalten sie nach Abschluss der Konversation nicht zurück.

Hosting

Die Seite wird als statische Dateien ausgeliefert. Ihr Browser tauscht Standard-Request-Header (User-Agent, Accept-Language, IP-Adresse) mit dem CDN / Webserver aus, der die Dateien hostet; dieser Austausch unterliegt den Datenschutzpraktiken des Hosting-Anbieters und liegt nicht in der Kontrolle von ForgeLM. Wir loggen diesen Verkehr selbst nicht.

Drittanbieter-Assets

Die Seite lädt:

  • Google Fonts — die Schriften Inter und JetBrains Mono. Google kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • jsDelivr — Mermaid 10.9.4 wird ausschließlich auf der Guide-Seite geladen, um die Architektur- / Flussdiagramme zu rendern. jsDelivr kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • Formspree — nur wenn Sie das Kontaktformular absenden, wie oben beschrieben.

Sonst wird nichts von Drittanbietern geladen. Kein Google Analytics, kein Facebook-Pixel, kein Hotjar, kein Mixpanel, kein wie-auch-immer-aktueller-Tracker.

Kinder

Diese Seite richtet sich an Entwickler, ML-Ingenieure und Compliance-Fachleute. Sie richtet sich nicht an Kinder unter 16.

Ihre Rechte (DSGVO / KVKK)

Wenn Sie der Meinung sind, dass wir personenbezogene Daten über Sie speichern, und Sie Auskunft, Berichtigung oder Löschung wünschen, schreiben Sie uns über das Kontaktformular (oder GitHub Issues); wir antworten innerhalb von 30 Tagen. Da die einzigen personenbezogenen Daten, die wir halten, das sind, was Sie im Kontaktformular gesendet haben, ist die Löschung einfach: Wir löschen den E-Mail-Thread.

Das Toolkit selbst

Das ForgeLM-Python-Paket — installiert über pip install forgelm — telefoniert nicht nach Hause. Es sammelt keine Telemetrie, sendet keine Nutzungsdaten irgendwohin und arbeitet offline (Air-Gap-fähig). Modelle, Datensätze und Tokenizer, die Sie in Ihrer YAML-Konfiguration referenzieren, werden nur auf Ihre ausdrückliche Anforderung und nur aus von Ihnen angegebenen Quellen heruntergeladen.

Datensubjekt-Rechte, die Sie in Ihrer eigenen Pipeline ausliefern können

Wenn Ihre Organisation ForgeLM einsetzt, liefert das Toolkit konkrete Primitive für die Datensubjekt-Rechte-Seite der DSGVO: forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Artikel 15), und forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Darüber hinaus ist ForgeLM mit ISO 27001 / SOC 2 Type II Kontrollen aligned — siehe die Compliance-Seite für die vollständige Nachweis-Karte.

Änderungen

Wenn wir diese Seite aktualisieren, ändert sich das Datum oben. Wesentliche Änderungen werden im CHANGELOG.md des Projekts vermerkt.

Fragen

Öffnen Sie das Kontaktformular oder eine GitHub Discussion.

" }); Object.assign(T.fr, { @@ -4371,7 +4371,7 @@ "privacy.header.eyebrow": "Confidentialité", "privacy.header.title": "Politique de confidentialité", "privacy.header.lead": "ForgeLM est un logiciel open source que vous exécutez sur votre propre infrastructure. Cette page couvre ce que ce site web fait — pas la boîte à outils elle-même.", - "privacy.body": "

Version courte

Ce site est du HTML statique. Nous n'exécutons pas d'analytics, de pixels de tracking, de trackers publicitaires, d'A/B tests, de fingerprinting ou de télémétrie tierce. Trois requêtes tierces peuvent se produire ici : Google Fonts est récupéré à chaque chargement de page (les polices Inter et JetBrains Mono) ; Mermaid est chargé depuis jsDelivr uniquement sur la page Guide (pour rendre les diagrammes) ; et Formspree uniquement lorsque vous soumettez le formulaire de contact. Les trois sont détaillés ci-dessous ; rien d'autre.

Ce que nous collectons

De la navigation normale

Rien. Aucun cookie n'est défini. Deux valeurs sont stockées dans localStorage sur votre appareil :

  • forgelm-lang — la langue d'interface choisie (en, tr, de, fr, es, zh).
  • forgelm-theme — votre thème de couleur choisi (dark ou light).

Les deux restent sur votre appareil et ne sont jamais transmises. Vider le stockage du navigateur les supprime.

Du formulaire de contact

Lorsque vous soumettez le formulaire sur la page Contact, les valeurs saisies (nom, e-mail, organisation, sujet, objet, message) sont envoyées à Formspree, qui les transmet au mainteneur du projet. Les pratiques de confidentialité de Formspree sont régies par leur propre politique sur formspree.io/legal/privacy-policy.

Votre e-mail est utilisé uniquement pour répondre à votre message. Nous ne l'ajoutons pas à une liste de diffusion, ne le partageons pas avec des tiers et ne le conservons pas après la résolution de la conversation.

Hébergement

Ce site est servi sous forme de fichiers statiques. Votre navigateur échangera des en-têtes de requête standard (User-Agent, Accept-Language, adresse IP) avec le CDN / serveur web qui héberge les fichiers ; cet échange est régi par les pratiques de confidentialité du fournisseur d'hébergement et n'est pas sous le contrôle de ForgeLM. Nous ne loggons pas ce trafic nous-mêmes.

Ressources tierces

Le site charge :

  • Google Fonts — les polices Inter et JetBrains Mono. Google peut logger les requêtes comme décrit dans sa politique de confidentialité.
  • jsDelivr — Mermaid 10.9.4 est chargé uniquement sur la page Guide pour rendre les diagrammes d'architecture / flux. jsDelivr peut logger les requêtes comme décrit dans sa politique de confidentialité.
  • Formspree — uniquement lorsque vous soumettez le formulaire de contact, comme décrit ci-dessus.

Rien d'autre n'est chargé depuis un tiers. Pas de Google Analytics, pas de pixel Facebook, pas de Hotjar, pas de Mixpanel, pas de tracker-du-moment-quel-qu'il-soit.

Enfants

Ce site s'adresse aux développeurs, ingénieurs ML et professionnels de la conformité. Il n'est pas destiné aux enfants de moins de 16 ans.

Vos droits (RGPD / KVKK)

Si vous pensez que nous détenons des données personnelles vous concernant et que vous voulez y accéder, les corriger ou les supprimer, écrivez-nous via le formulaire de contact (ou GitHub Issues) et nous répondrons dans les 30 jours. Comme les seules données personnelles que nous détenons sont celles que vous avez envoyées dans le formulaire de contact, la suppression est simple : nous supprimons le fil d'e-mail.

La boîte à outils elle-même

Le paquet Python ForgeLM — installé via pip install forgelm — ne « téléphone pas à la maison ». Il ne collecte pas de télémétrie, n'envoie pas de données d'usage où que ce soit et fonctionne hors-ligne (compatible air-gap). Les modèles, datasets et tokenizers que vous référencez dans votre config YAML ne sont téléchargés qu'à votre demande explicite et uniquement depuis les sources que vous spécifiez.

Droits des personnes concernées que vous pouvez livrer dans votre propre pipeline

Si votre organisation déploie ForgeLM, le toolkit fournit des primitives concrètes pour le volet droits des personnes concernées du RGPD : forgelm reverse-pii scanne le corpus à la recherche de résidus en clair ou de digests salés d'un identifiant de personne et signale chaque ligne où elle apparaît (Article 15 accès), et forgelm purge supprime ces lignes de vos corpus d'entraînement et estampille la suppression dans le log d'audit append-only (Article 17 effacement). De plus, ForgeLM est aligné avec les contrôles ISO 27001 / SOC 2 Type II — voir la page Compliance pour la cartographie complète des preuves.

Modifications

Si nous mettons cette page à jour, la date en haut change. Les changements importants sont notés dans le CHANGELOG.md du projet.

Questions

Ouvrez le formulaire de contact ou une GitHub Discussion.

" + "privacy.body": "

Version courte

Ce site est du HTML statique. Nous n'exécutons pas d'analytics, de pixels de tracking, de trackers publicitaires, d'A/B tests, de fingerprinting ou de télémétrie tierce. Trois requêtes tierces peuvent se produire ici : Google Fonts est récupéré à chaque chargement de page (les polices Inter et JetBrains Mono) ; Mermaid est chargé depuis jsDelivr uniquement sur la page Guide (pour rendre les diagrammes) ; et Formspree uniquement lorsque vous soumettez le formulaire de contact. Les trois sont détaillés ci-dessous ; rien d'autre.

Ce que nous collectons

De la navigation normale

Rien. Aucun cookie n'est défini. Deux valeurs sont stockées dans localStorage sur votre appareil :

  • forgelm-lang — la langue d'interface choisie (en, tr, de, fr, es, zh).
  • forgelm-theme — votre thème de couleur choisi (dark ou light).

Les deux restent sur votre appareil et ne sont jamais transmises. Vider le stockage du navigateur les supprime.

Du formulaire de contact

Lorsque vous soumettez le formulaire sur la page Contact, les valeurs saisies (nom, e-mail, organisation, sujet, objet, message) sont envoyées à Formspree, qui les transmet au mainteneur du projet. Les pratiques de confidentialité de Formspree sont régies par leur propre politique sur formspree.io/legal/privacy-policy.

Votre e-mail est utilisé uniquement pour répondre à votre message. Nous ne l'ajoutons pas à une liste de diffusion, ne le partageons pas avec des tiers et ne le conservons pas après la résolution de la conversation.

Hébergement

Ce site est servi sous forme de fichiers statiques. Votre navigateur échangera des en-têtes de requête standard (User-Agent, Accept-Language, adresse IP) avec le CDN / serveur web qui héberge les fichiers ; cet échange est régi par les pratiques de confidentialité du fournisseur d'hébergement et n'est pas sous le contrôle de ForgeLM. Nous ne loggons pas ce trafic nous-mêmes.

Ressources tierces

Le site charge :

  • Google Fonts — les polices Inter et JetBrains Mono. Google peut logger les requêtes comme décrit dans sa politique de confidentialité.
  • jsDelivr — Mermaid 10.9.4 est chargé uniquement sur la page Guide pour rendre les diagrammes d'architecture / flux. jsDelivr peut logger les requêtes comme décrit dans sa politique de confidentialité.
  • Formspree — uniquement lorsque vous soumettez le formulaire de contact, comme décrit ci-dessus.

Rien d'autre n'est chargé depuis un tiers. Pas de Google Analytics, pas de pixel Facebook, pas de Hotjar, pas de Mixpanel, pas de tracker-du-moment-quel-qu'il-soit.

Enfants

Ce site s'adresse aux développeurs, ingénieurs ML et professionnels de la conformité. Il n'est pas destiné aux enfants de moins de 16 ans.

Vos droits (RGPD / KVKK)

Si vous pensez que nous détenons des données personnelles vous concernant et que vous voulez y accéder, les corriger ou les supprimer, écrivez-nous via le formulaire de contact (ou GitHub Issues) et nous répondrons dans les 30 jours. Comme les seules données personnelles que nous détenons sont celles que vous avez envoyées dans le formulaire de contact, la suppression est simple : nous supprimons le fil d'e-mail.

La boîte à outils elle-même

Le paquet Python ForgeLM — installé via pip install forgelm — ne « téléphone pas à la maison ». Il ne collecte pas de télémétrie, n'envoie pas de données d'usage où que ce soit et fonctionne hors-ligne (compatible air-gap). Les modèles, datasets et tokenizers que vous référencez dans votre config YAML ne sont téléchargés qu'à votre demande explicite et uniquement depuis les sources que vous spécifiez.

Droits des personnes concernées que vous pouvez livrer dans votre propre pipeline

Si votre organisation déploie ForgeLM, le toolkit fournit des primitives concrètes pour le volet droits des personnes concernées du RGPD : forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, article 15), et forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne l'effacement dans le log d'audit append-only (une étape de l'effacement, article 17 ; les poids entraînés ne changent pas). De plus, ForgeLM est aligné avec les contrôles ISO 27001 / SOC 2 Type II — voir la page Compliance pour la cartographie complète des preuves.

Modifications

Si nous mettons cette page à jour, la date en haut change. Les changements importants sont notés dans le CHANGELOG.md du projet.

Questions

Ouvrez le formulaire de contact ou une GitHub Discussion.

" }); Object.assign(T.es, { @@ -4380,7 +4380,7 @@ "privacy.header.eyebrow": "Privacidad", "privacy.header.title": "Política de privacidad", "privacy.header.lead": "ForgeLM es software de código abierto que ejecuta en su propia infraestructura. Esta página cubre lo que hace este sitio web — no el toolkit en sí.", - "privacy.body": "

Versión corta

Este sitio es HTML estático. No ejecutamos analytics, píxeles de tracking, trackers publicitarios, A/B tests, fingerprinting o telemetría de terceros. Aquí pueden ocurrir tres solicitudes a terceros: Google Fonts se obtiene en cada carga de página (las tipografías Inter y JetBrains Mono); Mermaid se carga desde jsDelivr solo en la página Guía (para renderizar los diagramas); y Formspree solo cuando envía el formulario de contacto. Las tres están detalladas abajo; nada más.

Qué recopilamos

De la navegación normal

Nada. No se establecen cookies. Se almacenan dos valores en localStorage en su dispositivo:

  • forgelm-lang — su idioma de interfaz elegido (en, tr, de, fr, es, zh).
  • forgelm-theme — su tema de color elegido (dark o light).

Ambos quedan en su dispositivo y nunca se transmiten. Limpiar el almacenamiento del navegador los elimina.

Del formulario de contacto

Cuando envía el formulario en la página de Contacto, los valores que escribió (nombre, email, organización, tema, asunto, mensaje) se envían a Formspree, que los reenvía al mantenedor del proyecto. Las prácticas de privacidad de Formspree se rigen por su propia política en formspree.io/legal/privacy-policy.

Su email se usa solo para responder a su mensaje. No lo agregamos a una lista de correo, no lo compartimos con terceros y no lo retenemos después de que la conversación se resuelve.

Hosting

Este sitio se sirve como archivos estáticos. Su navegador intercambiará cabeceras de solicitud estándar (User-Agent, Accept-Language, dirección IP) con el CDN / servidor web que aloja los archivos; ese intercambio se rige por las prácticas de privacidad del proveedor de hosting y no está bajo el control de ForgeLM. No registramos este tráfico nosotros mismos.

Recursos de terceros

El sitio carga:

  • Google Fonts — las tipografías Inter y JetBrains Mono. Google puede registrar las solicitudes como se describe en su política de privacidad.
  • jsDelivr — Mermaid 10.9.4 se carga solo en la página Guía para renderizar los diagramas de arquitectura / flujo. jsDelivr puede registrar las solicitudes como se describe en su política de privacidad.
  • Formspree — solo cuando envía el formulario de contacto, como se describe arriba.

Nada más se carga desde un tercero. No hay Google Analytics, no hay píxel de Facebook, no hay Hotjar, no hay Mixpanel, no hay tracker-del-momento-sea-cual-sea.

Niños

Este sitio está dirigido a desarrolladores, ingenieros de ML y profesionales del cumplimiento. No está dirigido a niños menores de 16 años.

Sus derechos (RGPD / KVKK)

Si cree que tenemos datos personales sobre usted y desea acceder, corregir o eliminarlos, escríbanos vía el formulario de contacto (o GitHub Issues) y responderemos en 30 días. Dado que los únicos datos personales que tenemos son lo que envió en el formulario de contacto, la eliminación es directa: borramos el hilo de email.

El toolkit en sí

El paquete Python ForgeLM — instalado vía pip install forgelm — no telefonea a casa. No recopila telemetría, no envía datos de uso a ninguna parte y funciona offline (compatible con air-gap). Los modelos, datasets y tokenizers que referencia en su config YAML se descargan solo a petición explícita suya y solo desde fuentes que usted especifica.

Derechos del interesado que puede enviar en su propio pipeline

Si su organización despliega ForgeLM, el toolkit proporciona primitivas concretas para el lado de los derechos del interesado del RGPD: forgelm reverse-pii escanea el corpus en busca de residuos en texto claro o digests salados de un identificador de sujeto y reporta cada fila en la que aparece (Artículo 15 acceso), y forgelm purge elimina esas filas de sus corpus de entrenamiento y estampa la supresión en el log de auditoría append-only (Artículo 17 supresión). Además, ForgeLM está alineado con los controles ISO 27001 / SOC 2 Type II — vea la página de Compliance para el mapa completo de pruebas.

Cambios

Si actualizamos esta página, la fecha en la parte superior cambia. Los cambios materiales se anotan en el CHANGELOG.md del proyecto.

Preguntas

Abra el formulario de contacto o una GitHub Discussion.

" + "privacy.body": "

Versión corta

Este sitio es HTML estático. No ejecutamos analytics, píxeles de tracking, trackers publicitarios, A/B tests, fingerprinting o telemetría de terceros. Aquí pueden ocurrir tres solicitudes a terceros: Google Fonts se obtiene en cada carga de página (las tipografías Inter y JetBrains Mono); Mermaid se carga desde jsDelivr solo en la página Guía (para renderizar los diagramas); y Formspree solo cuando envía el formulario de contacto. Las tres están detalladas abajo; nada más.

Qué recopilamos

De la navegación normal

Nada. No se establecen cookies. Se almacenan dos valores en localStorage en su dispositivo:

  • forgelm-lang — su idioma de interfaz elegido (en, tr, de, fr, es, zh).
  • forgelm-theme — su tema de color elegido (dark o light).

Ambos quedan en su dispositivo y nunca se transmiten. Limpiar el almacenamiento del navegador los elimina.

Del formulario de contacto

Cuando envía el formulario en la página de Contacto, los valores que escribió (nombre, email, organización, tema, asunto, mensaje) se envían a Formspree, que los reenvía al mantenedor del proyecto. Las prácticas de privacidad de Formspree se rigen por su propia política en formspree.io/legal/privacy-policy.

Su email se usa solo para responder a su mensaje. No lo agregamos a una lista de correo, no lo compartimos con terceros y no lo retenemos después de que la conversación se resuelve.

Hosting

Este sitio se sirve como archivos estáticos. Su navegador intercambiará cabeceras de solicitud estándar (User-Agent, Accept-Language, dirección IP) con el CDN / servidor web que aloja los archivos; ese intercambio se rige por las prácticas de privacidad del proveedor de hosting y no está bajo el control de ForgeLM. No registramos este tráfico nosotros mismos.

Recursos de terceros

El sitio carga:

  • Google Fonts — las tipografías Inter y JetBrains Mono. Google puede registrar las solicitudes como se describe en su política de privacidad.
  • jsDelivr — Mermaid 10.9.4 se carga solo en la página Guía para renderizar los diagramas de arquitectura / flujo. jsDelivr puede registrar las solicitudes como se describe en su política de privacidad.
  • Formspree — solo cuando envía el formulario de contacto, como se describe arriba.

Nada más se carga desde un tercero. No hay Google Analytics, no hay píxel de Facebook, no hay Hotjar, no hay Mixpanel, no hay tracker-del-momento-sea-cual-sea.

Niños

Este sitio está dirigido a desarrolladores, ingenieros de ML y profesionales del cumplimiento. No está dirigido a niños menores de 16 años.

Sus derechos (RGPD / KVKK)

Si cree que tenemos datos personales sobre usted y desea acceder, corregir o eliminarlos, escríbanos vía el formulario de contacto (o GitHub Issues) y responderemos en 30 días. Dado que los únicos datos personales que tenemos son lo que envió en el formulario de contacto, la eliminación es directa: borramos el hilo de email.

El toolkit en sí

El paquete Python ForgeLM — instalado vía pip install forgelm — no telefonea a casa. No recopila telemetría, no envía datos de uso a ninguna parte y funciona offline (compatible con air-gap). Los modelos, datasets y tokenizers que referencia en su config YAML se descargan solo a petición explícita suya y solo desde fuentes que usted especifica.

Derechos del interesado que puede enviar en su propio pipeline

Si su organización despliega ForgeLM, el toolkit proporciona primitivas concretas para el lado de los derechos del interesado del RGPD: forgelm reverse-pii encuentra las filas que contienen un identificador que usted proporciona (un elemento de la respuesta a la solicitud de acceso del Artículo 15), y forgelm purge elimina filas por id de un corpus de entrenamiento y registra la supresión en el log de auditoría append-only (un paso de la supresión conforme al Artículo 17; los pesos entrenados no cambian). Además, ForgeLM está alineado con los controles ISO 27001 / SOC 2 Type II — vea la página de Compliance para el mapa completo de pruebas.

Cambios

Si actualizamos esta página, la fecha en la parte superior cambia. Los cambios materiales se anotan en el CHANGELOG.md del proyecto.

Preguntas

Abra el formulario de contacto o una GitHub Discussion.

" }); Object.assign(T.zh, { @@ -4389,7 +4389,7 @@ "privacy.header.eyebrow": "隐私", "privacy.header.title": "隐私政策", "privacy.header.lead": "ForgeLM 是你在自己基础设施上运行的开源软件。本页讲的是这个网站做了什么 — 不是 toolkit 本身。", - "privacy.body": "

简短版

本站是静态 HTML。我们不运行分析、跟踪像素、广告跟踪器、A/B 测试、指纹识别或任何第三方遥测。本站可能发生三种第三方请求:Google Fonts 在每次页面加载时获取 (Inter 和 JetBrains Mono 字体);Mermaid 仅在指南页面从 jsDelivr 加载 (用于渲染图表);Formspree 仅在你提交联系表单时。三者均在下方详述;此外没有别的。

我们收集什么

正常浏览时

什么也没有。不设置 cookie。在你设备的 localStorage 中存储两个值:

  • forgelm-lang — 你选择的界面语言 (en、tr、de、fr、es、zh)。
  • forgelm-theme — 你选择的色彩主题 (dark 或 light)。

两者都留在你设备上,从不发送出去。清除浏览器存储就能移除它们。

从联系表单

当你在联系页提交表单时,你填写的值 (姓名、邮箱、组织、主题、标题、消息) 会发送到 Formspree,Formspree 再转发给项目维护者。Formspree 的隐私实践由其自身政策管理:formspree.io/legal/privacy-policy。

你的邮箱仅用于回复你的消息。不会加入任何邮件列表,不会与第三方共享,在对话结束之后也不保留。

托管

本站以静态文件的方式提供。你的浏览器会与托管文件的 CDN / Web 服务器交换标准请求头 (User-Agent、Accept-Language、IP 地址);该交换由托管提供商的隐私实践管理,不在 ForgeLM 的控制之下。我们自己不记录这部分流量。

第三方资源

本站加载:

  • Google Fonts — Inter 和 JetBrains Mono 字体。Google 可能按照其隐私政策记录请求。
  • jsDelivr — Mermaid 10.9.4 仅在指南页面加载,用于渲染架构 / 流程图。jsDelivr 可能按照其隐私政策记录请求。
  • Formspree — 仅在你提交联系表单时,如上所述。

除此之外不从任何第三方加载。没有 Google Analytics、没有 Facebook 像素、没有 Hotjar、没有 Mixpanel、没有那个时髦的随便什么跟踪器。

儿童

本站面向开发者、ML 工程师和合规专业人员。不面向 16 岁以下儿童。

你的权利 (GDPR / KVKK)

如果你认为我们持有关于你的个人数据,并希望访问、更正或删除它,通过联系表单 (或 GitHub Issues) 写信给我们;我们会在 30 天内回复。由于我们持有的唯一个人数据就是你在联系表单中发送的内容,删除很简单:我们删除该邮件 thread。

Toolkit 本身

通过 pip install forgelm 安装的 ForgeLM Python 包不会回拨。不收集遥测,不向任何地方发送使用数据,可离线工作 (支持气隙)。你在 YAML 配置中引用的模型、数据集和 tokenizer 仅在你的明确请求下且仅从你指定的来源下载。

你可以在自己的管道中交付的数据主体权利

如果你的组织部署 ForgeLM,这套工具为 GDPR 数据主体权利那一面提供了具体原语:forgelm reverse-pii 扫描语料库,查找主体标识的明文残留或加盐 digest 匹配,并报告该主体出现的每一行 (第 15 条访问);forgelm purge 从你的训练语料中删除这些行,并将删除盖戳到 append-only 审计日志中 (第 17 条删除)。此外,ForgeLM 与 ISO 27001 / SOC 2 Type II 控制项保持对齐 — 见合规页面查看完整证据图。

变更

如果我们更新本页,顶部日期会变化。重要变更会记入项目的 CHANGELOG.md。

问题

打开联系表单或一个 GitHub Discussion。

" + "privacy.body": "

简短版

本站是静态 HTML。我们不运行分析、跟踪像素、广告跟踪器、A/B 测试、指纹识别或任何第三方遥测。本站可能发生三种第三方请求:Google Fonts 在每次页面加载时获取 (Inter 和 JetBrains Mono 字体);Mermaid 仅在指南页面从 jsDelivr 加载 (用于渲染图表);Formspree 仅在你提交联系表单时。三者均在下方详述;此外没有别的。

我们收集什么

正常浏览时

什么也没有。不设置 cookie。在你设备的 localStorage 中存储两个值:

  • forgelm-lang — 你选择的界面语言 (en、tr、de、fr、es、zh)。
  • forgelm-theme — 你选择的色彩主题 (dark 或 light)。

两者都留在你设备上,从不发送出去。清除浏览器存储就能移除它们。

从联系表单

当你在联系页提交表单时,你填写的值 (姓名、邮箱、组织、主题、标题、消息) 会发送到 Formspree,Formspree 再转发给项目维护者。Formspree 的隐私实践由其自身政策管理:formspree.io/legal/privacy-policy。

你的邮箱仅用于回复你的消息。不会加入任何邮件列表,不会与第三方共享,在对话结束之后也不保留。

托管

本站以静态文件的方式提供。你的浏览器会与托管文件的 CDN / Web 服务器交换标准请求头 (User-Agent、Accept-Language、IP 地址);该交换由托管提供商的隐私实践管理,不在 ForgeLM 的控制之下。我们自己不记录这部分流量。

第三方资源

本站加载:

  • Google Fonts — Inter 和 JetBrains Mono 字体。Google 可能按照其隐私政策记录请求。
  • jsDelivr — Mermaid 10.9.4 仅在指南页面加载,用于渲染架构 / 流程图。jsDelivr 可能按照其隐私政策记录请求。
  • Formspree — 仅在你提交联系表单时,如上所述。

除此之外不从任何第三方加载。没有 Google Analytics、没有 Facebook 像素、没有 Hotjar、没有 Mixpanel、没有那个时髦的随便什么跟踪器。

儿童

本站面向开发者、ML 工程师和合规专业人员。不面向 16 岁以下儿童。

你的权利 (GDPR / KVKK)

如果你认为我们持有关于你的个人数据,并希望访问、更正或删除它,通过联系表单 (或 GitHub Issues) 写信给我们;我们会在 30 天内回复。由于我们持有的唯一个人数据就是你在联系表单中发送的内容,删除很简单:我们删除该邮件 thread。

Toolkit 本身

通过 pip install forgelm 安装的 ForgeLM Python 包不会回拨。不收集遥测,不向任何地方发送使用数据,可离线工作 (支持气隙)。你在 YAML 配置中引用的模型、数据集和 tokenizer 仅在你的明确请求下且仅从你指定的来源下载。

你可以在自己的管道中交付的数据主体权利

如果你的组织部署 ForgeLM,这套工具为 GDPR 数据主体权利那一面提供了具体原语:forgelm reverse-pii 查找包含你所提供标识符的行 (对第 15 条访问请求答复的输入之一);forgelm purge 按 id 从训练语料中删除行,并将该删除记录到 append-only 审计日志中 (第 17 条所涉删除的其中一步,已训练的权重不会改变)。此外,ForgeLM 与 ISO 27001 / SOC 2 Type II 控制项保持对齐 — 见合规页面查看完整证据图。

变更

如果我们更新本页,顶部日期会变化。重要变更会记入项目的 CHANGELOG.md。

问题

打开联系表单或一个 GitHub Discussion。

" }); /* ── Terms page (terms.html) ─────────────────────────────── */ @@ -4469,7 +4469,7 @@ "compliance.iso.cookbook": "Für das vollständige Audit-Cookbook, einschließlich 8 Audit-Floor-Fragen und einer 93-Kontrollen-Abdeckungskarte, siehe docs/guides/iso_soc2_deployer_guide.md.", "features.gov.heading": "Compliance- & DSGVO-Tooling", "features.gov.body": "EU-AI-Act-Artikel 9-17, DSGVO-Artikel 15 + 17, ISO-27001- / SOC-2-Type-II-Alignment-Nachweise.", - "features.gov.access.title": "DSGVO-Artikel-15-Auskunft", + "features.gov.access.title": "Kennungssuche (Baustein für Art. 15)", "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern der Kennung einer betroffenen Person und listet die Zeilen der gescannten Dateien auf, die sie enthalten.", "features.gov.annex.title": "Annex-IV-Verifier", "features.gov.annex.body": "forgelm verify-annex-iv validiert die Annex-IV-§1-9-Schemavollständigkeit und den manifest_hash des Bundles zur Manipulationserkennung. (Audit-Ketten-Integrität ist der separate Befehl forgelm verify-audit.)", @@ -4479,7 +4479,7 @@ "features.gov.gguf.body": "forgelm verify-gguf validiert den GGUF-Magic-Header, parst den Metadatenblock und prüft das beim Export geschriebene .sha256-Sidecar (sofern vorhanden) erneut, bevor Sie ein quantisiertes Modell ausliefern.", "features.gov.iso.title": "ISO 27001 / SOC 2 Alignment", "features.gov.iso.body": "Audit-Trail-, Change-Management-, Daten-Lineage- und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Die Software ist aligned, nicht zertifiziert.", - "features.gov.purge.title": "DSGVO-Artikel-17-Löschung", + "features.gov.purge.title": "Zeilenlöschung (Schritt für Art. 17)", "features.gov.purge.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log; es ist ein Schritt einer Löschung nach Artikel 17 (trainierte Gewichte bleiben unverändert).", "features.eval.safetyeval.title": "Standalone-Sicherheits-Eval", "features.eval.safetyeval.body": "forgelm safety-eval führt Llama Guard gegen eine --probes-JSONL-Prompt-Datei aus (oder --default-probes für das gebündelte 51-Prompt-Set über 18 Schadkategorien) und gibt eine kategorienspezifische Sicherheitszusammenfassung aus (safety_results.json + safety_trend.jsonl).", @@ -4515,7 +4515,7 @@ "compliance.iso.cookbook": "Pour le cookbook d'audit complet, incluant 8 questions audit-floor et une cartographie de couverture de 93 contrôles, voir docs/guides/iso_soc2_deployer_guide.md.", "features.gov.heading": "Outillage Compliance & RGPD", "features.gov.body": "Articles 9-17 de l'EU AI Act, RGPD Articles 15 + 17, preuves d'alignement ISO 27001 / SOC 2 Type II.", - "features.gov.access.title": "Accès RGPD Article 15", + "features.gov.access.title": "Recherche d'identifiant (élément de l'article 15)", "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de correspondances en clair ou de digests salés de l'identifiant d'une personne concernée et liste les lignes des fichiers analysés qui le contiennent.", "features.gov.annex.title": "Vérificateur Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valide la complétude du schéma Annex IV §1-9 et le manifest_hash du bundle pour la détection de falsification. (L'intégrité de la chaîne d'audit est la commande séparée forgelm verify-audit.)", @@ -4525,7 +4525,7 @@ "features.gov.gguf.body": "forgelm verify-gguf valide l'en-tête magique GGUF, analyse le bloc de métadonnées et revérifie le fichier .sha256 écrit lors de l'export (s'il est présent) avant que vous n'expédiiez un modèle quantisé.", "features.gov.iso.title": "Alignement ISO 27001 / SOC 2", "features.gov.iso.body": "Preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain que l'auditeur du déployeur demande. Le logiciel est aligné, non certifié.", - "features.gov.purge.title": "Effacement RGPD Article 17", + "features.gov.purge.title": "Effacement de lignes (étape de l'article 17)", "features.gov.purge.body": "forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne l'effacement dans le log d'audit append-only ; c'est une étape d'un effacement au titre de l'article 17 (les poids entraînés ne changent pas).", "features.eval.safetyeval.title": "Évaluation sécurité autonome", "features.eval.safetyeval.body": "forgelm safety-eval exécute Llama Guard contre un fichier de prompts JSONL --probes (ou --default-probes pour le set bundlé de 51 prompts couvrant 18 catégories de préjudice) et émet un résumé de sécurité par catégorie (safety_results.json + safety_trend.jsonl).", @@ -4561,7 +4561,7 @@ "compliance.iso.cookbook": "Para el cookbook de auditoría completo, incluidas 8 preguntas audit-floor y un mapa de cobertura de 93 controles, vea docs/guides/iso_soc2_deployer_guide.md.", "features.gov.heading": "Tooling Compliance & RGPD", "features.gov.body": "Artículos 9-17 del EU AI Act, RGPD Artículos 15 + 17, pruebas de alineamiento ISO 27001 / SOC 2 Type II.", - "features.gov.access.title": "Acceso RGPD Artículo 15", + "features.gov.access.title": "Búsqueda de identificador (elemento del Artículo 15)", "features.gov.access.body": "forgelm reverse-pii escanea corpus JSONL en busca de coincidencias en texto claro o de digests salados de un identificador de interesado y lista las líneas de los archivos escaneados que lo contienen.", "features.gov.annex.title": "Verificador Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valida la completitud del esquema Annex IV §1-9 y el manifest_hash del bundle para la detección de manipulación. (La integridad de la cadena de auditoría es el comando separado forgelm verify-audit.)", @@ -4571,7 +4571,7 @@ "features.gov.gguf.body": "forgelm verify-gguf valida la cabecera mágica GGUF, analiza el bloque de metadatos y reverifica el sidecar .sha256 escrito durante el export (si existe) antes de que envíe un modelo cuantizado.", "features.gov.iso.title": "Alineamiento ISO 27001 / SOC 2", "features.gov.iso.body": "Pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro que pide el auditor del deployer. El software está alineado, no certificado.", - "features.gov.purge.title": "Supresión RGPD Artículo 17", + "features.gov.purge.title": "Supresión de filas (paso del Artículo 17)", "features.gov.purge.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra la supresión en el log de auditoría append-only; es un paso de una supresión conforme al Artículo 17 (los pesos entrenados no cambian).", "features.eval.safetyeval.title": "Evaluación de seguridad autónoma", "features.eval.safetyeval.body": "forgelm safety-eval ejecuta Llama Guard contra un archivo de prompts JSONL --probes (o --default-probes para el set bundleado de 51 prompts que cubre 18 categorías de daño) y emite un resumen de seguridad por categoría (safety_results.json + safety_trend.jsonl).", @@ -4607,7 +4607,7 @@ "compliance.iso.cookbook": "完整的审计 cookbook,包括 8 个 audit-floor 问题和 93 控制项覆盖图,见 docs/guides/iso_soc2_deployer_guide.md。", "features.gov.heading": "合规 & GDPR 工具", "features.gov.body": "EU AI Act 第 9-17 条、GDPR 第 15 + 17 条、ISO 27001 / SOC 2 Type II 对齐证据。", - "features.gov.access.title": "GDPR 第 15 条访问", + "features.gov.access.title": "标识符查找 (第 15 条的输入之一)", "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文或加盐摘要匹配,并列出所扫描文件中包含它的行。", "features.gov.annex.title": "Annex IV 验证器", "features.gov.annex.body": "forgelm verify-annex-iv 验证 Annex IV §1-9 模式完整性以及 bundle 的 manifest_hash 防篡改。(审计链完整性由独立的 forgelm verify-audit 命令负责。)", @@ -4617,7 +4617,7 @@ "features.gov.gguf.body": "forgelm verify-gguf 在你交付量化模型之前,校验 GGUF 魔数头、解析 metadata 块,并(若存在)根据 export 时写入的 .sha256 sidecar 文件重新校验。", "features.gov.iso.title": "ISO 27001 / SOC 2 对齐", "features.gov.iso.body": "deployer 的审计员需要的审计追踪、变更管理、数据血缘和供应链证据。软件是对齐,而不是被认证。", - "features.gov.purge.title": "GDPR 第 17 条删除", + "features.gov.purge.title": "行删除 (第 17 条的其中一步)", "features.gov.purge.body": "forgelm purge 按 id 从训练语料中删除行,并将该删除记录到 append-only 审计日志;它只是第 17 条所涉删除的其中一步(已训练的权重不会改变)。", "features.eval.safetyeval.title": "独立安全评估", "features.eval.safetyeval.body": "forgelm safety-eval 对 --probes JSONL 提示文件 (或用 --default-probes 调用打包的、覆盖 18 个危害类别的 51-提示集) 运行 Llama Guard,并产出按类别的安全摘要 (safety_results.json + safety_trend.jsonl)。", diff --git a/tests/test_reverse_pii.py b/tests/test_reverse_pii.py index 47883e3a..e3a36200 100644 --- a/tests/test_reverse_pii.py +++ b/tests/test_reverse_pii.py @@ -418,6 +418,53 @@ def test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir(self, "cross-tool audit correlation requires both subcommands to salt with the same per-output-dir salt" ) + @pytest.mark.parametrize("secret", [None, "s3cret-value-for-test"]) + def test_digests_correlate_end_to_end_in_the_default_plaintext_mode( + self, tmp_path: Path, capsys, monkeypatch, secret: str | None + ) -> None: + """The documented rule: purge's ``row_id_hash`` equals reverse-pii's ``query_hash`` when the value, the + ``--output-dir`` and the ``FORGELM_AUDIT_SECRET`` state are the same, with no ``--salt-source``; and + ``--salt-source`` is a *different* scan (digests inside the corpus) that finds nothing in a plaintext corpus.""" + from forgelm.cli.subcommands._purge import _run_purge_cmd + from forgelm.cli.subcommands._reverse_pii import _run_reverse_pii_cmd + + if secret: + monkeypatch.setenv("FORGELM_AUDIT_SECRET", secret) + corpus = tmp_path / "train.jsonl" + _seed_corpus(corpus, [{"id": "alice@example.com", "text": "mail alice@example.com"}]) + purge_args = SimpleNamespace( + row_id="alice@example.com", + corpus=str(corpus), + run_id=None, + kind=None, + check_policy=False, + output_dir=str(tmp_path), + config=None, + justification=None, + dry_run=True, + row_matches="one", + ) + with pytest.raises(SystemExit) as purge_exit: + _run_purge_cmd(purge_args, output_format="json") + assert purge_exit.value.code == 0 + purge_hash = json.loads(capsys.readouterr().out)["row_id_hash"] + + def scan(salt_source: str | None) -> dict: + args = _build_args( + query="alice@example.com", files=[str(corpus)], output_dir=str(tmp_path), salt_source=salt_source + ) + with pytest.raises(SystemExit) as scan_exit: + _run_reverse_pii_cmd(args, output_format="json") + assert scan_exit.value.code == 0 + return json.loads(capsys.readouterr().out) + + plaintext = scan(None) + assert plaintext["query_hash"] == purge_hash + assert plaintext["match_count"] == 1 + hash_mode = scan("env_var" if secret else "per_dir") + assert hash_mode["scan_mode"] == "hash" + assert hash_mode["match_count"] == 0, "--salt-source searches for digests, so a plaintext corpus has none" + # --------------------------------------------------------------------------- # §4 Test — Glob expansion: multiple files, deterministic ordering From d1f0968ce955670f2fb984f1d28a5e768fc86afa Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 03:39:00 +0300 Subject: [PATCH 16/25] fix(docs): gpu-cost and chat pages, PII detector table, masking opt-in claims, site language polish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two user-manual pages that earlier steps only partly corrected still described things the code does not do, and the pages around them overstated masking: - gpu-cost: the "run_complete" audit event, the model-card cost block, `--gpus`, `nvidia-smi` and the price table were invented. The page now describes what `_collect_resource_usage` records (device name from PyTorch, `train_runtime`, visible-device count, the real price table and `cost_source` values), where it lands (`compliance/compliance_report.json`, the JSON result, the console, `resource/` metrics), the CPU/MPS placeholder rate, the per-GPU rate, a calibration recipe that runs (output directory is a config field) with correct arithmetic, and a warning that `gpu_profiles` is not implemented. - chat: GGUF is not loadable, there is no seed or greedy mode (`/temperature 0` is rejected and `--temperature 0` fails every turn), history is the last 50 pairs not a token budget, the transcript and `/save` messages now match the REPL, and `--adapter`, `--backend` and `--trust-remote-code` are listed. The Turkish page no longer translates CLI output. - pii-masking: eight detectors, not nine; the German and French rows were "Steuer-ID / checksum" and "NIR / key validation" but both are shape-only (ID-card serial, NIR); the "Locales" table described locales that do not exist; the audit report carries counts, not row indices; running audit over masked output re-applies the same regexes. The `anon.jsonl` example name is now `masked.jsonl`. - Site: the home use-case card and the PII feature card say masking is opt-in (`--pii-mask`) in all six languages; retention hints say `0` stops the report and that the wizard writes a `retention:` block only when a value differs from its default; Article 15 card states that matching is literal and case-sensitive; German, French, Spanish and Chinese wording is tightened (overdue / "dépassé", "Default" as a noun, legacy layout, article citations, fragment punctuation). DE/FR/ES/ZH remain without native review. Co-Authored-By: Claude Sonnet 5.5 --- docs/guides/ingestion-tr.md | 7 +- docs/guides/ingestion.md | 5 +- docs/reference/usage-tr.md | 2 +- docs/reference/usage.md | 2 +- docs/usermanuals/en/data/ingestion.md | 2 +- docs/usermanuals/en/data/pii-masking.md | 41 ++++--- docs/usermanuals/en/deployment/chat.md | 70 +++++++----- docs/usermanuals/en/operations/gpu-cost.md | 125 ++++++++++++--------- docs/usermanuals/tr/data/ingestion.md | 2 +- docs/usermanuals/tr/data/pii-masking.md | 41 ++++--- docs/usermanuals/tr/deployment/chat.md | 71 ++++++------ docs/usermanuals/tr/operations/gpu-cost.md | 121 ++++++++++++-------- site/compliance.html | 2 +- site/features.html | 2 +- site/index.html | 4 +- site/js/translations.js | 110 +++++++++--------- 16 files changed, 344 insertions(+), 263 deletions(-) diff --git a/docs/guides/ingestion-tr.md b/docs/guides/ingestion-tr.md index fc73b77a..68f1779b 100644 --- a/docs/guides/ingestion-tr.md +++ b/docs/guides/ingestion-tr.md @@ -102,12 +102,13 @@ US SSN) chunk'lar JSONL'a inmeden önce redact etmek için `--pii-mask` ekleyin: ```bash -forgelm ingest ./customer_emails/ --output data/anon.jsonl --pii-mask +forgelm ingest ./customer_emails/ --output data/masked.jsonl --pii-mask ``` Tespit edilen span'ler `[REDACTED]` ile değiştirilir. Tespit regex -tabanlıdır — false positive'ler kasıtlıdır. Sonrasında -`forgelm audit` ile çıktıyı doğrulayın. +tabanlıdır — false positive'ler kasıtlıdır. Sonrasında çıktıyı +`forgelm audit` ile denetleyin: aynı regex'leri yeniden uygular, yani o pattern'lerin artık eşleşmediğini doğrular; +`--pii-ml` isim ve konum sayımı ekler. Maskelenmiş metin, geriye kalandan bir kişi yeniden tanımlanabildiği sürece hâlâ kişisel veridir. --- diff --git a/docs/guides/ingestion.md b/docs/guides/ingestion.md index 708ad92b..e174cc88 100644 --- a/docs/guides/ingestion.md +++ b/docs/guides/ingestion.md @@ -101,12 +101,13 @@ IBAN, and national IDs (TR Kimlik No, DE Personalausweis, FR INSEE, US SSN) before any chunk lands in the JSONL: ```bash -forgelm ingest ./customer_emails/ --output data/anon.jsonl --pii-mask +forgelm ingest ./customer_emails/ --output data/masked.jsonl --pii-mask ``` Detected spans are replaced with `[REDACTED]`. Detection is regex-based — false positives are intentional. Audit your output afterwards with -`forgelm audit` to verify. +`forgelm audit`: it re-applies the same regexes, so it confirms those patterns no longer match, and +`--pii-ml` adds a count of names and locations. Masked text is still personal data wherever a person can be re-identified from what remains. --- diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index 7bd9fa34..34ada6ba 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -170,7 +170,7 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ --output data/scan.jsonl # Yazmadan önce PII'yi maskele -forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl +forgelm ingest ./customer_emails/ --pii-mask --output data/masked.jsonl # Token-aware chunking — chunk'ları modelinizin vocab'ına göre boyutlandırır forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 4b9f1187..9fdd750d 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -187,7 +187,7 @@ forgelm ingest ./scan.pdf --strategy sliding --chunk-size 1024 --overlap 128 \ --output data/scan.jsonl # Mask PII before writing -forgelm ingest ./customer_emails/ --pii-mask --output data/anon.jsonl +forgelm ingest ./customer_emails/ --pii-mask --output data/masked.jsonl # Token-aware chunking — sizes chunks against your model's vocab forgelm ingest ./policies/ --recursive --output data/policies.jsonl \ diff --git a/docs/usermanuals/en/data/ingestion.md b/docs/usermanuals/en/data/ingestion.md index babb2d20..1fe04e3a 100644 --- a/docs/usermanuals/en/data/ingestion.md +++ b/docs/usermanuals/en/data/ingestion.md @@ -126,7 +126,7 @@ that appear most often: ::: :::tip -**Always audit after ingest.** A clean ingest doesn't mean a clean dataset. Run `forgelm audit data/output.jsonl` to check for cross-split leakage, near-duplicates, and PII the masking missed. +**Always audit after ingest.** A clean ingest doesn't mean a clean dataset. Run `forgelm audit data/output.jsonl` to check for cross-split leakage, near-duplicates, and PII the masking did not cover (`forgelm audit` re-applies the same regexes, so add `--pii-ml` to count names and locations). ::: ## See also diff --git a/docs/usermanuals/en/data/pii-masking.md b/docs/usermanuals/en/data/pii-masking.md index 580ccd7a..3b0cf5ad 100644 --- a/docs/usermanuals/en/data/pii-masking.md +++ b/docs/usermanuals/en/data/pii-masking.md @@ -5,7 +5,7 @@ description: Detect and redact emails, phones, credit cards, IBAN, and national # PII Masking -Personal data in your training set is a regulatory hazard (GDPR Article 5(1)(c) — data minimisation) and an operational hazard (the model memorises and emits it). ForgeLM's PII masker detects nine categories of PII and redacts them at ingest time, before chunks land in JSONL. +Personal data in your training set is a regulatory hazard (GDPR Article 5(1)(c) — data minimisation) and an operational hazard (the model memorises and emits it). ForgeLM's PII masker detects eight categories of PII and redacts them at ingest time, before chunks land in JSONL. ## What gets detected @@ -16,8 +16,8 @@ Personal data in your training set is a regulatory hazard (GDPR Article 5(1)(c) | **Credit card** | `4111-1111-1111-1111` | Issuer prefix (Visa/MC/Amex/Discover/JCB/UnionPay) + Luhn check — not Luhn alone, which every IMEI passes | | **IBAN** | `TR12 0006 4000 0011 2345 6789 01` | Country-aware checksum | | **National ID — Turkey** | 11-digit TC kimlik | Modulo-10 + modulo-11 checksums | -| **National ID — Germany** | Steuer-ID | Format + checksum | -| **National ID — France** | NIR (social security) | Format + key validation | +| **National ID — Germany** | ID-card serial (Personalausweis), e.g. `T22000129` | Shape only: a letter, 7-8 digits and an optional check character; no checksum | +| **National ID — France** | NIR (social security number) | Shape only: sex, year, month, department, commune and order digits plus an optional two-digit key; the key is not verified | | **US SSN** | `123-45-6789` | Format + reserved-block exclusions | These eight are the complete set (`_PII_PATTERNS` in `forgelm/data_audit/_pii_regex.py`). **IP addresses are not detected** — earlier versions of this page listed an "IPv4 / IPv6 (off by default; opt in)" row, but no such pattern and no such opt-in exists. If your GDPR assessment treats IP addresses as personal data, you need a separate control for them. @@ -46,7 +46,7 @@ The placeholder is consistent across the dataset, so a model can still learn tha ## The placeholder emitted :::warn -**There is one placeholder, not one per category.** Every detected span — email, phone, credit card, IBAN, TC kimlik, Steuer-ID, NIR, US SSN — is replaced with the single literal `[REDACTED]`. Earlier versions of this page published a nine-row table of per-category tags (`[EMAIL_REDACTED]`, `[PHONE_REDACTED]`, `[IP_REDACTED]`, …). None of those strings exist anywhere in the codebase. A downstream consumer that parses redaction tags to recover *which* category was masked cannot do so — that information survives only in the audit report's `pii_summary` counts. +**There is one placeholder, not one per category.** Every detected span — email, phone, credit card, IBAN, TC kimlik, German ID-card serial, NIR, US SSN — is replaced with the single literal `[REDACTED]`. Earlier versions of this page published a nine-row table of per-category tags (`[EMAIL_REDACTED]`, `[PHONE_REDACTED]`, `[IP_REDACTED]`, …). None of those strings exist anywhere in the codebase. A downstream consumer that parses redaction tags to recover *which* category was masked cannot do so — that information survives only in the audit report's `pii_summary` counts. ::: The default is `[REDACTED]` (`mask_pii`'s `replacement` parameter). The parallel secrets masker uses `[REDACTED-SECRET]` — see [Secrets Scrubbing](#/data/secrets). Note in the example above that the phone pattern does not always consume the full number; verify masking against your own formats before relying on it. @@ -55,8 +55,8 @@ The default is `[REDACTED]` (`mask_pii`'s `replacement` parameter). The parallel The PII regexes are deliberately tuned for **low false-positive rate**. They prefer to miss a borderline match (false negative) than to redact a non-PII string in your prose (false positive). Reasons: -1. False positives silently corrupt your data — replacing legitimate words with `[EMAIL_REDACTED]` ruins examples. -2. The audit step catches what masking missed; you can decide per-row whether to fix or drop. +1. False positives silently corrupt your data — replacing legitimate words with `[REDACTED]` ruins examples. +2. Running `forgelm audit` over the masked output re-applies the same regexes, so it confirms those patterns no longer match; it cannot find identifiers the patterns do not describe (names, for example — `--pii-ml` counts those). 3. Aggressive regexes have caused real-world ML pipeline outages (the Phase 11.5 incident is documented in the contributor [regex standard on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/regex.md)). If you need stricter detection — for instance, a high-stakes legal corpus — pair the masker with a manual review step. Don't push the regexes harder. @@ -67,20 +67,29 @@ To detect without modifying: ```shell $ forgelm audit data/policies.jsonl -⚠ PII: 18 emails, 4 phone, 2 IBAN (medium severity) +Data audit summary + Source : data/policies.jsonl + Total samples : 12240 + Splits : train + └─ train: n=12240 + PII : email=18, phone=4 + PII severity : worst tier = medium + by tier : medium=18, low=4 ``` -The audit report lists row indices and offsets, so you can inspect specific cases. +The audit report carries counts per category and split, not row indices or offsets. To find the rows that hold a specific identifier, search for it with `forgelm reverse-pii`. -## Locales +## Regional formats -| Locale | Phone | National ID | Notes | -|---|---|---|---| -| TR (default) | E.164 + Turkish formats | TC kimlik | Most heavily tuned. | -| DE | E.164 + German formats | Steuer-ID | | -| FR | E.164 + French formats | NIR | | -| US | E.164 + (xxx) xxx-xxxx | SSN with reserved-block exclusion | | -| Global | E.164 only | none | Fallback for unknown locales. | +| Format | What it matches | Checksum-validated | +|---|---|---| +| Phone (all countries) | One pattern: a `+` country prefix, or a parenthesised area code | No | +| TR national ID | 11 digits (TC kimlik) | Yes: modulo-10 and modulo-11 | +| DE national ID | ID-card serial: a letter, 7-8 digits, an optional check character | No | +| FR national ID | NIR: 13 digits with an optional two-digit key | No | +| US SSN | `NNN-NN-NNNN`, reserved blocks excluded | No | + +Only credit cards, IBANs and TC kimlik numbers are checksum-validated; the other families are matched by shape only, so treat a finding as a candidate to review rather than a confirmed identifier. The regex-based PII layer triggered by `forgelm ingest --pii-mask` (or the audit equivalent) detects all patterns in the table above diff --git a/docs/usermanuals/en/deployment/chat.md b/docs/usermanuals/en/deployment/chat.md index b223be2e..1e9de3f2 100644 --- a/docs/usermanuals/en/deployment/chat.md +++ b/docs/usermanuals/en/deployment/chat.md @@ -11,29 +11,35 @@ description: Sanity-check your fine-tuned model in a streaming REPL. ```shell $ forgelm chat ./checkpoints/customer-support -ForgeLM 0.7.0 — chat with checkpoints/customer-support -forgelm> how do I cancel my subscription? -You can cancel from Settings → Billing → Cancel subscription. Your access -continues until the end of the current billing period… +ForgeLM Chat — type your message and press Enter +Streaming : on | Temperature : 0.7 +Commands : /help /reset /save /temperature /system /exit +──────────────────────────────────────────────────────── -forgelm> /system You are a polite customer-support agent for a Turkish telecom. -[system prompt updated] +> how do I cancel my subscription? +Assistant: You can cancel from Settings → Billing → Cancel subscription. Your +access continues until the end of the current billing period… -forgelm> aboneliği nasıl iptal ederim? -Aboneliğinizi iptal etmek için Ayarlar → Faturalandırma → Aboneliği İptal Et -yolunu izleyebilirsiniz... +> /system You are a polite customer-support agent for a Turkish telecom. +[System prompt updated] + +> aboneliği nasıl iptal ederim? +Assistant: Aboneliğinizi iptal etmek için Ayarlar → Faturalandırma → Aboneliği +İptal Et yolunu izleyebilirsiniz... ``` +With `rich` installed, the banner is drawn inside a panel. A new system prompt applies to the turns that follow; it does not rewrite the conversation so far. + ## Slash commands | Command | What it does | |---|---| | `/reset` | Clear conversation history. | -| `/save ` | Save the conversation to JSONL. | -| `/system ` | Set or update the system prompt. | -| `/temperature ` | Set sampling temperature (0.0 to 2.0). | -| `/help` | Show this list. | -| `/exit` or `Ctrl+D` | Exit. | +| `/save [path]` | Save the conversation to JSONL; without a path it writes `chat_history_.jsonl` in the current directory. | +| `/system [prompt]` | Set or update the system prompt; without an argument it shows the current one. | +| `/temperature ` | Set sampling temperature: above 0 and up to 2.0 (`0` is rejected). | +| `/help` or `/?` | Show this list. | +| `/exit`, `/quit` or `Ctrl+D` | Exit. | ## Configuration @@ -41,13 +47,16 @@ yolunu izleyebilirsiniz... | Setting | Flag | Default | |---|---|---| +| Adapter to merge before chatting | `--adapter DIR` (not supported with `--backend unsloth`) | none | | Initial system prompt | `--system PROMPT` (change it later with `/system`) | none | | Sampling temperature | `--temperature` (change it later with `/temperature`) | `0.7` | | Maximum tokens per reply | `--max-new-tokens` | `512` | | Streaming output | `--no-stream` turns it off | on | -| Quantized loading (CUDA with bitsandbytes only; ignored otherwise) | `--load-in-4bit` (NF4) or `--load-in-8bit`; 4-bit wins if both are given | off | +| Quantized loading (CUDA with bitsandbytes only; ignored otherwise) | `--load-in-4bit` (NF4) or `--load-in-8bit`; with `--backend unsloth`, 4-bit wins if both are given and is the default when neither is | off | +| Model backend | `--backend transformers` or `--backend unsloth` | `transformers` | +| Run code bundled with the model | `--trust-remote-code` | off | -Top-p, top-k and the repetition penalty are fixed (0.9 / 50 / 1.1) and cannot be set, and there is no history file that persists across sessions; write a conversation out with `/save`. +Top-p, top-k and the repetition penalty are fixed (0.9 / 50 / 1.1) and cannot be set. Chat always samples: there is no seed and no greedy mode. There is no history file that persists across sessions; write a conversation out with `/save`. The command-line flag accepts a temperature of `0.0`, but sampling rejects it, so every turn then fails with a `[Generation error: …]` line; use a value above 0. ## Loading a checkpoint @@ -56,7 +65,8 @@ Top-p, top-k and the repetition penalty are fixed (0.9 / 50 / 1.1) and cannot be - A directory with adapter weights (LoRA): `./checkpoints/run/` - A merged checkpoint directory: `./checkpoints/run/merged/` - A HuggingFace model ID: `Qwen/Qwen2.5-7B-Instruct` -- A GGUF file: `./model.gguf` (uses llama.cpp under the hood) + +GGUF files are not loaded by `forgelm chat`: the command loads Hugging Face checkpoints only. To try a GGUF export, run it in Ollama or llama.cpp (see [Deploy Targets](#/deployment/deploy-targets)). For LoRA checkpoints, pass the base model as the positional argument and the adapter directory via `--adapter`: @@ -71,7 +81,7 @@ $ forgelm chat "Qwen/Qwen2.5-7B" --adapter ./checkpoints/run/ Under the planned `--safety` flag (preview only — see the note above), every prompt and response would be screened by Llama Guard: ```text -forgelm> [adversarial prompt] +> [adversarial prompt] [Llama Guard flagged S2 (non-violent crimes) — refusing] I can't help with that. Try rephrasing the question. ``` @@ -80,28 +90,26 @@ Useful when probing the model for jailbreaks before deployment. Off by default f ## Multi-turn handling -Conversation history is preserved within a session. The model sees the full chat (or as much as fits in `max_length`): +Conversation history is preserved within a session. With every prompt the model receives the system prompt and the last 50 question-and-answer pairs; older pairs stay in memory (and in a `/save` file) but are no longer sent. The limit counts pairs, not tokens: ```text -forgelm> what's the capital of Türkiye? -Ankara. +> what's the capital of Türkiye? +Assistant: Ankara. -forgelm> what's its population? -[uses prior context: "its" refers to Ankara] -About 5.7 million as of recent estimates... +> what's its population? +Assistant: About 5.7 million as of recent estimates... ``` -To start fresh: `/reset`. +The second answer is only possible because the first turn is sent along with it ("its" refers to Ankara). To start fresh: `/reset`. ## Saving and replaying ```shell -forgelm> /save sessions/qa-1.jsonl -[saved 6 turns to sessions/qa-1.jsonl] +> /save sessions/qa-1.jsonl +[History saved → sessions/qa-1.jsonl (7 messages)] ``` -The saved JSONL is replayable by feeding it back through a fresh -session's prompt history (replay tooling is roadmapped for a future Pro CLI release). +The file holds one `{"role": ..., "content": ...}` message per line: the system prompt first (when one is set), then every user and assistant message of the session. The system prompt written is the one in force when you save, even if you changed it mid-conversation. There is no command to load a saved file back into a session (replay tooling is roadmapped for a future Pro CLI release). Sessions are useful for: - Reproducing a bug you found during testing. @@ -125,11 +133,11 @@ Win rate v2 vs v1: 0.62 (sig p=0.04) ## Common pitfalls :::warn -**Treating `/temperature 0` as deterministic.** It's near-deterministic, but tiebreaks in argmax sampling can still produce minor variance. For exact reproducibility, set `seed:` in YAML. +**Treating chat as reproducible.** Chat always samples, and it has no seed and no greedy mode (`/temperature 0` is rejected), so the same prompt can give different answers. Use it to explore; keep a `/save` transcript of what the model actually said, and use the evaluation commands for repeatable checks. ::: :::warn -**Long conversations exceeding context.** Once history exceeds `max_length`, ForgeLM drops oldest messages first. The conversation can lose continuity unexpectedly. Use `/reset` periodically for long testing sessions. +**Long conversations exceeding context.** Only the last 50 pairs are sent, but nothing counts tokens: a long system prompt, long answers or a small-context model can still overflow, and the turn then fails with a `[Generation error: …]` line (the turn is not added to the history). Use `/reset` periodically for long testing sessions. ::: :::tip diff --git a/docs/usermanuals/en/operations/gpu-cost.md b/docs/usermanuals/en/operations/gpu-cost.md index 2e06488a..8bedd83c 100644 --- a/docs/usermanuals/en/operations/gpu-cost.md +++ b/docs/usermanuals/en/operations/gpu-cost.md @@ -1,93 +1,114 @@ --- title: GPU Cost Estimation -description: Auto-detect across 16 GPU profiles and track per-run cost against your hourly rate. +description: Detect the GPU, price the run from your hourly rate or a built-in table, and record the cost of every run. --- # GPU Cost Estimation -> **Status:** GPU detection + per-run duration + audit-log stamping ship -> today, and `training.gpu_cost_per_hour` is the one cost setting. A -> config-driven `cost_tracking:` block (per-GPU rate tables, alert / halt -> thresholds) is **tracked on the roadmap and not yet implemented** — +> **Status:** GPU detection, the training duration and a cost estimate are +> recorded for every run today, and `training.gpu_cost_per_hour` is the one cost +> setting. A config-driven `cost_tracking:` block (per-GPU rate tables, alert / +> halt thresholds) is **tracked on the roadmap and not yet implemented** — > `forgelm/config.py` has no `output.cost_tracking` surface, so no example on > this page uses one. The deferral is tracked in the > [risks-and-decisions roadmap on GitHub](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). -ForgeLM detects the GPU you're running on, looks up its profile (memory, compute, typical hourly rate), and tracks per-run cost. After every run, the audit log records exactly how much GPU time was used and what it cost. +When a training run finishes, ForgeLM records how long it trained, on how many GPUs, and what that cost at an hourly rate: yours if you set one, otherwise a built-in price for the detected GPU. The estimate is written next to the other run results; it is not a bill, and it does not stop or alert anything. ## How detection works -On startup, `forgelm` reads: -- `nvidia-smi --query-gpu=name,memory.total,...` for hardware identification. -- The matching hourly rate from the built-in GPU price table in `forgelm/trainer.py`. +After training, ForgeLM asks PyTorch for the first CUDA device's name (`torch.cuda.get_device_name(0)`), the number of CUDA devices the process can see, and the peak memory allocated on the first device. It takes the training time from the Hugging Face trainer's `train_runtime`. It does not call `nvidia-smi`. Without a CUDA GPU none of the GPU fields are recorded. -Supported GPUs include: +The rate comes from `training.gpu_cost_per_hour` when set, and otherwise from the price table `_GPU_PRICING` in `forgelm/trainer.py`, which is matched against the exact device name that PyTorch reports: -| Class | Models | +| Class | Recognised names | |---|---| -| Datacenter | A100 40 GB / 80 GB, H100 80 GB, H200, L40S | -| Workstation | RTX 6000 Ada, RTX A6000 | -| Consumer | RTX 4090, RTX 4080, RTX 3090, RTX 3080 | -| Cloud-only | T4, V100, A10G | -| Apple | M1/M2/M3 Max (CPU/MPS fallback) | +| Tesla | Tesla T4, Tesla P100, Tesla V100, Tesla K80 | +| NVIDIA data center | NVIDIA A10G, NVIDIA A100-SXM4-40GB, NVIDIA A100-SXM4-80GB, NVIDIA A100 80GB PCIe, NVIDIA H100 80GB HBM3, NVIDIA H100 SXM5 80GB, NVIDIA H200, NVIDIA L4, NVIDIA L40S, NVIDIA B200 | +| GeForce (self-hosted) | NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 4090 | -If your GPU isn't recognised, ForgeLM logs a warning and falls back to a generic profile (no cost estimation, but training works). +The prices in that table are approximate on-demand cloud averages (for the GeForce cards, an estimate of electricity plus amortisation). A GPU that is not in the table, for example `NVIDIA RTX A6000` or `NVIDIA H100 PCIe`, is **not** reported with a warning: the run simply has no estimated cost unless you set `training.gpu_cost_per_hour`. ## Configuring your hourly rate ```yaml training: - gpu_cost_per_hour: 1.10 # USD per hour for the training GPU + gpu_cost_per_hour: 1.10 # USD per hour, per GPU ``` -Set `training.gpu_cost_per_hour` to your real rate. When it is unset, ForgeLM falls back to its built-in price table for the detected GPU; the run's `resource_usage` records which one it used (`cost_source`: `user_config`, `auto_detected` for an exact GPU-name match, or `fuzzy_match` for a partial-name match; the key is absent when no rate could be resolved). Without a CUDA GPU there is no detected GPU, so set `training.gpu_cost_per_hour` explicitly. There is one rate per run: per-GPU-model rate tables are not supported. +Set `training.gpu_cost_per_hour` to your real rate. When it is unset, ForgeLM falls back to its built-in price table for the detected GPU; the run's `resource_usage` records which one it used in `cost_source`: + +| `cost_source` | Meaning | +|---|---| +| `user_config` | The rate came from `training.gpu_cost_per_hour` (`0` is accepted and records a zero cost). | +| `auto_detected` | The device name matched a table entry exactly. | +| `fuzzy_match` | The device name partly matched a table entry. This is a guess: `NVIDIA L40`, for instance, is priced as `NVIDIA L40S`. | + +When no rate resolves, `cost_source`, `gpu_cost_per_hour_usd` and `estimated_cost_usd` are absent. + +:::warn +**Machines without a CUDA GPU (CPU, Apple silicon).** There is no device name, and an empty name partly matches the first table entry, so the run is recorded at a placeholder `0.15` USD per hour with `cost_source: fuzzy_match`. Do not read that as a real cost; set `training.gpu_cost_per_hour` explicitly on such machines (`0` if the run is free to you). +::: + +There is one rate per run, and it is per GPU: per-GPU-model rate tables are not supported. ## Output -After each run, the audit log records: +The estimate is recorded in four places: + +- **`compliance/compliance_report.json`**, under `resource_usage`, in the run's `training.output_dir`, written when the run completes. +- **The `--output-format json` result**, as `resource_usage` and a top-level `estimated_cost_usd`. +- **The console**, as an `Estimated training cost` line and a `GPU-hours` line that names the pricing source (text mode). +- **The run metrics**: every numeric field is copied in as `resource/` (for example `resource/estimated_cost_usd`), so it reaches your experiment tracker and the `metrics_summary` of the `pipeline.completed` audit event. + +A 4-GPU run that trained for 1892 seconds at `gpu_cost_per_hour: 1.10` records: ```json { - "event": "run_complete", - "ts": "2026-04-29T14:33:10Z", - "duration_seconds": 1892, - "gpu_profile": "A100_80GB", - "gpus_used": 1, - "estimated_cost_usd": 0.578, - "rate_per_hour": 1.10, - "currency": "USD" + "resource_usage": { + "gpu_model": "NVIDIA A100-SXM4-80GB", + "peak_vram_gb": 41.7, + "gpu_count": 4, + "training_duration_seconds": 1892.0, + "gpu_hours": 2.102, + "cost_source": "user_config", + "gpu_cost_per_hour_usd": 1.1, + "estimated_cost_usd": 2.3124 + } } ``` -The model card cites it: - -```markdown -## Training cost - -This model was trained for 31m 32s on 1× A100 80GB, -estimated at $0.58 USD at the configured rate. -``` +The model card does not include the cost. ## Pre-flight cost estimation -Before starting a long training run, run a short calibration (1-2 steps with `training.max_steps: 2`), capture the resulting `gpu_hours` from the per-run `compliance_report.json`, and multiply by your provider's hourly rate. (A dedicated `--estimate-cost` flag was discussed but not shipped; the resource-tracking path emits actuals only.) +Before starting a long training run, run a short calibration with a hard step cap, read `gpu_hours` from its `resource_usage`, scale it to the full step count and multiply by your hourly rate. A dedicated `--estimate-cost` flag was discussed but not shipped; the resource-tracking path records actuals only. + +```yaml +# configs/calibration.yaml — your real config, plus: +training: + max_steps: 20 + output_dir: /tmp/calib +``` ```shell -$ forgelm --config configs/calibration.yaml --output-dir /tmp/calib -$ jq '.resource_usage.gpu_hours' /tmp/calib/compliance_report.json +$ forgelm --config configs/calibration.yaml +$ jq '.resource_usage.gpu_hours' /tmp/calib/compliance/compliance_report.json 0.034 -$ python -c "print(0.034 * (3 / 0.034) * 1.10)" # 6h training at $1.10/hr -$7.15 +$ python -c "print(0.034 * (3000 / 20) * 1.10)" # 3,000 steps at $1.10 per GPU-hour +5.61 ``` -The calibration approach is typically within 20% of actual. +Treat the result as a rough figure. The first steps include warm-up, and the mix of sequence lengths in a few steps rarely matches the whole dataset, so run enough steps to see a stable step time before scaling it. ## Multi-GPU and distributed -For multi-GPU training, ForgeLM multiplies the single `training.gpu_cost_per_hour` rate by the GPU count (`gpu_hours` is the training time times the GPU count). +`gpu_hours` is the training time multiplied by the number of CUDA devices visible to the process, and the single `training.gpu_cost_per_hour` rate is applied per GPU. A 4×A100 run for 2 hours = 4 × 2 × $1.10 = $8.80, regardless of whether you use ZeRO or FSDP. +If the job uses fewer GPUs than the machine has, restrict the devices it can see (for example with `CUDA_VISIBLE_DEVICES`), or the estimate counts GPUs the run never used. + ## Cost alerts (roadmap — not yet implemented) For runs that may run away, the planned `cost_tracking` block would support @@ -95,36 +116,38 @@ threshold-based alerts (a webhook fires when an alert threshold is crossed) and halts (training stops at a halt threshold). No such setting exists in `forgelm/config.py` today. -When implemented, the alert will fire the configured webhook (see [Webhooks](#/operations/webhooks)) — useful in CI to catch a misconfigured run before it spends a week of budget overnight. Until then, monitor cost manually via the audit log + a budget-side guardrail in your scheduler. +When implemented, the alert will fire the configured webhook (see [Webhooks](#/operations/webhooks)) — useful in CI to catch a misconfigured run before it spends a week of budget overnight. Until then, monitor cost manually via the recorded `resource_usage` and a budget-side guardrail in your scheduler. ## Custom GPU profiles -To add a GPU not in the default profile: +:::warn +**Not implemented.** ForgeLM has no `gpu_profiles` setting and reads no `gpu_profiles.local.yaml`; the snippet below is the planned shape only and a config that contains it is rejected. Today the way to price a GPU that is not in the table is `training.gpu_cost_per_hour`, which works for any GPU. +::: + +To add a GPU not in the built-in table (planned): ```yaml gpu_profiles: custom: - name: "AcmeNVIDIA-XYZ" - pattern: "AcmeNVIDIA-XYZ" # match against nvidia-smi name + pattern: "AcmeNVIDIA-XYZ" # match against the device name memory_gb: 96 compute_capability: 9.0 tensor_cores: true ``` -Drop this in your project root as `gpu_profiles.local.yaml`; ForgeLM auto-merges it. - ## Common pitfalls :::warn -**Hourly rate as gospel.** ForgeLM's defaults are reasonable averages — your actual cloud bill depends on instance type, region, spot vs on-demand, and discounts. Override with your real rate. +**Hourly rate as gospel.** ForgeLM's defaults are rough averages — your actual cloud bill depends on instance type, region, spot vs on-demand, and discounts. Override with your real rate. ::: :::warn -**Multi-tenancy on shared GPUs.** If multiple jobs share a GPU (rare for training, common for inference), the cost-tracking multiplies rather than divides. Use `--gpus N` to declare actual allocation. +**Shared GPUs.** The estimate bills the whole training time on every visible GPU; there is no setting for a fractional or shared allocation. If other jobs share the GPU, divide the figure yourself. ::: :::tip -**Track cost over time.** The audit log includes cost for every run; trend it over weeks to catch creeping overruns. A 2× cost increase for the same training time is usually a sign that data grew or hyperparameters drifted. +**Track cost over time.** Every run records its cost; trend it over weeks to catch creeping overruns. A 2× cost increase for the same training time is usually a sign that data grew or hyperparameters drifted. ::: ## See also diff --git a/docs/usermanuals/tr/data/ingestion.md b/docs/usermanuals/tr/data/ingestion.md index caa97460..63421c4e 100644 --- a/docs/usermanuals/tr/data/ingestion.md +++ b/docs/usermanuals/tr/data/ingestion.md @@ -125,7 +125,7 @@ Yetkili liste için `forgelm ingest --help`. En sık görülenler: ::: :::tip -**Ingest'ten sonra her zaman audit yapın.** Temiz ingest temiz dataset anlamına gelmez. Split-arası sızıntı, near-duplicate ve maskelemenin kaçırdığı PII için `forgelm audit data/output.jsonl` koşturun. +**Ingest'ten sonra her zaman audit yapın.** Temiz ingest temiz dataset anlamına gelmez. Split-arası sızıntı, near-duplicate ve maskelemenin kapsamadığı PII için `forgelm audit data/output.jsonl` koşturun (`forgelm audit` aynı regex'leri yeniden uygular; isim ve konumları saymak için `--pii-ml` ekleyin). ::: ## Bkz. diff --git a/docs/usermanuals/tr/data/pii-masking.md b/docs/usermanuals/tr/data/pii-masking.md index 8683adc0..358b551b 100644 --- a/docs/usermanuals/tr/data/pii-masking.md +++ b/docs/usermanuals/tr/data/pii-masking.md @@ -5,7 +5,7 @@ description: E-posta, telefon, kredi kartı, IBAN ve ulusal kimlikleri ingest s # PII Maskeleme -Eğitim setinizdeki kişisel veri hem regülatif tehlikedir (GDPR Md. 5(1)(c) — veri minimizasyonu) hem operasyonel tehlikedir (model ezberler ve geri verir). ForgeLM'in PII maskeleyicisi dokuz kategori PII'yi tespit eder ve ingest zamanında, satırlar JSONL'a düşmeden redakte eder. +Eğitim setinizdeki kişisel veri hem regülatif tehlikedir (GDPR Md. 5(1)(c) — veri minimizasyonu) hem operasyonel tehlikedir (model ezberler ve geri verir). ForgeLM'in PII maskeleyicisi sekiz kategori PII'yi tespit eder ve ingest zamanında, satırlar JSONL'a düşmeden redakte eder. ## Tespit edilenler @@ -16,8 +16,8 @@ Eğitim setinizdeki kişisel veri hem regülatif tehlikedir (GDPR Md. 5(1)(c) | **Kredi kartı** | `4111-1111-1111-1111` | Issuer öneki (Visa/MC/Amex/Discover/JCB/UnionPay) + Luhn — yalnızca Luhn değil; her IMEI Luhn'dan geçer | | **IBAN** | `TR12 0006 4000 0011 2345 6789 01` | Ülke-bilinçli checksum | | **Ulusal ID — Türkiye** | 11 haneli TC kimlik | Modulo-10 + modulo-11 checksum | -| **Ulusal ID — Almanya** | Steuer-ID | Format + checksum | -| **Ulusal ID — Fransa** | NIR (sosyal güvenlik) | Format + key validation | +| **Ulusal ID — Almanya** | Kimlik kartı seri numarası (Personalausweis), örn. `T22000129` | Yalnızca şekil: bir harf, 7-8 rakam ve isteğe bağlı bir kontrol karakteri; checksum yok | +| **Ulusal ID — Fransa** | NIR (sosyal güvenlik numarası) | Yalnızca şekil: cinsiyet, yıl, ay, departman, komün ve sıra rakamları ile isteğe bağlı iki haneli anahtar; anahtar doğrulanmaz | | **US SSN** | `123-45-6789` | Format + reserved-block dışlama | Bu sekizi tam kümedir (`forgelm/data_audit/_pii_regex.py` içindeki `_PII_PATTERNS`). **IP adresleri tespit edilmez** — bu sayfanın önceki sürümleri bir "IPv4 / IPv6 (varsayılan kapalı; opt-in)" satırı listeliyordu, ancak böyle bir pattern ve böyle bir opt-in mevcut değil. GDPR değerlendirmeniz IP adreslerini kişisel veri sayıyorsa, bunun için ayrı bir kontrole ihtiyacınız var. @@ -46,7 +46,7 @@ Placeholder dataset boyunca tutarlı; model "şu slot'a maskelenmiş *bir şey* ## Yayınlanan placeholder :::warn -**Kategori başına değil, tek bir placeholder vardır.** Tespit edilen her span — e-posta, telefon, kredi kartı, IBAN, TC kimlik, Steuer-ID, NIR, US SSN — tek bir düz `[REDACTED]` ile değiştirilir. Bu sayfanın önceki sürümleri kategori başına etiketlerden oluşan dokuz satırlık bir tablo yayımlıyordu (`[EMAIL_REDACTED]`, `[PHONE_REDACTED]`, `[IP_REDACTED]`, …). Bu string'lerin hiçbiri kod tabanının hiçbir yerinde mevcut değil. Hangi kategorinin maskelendiğini geri elde etmek için redaksiyon etiketlerini parse eden bir tüketici bunu yapamaz — o bilgi yalnızca audit raporundaki `pii_summary` sayımlarında yaşar. +**Kategori başına değil, tek bir placeholder vardır.** Tespit edilen her span — e-posta, telefon, kredi kartı, IBAN, TC kimlik, Alman kimlik kartı seri numarası, NIR, US SSN — tek bir düz `[REDACTED]` ile değiştirilir. Bu sayfanın önceki sürümleri kategori başına etiketlerden oluşan dokuz satırlık bir tablo yayımlıyordu (`[EMAIL_REDACTED]`, `[PHONE_REDACTED]`, `[IP_REDACTED]`, …). Bu string'lerin hiçbiri kod tabanının hiçbir yerinde mevcut değil. Hangi kategorinin maskelendiğini geri elde etmek için redaksiyon etiketlerini parse eden bir tüketici bunu yapamaz — o bilgi yalnızca audit raporundaki `pii_summary` sayımlarında yaşar. ::: Varsayılan `[REDACTED]`'dır (`mask_pii`'nin `replacement` parametresi). Paralel sır maskeleyici `[REDACTED-SECRET]` kullanır — bkz. [Sır Temizleme](#/data/secrets). Yukarıdaki örnekte telefon pattern'ının numaranın tamamını her zaman tüketmediğine dikkat edin; maskelemeye güvenmeden önce kendi formatlarınıza karşı doğrulayın. @@ -55,8 +55,8 @@ Varsayılan `[REDACTED]`'dır (`mask_pii`'nin `replacement` parametresi). Parale PII regex'leri bilinçli olarak **düşük false-positive oran** için ayarlanır. Sınırda eşleşmeyi atlamayı (false negative) prose'unuzdaki PII olmayan string'i redakte etmeye (false positive) tercih eder. Sebepler: -1. False positive sessizce verinizi bozar — gerçek kelimeleri `[EMAIL_REDACTED]` ile değiştirmek örnekleri mahveder. -2. Audit aşaması maskelemenin kaçırdığını yakalar; satır başına düzeltme veya düşürme kararı sizde. +1. False positive sessizce verinizi bozar — gerçek kelimeleri `[REDACTED]` ile değiştirmek örnekleri mahveder. +2. Maskelenmiş çıktı üzerinde `forgelm audit` koşturmak aynı regex'leri yeniden uygular; bu yüzden o pattern'lerin artık eşleşmediğini doğrular, ama pattern'lerin tarif etmediği tanımlayıcıları (örneğin isimleri) bulamaz — onları `--pii-ml` sayar. 3. Agresif regex'ler gerçek-dünya ML pipeline kesintilerine yol açtı (Phase 11.5 olayı [GitHub'daki katkıda bulunan regex standardında](https://github.com/HodeTech/ForgeLM/blob/main/docs/standards/regex.md) belgelenmiştir). Daha katı tespit gerekirse — örneğin yüksek-stake bir hukuk corpus'u — maskeleyiciyi manuel inceleme adımıyla birleştirin. Regex'leri zorlamayın. @@ -67,20 +67,29 @@ Modifiye etmeden tespit için: ```shell $ forgelm audit data/policies.jsonl -⚠ PII: 18 e-posta, 4 telefon, 2 IBAN (orta seviye) +Data audit summary + Source : data/policies.jsonl + Total samples : 12240 + Splits : train + └─ train: n=12240 + PII : email=18, phone=4 + PII severity : worst tier = medium + by tier : medium=18, low=4 ``` -Audit raporu satır indisleri ve offset'leri listeler; spesifik vakaları inceleyebilirsiniz. +Audit raporu kategori ve split başına sayımları taşır; satır indislerini ya da offset'leri değil. Belirli bir tanımlayıcıyı içeren satırları bulmak için onu `forgelm reverse-pii` ile arayın. -## Locale'ler +## Bölgesel formatlar -| Locale | Telefon | Ulusal ID | Notlar | -|---|---|---|---| -| TR (varsayılan) | E.164 + Türkiye formatları | TC kimlik | En çok ayarlanan. | -| DE | E.164 + Almanya formatları | Steuer-ID | | -| FR | E.164 + Fransa formatları | NIR | | -| US | E.164 + (xxx) xxx-xxxx | Reserved-block dışlamalı SSN | | -| Global | Sadece E.164 | yok | Bilinmeyen locale fallback. | +| Format | Neyi eşleştirir | Checksum doğrulaması | +|---|---|---| +| Telefon (tüm ülkeler) | Tek bir pattern: `+` ülke öneki ya da parantezli alan kodu | Hayır | +| TR ulusal ID | 11 rakam (TC kimlik) | Evet: modulo-10 ve modulo-11 | +| DE ulusal ID | Kimlik kartı seri numarası: bir harf, 7-8 rakam, isteğe bağlı bir kontrol karakteri | Hayır | +| FR ulusal ID | NIR: isteğe bağlı iki haneli anahtarla 13 rakam | Hayır | +| US SSN | `NNN-NN-NNNN`, reserved block'lar hariç | Hayır | + +Yalnızca kredi kartları, IBAN'lar ve TC kimlik numaraları checksum ile doğrulanır; diğer aileler yalnızca şekle göre eşleştirilir. Bu yüzden bir bulguyu doğrulanmış bir tanımlayıcı değil, gözden geçirilecek bir aday sayın. `forgelm ingest --pii-mask` (veya audit eşdeğeri) tarafından tetiklenen regex-bazlı PII katmanı, yukarıdaki tablodaki tüm diff --git a/docs/usermanuals/tr/deployment/chat.md b/docs/usermanuals/tr/deployment/chat.md index f2d0941d..08357564 100644 --- a/docs/usermanuals/tr/deployment/chat.md +++ b/docs/usermanuals/tr/deployment/chat.md @@ -11,29 +11,35 @@ description: Akışlı REPL'de fine-tuned modelinizi sınayın. ```shell $ forgelm chat ./checkpoints/customer-support -ForgeLM 0.7.0 — checkpoints/customer-support ile sohbet -forgelm> aboneliği nasıl iptal ederim? -Aboneliğinizi Ayarlar → Faturalandırma → Aboneliği İptal Et adımlarıyla -iptal edebilirsiniz. Erişiminiz mevcut faturalama döneminin sonuna kadar -devam eder… +ForgeLM Chat — type your message and press Enter +Streaming : on | Temperature : 0.7 +Commands : /help /reset /save /temperature /system /exit +──────────────────────────────────────────────────────── -forgelm> /system Türk telekom için kibar bir müşteri-destek temsilcisisin. -[system prompt güncellendi] +> aboneliği nasıl iptal ederim? +Assistant: Aboneliğinizi Ayarlar → Faturalandırma → Aboneliği İptal Et +adımlarıyla iptal edebilirsiniz. Erişiminiz mevcut faturalama döneminin +sonuna kadar devam eder… -forgelm> tarifeyi nasıl yükseltirim? -Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... +> /system Türk telekom için kibar bir müşteri-destek temsilcisisin. +[System prompt updated] + +> tarifeyi nasıl yükseltirim? +Assistant: Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... ``` +CLI çıktısı yalnızca İngilizcedir; yukarıdaki banner ve köşeli parantezli mesajlar bu yüzden çevrilmemiştir. `rich` kuruluysa banner bir panel içinde çizilir. Yeni bir system prompt sonraki turlara uygulanır; o ana kadarki konuşmayı yeniden yazmaz. + ## Slash komutları | Komut | Yaptığı | |---|---| | `/reset` | Konuşma geçmişini temizle. | -| `/save ` | Konuşmayı JSONL'a kaydet. | -| `/system ` | System prompt'u ayarla veya güncelle. | -| `/temperature ` | Sampling sıcaklığını ayarla (0.0 - 2.0). | -| `/help` | Bu listeyi göster. | -| `/exit` veya `Ctrl+D` | Çık. | +| `/save [yol]` | Konuşmayı JSONL'a kaydet; yol verilmezse geçerli dizine `chat_history_.jsonl` yazar. | +| `/system [prompt]` | System prompt'u ayarla veya güncelle; argümansız çağrılırsa mevcut olanı gösterir. | +| `/temperature ` | Sampling sıcaklığını ayarla: 0'ın üstünde ve 2.0'a kadar (`0` reddedilir). | +| `/help` veya `/?` | Bu listeyi göster. | +| `/exit`, `/quit` veya `Ctrl+D` | Çık. | ## Konfigürasyon @@ -41,13 +47,16 @@ Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... | Ayar | Flag | Varsayılan | |---|---|---| +| Sohbetten önce birleştirilecek adapter | `--adapter DIR` (`--backend unsloth` ile desteklenmez) | yok | | Başlangıç system prompt'u | `--system PROMPT` (sonra `/system` ile değiştirin) | yok | | Sampling temperature | `--temperature` (sonra `/temperature` ile değiştirin) | `0.7` | | Yanıt başına maksimum token | `--max-new-tokens` | `512` | | Streaming çıktı | `--no-stream` kapatır | açık | -| Quantize yükleme (yalnızca CUDA + bitsandbytes; aksi halde yok sayılır) | `--load-in-4bit` (NF4) veya `--load-in-8bit`; ikisi de verilirse 4-bit kazanır | kapalı | +| Quantize yükleme (yalnızca CUDA + bitsandbytes; aksi halde yok sayılır) | `--load-in-4bit` (NF4) veya `--load-in-8bit`; `--backend unsloth` ile ikisi de verilirse 4-bit kazanır, hiçbiri verilmezse 4-bit varsayılandır | kapalı | +| Model backend'i | `--backend transformers` veya `--backend unsloth` | `transformers` | +| Modelle birlikte gelen kodu çalıştırma | `--trust-remote-code` | kapalı | -Top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz. Oturumlar arası kalıcı bir history dosyası da yoktur; bir konuşmayı `/save` ile kaydedin. +Top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz. Chat her zaman sampling yapar: seed yoktur ve greedy mod yoktur. Oturumlar arası kalıcı bir history dosyası da yoktur; bir konuşmayı `/save` ile kaydedin. Komut satırı flag'i `0.0` sıcaklığı kabul eder, ancak sampling onu reddeder; bu yüzden her tur bir `[Generation error: …]` satırıyla başarısız olur. 0'ın üstünde bir değer kullanın. ## Checkpoint yükleme @@ -56,7 +65,8 @@ Top-p, top-k ve repetition penalty sabittir (0.9 / 50 / 1.1) ve ayarlanamaz. Otu - LoRA adapter ağırlıklı dizin: `./checkpoints/run/` - Birleştirilmiş checkpoint dizini: `./checkpoints/run/merged/` - HuggingFace model ID: `Qwen/Qwen2.5-7B-Instruct` -- GGUF dosyası: `./model.gguf` (alta llama.cpp) + +GGUF dosyalarını `forgelm chat` yüklemez: komut yalnızca Hugging Face checkpoint'lerini yükler. Bir GGUF çıktısını denemek için onu Ollama veya llama.cpp'de çalıştırın (bkz. [Deploy Hedefleri](#/deployment/deploy-targets)). LoRA checkpoint'lerinde base modeli pozisyonel argüman olarak verin ve adapter dizinini `--adapter` ile belirtin: @@ -71,7 +81,7 @@ $ forgelm chat "Qwen/Qwen2.5-7B" --adapter ./checkpoints/run/ Planlanan `--safety` bayrağı altında (yalnız önizleme — yukarıdaki nota bakın) her prompt ve yanıt Llama Guard tarafından taranır: ```text -forgelm> [adversarial prompt] +> [adversarial prompt] [Llama Guard S2 (şiddet içermeyen suç) flagledi — reddediliyor] Bunda yardım edemem. Soruyu farklı şekilde sormayı deneyin. ``` @@ -80,29 +90,26 @@ Deployment öncesi modeli jailbreak için sondalarken faydalı. Sıradan sohbett ## Multi-turn yönetimi -Konuşma geçmişi bir oturum içinde korunur. Model tüm chat'i (veya `max_length`'a sığanı) görür: +Konuşma geçmişi bir oturum içinde korunur. Model her prompt'ta system prompt'u ve son 50 soru-cevap çiftini alır; daha eski çiftler bellekte (ve `/save` dosyasında) kalır ama artık gönderilmez. Sınır token'ı değil, çiftleri sayar: ```text -forgelm> Türkiye'nin başkenti neresi? -Ankara. +> Türkiye'nin başkenti neresi? +Assistant: Ankara. -forgelm> nüfusu ne kadar? -[önceki bağlamı kullanır: "nüfusu" Ankara'ya işaret ediyor] -Yaklaşık 5.7 milyon, son tahminlere göre... +> nüfusu ne kadar? +Assistant: Yaklaşık 5.7 milyon, son tahminlere göre... ``` -Sıfırdan başlamak için: `/reset`. +İkinci yanıt, ilk tur onunla birlikte gönderildiği için mümkündür ("nüfusu" Ankara'ya işaret ediyor). Sıfırdan başlamak için: `/reset`. ## Kaydetme ve oynatma ```shell -forgelm> /save sessions/qa-1.jsonl -[6 tur sessions/qa-1.jsonl'a kaydedildi] +> /save sessions/qa-1.jsonl +[History saved → sessions/qa-1.jsonl (7 messages)] ``` -Kaydedilen JSONL, yeni bir oturumun prompt history'sine geri -besleyerek tekrar oynatılabilir (replay tooling gelecekteki bir Pro CLI sürümü için -roadmap'tedir). +Dosya, satır başına bir `{"role": ..., "content": ...}` mesajı içerir: önce (ayarlıysa) system prompt, ardından oturumun tüm kullanıcı ve asistan mesajları. Yazılan system prompt, konuşma sırasında değiştirmiş olsanız bile kaydettiğiniz andaki geçerli olandır. Kaydedilen bir dosyayı bir oturuma geri yükleyen bir komut yoktur (replay tooling gelecekteki bir Pro CLI sürümü için roadmap'tedir). Oturumlar şu durumlar için faydalı: - Test sırasında bulduğunuz bir bug'ı yeniden üretmek. @@ -126,11 +133,11 @@ v1 vs v2 win rate: 0.62 (sig p=0.04) ## Sık hatalar :::warn -**`/temperature 0`'ı determinist saymak.** Yakın-determinist ama argmax sampling'deki eşitlik bozma hâlâ küçük varyans üretebilir. Tam yeniden üretilebilirlik için YAML'da `seed:` ayarlayın. +**Chat'i yeniden üretilebilir saymak.** Chat her zaman sampling yapar; seed'i ve greedy modu yoktur (`/temperature 0` reddedilir), bu yüzden aynı prompt farklı yanıtlar verebilir. Keşif için kullanın; modelin gerçekte ne söylediğini `/save` dökümüyle saklayın ve tekrarlanabilir kontroller için değerlendirme komutlarını kullanın. ::: :::warn -**Context'i aşan uzun konuşmalar.** Geçmiş `max_length`'i aşınca ForgeLM en eski mesajları düşürür. Konuşma süreklilik kaybedebilir. Uzun test oturumlarında periyodik `/reset` kullanın. +**Context'i aşan uzun konuşmalar.** Yalnızca son 50 çift gönderilir, ancak hiçbir şey token saymaz: uzun bir system prompt, uzun yanıtlar ya da küçük context'li bir model yine de taşabilir; tur o zaman bir `[Generation error: …]` satırıyla başarısız olur (tur geçmişe eklenmez). Uzun test oturumlarında periyodik `/reset` kullanın. ::: :::tip diff --git a/docs/usermanuals/tr/operations/gpu-cost.md b/docs/usermanuals/tr/operations/gpu-cost.md index 90042fa4..ed189746 100644 --- a/docs/usermanuals/tr/operations/gpu-cost.md +++ b/docs/usermanuals/tr/operations/gpu-cost.md @@ -1,93 +1,114 @@ --- title: GPU Maliyet Tahmini -description: 16 GPU profilinde otomatik tespit ve saatlik tarifenize göre koşu başı maliyet izleme. +description: GPU'yu tespit eder, koşuyu sizin saatlik tarifenizle ya da yerleşik bir tabloyla fiyatlandırır ve her koşunun maliyetini kaydeder. --- # GPU Maliyet Tahmini -> **Durum:** GPU tespiti + koşu süresi + audit-log damgalama bugün -> gönderiliyor ve `training.gpu_cost_per_hour` tek maliyet ayarıdır. +> **Durum:** GPU tespiti, eğitim süresi ve maliyet tahmini bugün her koşu için +> kaydedilir ve `training.gpu_cost_per_hour` tek maliyet ayarıdır. > Konfigürasyon-tabanlı bir `cost_tracking:` bloğu (GPU başına tarife > tabloları, alarm / durdurma eşikleri) **yol haritasında ve henüz > uygulanmadı** — `forgelm/config.py`'de `output.cost_tracking` yüzeyi > yok, bu yüzden bu sayfadaki hiçbir örnek onu kullanmaz. Erteleme için bkz. > [GitHub'daki risks-and-decisions yol haritası](https://github.com/HodeTech/ForgeLM/blob/main/docs/roadmap/risks-and-decisions.md). -ForgeLM, üzerinde koştuğunuz GPU'yu tespit eder, profilini (bellek, compute, tipik saatlik tarife) bulur ve koşu başı maliyeti izler. Her koşunun ardından audit log tam olarak ne kadar GPU zamanı kullanıldığını ve maliyetini kaydeder. +Bir eğitim koşusu bittiğinde ForgeLM, ne kadar süre eğitildiğini, kaç GPU üzerinde koştuğunu ve bunun saatlik bir tarifeyle ne tuttuğunu kaydeder: tarifeyi siz verdiyseniz sizinki, vermediyseniz tespit edilen GPU için yerleşik bir fiyat kullanılır. Tahmin diğer koşu sonuçlarının yanına yazılır; fatura değildir ve hiçbir şeyi durdurmaz ya da uyarmaz. ## Tespit nasıl çalışır -Başlangıçta `forgelm` şunları okur: -- Donanım tanımlama için `nvidia-smi --query-gpu=name,memory.total,...`. -- `forgelm/trainer.py` içindeki yerleşik GPU fiyat tablosundan eşleşen saatlik ücret. +Eğitimden sonra ForgeLM, PyTorch'tan ilk CUDA aygıtının adını (`torch.cuda.get_device_name(0)`), sürecin görebildiği CUDA aygıtı sayısını ve ilk aygıtta ayrılan tepe belleği ister. Eğitim süresini Hugging Face trainer'ının `train_runtime` değerinden alır. `nvidia-smi` çağırmaz. CUDA GPU yoksa GPU alanlarının hiçbiri kaydedilmez. -Desteklenen GPU'lar: +Tarife, ayarlıysa `training.gpu_cost_per_hour`'dan; değilse `forgelm/trainer.py` içindeki `_GPU_PRICING` fiyat tablosundan gelir ve PyTorch'un bildirdiği tam aygıt adıyla eşleştirilir: -| Sınıf | Modeller | +| Sınıf | Tanınan adlar | |---|---| -| Datacenter | A100 40 GB / 80 GB, H100 80 GB, H200, L40S | -| Workstation | RTX 6000 Ada, RTX A6000 | -| Consumer | RTX 4090, RTX 4080, RTX 3090, RTX 3080 | -| Sadece-cloud | T4, V100, A10G | -| Apple | M1/M2/M3 Max (CPU/MPS fallback) | +| Tesla | Tesla T4, Tesla P100, Tesla V100, Tesla K80 | +| NVIDIA data center | NVIDIA A10G, NVIDIA A100-SXM4-40GB, NVIDIA A100-SXM4-80GB, NVIDIA A100 80GB PCIe, NVIDIA H100 80GB HBM3, NVIDIA H100 SXM5 80GB, NVIDIA H200, NVIDIA L4, NVIDIA L40S, NVIDIA B200 | +| GeForce (kendi sunucunuz) | NVIDIA GeForce RTX 3090, NVIDIA GeForce RTX 4090 | -GPU'nuz tanınmazsa ForgeLM uyarı bırakır ve generic profile fallback yapar (maliyet tahmini yok ama eğitim çalışır). +Tablodaki fiyatlar yaklaşık on-demand cloud ortalamalarıdır (GeForce kartlarında elektrik artı amortisman tahmini). Tabloda olmayan bir GPU için, örneğin `NVIDIA RTX A6000` ya da `NVIDIA H100 PCIe`, **uyarı verilmez**: `training.gpu_cost_per_hour` ayarlamadıysanız koşunun tahmini bir maliyeti olmaz. ## Saatlik tarife konfigürasyonu ```yaml training: - gpu_cost_per_hour: 1.10 # eğitim GPU'su için saatlik USD + gpu_cost_per_hour: 1.10 # GPU başına saatlik USD ``` -`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını yazar (`cost_source`: `user_config`; tam GPU adı eşleşmesi için `auto_detected`; kısmi ad eşleşmesi için `fuzzy_match`; tarife belirlenemediyse anahtar bulunmaz). CUDA GPU yoksa tespit edilen bir GPU da yoktur, bu yüzden `training.gpu_cost_per_hour`'u açıkça ayarlayın. Koşu başına tek bir tarife vardır: GPU modeli başına tarife tabloları desteklenmez. +`training.gpu_cost_per_hour`'u gerçek tarifenize ayarlayın. Ayarlı değilse ForgeLM tespit edilen GPU için yerleşik fiyat tablosuna döner; koşunun `resource_usage` kaydı hangisinin kullanıldığını `cost_source` alanında yazar: + +| `cost_source` | Anlamı | +|---|---| +| `user_config` | Tarife `training.gpu_cost_per_hour`'dan geldi (`0` kabul edilir ve sıfır maliyet kaydeder). | +| `auto_detected` | Aygıt adı bir tablo girdisiyle tam eşleşti. | +| `fuzzy_match` | Aygıt adı bir tablo girdisiyle kısmen eşleşti. Bu bir tahmindir: örneğin `NVIDIA L40`, `NVIDIA L40S` fiyatıyla hesaplanır. | + +Hiçbir tarife belirlenemediğinde `cost_source`, `gpu_cost_per_hour_usd` ve `estimated_cost_usd` bulunmaz. + +:::warn +**CUDA GPU'su olmayan makineler (CPU, Apple silicon).** Aygıt adı yoktur ve boş bir ad tablonun ilk girdisiyle kısmen eşleşir; bu yüzden koşu, `cost_source: fuzzy_match` ile yer tutucu bir `0.15` USD/saat tarifesiyle kaydedilir. Bunu gerçek bir maliyet saymayın; böyle makinelerde `training.gpu_cost_per_hour`'u açıkça ayarlayın (koşu size ücretsizse `0`). +::: + +Koşu başına tek bir tarife vardır ve bu tarife GPU başınadır: GPU modeli başına tarife tabloları desteklenmez. ## Çıktı -Her koşunun ardından audit log şunları kaydeder: +Tahmin dört yerde kaydedilir: + +- **`compliance/compliance_report.json`**, `resource_usage` altında; koşunun `training.output_dir` dizininde, koşu tamamlandığında yazılır. +- **`--output-format json` sonucu**: `resource_usage` ve üst düzey `estimated_cost_usd` olarak. +- **Konsol**: `Estimated training cost` satırı ve fiyat kaynağını belirten bir `GPU-hours` satırı (text modunda). +- **Koşu metrikleri**: her sayısal alan `resource/` olarak kopyalanır (örneğin `resource/estimated_cost_usd`); böylece deney izleyicinize ve `pipeline.completed` audit olayının `metrics_summary` alanına ulaşır. + +`gpu_cost_per_hour: 1.10` ile 1892 saniye eğitilen 4 GPU'lu bir koşu şunu kaydeder: ```json { - "event": "run_complete", - "ts": "2026-04-29T14:33:10Z", - "duration_seconds": 1892, - "gpu_profile": "A100_80GB", - "gpus_used": 1, - "estimated_cost_usd": 0.578, - "rate_per_hour": 1.10, - "currency": "USD" + "resource_usage": { + "gpu_model": "NVIDIA A100-SXM4-80GB", + "peak_vram_gb": 41.7, + "gpu_count": 4, + "training_duration_seconds": 1892.0, + "gpu_hours": 2.102, + "cost_source": "user_config", + "gpu_cost_per_hour_usd": 1.1, + "estimated_cost_usd": 2.3124 + } } ``` -Model card atıf yapar: - -```markdown -## Eğitim maliyeti - -Bu model 1× A100 80GB üzerinde 31m 32s eğitildi, -konfigüre tarifeyle tahmini $0.58 USD. -``` +Model card maliyeti içermez. ## Uçuş öncesi maliyet tahmini -Uzun bir eğitim koşusuna başlamadan önce kısa bir kalibrasyon (1-2 adım, `training.max_steps: 2` ile) koşturun, koşum-başına `compliance_report.json`'dan ortaya çıkan `gpu_hours`'u yakalayın ve sağlayıcınızın saatlik tarifesiyle çarpın. (Özel bir `--estimate-cost` flag'i tartışıldı ama ship edilmedi; resource-tracking yolu yalnızca gerçek değerleri yayar.) +Uzun bir eğitim koşusuna başlamadan önce sert bir adım sınırıyla kısa bir kalibrasyon koşturun, `resource_usage` içindeki `gpu_hours`'u okuyun, tam adım sayısına ölçekleyin ve saatlik tarifenizle çarpın. Özel bir `--estimate-cost` flag'i tartışıldı ama ship edilmedi; resource-tracking yolu yalnızca gerçek değerleri kaydeder. + +```yaml +# configs/calibration.yaml — gerçek config'iniz ve ek olarak: +training: + max_steps: 20 + output_dir: /tmp/calib +``` ```shell -$ forgelm --config configs/calibration.yaml --output-dir /tmp/calib -$ jq '.resource_usage.gpu_hours' /tmp/calib/compliance_report.json +$ forgelm --config configs/calibration.yaml +$ jq '.resource_usage.gpu_hours' /tmp/calib/compliance/compliance_report.json 0.034 -$ python -c "print(0.034 * (3 / 0.034) * 1.10)" # 6h eğitim, $1.10/sa -$7.15 +$ python -c "print(0.034 * (3000 / 20) * 1.10)" # 3.000 adım, GPU-saat başına $1.10 +5.61 ``` -Kalibrasyon yaklaşımı genelde gerçeğin %20 içindedir. +Sonucu kaba bir rakam olarak değerlendirin. İlk adımlar ısınma içerir ve birkaç adımdaki dizi uzunluğu karışımı nadiren tüm veri setininkiyle örtüşür; ölçeklemeden önce kararlı bir adım süresi görecek kadar adım koşturun. ## Çoklu-GPU ve dağıtık -Çoklu-GPU eğitiminde ForgeLM tek `training.gpu_cost_per_hour` tarifesini GPU sayısıyla çarpar (`gpu_hours`, eğitim süresi çarpı GPU sayısıdır). +`gpu_hours`, eğitim süresinin sürecin görebildiği CUDA aygıtı sayısıyla çarpımıdır ve tek `training.gpu_cost_per_hour` tarifesi GPU başına uygulanır. 4×A100 koşusu 2 saat = 4 × 2 × $1.10 = $8.80; ZeRO veya FSDP kullanmak fark etmez. +İş, makinedeki GPU'ların bir kısmını kullanıyorsa sürecin görebileceği aygıtları sınırlayın (örneğin `CUDA_VISIBLE_DEVICES` ile); aksi halde tahmin, koşunun hiç kullanmadığı GPU'ları da sayar. + ## Maliyet alarmları (yol haritası — henüz uygulanmadı) Kontrolden çıkabilecek koşular için planlanan `cost_tracking` bloğu eşik @@ -95,36 +116,38 @@ tabanlı alarmı (alarm eşiği aşılınca webhook tetiklenir) ve durdurmayı (durdurma eşiğinde eğitim durur) destekleyecek. Böyle bir ayar bugün `forgelm/config.py`'de yoktur. -İmplemente edildiğinde, alarm konfigüre webhook'u fırlatır (bkz. [Webhook'lar](#/operations/webhooks)) — yanlış konfigüre koşunun gece bütçeyi haftalık bitirmesini CI'da yakalamak için faydalı. O zamana kadar maliyeti audit log + zamanlayıcınızdaki bütçe-tarafı koruyucu ile manuel izleyin. +İmplemente edildiğinde, alarm konfigüre webhook'u fırlatır (bkz. [Webhook'lar](#/operations/webhooks)) — yanlış konfigüre koşunun gece bütçeyi haftalık bitirmesini CI'da yakalamak için faydalı. O zamana kadar maliyeti kaydedilen `resource_usage` ve zamanlayıcınızdaki bütçe-tarafı koruyucu ile manuel izleyin. ## Özel GPU profilleri -Varsayılan profilde olmayan GPU eklemek için: +:::warn +**Uygulanmadı.** ForgeLM'de `gpu_profiles` ayarı yoktur ve `gpu_profiles.local.yaml` okunmaz; aşağıdaki snippet yalnızca planlanan biçimdir ve onu içeren bir config reddedilir. Bugün tabloda olmayan bir GPU'yu fiyatlandırmanın yolu, her GPU için çalışan `training.gpu_cost_per_hour`'dur. +::: + +Yerleşik tabloda olmayan bir GPU eklemek için (planlanan): ```yaml gpu_profiles: custom: - name: "AcmeNVIDIA-XYZ" - pattern: "AcmeNVIDIA-XYZ" # nvidia-smi name'ye karşı eşleştir + pattern: "AcmeNVIDIA-XYZ" # aygıt adına karşı eşleştir memory_gb: 96 compute_capability: 9.0 tensor_cores: true ``` -Bunu proje kökünüze `gpu_profiles.local.yaml` olarak bırakın; ForgeLM otomatik birleştirir. - ## Sık hatalar :::warn -**Saatlik tarifeyi mutlak doğru saymak.** ForgeLM'in varsayılanları makul ortalamalar — gerçek cloud faturanız instance tipi, bölge, spot vs on-demand ve indirimlerinize bağlı. Kendi gerçek tarifenizle override edin. +**Saatlik tarifeyi mutlak doğru saymak.** ForgeLM'in varsayılanları kaba ortalamalar — gerçek cloud faturanız instance tipi, bölge, spot vs on-demand ve indirimlerinize bağlı. Kendi gerçek tarifenizle override edin. ::: :::warn -**Paylaşımlı GPU'larda multi-tenancy.** Birden çok iş bir GPU'yu paylaşıyorsa (eğitimde nadir, inference'ta yaygın), maliyet izleme bölmek yerine çarpıyor. Gerçek tahsisi belirtmek için `--gpus N` kullanın. +**Paylaşımlı GPU'lar.** Tahmin, görünen her GPU'daki eğitim süresinin tamamını faturalar; kesirli ya da paylaşımlı tahsis için bir ayar yoktur. GPU'yu başka işlerle paylaşıyorsanız rakamı kendiniz bölün. ::: :::tip -**Maliyeti zaman içinde izleyin.** Audit log her koşunun maliyetini içerir; haftalar boyunca trend alın, sürünen aşımları yakalayın. Aynı eğitim süresinde 2× maliyet artışı genelde verinin büyüdüğü veya hyperparam'ların kaydığı işaretidir. +**Maliyeti zaman içinde izleyin.** Her koşu maliyetini kaydeder; haftalar boyunca trend alın, sürünen aşımları yakalayın. Aynı eğitim süresinde 2× maliyet artışı genelde verinin büyüdüğü veya hyperparam'ların kaydığı işaretidir. ::: ## Bkz. diff --git a/site/compliance.html b/site/compliance.html index 333d87ed..aeb186fa 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -202,7 +202,7 @@

PII redaction at ingest

GDPR Art. 15

Right of access

-

forgelm reverse-pii scans JSONL corpora for plaintext occurrences, or for salted SHA-256 digests written by an external masking pipeline, of a data-subject identifier and reports every line, in the files you scan, that contains it. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.

+

forgelm reverse-pii searches JSONL corpora for a data-subject identifier, as plain text or as a salted SHA-256 digest written by an external masking pipeline, and reports every line in the scanned files that contains it. The match is literal and case-sensitive. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.

forgelm reverse-pii
diff --git a/site/features.html b/site/features.html index c1e690aa..cc574d69 100644 --- a/site/features.html +++ b/site/features.html @@ -420,7 +420,7 @@

Row erasure (Article 17 step)

Identifier search (Article 15 input)

-

forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.

+

forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it: an input to an Article 15 response.

diff --git a/site/index.html b/site/index.html index 157705b2..02e92654 100644 --- a/site/index.html +++ b/site/index.html @@ -536,7 +536,7 @@

ISO 27001 / SOC 2 alignment

GDPR rights tooling

-

Tooling for GDPR data-subject requests: forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

+

forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge deletes rows by id from training corpora (one step of an Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.

forgelm purge forgelm reverse-pii @@ -637,7 +637,7 @@

Use cases that actually ship

Domain expert from policy PDFs

-

Drop your regulatory corpus into forgelm ingest, audit it, and SFT a model that answers in your domain's voice — with common PII formats masked at ingest, before training.

+

Drop your regulatory corpus into forgelm ingest, audit it, and SFT a model that answers in your domain's voice — with common PII formats masked at ingest (opt in with --pii-mask), before training.

forgelm quickstart domain-expert
diff --git a/site/js/translations.js b/site/js/translations.js index 68ad29f4..ba52dbb3 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -390,7 +390,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 alignment", "home.cap.iso.body": "Audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor asks for. Software is aligned, not certified.", "home.cap.gdpr.title": "GDPR rights tooling", - "home.cap.gdpr.body": "Tooling for GDPR data-subject requests: forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge removes rows from training corpora (one step of Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", + "home.cap.gdpr.body": "forgelm reverse-pii finds the lines that contain an identifier you supply (an input to an Article 15 response); forgelm purge deletes rows by id from training corpora (one step of an Article 17 erasure; trained weights are not changed). Both are recorded in the audit log.", "home.cap.see_all": "See the full feature matrix", "home.yaml.eyebrow": "A real config — not a screenshot", "home.yaml.title": "One YAML, end-to-end run", @@ -398,7 +398,7 @@ "home.uses.eyebrow": "Built for these workflows", "home.uses.title": "Use cases that actually ship", "home.use.domain.title": "Domain expert from policy PDFs", - "home.use.domain.body": "Drop your regulatory corpus into forgelm ingest, audit it, and SFT a model that answers in your domain's voice — with common PII formats masked at ingest, before training.", + "home.use.domain.body": "Drop your regulatory corpus into forgelm ingest, audit it, and SFT a model that answers in your domain's voice — with common PII formats masked at ingest (opt in with --pii-mask), before training.", "home.use.support.title": "Customer support assistant", "home.use.support.body": "SFT on past tickets (and DPO on chosen/rejected agent feedback when you have it), Llama Guard safety evaluation gate, auto-revert if helpfulness drops. Ship to vLLM with one command.", "home.use.code.title": "Code copilot fine-tune", @@ -496,7 +496,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 hizalaması", "home.cap.iso.body": "Deployer denetçisinin sorduğu audit izi, change-management, veri kökenleri ve tedarik zinciri kanıtları. Yazılım sertifikalı değil, hizalanmıştır.", "home.cap.gdpr.title": "GDPR hakları araçları", - "home.cap.gdpr.body": "GDPR veri sahibi talepleri için araçlar: forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 yanıtının bir girdisi); forgelm purge satırları eğitim corpus'larından siler (Madde 17 kapsamındaki silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", + "home.cap.gdpr.body": "forgelm reverse-pii verdiğiniz bir tanımlayıcıyı içeren satırları bulur (Madde 15 yanıtının bir girdisi); forgelm purge eğitim corpus'larından satırları id ile siler (Madde 17 kapsamındaki silmenin bir adımı; eğitilmiş ağırlıklar değişmez). İkisi de audit log'a kaydedilir.", "home.cap.see_all": "Tüm özellik matrisini gör", "home.yaml.eyebrow": "Gerçek bir konfigürasyon — ekran görüntüsü değil", "home.yaml.title": "Bir YAML, uçtan uca koşu", @@ -504,7 +504,7 @@ "home.uses.eyebrow": "Bu iş akışları için tasarlandı", "home.uses.title": "Gerçekten üretime çıkan kullanım senaryoları", "home.use.domain.title": "Politika PDF'lerinden alan uzmanı", - "home.use.domain.body": "Regülasyon korpusunuzu forgelm ingest'e bırakın, denetleyin ve alanınızın diliyle cevap veren bir modeli SFT'leyin — yaygın PII biçimleri ingest sırasında, eğitimden önce maskelenir.", + "home.use.domain.body": "Regülasyon korpusunuzu forgelm ingest'e bırakın, denetleyin ve alanınızın diliyle cevap veren bir modeli SFT'leyin — yaygın PII biçimleri ingest sırasında, eğitimden önce maskelenir (--pii-mask ile açılır).", "home.use.support.title": "Müşteri destek asistanı", "home.use.support.body": "Geçmiş ticket'larda SFT (eğer chosen/rejected agent geri bildirimi varsa DPO eklenebilir), Llama Guard güvenlik değerlendirme kapısı, yardımseverlik düşerse otomatik artefakt silme. Tek komutla vLLM'e gönderim.", "home.use.code.title": "Kod copilot fine-tune", @@ -602,7 +602,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 Alignment", "home.cap.iso.body": "Audit-Trail, Change-Management, Daten-Lineage und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Software ist aligned, nicht zertifiziert.", "home.cap.gdpr.title": "DSGVO-Rechtetooling", - "home.cap.gdpr.body": "Werkzeuge für DSGVO-Betroffenenanfragen: forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Artikel 15); forgelm purge entfernt Zeilen aus Trainingskorpora (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Beides wird im Audit-Log festgehalten.", + "home.cap.gdpr.body": "forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Art. 15 DSGVO); forgelm purge löscht Zeilen anhand ihrer ID aus Trainingskorpora (ein Schritt der Löschung nach Art. 17 DSGVO; trainierte Gewichte bleiben unverändert). Beide Vorgänge werden im Audit-Log festgehalten.", "home.cap.see_all": "Vollständige Funktionsmatrix anzeigen", "home.yaml.eyebrow": "Eine echte Konfiguration — kein Screenshot", "home.yaml.title": "Ein YAML, ein Ende-zu-Ende-Lauf", @@ -610,7 +610,7 @@ "home.uses.eyebrow": "Gebaut für diese Workflows", "home.uses.title": "Use-Cases, die wirklich in Produktion gehen", "home.use.domain.title": "Domänen-Experte aus Policy-PDFs", - "home.use.domain.body": "Werfen Sie Ihren regulatorischen Korpus in forgelm ingest, auditieren Sie ihn und SFTen Sie ein Modell, das in der Sprache Ihrer Domäne antwortet — mit beim Ingest, vor dem Training, maskierten gängigen PII-Formaten.", + "home.use.domain.body": "Werfen Sie Ihren regulatorischen Korpus in forgelm ingest, auditieren Sie ihn und SFTen Sie ein Modell, das in der Sprache Ihrer Domäne antwortet — mit beim Ingest, vor dem Training, maskierten gängigen PII-Formaten (per --pii-mask zu aktivieren).", "home.use.support.title": "Kundensupport-Assistent", "home.use.support.body": "SFT auf vergangenen Tickets (und DPO auf gewählten/abgelehnten Agenten-Feedback, falls verfügbar), Llama-Guard-Sicherheitsbewertungs-Gate, automatisches Verwerfen der Artefakte, wenn die Hilfsbereitschaft sinkt. Mit einem Befehl an vLLM ausliefern.", "home.use.code.title": "Code-Copilot Fine-Tune", @@ -708,7 +708,7 @@ "home.cap.iso.title": "Alignement ISO 27001 / SOC 2", "home.cap.iso.body": "Piste d'audit, gestion du changement, lignée des données et preuves de chaîne logistique demandées par l'auditeur du déployeur. Le logiciel est aligné, pas certifié.", "home.cap.gdpr.title": "Outils de droits RGPD", - "home.cap.gdpr.body": "Outils pour les demandes des personnes concernées au titre du RGPD : forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, article 15) ; forgelm purge supprime des lignes des corpus d'entraînement (une étape de l'effacement, article 17 ; les poids entraînés ne changent pas). Les deux sont consignés dans le journal d'audit.", + "home.cap.gdpr.body": "forgelm reverse-pii trouve les lignes qui contiennent un identifiant que vous fournissez (un élément de la réponse à une demande d'accès, article 15) ; forgelm purge supprime des lignes des corpus d'entraînement d'après leur identifiant (une étape de l'effacement, article 17 ; les poids entraînés ne changent pas). Les deux opérations sont consignées dans le journal d'audit.", "home.cap.see_all": "Voir la matrice complète des fonctionnalités", "home.yaml.eyebrow": "Une vraie configuration — pas une capture d'écran", "home.yaml.title": "Un YAML, exécution de bout en bout", @@ -716,7 +716,7 @@ "home.uses.eyebrow": "Conçu pour ces workflows", "home.uses.title": "Cas d'usage qui livrent vraiment", "home.use.domain.title": "Expert métier à partir de PDF de politiques", - "home.use.domain.body": "Déposez votre corpus réglementaire dans forgelm ingest, auditez-le, et fine-tunez en SFT un modèle qui répond avec la voix de votre domaine — avec les formats PII courants masqués à l'ingestion, avant l'entraînement.", + "home.use.domain.body": "Déposez votre corpus réglementaire dans forgelm ingest, auditez-le, et fine-tunez en SFT un modèle qui répond avec la voix de votre domaine — avec les formats PII courants masqués à l'ingestion, avant l'entraînement (à activer avec --pii-mask).", "home.use.support.title": "Assistant de support client", "home.use.support.body": "SFT sur tickets passés (et DPO sur feedbacks choisis/rejetés des agents si disponibles), gate d'évaluation sécurité Llama Guard, suppression automatique des artefacts si la serviabilité baisse. Livraison à vLLM en une commande.", "home.use.code.title": "Fine-tune copilote de code", @@ -814,7 +814,7 @@ "home.cap.iso.title": "Alineación con ISO 27001 / SOC 2", "home.cap.iso.body": "Pista de auditoría, gestión del cambio, linaje de datos y evidencias de cadena de suministro que pide el auditor del implementador. El software está alineado, no certificado.", "home.cap.gdpr.title": "Herramientas de derechos RGPD", - "home.cap.gdpr.body": "Herramientas para solicitudes de interesados del RGPD: forgelm reverse-pii encuentra las filas que contienen un identificador que usted proporciona (un elemento de la respuesta a la solicitud de acceso del Artículo 15); forgelm purge elimina filas de los corpus de entrenamiento (un paso de la supresión del Artículo 17; los pesos entrenados no cambian). Ambos quedan registrados en el log de auditoría.", + "home.cap.gdpr.body": "forgelm reverse-pii encuentra las líneas que contienen un identificador que usted proporciona (un elemento de la respuesta a una solicitud de acceso, artículo 15); forgelm purge elimina filas por id de los corpus de entrenamiento (un paso de la supresión, artículo 17; los pesos entrenados no cambian). Ambas operaciones quedan registradas en el log de auditoría.", "home.cap.see_all": "Ver la matriz completa de características", "home.yaml.eyebrow": "Una configuración real — no una captura", "home.yaml.title": "Un YAML, ejecución de extremo a extremo", @@ -822,7 +822,7 @@ "home.uses.eyebrow": "Construido para estos flujos", "home.uses.title": "Casos de uso que realmente se despliegan", "home.use.domain.title": "Experto de dominio a partir de PDFs regulatorios", - "home.use.domain.body": "Suelte su corpus regulatorio en forgelm ingest, audítelo y haga SFT a un modelo que responda con la voz de su dominio — con los formatos de PII comunes enmascarados en la ingestión, antes del entrenamiento.", + "home.use.domain.body": "Suelte su corpus regulatorio en forgelm ingest, audítelo y haga SFT a un modelo que responda con la voz de su dominio — con los formatos de PII comunes enmascarados en la ingestión, antes del entrenamiento (se activa con --pii-mask).", "home.use.support.title": "Asistente de soporte al cliente", "home.use.support.body": "SFT sobre tickets pasados (y DPO sobre feedback elegido/rechazado de agentes si está disponible), puerta de evaluación de seguridad Llama Guard, descarte automático de artefactos si la utilidad baja. Despliegue a vLLM con un solo comando.", "home.use.code.title": "Fine-tune de copiloto de código", @@ -920,7 +920,7 @@ "home.cap.iso.title": "ISO 27001 / SOC 2 对齐", "home.cap.iso.body": "部署方审计员要求的审计轨迹、变更管理、数据沿袭和供应链证据。软件是对齐的,不是认证的。", "home.cap.gdpr.title": "GDPR 权利工具", - "home.cap.gdpr.body": "针对 GDPR 数据主体请求的工具:forgelm reverse-pii 查找包含你所提供标识符的行(对第 15 条访问请求答复的输入之一),forgelm purge 从训练语料中删除行(第 17 条所涉删除的其中一步,已训练的权重不会改变)。两者都会记录到审计日志。", + "home.cap.gdpr.body": "forgelm reverse-pii 查找包含你所提供标识符的行(可作为答复第 15 条访问请求的材料之一);forgelm purge 按 id 从训练语料中删除行(只是第 17 条删除流程中的一步,已训练的权重不会改变)。两者都会记录到审计日志。", "home.cap.see_all": "查看完整功能矩阵", "home.yaml.eyebrow": "真实配置 — 不是截图", "home.yaml.title": "一个 YAML,端到端运行", @@ -928,7 +928,7 @@ "home.uses.eyebrow": "为这些工作流而构建", "home.uses.title": "真正能上线的用例", "home.use.domain.title": "从政策 PDF 中得到的领域专家", - "home.use.domain.body": "把你的法规语料投入 forgelm ingest、审计它、然后 SFT 一个能用你领域口吻回答的模型 — 常见 PII 格式在 ingest 时,训练之前,就被屏蔽。", + "home.use.domain.body": "把你的法规语料投入 forgelm ingest、审计它、然后 SFT 一个能用你领域口吻回答的模型 — 常见 PII 格式在 ingest 时,训练之前,就被屏蔽(需通过 --pii-mask 开启)。", "home.use.support.title": "客户支持助手", "home.use.support.body": "在过往工单上 SFT (有 chosen/rejected 工单反馈时可叠加 DPO),Llama Guard 安全评估 gate,有用度下降时自动丢弃训练产物。一条命令交付到 vLLM。", "home.use.code.title": "代码 copilot 微调", @@ -983,7 +983,7 @@ "features.data.multi.title": "Multi-format ingest", "features.data.multi.body": "PDF, DOCX, EPUB, TXT, Markdown → SFT-ready JSONL. Markdown-aware splitter; DOCX tables emit as Markdown tables; PDF header/footer dedup.", "features.data.pii.title": "PII masking", - "features.data.pii.body": "Email, phone, credit card (Luhn-validated), IBAN, national IDs (TR / DE / FR / US-SSN). Masks at ingest time, before chunks land in JSONL.", + "features.data.pii.body": "Email, phone, credit card (Luhn-validated), IBAN, national IDs (TR / DE / FR / US-SSN). Opt in with --pii-mask: masks at ingest time, before chunks land in JSONL.", "features.data.secrets.title":"Secrets scrubbing", "features.data.secrets.body": "AWS keys, GitHub PATs, Slack / OpenAI / Google tokens, JWTs, full PEM private-key blocks (BEGIN…END), Azure storage strings — redacted with [REDACTED-SECRET].", "features.data.dup.title": "Near-duplicate detection", @@ -1035,7 +1035,7 @@ "features.gov.purge.title": "Row erasure (Article 17 step)", "features.gov.purge.body": "forgelm purge deletes rows by id from a training corpus and records the erasure in the append-only audit log; it is one step of an Article 17 erasure (trained weights are not changed).", "features.gov.access.title": "Identifier search (Article 15 input)", - "features.gov.access.body": "forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it.", + "features.gov.access.body": "forgelm reverse-pii scans JSONL corpora for plaintext or salted-digest matches of a data-subject identifier and lists the lines in the scanned files that contain it: an input to an Article 15 response.", "features.gov.annex.title": "Annex IV verifier", "features.gov.annex.body": "forgelm verify-annex-iv validates the Annex IV §1-9 schema completeness and the bundle's manifest_hash for tampering detection. (Audit-chain integrity is the separate forgelm verify-audit command.)", "features.gov.gguf.title": "GGUF integrity verifier", @@ -1088,7 +1088,7 @@ "features.data.multi.title": "Çoklu format ingest'i", "features.data.multi.body": "PDF, DOCX, EPUB, TXT, Markdown → SFT-hazır JSONL. Markdown-bilen splitter; DOCX tabloları Markdown tablo olarak; PDF header/footer dedup.", "features.data.pii.title": "PII maskeleme", - "features.data.pii.body": "E-posta, telefon, kredi kartı (Luhn doğrulamalı), IBAN, ulusal kimlik (TR / DE / FR / US-SSN). Chunk'lar JSONL'a düşmeden ingest sırasında maskelenir.", + "features.data.pii.body": "E-posta, telefon, kredi kartı (Luhn doğrulamalı), IBAN, ulusal kimlik (TR / DE / FR / US-SSN). Chunk'lar JSONL'a düşmeden ingest sırasında maskelenir (--pii-mask ile açılır).", "features.data.secrets.title":"Sırların temizlenmesi", "features.data.secrets.body": "AWS anahtarları, GitHub PAT'leri, Slack / OpenAI / Google token'ları, JWT'ler, tam PEM özel anahtar blokları (BEGIN…END), Azure storage string'leri — [REDACTED-SECRET] ile maskelenir.", "features.data.dup.title": "Near-duplicate tespiti", @@ -1140,7 +1140,7 @@ "features.gov.purge.title": "Satır silme (Madde 17 adımı)", "features.gov.purge.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve silmeyi append-only audit log'a kaydeder; Madde 17 kapsamındaki silmenin bir adımıdır (eğitilmiş ağırlıklar değişmez).", "features.gov.access.title": "Tanımlayıcı arama (Madde 15 girdisi)", - "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da tuzlanmış digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler.", + "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da tuzlanmış digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler: Madde 15 yanıtının bir girdisi.", "features.gov.annex.title": "Annex IV doğrulayıcı", "features.gov.annex.body": "forgelm verify-annex-iv Annex IV §1-9 şema tamlığını ve bundle'ın manifest_hash'ini tampering tespiti için doğrular. (Audit-chain bütünlüğü ayrı bir forgelm verify-audit komutudur.)", "features.gov.gguf.title": "GGUF bütünlük doğrulayıcı", @@ -1193,7 +1193,7 @@ "features.data.multi.title": "Multi-Format-Ingest", "features.data.multi.body": "PDF, DOCX, EPUB, TXT, Markdown → SFT-fertiges JSONL. Markdown-bewusster Splitter; DOCX-Tabellen werden als Markdown-Tabellen ausgegeben; PDF-Header/Footer-Dedup.", "features.data.pii.title": "PII-Maskierung", - "features.data.pii.body": "E-Mail, Telefon, Kreditkarte (Luhn-validiert), IBAN, nationale IDs (TR / DE / FR / US-SSN). Maskiert beim Ingest, bevor Chunks im JSONL landen.", + "features.data.pii.body": "E-Mail, Telefon, Kreditkarte (Luhn-validiert), IBAN, nationale IDs (TR / DE / FR / US-SSN). Maskiert beim Ingest, bevor Chunks im JSONL landen (mit --pii-mask zu aktivieren).", "features.data.secrets.title":"Secrets-Bereinigung", "features.data.secrets.body": "AWS-Schlüssel, GitHub-PATs, Slack-/OpenAI-/Google-Tokens, JWTs, vollständige PEM-Privatschlüssel-Blöcke (BEGIN…END), Azure-Storage-Strings — geschwärzt mit [REDACTED-SECRET].", "features.data.dup.title": "Near-Duplicate-Erkennung", @@ -1272,7 +1272,7 @@ "features.data.multi.title": "Ingestion multi-format", "features.data.multi.body": "PDF, DOCX, EPUB, TXT, Markdown → JSONL prêt pour SFT. Splitter conscient du Markdown ; les tables DOCX deviennent des tables Markdown ; dédoublonnage des en-têtes/pieds PDF.", "features.data.pii.title": "Masquage des PII", - "features.data.pii.body": "E-mail, téléphone, carte de crédit (validée Luhn), IBAN, identifiants nationaux (TR / DE / FR / US-SSN). Masqués au moment de l'ingestion, avant que les chunks atterrissent dans le JSONL.", + "features.data.pii.body": "E-mail, téléphone, carte de crédit (validée Luhn), IBAN, identifiants nationaux (TR / DE / FR / US-SSN). Masqués au moment de l'ingestion, avant que les chunks atterrissent dans le JSONL (à activer avec --pii-mask).", "features.data.secrets.title":"Nettoyage des secrets", "features.data.secrets.body": "Clés AWS, PATs GitHub, jetons Slack / OpenAI / Google, JWTs, blocs PEM de clé privée complets (BEGIN…END), chaînes Azure storage — masqués par [REDACTED-SECRET].", "features.data.dup.title": "Détection de quasi-doublons", @@ -1351,7 +1351,7 @@ "features.data.multi.title": "Ingestión multi-formato", "features.data.multi.body": "PDF, DOCX, EPUB, TXT, Markdown → JSONL listo para SFT. Splitter consciente de Markdown; las tablas DOCX salen como tablas Markdown; dedup de cabecera/pie en PDF.", "features.data.pii.title": "Enmascaramiento PII", - "features.data.pii.body": "Email, teléfono, tarjeta de crédito (validada con Luhn), IBAN, IDs nacionales (TR / DE / FR / US-SSN). Enmascarados en el momento de la ingestión, antes de que los chunks lleguen al JSONL.", + "features.data.pii.body": "Email, teléfono, tarjeta de crédito (validada con Luhn), IBAN, IDs nacionales (TR / DE / FR / US-SSN). Enmascarados en el momento de la ingestión, antes de que los chunks lleguen al JSONL (se activa con --pii-mask).", "features.data.secrets.title":"Limpieza de secretos", "features.data.secrets.body": "Claves AWS, PATs de GitHub, tokens de Slack / OpenAI / Google, JWTs, bloques PEM de clave privada completos (BEGIN…END), strings de Azure storage — redactados con [REDACTED-SECRET].", "features.data.dup.title": "Detección de cuasi-duplicados", @@ -1430,7 +1430,7 @@ "features.data.multi.title": "多格式 ingest", "features.data.multi.body": "PDF、DOCX、EPUB、TXT、Markdown → SFT 就绪的 JSONL。Markdown 感知 splitter;DOCX 表格输出为 Markdown 表格;PDF 页眉/页脚去重。", "features.data.pii.title": "PII 屏蔽", - "features.data.pii.body": "邮箱、电话、信用卡 (Luhn 校验)、IBAN、国家身份证 (TR / DE / FR / US-SSN)。在 ingest 时,chunk 进入 JSONL 之前完成屏蔽。", + "features.data.pii.body": "邮箱、电话、信用卡 (Luhn 校验)、IBAN、国家身份证 (TR / DE / FR / US-SSN)。在 ingest 时,chunk 进入 JSONL 之前完成屏蔽(需通过 --pii-mask 开启)。", "features.data.secrets.title":"Secrets 清洗", "features.data.secrets.body": "AWS 密钥、GitHub PATs、Slack / OpenAI / Google token、JWT、完整 PEM 私钥块 (BEGIN…END)、Azure storage 字符串 — 用 [REDACTED-SECRET] 替换。", "features.data.dup.title": "近重复检测", @@ -1811,7 +1811,7 @@ "wizard.synthetic.api_key_env.label": "Teacher API key env var", "wizard.synthetic.api_key_env.hint": "Environment variable name carrying the teacher API key. Empty = local teacher model.", "wizard.retention.title": "Retention horizons (GDPR Art. 5+17)", - "wizard.retention.hint": "Ages at which forgelm purge --check-policy reports artefacts as overdue; nothing is deleted.", + "wizard.retention.hint": "Ages at which forgelm purge --check-policy reports artefacts as overdue; nothing is deleted. Set a value to 0 to stop reporting that artefact. The retention block is written to the YAML only when a value differs from its default; without it, --check-policy checks nothing.", "wizard.retention.audit_log.label": "Audit log retention (days)", "wizard.retention.audit_log.hint": "How long before forgelm purge --check-policy reports audit_log.jsonl as overdue. Default 1825 (5 years) is a ForgeLM choice, not a legal period.", "wizard.retention.staging.label": "Staging horizon (days)", @@ -2209,7 +2209,7 @@ "wizard.synthetic.api_key_env.label": "Teacher API key env değişkeni", "wizard.synthetic.api_key_env.hint": "Teacher API anahtarını taşıyan ortam değişkeni adı. Boş = yerel teacher modeli.", "wizard.retention.title": "Saklama ufukları (GDPR Madde 5+17)", - "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları süresi geçmiş olarak raporladığı yaşlar; hiçbir şey silinmez.", + "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları süresi geçmiş olarak raporladığı yaşlar; hiçbir şey silinmez. Bir değeri 0 yapmak o artefact'ın raporlanmasını durdurur. retention bloğu YAML'a yalnızca bir değer varsayılandan farklıysa yazılır; blok yoksa --check-policy hiçbir şeyi kontrol etmez.", "wizard.retention.audit_log.label": "Audit log saklama (gün)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra süresi geçmiş olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", "wizard.retention.staging.label": "Staging ufku (gün)", @@ -2603,11 +2603,11 @@ "wizard.synthetic.api_key_env.label": "Teacher API-Key Env-Variable", "wizard.synthetic.api_key_env.hint": "Env-Variable mit dem Teacher-API-Key. Leer = lokales Teacher-Modell.", "wizard.retention.title": "Aufbewahrungsfristen (DSGVO Art. 5+17)", - "wizard.retention.hint": "Alter, ab dem forgelm purge --check-policy Artefakte als überfällig meldet; nichts wird gelöscht.", + "wizard.retention.hint": "Alter, ab dem forgelm purge --check-policy Artefakte als überfällig meldet; nichts wird gelöscht. Ein Wert von 0 beendet die Meldung für dieses Artefakt. Der retention-Block wird nur dann in die YAML geschrieben, wenn ein Wert vom Default abweicht; ohne ihn prüft --check-policy nichts.", "wizard.retention.audit_log.label": "Audit-Log Aufbewahrung (Tage)", - "wizard.retention.audit_log.hint": "Nach wie vielen Tagen forgelm purge --check-policy audit_log.jsonl als überfällig meldet. Default 1825 (5 Jahre) ist eine ForgeLM-Wahl, keine gesetzliche Frist.", + "wizard.retention.audit_log.hint": "Nach wie vielen Tagen forgelm purge --check-policy audit_log.jsonl als überfällig meldet. Der Default 1825 (5 Jahre) ist eine Vorgabe von ForgeLM, keine gesetzliche Frist.", "wizard.retention.staging.label": "Staging-Horizont (Tage)", - "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging.*/-Verzeichnis (oder das ältere final_model.staging/) als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", + "wizard.retention.staging.hint": "Ab welchem Alter forgelm purge --check-policy ein final_model.staging.*/-Verzeichnis (oder das frühere final_model.staging/ im Legacy-Layout) als überfällig meldet; nichts wird automatisch gelöscht. Default 7.", "wizard.retention.ephemeral.label": "Ephemere Artefakte (Tage)", "wizard.retention.ephemeral.hint": "Nach wie vielen Tagen seit dem ersten Ereignis im Audit-Log (ohne Audit-Log: Dateialter) forgelm purge --check-policy compliance/ und data_audit_report.json als überfällig meldet. Default 90.", "wizard.retention.raw_documents.label": "Rohdokumente (Tage)", @@ -2997,15 +2997,15 @@ "wizard.synthetic.api_key_env.label": "Variable env clé API teacher", "wizard.synthetic.api_key_env.hint": "Nom variable env portant clé API teacher. Vide = modèle teacher local.", "wizard.retention.title": "Horizons de rétention (RGPD Art. 5+17)", - "wizard.retention.hint": "Seuils d'âge à partir desquels forgelm purge --check-policy signale des artefacts comme dépassés ; rien n'est supprimé.", + "wizard.retention.hint": "Seuils d'âge à partir desquels forgelm purge --check-policy signale des artefacts comme ayant dépassé leur durée de conservation ; rien n'est supprimé. Une valeur de 0 arrête le signalement de cet artefact. Le bloc retention n'est écrit dans le YAML que si une valeur diffère de sa valeur par défaut ; sans lui, --check-policy ne vérifie rien.", "wizard.retention.audit_log.label": "Rétention audit log (jours)", - "wizard.retention.audit_log.hint": "Après combien de jours forgelm purge --check-policy signale audit_log.jsonl comme dépassé. Default 1825 (5 ans) est un choix de ForgeLM, pas un délai légal.", + "wizard.retention.audit_log.hint": "Nombre de jours au-delà duquel forgelm purge --check-policy signale audit_log.jsonl comme ayant dépassé sa durée de conservation. Par défaut : 1825 (5 ans), un choix de ForgeLM et non un délai légal.", "wizard.retention.staging.label": "Horizon staging (jours)", - "wizard.retention.staging.hint": "À partir de quel âge forgelm purge --check-policy signale un répertoire final_model.staging.*/ (ou l'ancien final_model.staging/) comme dépassé ; rien n'est supprimé automatiquement. Default 7.", + "wizard.retention.staging.hint": "Âge à partir duquel forgelm purge --check-policy signale un répertoire final_model.staging.*/ (ou l'ancien format final_model.staging/) comme ayant dépassé sa durée de conservation ; rien n'est supprimé automatiquement. Par défaut : 7.", "wizard.retention.ephemeral.label": "Artefacts éphémères (jours)", - "wizard.retention.ephemeral.hint": "Après combien de jours depuis le premier événement du log d'audit (à défaut de log : l'âge du fichier) forgelm purge --check-policy signale compliance/ et data_audit_report.json comme dépassés. Default 90.", + "wizard.retention.ephemeral.hint": "Nombre de jours, comptés depuis le premier événement du journal d'audit (à défaut de journal : l'âge du fichier), au-delà duquel forgelm purge --check-policy signale compliance/ et data_audit_report.json comme ayant dépassé leur durée de conservation. Par défaut : 90.", "wizard.retention.raw_documents.label": "Documents bruts (jours)", - "wizard.retention.raw_documents.hint": "Après combien de jours depuis le premier événement du log d'audit (à défaut de log : l'âge du fichier) forgelm purge --check-policy signale raw_documents/ ou ingestion_output/ comme dépassés. Default 90.", + "wizard.retention.raw_documents.hint": "Nombre de jours, comptés depuis le premier événement du journal d'audit (à défaut de journal : l'âge du fichier), au-delà duquel forgelm purge --check-policy signale raw_documents/ ou ingestion_output/ comme ayant dépassé leur durée de conservation. Par défaut : 90.", "wizard.retention.enforce.label": "Mode de politique (réservé)", "wizard.retention.enforce.hint": "Réservé : aucune commande ForgeLM n'agit encore dessus, donc les trois modes se comportent de la même façon. Les horizons de rétention ne sont signalés que par forgelm purge --check-policy ; rien n'est supprimé automatiquement.", "wizard.retention.enforce.log_only": "log_only — défaut (aucune commande n'en tient compte)", @@ -3391,15 +3391,15 @@ "wizard.synthetic.api_key_env.label": "Variable env clave API teacher", "wizard.synthetic.api_key_env.hint": "Nombre variable env con clave API teacher. Vacío = modelo teacher local.", "wizard.retention.title": "Horizontes de retención (RGPD Art. 5+17)", - "wizard.retention.hint": "Antigüedad a partir de la cual forgelm purge --check-policy reporta los artefactos como vencidos; no se elimina nada.", + "wizard.retention.hint": "Antigüedad a partir de la cual forgelm purge --check-policy señala los artefactos como vencidos; no se elimina nada. Un valor de 0 deja de señalar ese artefacto. El bloque retention solo se escribe en el YAML si algún valor difiere de su valor por defecto; sin él, --check-policy no comprueba nada.", "wizard.retention.audit_log.label": "Retención audit log (días)", - "wizard.retention.audit_log.hint": "Tras cuántos días forgelm purge --check-policy reporta audit_log.jsonl como vencido. Default 1825 (5 años) es una elección de ForgeLM, no un plazo legal.", + "wizard.retention.audit_log.hint": "Días tras los cuales forgelm purge --check-policy señala audit_log.jsonl como vencido. El valor por defecto, 1825 (5 años), es una elección de ForgeLM, no un plazo legal.", "wizard.retention.staging.label": "Horizonte staging (días)", - "wizard.retention.staging.hint": "A partir de qué antigüedad forgelm purge --check-policy reporta un directorio final_model.staging.*/ (o el antiguo final_model.staging/) como vencido; no se elimina nada automáticamente. Default 7.", + "wizard.retention.staging.hint": "Antigüedad a partir de la cual forgelm purge --check-policy señala como vencido un directorio final_model.staging.*/ (o el formato heredado final_model.staging/); no se elimina nada automáticamente. Valor por defecto: 7.", "wizard.retention.ephemeral.label": "Artefactos efímeros (días)", - "wizard.retention.ephemeral.hint": "Tras cuántos días desde el primer evento del log de auditoría (sin log: la antigüedad del archivo) forgelm purge --check-policy reporta compliance/ y data_audit_report.json como vencidos. Default 90.", + "wizard.retention.ephemeral.hint": "Días desde el primer evento del log de auditoría (si no hay log, la antigüedad del archivo) tras los cuales forgelm purge --check-policy señala compliance/ y data_audit_report.json como vencidos. Valor por defecto: 90.", "wizard.retention.raw_documents.label": "Documentos crudos (días)", - "wizard.retention.raw_documents.hint": "Tras cuántos días desde el primer evento del log de auditoría (sin log: la antigüedad del archivo) forgelm purge --check-policy reporta raw_documents/ o ingestion_output/ como vencidos. Default 90.", + "wizard.retention.raw_documents.hint": "Días desde el primer evento del log de auditoría (si no hay log, la antigüedad del archivo) tras los cuales forgelm purge --check-policy señala raw_documents/ o ingestion_output/ como vencidos. Valor por defecto: 90.", "wizard.retention.enforce.label": "Modo de política (reservado)", "wizard.retention.enforce.hint": "Reservado: ningún comando de ForgeLM actúa aún sobre esto, así que los tres modos se comportan igual. Los horizontes de retención solo los informa forgelm purge --check-policy; nada se elimina automáticamente.", "wizard.retention.enforce.log_only": "log_only — default (ningún comando lo usa)", @@ -3785,11 +3785,11 @@ "wizard.synthetic.api_key_env.label": "Teacher API 密钥环境变量", "wizard.synthetic.api_key_env.hint": "承载 teacher API 密钥的环境变量名。空 = 本地 teacher 模型。", "wizard.retention.title": "保留期限(GDPR 第 5+17 条)", - "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已超期的期限(天数);不会删除任何内容。", + "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已超期的期限(天数);不会删除任何内容。将某一项设为 0 即不再报告该 artefact。仅当某个值与默认值不同时,retention 块才会写入 YAML;没有该块时,--check-policy 不检查任何内容。", "wizard.retention.audit_log.label": "Audit log 保留(天)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已超期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", "wizard.retention.staging.label": "Staging 期限(天)", - "wizard.retention.staging.hint": "自该运行的第一条审计事件起,final_model.staging.*/(或旧的 final_model.staging/)目录满多少天后,forgelm purge --check-policy 会将其报告为已超期;不会自动删除任何内容。默认 7。", + "wizard.retention.staging.hint": "forgelm purge --check-policy 在 final_model.staging.*/(或旧版 final_model.staging/)目录满多少天时将其报告为已超期;不会自动删除任何内容。默认 7。", "wizard.retention.ephemeral.label": "临时 artefact(天)", "wizard.retention.ephemeral.hint": "自审计日志第一条事件起(无审计日志时按文件修改时间)满多少天后,forgelm purge --check-policy 会将 compliance/ 和 data_audit_report.json 报告为已超期。默认 90。", "wizard.retention.raw_documents.label": "原始文档(天)", @@ -3870,7 +3870,7 @@ "compliance.gdpr.title": "PII redaction at ingest", "compliance.gdpr.body": "--pii-mask redacts emails, phones, credit cards (Luhn), IBAN, national IDs (TR/DE/FR/US-SSN) before chunks land in JSONL. It reduces what reaches training and supports data minimisation (Article 5(1)(c)); names and free text are not detected, so it is an aid, not a guarantee.", "compliance.gdpr15.title": "Right of access", - "compliance.gdpr15.body": "forgelm reverse-pii scans JSONL corpora for plaintext occurrences, or for salted SHA-256 digests written by an external masking pipeline, of a data-subject identifier and reports every line, in the files you scan, that contains it. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.", + "compliance.gdpr15.body": "forgelm reverse-pii searches JSONL corpora for a data-subject identifier, as plain text or as a salted SHA-256 digest written by an external masking pipeline, and reports every line in the scanned files that contains it. The match is literal and case-sensitive. It is an input to your Article 15 response, which must also give a copy of the data and the Article 15(1)(a)-(h) information; raw sources and the trained model are not searched.", "compliance.gdpr17.title": "Right to erasure", "compliance.gdpr17.body": "forgelm purge deletes rows by id from a training corpus and records data.erasure_requested + data.erasure_completed events in the append-only audit log. It is one step of an Article 17 erasure: trained weights, source documents and copies elsewhere are not changed.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", @@ -3929,7 +3929,7 @@ "compliance.gdpr.title": "Ingest'te PII redaksiyonu", "compliance.gdpr.body": "--pii-mask chunk'lar JSONL'a düşmeden e-posta, telefon, kredi kartı (Luhn), IBAN, ulusal kimlik (TR/DE/FR/US-SSN) maskeler. Eğitime ulaşanı azaltır ve veri minimizasyonunu (Madde 5(1)(c)) destekler; isimler ve serbest metin tespit edilmez, bu yüzden bir yardımcıdır, garanti değildir.", "compliance.gdpr15.title": "Erişim hakkı", - "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz metin olarak geçtiği yerleri veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", + "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz metin olarak geçtiği yerleri veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Eşleşme birebir ve büyük/küçük harfe duyarlıdır. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", "compliance.gdpr17.title": "Silme hakkı", "compliance.gdpr17.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını kaydeder. Madde 17 kapsamındaki silmenin bir adımıdır: eğitilmiş ağırlıklar, kaynak dokümanlar ve başka yerlerdeki kopyalar değişmez.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", @@ -4452,9 +4452,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.de, { "compliance.gdpr15.title": "Auskunftsrecht", - "compliance.gdpr15.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext-Vorkommen oder nach von einer externen Maskierungs-Pipeline geschriebenen gesalzenen SHA-256-Digests der Kennung einer betroffenen Person und meldet jede Zeile der gescannten Dateien, die sie enthält. Das ist ein Baustein Ihrer Auskunft nach Artikel 15 DSGVO, die außerdem eine Datenkopie und die Informationen nach Artikel 15 Abs. 1 Buchst. a bis h enthalten muss; Rohquellen und das trainierte Modell werden nicht durchsucht.", + "compliance.gdpr15.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach der Kennung einer betroffenen Person – im Klartext oder als gesalzenen SHA-256-Digest, den eine externe Maskierungs-Pipeline geschrieben hat – und meldet jede Zeile der durchsuchten Dateien, die sie enthält. Die Suche ist wortgetreu und unterscheidet Groß- und Kleinschreibung. Das ist ein Baustein Ihrer Auskunft nach Art. 15 DSGVO, die außerdem eine Kopie der Daten und die Informationen nach Art. 15 Abs. 1 Buchst. a bis h DSGVO enthalten muss; Rohquellen und das trainierte Modell werden nicht durchsucht.", "compliance.gdpr17.title": "Recht auf Löschung", - "compliance.gdpr17.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert data.erasure_requested + data.erasure_completed im Append-only-Audit-Log. Es ist ein Schritt einer Löschung nach Artikel 17: trainierte Gewichte, Quelldokumente und Kopien an anderen Orten bleiben unverändert.", + "compliance.gdpr17.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Ereignisse data.erasure_requested + data.erasure_completed im Append-only-Audit-Log. Das ist ein Schritt einer Löschung nach Art. 17 DSGVO: Trainierte Gewichte, Quelldokumente und Kopien an anderen Orten bleiben unverändert.", "compliance.iso.title": "Aligned, nicht zertifiziert", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "Software kann nicht ISO 27001 oder SOC 2 zertifiziert werden — nur Organisationen. ForgeLM ist aligned mit den Kontrollen, nach denen ein Auditor den Deployer fragt: Es produziert die Audit-Trail-, Change-Management-, Daten-Lineage- und Supply-Chain-Nachweise auf jedem Lauf, bereit zur Aufnahme in Ihren Audit-Ordner.", @@ -4470,7 +4470,7 @@ "features.gov.heading": "Compliance- & DSGVO-Tooling", "features.gov.body": "EU-AI-Act-Artikel 9-17, DSGVO-Artikel 15 + 17, ISO-27001- / SOC-2-Type-II-Alignment-Nachweise.", "features.gov.access.title": "Kennungssuche (Baustein für Art. 15)", - "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern der Kennung einer betroffenen Person und listet die Zeilen der gescannten Dateien auf, die sie enthalten.", + "features.gov.access.body": "forgelm reverse-pii durchsucht JSONL-Korpora nach Klartext- oder gesalzenen Digest-Treffern der Kennung einer betroffenen Person und listet die Zeilen der durchsuchten Dateien auf, die sie enthalten – ein Baustein der Auskunft nach Art. 15 DSGVO.", "features.gov.annex.title": "Annex-IV-Verifier", "features.gov.annex.body": "forgelm verify-annex-iv validiert die Annex-IV-§1-9-Schemavollständigkeit und den manifest_hash des Bundles zur Manipulationserkennung. (Audit-Ketten-Integrität ist der separate Befehl forgelm verify-audit.)", "features.gov.approve.title": "Artikel-14-Staging-Gate", @@ -4480,7 +4480,7 @@ "features.gov.iso.title": "ISO 27001 / SOC 2 Alignment", "features.gov.iso.body": "Audit-Trail-, Change-Management-, Daten-Lineage- und Supply-Chain-Nachweise, nach denen der Auditor des Deployers fragt. Die Software ist aligned, nicht zertifiziert.", "features.gov.purge.title": "Zeilenlöschung (Schritt für Art. 17)", - "features.gov.purge.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log; es ist ein Schritt einer Löschung nach Artikel 17 (trainierte Gewichte bleiben unverändert).", + "features.gov.purge.body": "forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log; das ist ein Schritt einer Löschung nach Art. 17 DSGVO (trainierte Gewichte bleiben unverändert).", "features.eval.safetyeval.title": "Standalone-Sicherheits-Eval", "features.eval.safetyeval.body": "forgelm safety-eval führt Llama Guard gegen eine --probes-JSONL-Prompt-Datei aus (oder --default-probes für das gebündelte 51-Prompt-Set über 18 Schadkategorien) und gibt eine kategorienspezifische Sicherheitszusammenfassung aus (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "Bibliotheks-API", @@ -4498,9 +4498,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.fr, { "compliance.gdpr15.title": "Droit d'accès", - "compliance.gdpr15.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche d'occurrences en clair, ou de digests SHA-256 salés écrits par un pipeline de masquage externe, de l'identifiant d'une personne concernée et signale chaque ligne des fichiers analysés qui le contient. C'est un élément de votre réponse à une demande d'accès (article 15), qui doit aussi fournir une copie des données et les informations de l'article 15, paragraphe 1, points a) à h) ; les sources brutes et le modèle entraîné ne sont pas parcourus.", + "compliance.gdpr15.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de l'identifiant d'une personne concernée, en clair ou sous forme d'empreinte SHA-256 salée écrite par un pipeline de masquage externe, et signale chaque ligne des fichiers analysés qui le contient. La recherche est littérale et sensible à la casse. C'est un élément de votre réponse à une demande d'accès (article 15) ; cette réponse doit aussi fournir une copie des données et les informations visées à l'article 15, paragraphe 1, points a) à h). Les sources brutes et le modèle entraîné ne sont pas parcourus.", "compliance.gdpr17.title": "Droit à l'effacement", - "compliance.gdpr17.body": "forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne les événements data.erasure_requested + data.erasure_completed dans le log d'audit append-only. C'est une étape d'un effacement au titre de l'article 17 : les poids entraînés, les documents sources et les copies conservées ailleurs ne sont pas modifiés.", + "compliance.gdpr17.body": "forgelm purge supprime des lignes d'un corpus d'entraînement d'après leur identifiant (id) et consigne les événements data.erasure_requested + data.erasure_completed dans le journal d'audit append-only. C'est une étape de l'effacement demandé au titre de l'article 17 : les poids entraînés, les documents sources et les copies conservées ailleurs ne sont pas modifiés.", "compliance.iso.title": "Aligné, non certifié", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "Le logiciel ne peut pas être certifié ISO 27001 ou SOC 2 — seules les organisations le peuvent. ForgeLM est aligné avec les contrôles qu'un auditeur demande au déployeur : il produit les preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain à chaque exécution, prêtes à être déposées dans votre classeur d'audit.", @@ -4516,7 +4516,7 @@ "features.gov.heading": "Outillage Compliance & RGPD", "features.gov.body": "Articles 9-17 de l'EU AI Act, RGPD Articles 15 + 17, preuves d'alignement ISO 27001 / SOC 2 Type II.", "features.gov.access.title": "Recherche d'identifiant (élément de l'article 15)", - "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de correspondances en clair ou de digests salés de l'identifiant d'une personne concernée et liste les lignes des fichiers analysés qui le contiennent.", + "features.gov.access.body": "forgelm reverse-pii parcourt des corpus JSONL à la recherche de l'identifiant d'une personne concernée, en clair ou sous forme d'empreinte salée, et liste les lignes des fichiers analysés qui le contiennent : un élément de la réponse à une demande d'accès (article 15).", "features.gov.annex.title": "Vérificateur Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valide la complétude du schéma Annex IV §1-9 et le manifest_hash du bundle pour la détection de falsification. (L'intégrité de la chaîne d'audit est la commande séparée forgelm verify-audit.)", "features.gov.approve.title": "Gate de staging Article 14", @@ -4526,7 +4526,7 @@ "features.gov.iso.title": "Alignement ISO 27001 / SOC 2", "features.gov.iso.body": "Preuves d'audit-trail, de gestion du changement, de traçabilité des données et de supply chain que l'auditeur du déployeur demande. Le logiciel est aligné, non certifié.", "features.gov.purge.title": "Effacement de lignes (étape de l'article 17)", - "features.gov.purge.body": "forgelm purge supprime, par identifiant (id), des lignes d'un corpus d'entraînement et consigne l'effacement dans le log d'audit append-only ; c'est une étape d'un effacement au titre de l'article 17 (les poids entraînés ne changent pas).", + "features.gov.purge.body": "forgelm purge supprime des lignes d'un corpus d'entraînement d'après leur identifiant (id) et consigne l'effacement dans le journal d'audit append-only ; c'est une étape de l'effacement demandé au titre de l'article 17 (les poids entraînés ne changent pas).", "features.eval.safetyeval.title": "Évaluation sécurité autonome", "features.eval.safetyeval.body": "forgelm safety-eval exécute Llama Guard contre un fichier de prompts JSONL --probes (ou --default-probes pour le set bundlé de 51 prompts couvrant 18 catégories de préjudice) et émet un résumé de sécurité par catégorie (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "API bibliothèque", @@ -4544,9 +4544,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.es, { "compliance.gdpr15.title": "Derecho de acceso", - "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL en busca de apariciones en texto claro, o de digests SHA-256 salados escritos por un pipeline de enmascaramiento externo, de un identificador de interesado y reporta cada línea de los archivos escaneados que lo contiene. Es un elemento de su respuesta a una solicitud de acceso (Artículo 15), que además debe incluir una copia de los datos y la información del Artículo 15, apartado 1, letras a) a h); no se buscan las fuentes sin procesar ni el modelo entrenado.", + "compliance.gdpr15.body": "forgelm reverse-pii escanea corpus JSONL en busca del identificador de un interesado, en texto claro o como hash SHA-256 con sal escrito por un pipeline de enmascaramiento externo, e indica cada línea de los archivos escaneados que lo contiene. La búsqueda es literal y distingue mayúsculas de minúsculas. Es un elemento de su respuesta a una solicitud de acceso (artículo 15); esa respuesta debe incluir además una copia de los datos y la información contemplada en el artículo 15, apartado 1, letras a) a h). No se examinan las fuentes sin procesar ni el modelo entrenado.", "compliance.gdpr17.title": "Derecho de supresión", - "compliance.gdpr17.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra los eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only. Es un paso de una supresión conforme al Artículo 17: los pesos entrenados, los documentos de origen y las copias que existan en otros lugares no cambian.", + "compliance.gdpr17.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra los eventos data.erasure_requested + data.erasure_completed en el log de auditoría append-only. Es un paso de la supresión solicitada conforme al artículo 17: los pesos entrenados, los documentos de origen y las copias que existan en otros lugares no cambian.", "compliance.iso.title": "Alineado, no certificado", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "El software no puede ser certificado ISO 27001 o SOC 2 — solo las organizaciones pueden. ForgeLM está alineado con los controles que un auditor le pregunta al deployer: produce las pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro en cada ejecución, listas para incorporarse a su carpeta de auditoría.", @@ -4562,7 +4562,7 @@ "features.gov.heading": "Tooling Compliance & RGPD", "features.gov.body": "Artículos 9-17 del EU AI Act, RGPD Artículos 15 + 17, pruebas de alineamiento ISO 27001 / SOC 2 Type II.", "features.gov.access.title": "Búsqueda de identificador (elemento del Artículo 15)", - "features.gov.access.body": "forgelm reverse-pii escanea corpus JSONL en busca de coincidencias en texto claro o de digests salados de un identificador de interesado y lista las líneas de los archivos escaneados que lo contienen.", + "features.gov.access.body": "forgelm reverse-pii escanea corpus JSONL en busca de coincidencias en texto claro o de hashes con sal del identificador de un interesado y lista las líneas de los archivos escaneados que lo contienen: un elemento de la respuesta a una solicitud de acceso (artículo 15).", "features.gov.annex.title": "Verificador Annex IV", "features.gov.annex.body": "forgelm verify-annex-iv valida la completitud del esquema Annex IV §1-9 y el manifest_hash del bundle para la detección de manipulación. (La integridad de la cadena de auditoría es el comando separado forgelm verify-audit.)", "features.gov.approve.title": "Gate de staging del Artículo 14", @@ -4572,7 +4572,7 @@ "features.gov.iso.title": "Alineamiento ISO 27001 / SOC 2", "features.gov.iso.body": "Pruebas de pista de auditoría, gestión de cambios, trazabilidad de datos y cadena de suministro que pide el auditor del deployer. El software está alineado, no certificado.", "features.gov.purge.title": "Supresión de filas (paso del Artículo 17)", - "features.gov.purge.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra la supresión en el log de auditoría append-only; es un paso de una supresión conforme al Artículo 17 (los pesos entrenados no cambian).", + "features.gov.purge.body": "forgelm purge elimina filas por id de un corpus de entrenamiento y registra la supresión en el log de auditoría append-only; es un paso de la supresión solicitada conforme al artículo 17 (los pesos entrenados no cambian).", "features.eval.safetyeval.title": "Evaluación de seguridad autónoma", "features.eval.safetyeval.body": "forgelm safety-eval ejecuta Llama Guard contra un archivo de prompts JSONL --probes (o --default-probes para el set bundleado de 51 prompts que cubre 18 categorías de daño) y emite un resumen de seguridad por categoría (safety_results.json + safety_trend.jsonl).", "features.ent.api.title": "API de biblioteca", @@ -4590,9 +4590,9 @@ /* ── Phase 22 / pass-4: backfill missing compliance + features keys (was H6 i18n gap) ── */ Object.assign(T.zh, { "compliance.gdpr15.title": "访问权", - "compliance.gdpr15.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文,或由外部掩码流水线写入的该标识符的加盐 SHA-256 摘要,并报告所扫描文件中包含它的每一行。它只是对第 15 条访问请求答复的输入之一,该答复还必须提供数据副本和第 15 条第 1 款(a)至(h)项信息;不会搜索原始来源或已训练的模型。", + "compliance.gdpr15.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文,或由外部脱敏流水线写入的该标识符的加盐 SHA-256 摘要,并报告所扫描文件中包含它的每一行。匹配是逐字的,且区分大小写。其结果只是答复第 15 条访问请求的材料之一;该答复还必须提供个人数据副本以及第 15 条第 1 款第 (a) 至 (h) 项所列信息。不会搜索原始来源或已训练的模型。", "compliance.gdpr17.title": "删除权", - "compliance.gdpr17.body": "forgelm purge 按 id 从训练语料中删除行,并将 data.erasure_requested + data.erasure_completed 事件记录到 append-only 审计日志。它只是第 17 条所涉删除的其中一步:已训练的权重、源文档和其他位置的副本不会改变。", + "compliance.gdpr17.body": "forgelm purge 按 id 从训练语料中删除行,并将 data.erasure_requested + data.erasure_completed 事件记录到 append-only 审计日志。它只是第 17 条删除流程中的一步:已训练的权重、源文档和其他位置的副本不会改变。", "compliance.iso.title": "对齐,而非认证", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", "compliance.iso.lead": "软件无法获得 ISO 27001 或 SOC 2 认证 — 只有组织可以。ForgeLM 与审计员向 deployer 询问的控制项保持对齐:它在每次运行时产出审计追踪、变更管理、数据血缘和供应链证据,可直接放入你的审计文件夹。", @@ -4608,7 +4608,7 @@ "features.gov.heading": "合规 & GDPR 工具", "features.gov.body": "EU AI Act 第 9-17 条、GDPR 第 15 + 17 条、ISO 27001 / SOC 2 Type II 对齐证据。", "features.gov.access.title": "标识符查找 (第 15 条的输入之一)", - "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文或加盐摘要匹配,并列出所扫描文件中包含它的行。", + "features.gov.access.body": "forgelm reverse-pii 扫描 JSONL 语料库,查找数据主体标识符的明文或加盐摘要匹配,并列出所扫描文件中包含它的行,可作为答复第 15 条访问请求的材料之一。", "features.gov.annex.title": "Annex IV 验证器", "features.gov.annex.body": "forgelm verify-annex-iv 验证 Annex IV §1-9 模式完整性以及 bundle 的 manifest_hash 防篡改。(审计链完整性由独立的 forgelm verify-audit 命令负责。)", "features.gov.approve.title": "第 14 条 staging gate", @@ -4618,7 +4618,7 @@ "features.gov.iso.title": "ISO 27001 / SOC 2 对齐", "features.gov.iso.body": "deployer 的审计员需要的审计追踪、变更管理、数据血缘和供应链证据。软件是对齐,而不是被认证。", "features.gov.purge.title": "行删除 (第 17 条的其中一步)", - "features.gov.purge.body": "forgelm purge 按 id 从训练语料中删除行,并将该删除记录到 append-only 审计日志;它只是第 17 条所涉删除的其中一步(已训练的权重不会改变)。", + "features.gov.purge.body": "forgelm purge 按 id 从训练语料中删除行,并将该删除记录到 append-only 审计日志;它只是第 17 条删除流程中的一步(已训练的权重不会改变)。", "features.eval.safetyeval.title": "独立安全评估", "features.eval.safetyeval.body": "forgelm safety-eval 对 --probes JSONL 提示文件 (或用 --default-probes 调用打包的、覆盖 18 个危害类别的 51-提示集) 运行 Llama Guard,并产出按类别的安全摘要 (safety_results.json + safety_trend.jsonl)。", "features.ent.api.title": "库 API", From aaf63c4fdaceb11ad386d307071806880817091c Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 03:50:10 +0300 Subject: [PATCH 17/25] fix(docs): retention wording, KVKK, human-oversight, QMS SOPs and design note against the code Remaining statements that the code or the law does not support, found while re-checking the documentation a second time: - Retention: the configuration reference intro is split and now says that without a `retention:` block nothing is checked and that `retention: {}` switches the defaults on; the heading drops "Article 17 erasure horizons". `raw_documents_retention_days` documents that ForgeLM creates neither `raw_documents/` nor `ingestion_output/`; the field descriptions say how age is computed; the "quarterly review cadence" / "ingestion window" justifications are gone; "no command reads it" is "no command acts on it" everywhere. - gdpr.md: a zero-match scan is no longer "a valid answer" to an access request, KVKK Articles 7, 11(1)(e) and 13(2) are cited for what they say, the id-assignment script now tells the reader to retire the original corpus file, and the site wizard retention accordion cites Article 5(1)(e) instead of "5+17". - human-oversight: `staging_ttl_days` is described as a report, not an expiry, and removing a pending staging directory leaves the approval undecided; the unsourced "always quorum >= 2" rule is replaced by what AI Act Article 14(5) actually requires. - QMS: the "minimum 5 years" retention lines for training data and compliance artefacts are replaced by a policy-set period (storage limitation, Article 18 is separate); the encryption SOP no longer says salt rotation affects `verify-audit` or that salted digests need no encryption. - Design note: temp-file recovery, the operator-field legal basis, the memorisation trigger and the Article 19 recommendation match the code; the trainer pre-flight gate and the audit retention notes are marked as never built; section 12 lists the exit-2 flag errors, the salt rule and the memorisation trigger. - GDPR Article 25(2), not "AI Act Article 10 privacy", is the cited basis for the `include_eval_samples` defaults; the notebook and `--pii-ml-language` advice use `xx` for Turkish; the bare `--row-id` error no longer invents an Article 17 rule; the purge synopsis and `--output-dir` help cover `--check-policy` and the run-mode default. - Turkish glossary: veri sahibi / veri sorumlusu / salt / digest are fixed in localization.md and applied to the pages and the site. Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 29 +++++++++------- docs/guides/gdpr_erasure-tr.md | 14 ++++---- docs/guides/gdpr_erasure.md | 8 ++--- docs/qms/encryption_at_rest-tr.md | 17 ++++++---- docs/qms/encryption_at_rest.md | 16 +++++---- docs/qms/sop_data_management-tr.md | 2 +- docs/qms/sop_data_management.md | 2 +- docs/qms/sop_model_training-tr.md | 2 +- docs/qms/sop_model_training.md | 2 +- docs/reference/configuration-tr.md | 33 ++++++++++--------- docs/reference/configuration.md | 32 +++++++++--------- docs/reference/purge_subcommand-tr.md | 6 ++-- docs/reference/purge_subcommand.md | 6 ++-- docs/standards/localization.md | 3 ++ .../usermanuals/en/compliance/gdpr-erasure.md | 6 ++-- docs/usermanuals/en/compliance/gdpr.md | 8 ++--- .../en/compliance/human-oversight.md | 4 +-- docs/usermanuals/en/evaluation/safety.md | 4 +-- .../usermanuals/tr/compliance/gdpr-erasure.md | 10 +++--- docs/usermanuals/tr/compliance/gdpr.md | 8 ++--- .../tr/compliance/human-oversight.md | 4 +-- docs/usermanuals/tr/evaluation/safety.md | 4 +-- .../usermanuals/tr/reference/configuration.md | 2 +- forgelm/cli/_config_load.py | 2 +- forgelm/cli/_parser.py | 2 +- forgelm/cli/subcommands/_purge.py | 2 +- forgelm/config.py | 20 +++++------ forgelm/data_audit/_types.py | 3 +- notebooks/data_curation.ipynb | 2 +- site/js/translations.js | 16 ++++----- site/js/wizard.js | 4 +-- 31 files changed, 146 insertions(+), 127 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 40e9dfdb..05564c2c 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -139,7 +139,7 @@ class RetentionConfig(BaseModel): enforce: Literal["log_only", "warn_on_excess", "block_on_excess"] = "log_only" ``` -`ForgeConfig` grows an optional `retention: Optional[RetentionConfig] = None` field. When unset, the existing default-permissive behaviour holds (no retention check). When set, the trainer's pre-flight gate runs the check. +`ForgeConfig` grows an optional `retention: Optional[RetentionConfig] = None` field. When unset, the existing default-permissive behaviour holds (no retention check). When set, `forgelm purge --check-policy` reports against it. The trainer pre-flight gate this section originally described was never built (§12). ### 3.3 mtime caveat (and the in-band age field) @@ -154,9 +154,9 @@ The `--check-policy` JSON output includes `age_source ∈ {audit, mtime}` per ar Three sites: -1. **`ForgeTrainer.train()` pre-flight** (before any data is read): scan `output_dir` for artefacts whose `mtime` exceeds the retention window; emit per-violation audit events; honour `enforce`. +1. **`ForgeTrainer.train()` pre-flight** *(designed; not implemented, see §12)* (before any data is read): scan `output_dir` for artefacts whose `mtime` exceeds the retention window; emit per-violation audit events; honour `enforce`. 2. **`forgelm purge --check-policy` (no --row-id / --run-id)**: read-only scan; reports violations as a structured summary (text + JSON output formats). -3. **`forgelm audit`** (existing subcommand): when `retention` is set in the loaded config, the audit report's `notes` block lists violations as a friendly heads-up. +3. **`forgelm audit`** (existing subcommand) *(designed; not implemented, see §12)*: when `retention` is set in the loaded config, the audit report's `notes` block lists violations as a friendly heads-up. ### 3.5 What "violation" means @@ -205,9 +205,9 @@ Standard ForgeLM 0/1/2/3/4 contract — all codes from `forgelm.cli._exit_codes` | Code | Constant | Meaning | |---|---|---| | 0 | `EXIT_SUCCESS` | Erasure completed (or `--dry-run` reported successfully). | -| 1 | `EXIT_CONFIG_ERROR` | Config-level error: missing flag, flag combination, unknown row-id (corpus mode), unknown run-id (run mode), `--row-id` directory mode without `--row-matches=all`, multi-row match without `--row-matches=all`. (`--yes` is not implemented — see §3.5 for the consent model; the shipped `forgelm purge` is non-interactive on every invocation, so there is no "missing `--yes` on a non-TTY" failure mode.) | +| 1 | `EXIT_CONFIG_ERROR` | Config-level error: unknown row-id (corpus mode), unknown run-id (run mode), `--row-id` directory mode without `--row-matches=all`, multi-row match without `--row-matches=all`. Flag combinations the argument parser rejects exit 2 instead (§12). (`--yes` is not implemented — see §3.5 for the consent model; the shipped `forgelm purge` is non-interactive on every invocation, so there is no "missing `--yes` on a non-TTY" failure mode.) | | 2 | `EXIT_TRAINING_ERROR` | Runtime / I/O failure: corpus unreadable, audit-log write failure, atomic-rename failure, partial commit recovery. | -| 3 | `EXIT_EVAL_FAILURE` | Reserved for the trainer pre-flight gate when `enforce: block_on_excess` is configured (see §3.4 in the spirit of the master `error-handling.md` exit-code contract). **Not** used by `forgelm purge --check-policy` — that path is a report, not a gate; see §10 Q5 + the paragraph below. | +| 3 | `EXIT_EVAL_FAILURE` | Reserved for the trainer pre-flight gate when `enforce: block_on_excess` is configured (no such gate exists today; see §3.4 in the spirit of the master `error-handling.md` exit-code contract). **Not** used by `forgelm purge --check-policy` — that path is a report, not a gate; see §10 Q5 + the paragraph below. | | 4 | `EXIT_AWAITING_APPROVAL` | Reserved for its trainer-pipeline meaning per `docs/standards/error-handling.md` ("Training + evals passed, but `require_human_approval: true` — staged, awaiting human sign-off"). **Not** reused by `forgelm purge`. Since the shipped `forgelm purge` is non-interactive (no prompt, no `--yes` flag), there is no "operator declined the prompt" path; consent is recorded in the operator-supplied `--justification` and the audit chain. | **`--check-policy` exits 0 on a successful report** (per §10 Q5 — the resolved decision: report-not-gate semantic). Operators who want a CI gate use `--output-format json` and test the report themselves (for example `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`; `count` is absent when no `retention:` block was loaded); CI then branches on that rather than on the exit code. This keeps the public contract `docs/standards/error-handling.md` (the exit-code table in that standard) consistent across every ForgeLM subcommand: a code-3 from `forgelm purge` always means "trainer pre-flight gate failed", never "report found something". @@ -223,7 +223,7 @@ Standard ForgeLM 0/1/2/3/4 contract — all codes from `forgelm.cli._exit_codes` 5. `os.fsync` + `os.replace` to swap. 6. **Audit `data.erasure_completed` LAST** — same fields plus `bytes_freed`, `pre_erasure_line_number`, `files_modified=[corpus_path]`. -Recovery path for a step-5 failure (rename fails after the temp file is fully written): the `data.erasure_requested` event is in the chain, the temp file is left on disk for forensic inspection, and a `data.erasure_failed` event is appended carrying the original error. An operator inspecting the log sees the request → failure pair and can manually decide whether to retry or revert. +Recovery path for a step-5 failure (rename fails after the temp file is fully written): the `data.erasure_requested` event is in the chain, the temp file is removed on a best-effort basis (the corpus itself is untouched), and a `data.erasure_failed` event is appended carrying the original error. An operator inspecting the log sees the request → failure pair and can manually decide whether to retry or revert. **Recovery path for a step-6 failure** (rename succeeded, completion event write fails): the row is gone from the corpus but the chain has only the request event. This is detectable: a `request` event without a matching `completed` / `failed` event within the same minute is the operator's signal to verify the corpus state and append a manual `data.erasure_completed` (or `_failed`) event by hand. Phase 21 ships a `tests/test_gdpr_erasure.py::test_partial_commit_emits_failed_event` test that simulates this race + asserts the chain stays interpretable. @@ -256,9 +256,9 @@ Six new events in `audit_event_catalog.md` — three core erasure events plus th | `data.erasure_requested` | First step of any `forgelm purge --row-id` / `--run-id` invocation, *before* any deletion (see §4.4 commit ordering). If deletion then fails, the request record stays in the chain so a forensic reviewer can see the intent. `--check-policy` is read-only and does not emit this event. | `target_kind` ∈ {row, staging, artefacts}, `target_id` (hashed per §5.4 for row mode), `salt_source` ∈ {`"env_var"`, `"per_dir"`} (row mode only; see §5.4), `corpus_path` (corpus mode), `output_dir` (run mode), `justification`, `dry_run` | | `data.erasure_completed` | Successful deletion finishes. | All `data.erasure_requested` fields + `bytes_freed`, `files_modified` (list), `pre_erasure_line_number` (corpus mode) | | `data.erasure_failed` | Deletion failed (atomic rename failure, permission denied, I/O error). | All `data.erasure_requested` fields + `error_class`, `error_message` (already-redacted via §5.4 policy + `_http`'s mask helper) | -| `data.erasure_warning_memorisation` | Emitted alongside `data.erasure_completed` when `target_kind="row"` AND a `/final_model/` exists for any run that consumed this corpus. Signals that the row is gone from the corpus but may still be memorised in trained weights. | All `data.erasure_completed` fields + `affected_run_ids` (list of run ids whose final_model used this corpus), `recommendation` (literal `"full retraining without the erased row is the only proper mitigation"`) | +| `data.erasure_warning_memorisation` | Emitted alongside `data.erasure_completed` when `target_kind="row"` AND a `/final_model/` exists. The trigger is that directory's existence alone: ForgeLM does not check which run (if any) trained on this corpus or this row. Signals that the row is gone from the corpus but may still be memorised in trained weights. | All `data.erasure_completed` fields + `affected_run_ids` (list of run ids whose final_model used this corpus), `recommendation` (literal `"full retraining without the erased row is the only proper mitigation"`) | | `data.erasure_warning_synthetic_data_present` | Emitted alongside `data.erasure_completed` when `target_kind="row"` AND any `/synthetic_data*.jsonl` exists. Signals that the row may have produced derivative synthetic snippets that the row→snippet mapping no longer connects. | All `data.erasure_completed` fields + `synthetic_files` (list of paths), `recommendation` (literal `"regenerate synthetic data after erasure to ensure no derivatives reference the erased row"`) | -| `data.erasure_warning_external_copies` | Emitted alongside `data.erasure_completed` when the loaded config has a non-empty `webhook` block. Signals that downstream consumers may have received notices that referenced the now-erased data; ForgeLM's local erasure does not propagate. | All `data.erasure_completed` fields + `webhook_targets` (list of redacted URLs), `recommendation` (literal `"propagate the erasure to downstream processors per Article 17(2)"`) | +| `data.erasure_warning_external_copies` | Emitted alongside `data.erasure_completed` when the loaded config has a non-empty `webhook` block. Signals that downstream consumers may have received notices that referenced the now-erased data; ForgeLM's local erasure does not propagate. | All `data.erasure_completed` fields + `webhook_targets` (list of redacted URLs), `recommendation` (literal `"tell the recipients the data was disclosed to about the erasure, per Article 19 (and Article 17(2) where the data was made public)"`) | ### 5.2 Chain-integrity post-erasure @@ -296,7 +296,7 @@ The audit log itself is append-only, but Article 5(1)(c) (data minimisation) sti | `pre_erasure_line_number` | int | None | clear | | `error_class` | exception class name | None | clear | | `error_message` | str | **Possibly personal** (Python tracebacks include `repr()` of objects, which may include row content) | route through `forgelm._http._mask_secrets_in_text` first; redact obvious PII (email regex, phone regex) via the existing `forgelm.data_audit._pii_regex` mask helpers | -| `operator` (carried on every AuditLogger event) | `FORGELM_OPERATOR` or `getuser()@host` | **Personal** | clear (Article 12 record-keeping requires identifiable operator; this is the explicit Article 5(1)(b) lawful basis) | +| `operator` (carried on every AuditLogger event) | `FORGELM_OPERATOR` or `getuser()@host` | **Personal** | clear (the record must say who acted, so the operator stays identifiable; the controller needs its own documented Article 6 basis for keeping that identity in the log) | The take-away: **the only field this design hashes by default is `target_id` in row mode**. Everything else either is genuinely operational, is the operator's responsibility (justification), or is masked through the existing Wave 1 `_mask_secrets_in_text` helper. Phase 21 ships a unit test `tests/test_gdpr_erasure.py::test_target_id_hashed_in_audit_event` that asserts the raw row id never appears in the persisted JSONL after a `forgelm purge --row-id ali@example.com --corpus train.jsonl` run. @@ -324,7 +324,7 @@ in three scopes: --kind {staging,artefacts}` deletes the model staging directory for a specific run and the compliance files whose names embed the run id (as shipped, not the standard bundle; see the purge reference). -3. **Retention policy enforcement.** Configure `retention.*` in your YAML; +3. **Retention reporting.** Configure `retention.*` in your YAML; `forgelm purge --check-policy` lists artefacts that have overstayed the declared horizon. @@ -444,7 +444,7 @@ The closure plan §15.5 lists three open items that intersect Phase 21: ### Q5. Should `--check-policy` return non-zero when violations are found? -**Decision:** no. `--check-policy` is a *report*, not a *gate*. The CI gate is the trainer pre-flight (when `enforce: block_on_excess` is configured). Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and test the report themselves (see §4.3). +**Decision:** no. `--check-policy` is a *report*, not a *gate*. The trainer pre-flight gate that was meant to be the CI gate (`enforce: block_on_excess`) was never built, so CI tests the JSON report. Mixing gate + report semantics into one exit code makes the tool harder to script. Operators who want a CI gate use `--output-format json` and test the report themselves (see §4.3). ### Q6. Should we delete the synthetic-data snapshots on row erasure? @@ -467,7 +467,7 @@ The closure plan §15.5 lists three open items that intersect Phase 21: The design above shipped in v0.5.5, with the deviations listed after this list: -- **`RetentionConfig`** (§3): pinned in `forgelm/config.py` as a top-level `retention:` block with five fields (`audit_log_retention_days`, `staging_ttl_days`, `ephemeral_artefact_retention_days`, `raw_documents_retention_days`, `enforce`). `EvaluationConfig.staging_ttl_days` deprecated with `DeprecationWarning` + alias-forwarding to `retention.staging_ttl_days`. +- **`RetentionConfig`** (§3): pinned in `forgelm/config.py` as a top-level `retention:` block with five fields (`audit_log_retention_days`, `staging_ttl_days`, `ephemeral_artefact_retention_days`, `raw_documents_retention_days`, `enforce`). `EvaluationConfig.staging_ttl_days` was deprecated with `DeprecationWarning` + alias-forwarding to `retention.staging_ttl_days` and has since been removed (v0.8.0); §3.1 describes that historical window. - **`forgelm purge` subcommand** (§4): pinned in `forgelm/cli/subcommands/_purge.py` with `--row-id`, `--corpus`, `--run-id`, `--kind`, `--check-policy`, atomic-rewrite + recovery semantics. - **Audit events** (§5): six new events live in `audit_event_catalog.md` — `data.erasure_requested`, `data.erasure_completed`, `data.erasure_failed`, plus three operator-warning events. - **Tests** (§7): `tests/test_gdpr_erasure.py` ships the regression coverage (unknown row id, multi-match refusal, partial-commit recovery, retention horizon enforcement, audit-PII redaction, etc.). @@ -480,4 +480,9 @@ The design above shipped in v0.5.5, with the deviations listed after this list: - The legacy `final_model.staging/` is purged only when it holds a `staging_run_id` marker equal to the run id. Nothing in ForgeLM writes that marker. - `--check-policy` ages the raw-documents directories (`raw_documents/`, `ingestion_output/`) from the audit log's first event, not per run; it reads `--config` only when given (no `./forgelm.yaml` discovery) and, without a `retention:` block, reports that nothing was checked. `--dry-run` combined with `--check-policy` is not rejected. - The `data.erasure_warning_external_copies` event is emitted in row mode only, and only when `--config` is given; run mode never loads the config. +- Mode flags that conflict (for example `--row-id` with `--run-id`, or `--kind logs`) are rejected by the argument parser, which exits 2 without a JSON envelope; other configuration errors exit 1. A run id is not looked up in the audit log: it is "unknown" only when no matching artefact exists. +- With `FORGELM_AUDIT_SECRET` set, the per-directory salt is XOR-combined with the first 16 bytes of the secret (`salt_source=env_var`), so a row's `target_id` changes when the secret rotates. `forgelm reverse-pii` derives its `query_hash` the same way in its default plaintext mode; `--salt-source` switches it to a hash-mask scan that looks for digests inside the corpus. +- `data.erasure_warning_memorisation` fires whenever `/final_model/` exists; the check does not tie that model to the erased row. +- The recovery paths of §4.4 assume the temp file survives a failed rename; the shipped code removes it (best effort) instead. +- Neither the trainer pre-flight gate nor the `forgelm audit` retention notes of §3.4 exist. - The `recommendation` field listed for the new events in §5.1 is not emitted, and the test table in §7 predates the shipped tests (for example, a dry-run appends audit events, and erasing the same row twice exits 1 with `NoMatchingRow`); `tests/test_gdpr_erasure.py` is the source of truth. diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 05c08757..62f2399b 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -24,9 +24,9 @@ - Modelleri yeniden eğitmez. Bir satırı corpus'tan kaldırmak onu zaten eğitilmiş ağırlıklardan unutturmaz — silinmiş satır olmadan tam yeniden eğitim tek doğru çözüm. Tool, `--output-dir` altında doğrudan bir `final_model/` dizini mevcut olduğunda `data.erasure_warning_memorisation` yayar (`--output-dir` varsayılan olarak corpus'un dizinidir; bu yüzden eğitim çıktı dizininizi verin); böylece operatör boşluğu görür. Corpus'u hangi koşumların tükettiğini ayırt edemez. - Audit log'un kendisinden entry silmez. Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Bir Madde 17(1) gerekçesi geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki talepleri kapsar. Hiçbir Madde 17(1) gerekçesi geçerli değilse (örneğin girdiler toplandıkları amaç için hâlâ gereklidir) silme gerekmez; ancak talebe yine de Madde 12 süresi içinde gerekçeli bir yanıt verilir ve şikâyet hakkı bildirilir (Madde 12(4)). -- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve ilgili kişi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). Tool, row modunda `--config` ile verdiğiniz config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. +- Downstream tüketicilere silme bildirimi göndermez. Webhook receiver'ları, dataset mirror'ları, deploy edilmiş model endpoint'leri ve yayınlanmış checkpoint'ten türetilmiş üçüncü-taraf fine-tune'lar operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve veri sahibi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). Tool, row modunda `--config` ile verdiğiniz config webhook block'u içerdiğinde `data.erasure_warning_external_copies` yayar; böylece audit log'u sorgulayan downstream tüketici açık hatırlatmayı görür. - Yedekleri silmez. Operatörün `` sınırı dışındaki replikalar, snapshot'lar ve storage backup'lar altyapı tarafı endişelerdir. -- Yanıtın tamamını vermez. Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); ilgili kişiye verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). +- Yanıtın tamamını vermez. Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); veri sahibine verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). ## Üç mod @@ -80,12 +80,12 @@ Salt-okunur tarama. Output dizinini gezer, her artefact'ın yaşını (canonical retention: audit_log_retention_days: 1825 # 5 yıl (ForgeLM varsayılanı, yasal süre değildir) staging_ttl_days: 7 # purge --check-policy'nin staging dizinini süresi geçmiş saydığı yaş - ephemeral_artefact_retention_days: 90 # üç-aylık review cadence'i - raw_documents_retention_days: 90 # data audit yeniden çalıştırmadan önce ingestion penceresi - enforce: log_only # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only / warn_on_excess / block_on_excess) + ephemeral_artefact_retention_days: 90 # compliance/ ve data_audit_report.json, audit log'un ilk olayından itibaren yaşlandırılır + raw_documents_retention_days: 90 # yalnızca /raw_documents/ ve ingestion_output/ (ForgeLM bunları oluşturmaz) + enforce: log_only # ayrılmış: henüz hiçbir komut onu dikkate almaz (log_only / warn_on_excess / block_on_excess) ``` -Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapatır (her iki durumda da hiçbir şey silinmez). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur; kaç tane bulursa bulsun 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. `enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz, bu yüzden üç değer de `log_only` gibi davranır. +Herhangi bir horizon'u `0` olarak ayarlamak o artefact kind'ı için raporu kapatır (her iki durumda da hiçbir şey silinmez). Horizon'lar `forgelm purge --check-policy` ile raporlanır (salt-okunur; kaç tane bulursa bulsun 0 ile çıkar); hiçbir şey otomatik silinmez ya da engellenmez. ForgeLM `raw_documents/` veya `ingestion_output/` dizinlerini oluşturmaz; `raw_documents_retention_days` yalnızca kaynak dokümanları orada tutuyorsanız bir etki yapar. `enforce` ayrılmıştır: henüz hiçbir komut onu dikkate almaz, bu yüzden üç değer de `log_only` gibi davranır. ## Kaldırıldı: `evaluation.staging_ttl_days` @@ -187,6 +187,6 @@ glob); `2` = runtime hatası (mid-scan I/O failure). JSON envelope ## Bkz. -- `docs/qms/sop_data_management.md` — retention + erasure prosedürleri dahil tam data-lifecycle SOP. +- `docs/qms/sop_data_management.md` — veri yaşam döngüsü SOP'si (toplama, kalite, saklama); bir saklama ya da silme prosedürü içermez, bunu bu kılavuzdan yola çıkarak yazın. - `docs/usermanuals/tr/compliance/overview.md` — buraya link veren operator-facing compliance overview. - [`docs/design/gdpr_erasure.md`](../design/gdpr_erasure.md) — bu implementation'ın gerçekleştirdiği Phase 20 design dokümanı. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index c5a028a1..c864c4d8 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -79,12 +79,12 @@ Read-only scan. Walks the output directory, compares each artefact's age (canoni retention: audit_log_retention_days: 1825 # 5 years (ForgeLM default, not a legal period) staging_ttl_days: 7 # age at which `purge --check-policy` reports a staging dir as overdue - ephemeral_artefact_retention_days: 90 # quarterly review cadence - raw_documents_retention_days: 90 # ingestion-window before re-running data audit + ephemeral_artefact_retention_days: 90 # compliance/ and data_audit_report.json, aged from the audit log's first event + raw_documents_retention_days: 90 # only /raw_documents/ and ingestion_output/ (ForgeLM does not create them) enforce: log_only # reserved: no command acts on it yet (log_only / warn_on_excess / block_on_excess) ``` -Setting any horizon to `0` turns the report off for that artefact kind (nothing is deleted either way). The horizons are reported by `forgelm purge --check-policy` (read-only; exits 0 however many it finds); nothing is deleted or blocked automatically. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. +Setting any horizon to `0` turns the report off for that artefact kind (nothing is deleted either way). The horizons are reported by `forgelm purge --check-policy` (read-only; exits 0 however many it finds); nothing is deleted or blocked automatically. ForgeLM does not create `raw_documents/` or `ingestion_output/`; `raw_documents_retention_days` has an effect only if you keep source documents there. `enforce` is reserved: no command acts on it yet, so all three values behave like `log_only`. ## Removed: `evaluation.staging_ttl_days` @@ -184,6 +184,6 @@ for the JSON envelope schema. ## See also -- `docs/qms/sop_data_management.md` — the full data-lifecycle SOP including the retention + erasure procedures. +- `docs/qms/sop_data_management.md` — the data-lifecycle SOP (collection, quality, storage); it does not contain a retention or erasure procedure, so write one from this guide. - `docs/usermanuals/en/compliance/overview.md` — the operator-facing compliance overview that links here. - [`docs/design/gdpr_erasure.md`](../design/gdpr_erasure.md) — the Phase 20 design document that this implementation realises. diff --git a/docs/qms/encryption_at_rest-tr.md b/docs/qms/encryption_at_rest-tr.md index 7a0742a9..fc57a9c1 100644 --- a/docs/qms/encryption_at_rest-tr.md +++ b/docs/qms/encryption_at_rest-tr.md @@ -25,7 +25,7 @@ Aşağıdaki ForgeLM-ürettiği asset sınıfları: | Model ağırlıkları — final | `/final_model/` | Eğitim verisi memorisation; rekabetçi moat | | Model ağırlıkları — staging | `/final_model.staging./` | Final ile aynı, artı pre-approval state | | Audit log | `/audit_log.jsonl` + `audit_log.jsonl.manifest.json` (genesis pin) | Operatör kimliği geçmişi; yapılandırma hashleri; uyumluluk için chain-of-custody — satır-seviyesi bütünlük per-line `_hmac` + `prev_hash` ile in-band'dir, ayrı bir `.sha256` sidecar yok | -| Per-output-dir salt | `/.forgelm_audit_salt` | Düşük-entropi identifier'ların salt'sız SHA-256'sı brute-force edilebilir — salt audit hash'ine wordlist direnci veren secret'tir | +| Per-output-dir salt | `/.forgelm_audit_salt` | Düşük-entropi identifier'ların salt'sız SHA-256'sı brute-force edilebilir — salt hazır bir wordlist'i işe yaramaz kılar, ancak hem log'a hem salt'a sahip olan biri tahminleri yine de deneyebilir | | Eğitim corpusu | (operatör-tedarikli; tipik olarak `data/*.jsonl`) | PII; ticari sırlar; müşteri verisi | | Quickstart-rendered config | (operatör path'i) | HF token'ları, webhook URL'leri, secret env-var isimleri | | Uyumluluk paketi (Annex IV) | `/compliance/` ZIP | Yukarıdaki her artefaktın aggregate formu | @@ -138,9 +138,10 @@ karşı offline dictionary saldırısı çalıştırmasına izin verir. - Sadece audit log'un şifreleme posture'ı ile eşleşen substrate'lere yedeklenir (HİÇBİR ZAMAN debug bucket'a plaintext). - Sadece karşılık gelen `` decommissioned olduğunda - rotate edilir — output-dir-ortasında rotate etmek `forgelm - verify-audit`'in önceki olaylar için salted-hash varsayımlarını - geçersiz kılar. + rotate edilir — output-dir-ortasında rotate etmek önceki ve sonraki + digest'ler arasındaki korelasyonu bozar (sonraki bir `forgelm + reverse-pii` sorgusu önceki bir `forgelm purge` olayıyla eşleşmez). + Salt'ı kullanmayan `forgelm verify-audit`'i etkilemez. **Defence-in-depth:** `FORGELM_AUDIT_SECRET`'i KMS'inizden export et böylece on-disk salt iki-bileşenli anahtarın yarısı haline gelir. @@ -215,9 +216,11 @@ ForgeLM kendisi şifreleme uygulamaz ama ŞUNLARI YAPAR: 3. **Şifreleme-sonrası bütünlüğü doğrula.** `forgelm verify-audit` chain-after-decryption'ın chain-as-emitted ile eşleştiğini teyit eder; substrate-seviyesi herhangi bir bozulma tespit edilebilirdir. -4. **Düşük-entropi identifier'ları hash'le.** Salted SHA-256, audit - chain'in kendisinin atıl-durumda gizli olması için şifrelenmesine - ihtiyaç olmadığı anlamına gelir — yalnız salt'ın. +4. **Düşük-entropi identifier'ları takma adlandır (pseudonymise).** Salted SHA-256 + bir identifier'ı anonim bir değere değil, bir takma ada dönüştürür (GDPR + Madde 4(5)): log'a ve salt'a sahip olan herkes tahmin ettiği bir identifier'dan + digest'i yeniden hesaplayabilir. Audit log'u ve salt'ı aynı standartta + şifreleyin ve digest'leri kişisel veri sayın. ## 6. Doğrulama checklist'i diff --git a/docs/qms/encryption_at_rest.md b/docs/qms/encryption_at_rest.md index 2c74a433..579bdf7b 100644 --- a/docs/qms/encryption_at_rest.md +++ b/docs/qms/encryption_at_rest.md @@ -24,7 +24,7 @@ The following ForgeLM-produced asset classes: | Model weights — final | `/final_model/` | Memorisation of training data; competitive moat | | Model weights — staging | `/final_model.staging./` | Same as final, plus pre-approval state | | Audit log | `/audit_log.jsonl` + `audit_log.jsonl.manifest.json` (genesis pin) | Operator identity history; configuration hashes; chain-of-custody for compliance — line-level integrity is in-band via per-line `_hmac` + `prev_hash`, no separate `.sha256` sidecar | -| Per-output-dir salt | `/.forgelm_audit_salt` | Unsalted SHA-256 of low-entropy identifiers (emails, phones) is brute-forcible — the salt is the secret that gives the audit hash its wordlist resistance | +| Per-output-dir salt | `/.forgelm_audit_salt` | Unsalted SHA-256 of low-entropy identifiers (emails, phones) is brute-forcible — the salt makes a precomputed wordlist useless, but anyone who holds both the log and the salt can still test guesses | | Training corpus | (operator-supplied; typically `data/*.jsonl`) | PII; trade secrets; client data | | Quickstart-rendered config | (operator path) | HF tokens, webhook URLs, secret env-var names | | Compliance bundle (Annex IV) | `/compliance/` ZIP | Aggregated form of every above artefact | @@ -134,8 +134,10 @@ to recover the cleartext identifiers ForgeLM intentionally hashed. - Backed up only to substrates that match the audit log's encryption posture (NEVER plaintext to a debug bucket). - Rotated only when the corresponding `` is decommissioned - — rotating mid-output-dir invalidates `forgelm verify-audit`'s - salted-hash assumptions for prior events. + — rotating mid-output-dir breaks the correlation between earlier and + later digests (a later `forgelm reverse-pii` query no longer matches an + earlier `forgelm purge` event). It does not affect + `forgelm verify-audit`, which does not use the salt. **Defence-in-depth:** export `FORGELM_AUDIT_SECRET` from your KMS so the on-disk salt becomes one half of a two-component key. An attacker @@ -206,9 +208,11 @@ ForgeLM does not implement encryption itself but DOES: 3. **Verify post-encryption integrity.** `forgelm verify-audit` confirms the chain-after-decryption matches the chain-as-emitted; any substrate-level corruption is detectable. -4. **Hash low-entropy identifiers.** Salted SHA-256 means the audit - chain itself does not need encryption to be confidential at rest - — only the salt does. +4. **Pseudonymise low-entropy identifiers.** Salted SHA-256 turns an + identifier into a pseudonym (GDPR Article 4(5)), not an anonymous value: + anyone holding the log and the salt can recompute a digest from a + guessed identifier. Encrypt the audit log and the salt to the same + standard and treat the digests as personal data. ## 6. Verification checklist diff --git a/docs/qms/sop_data_management-tr.md b/docs/qms/sop_data_management-tr.md index 782550cd..3ca07953 100644 --- a/docs/qms/sop_data_management-tr.md +++ b/docs/qms/sop_data_management-tr.md @@ -112,7 +112,7 @@ Manuel kontroller: - Eğitim verisi sürüm kontrollü veya immutable depolamada saklanır - SHA-256 fingerprint her eğitim koşumu için `data_provenance.json`'da kaydedilir -- Eğitim verisi en az **5 yıl** model artefaktlarıyla birlikte saklanır +- Eğitim verisi model artefaktlarıyla birlikte yalnızca saklama politikanızın belirlediği ve gerekçelendirdiği süre boyunca saklanır: kişisel veri yalnızca gerektiği kadar tutulabilir (GDPR Madde 5(1)(e)) ve EU AI Act Madde 18 gibi bir dokümantasyon saklama yükümlülüğü kişisel veriyi saklamayı tek başına haklı kılmaz. `data_provenance.json` içindeki SHA-256 parmak izi satır içeriği taşımaz ve corpus'tan daha uzun süre saklanabilir - Erişim yetkili ML ekibi üyeleriyle sınırlıdır ## 7. İnceleme diff --git a/docs/qms/sop_data_management.md b/docs/qms/sop_data_management.md index 6bf825cc..6bd4713b 100644 --- a/docs/qms/sop_data_management.md +++ b/docs/qms/sop_data_management.md @@ -111,7 +111,7 @@ Manual checks: - Training data stored in version-controlled or immutable storage - SHA-256 fingerprint recorded in `data_provenance.json` for every training run -- Retain training data for minimum **5 years** alongside model artifacts +- Retain training data alongside model artifacts only for the period your retention policy sets and justifies: personal data may be kept only as long as necessary (GDPR Article 5(1)(e)), and a documentation-retention duty such as EU AI Act Article 18 does not by itself justify keeping personal data. The SHA-256 fingerprint in `data_provenance.json` holds no row content and can be kept longer than the corpus - Access restricted to authorized ML team members ## 7. Review diff --git a/docs/qms/sop_model_training-tr.md b/docs/qms/sop_model_training-tr.md index 58031c32..6f373fdd 100644 --- a/docs/qms/sop_model_training-tr.md +++ b/docs/qms/sop_model_training-tr.md @@ -63,7 +63,7 @@ Manuel: ### 4.5 Kayıt Tutma -- Uyumluluk artefaktları en az **5 yıl** saklanır (veya regülasyon gerektirdiği kadar) +- Uyumluluk artefaktları saklama politikanızın belirlediği süre boyunca saklanır (örneğin EU AI Act Madde 18, yüksek riskli sistem sağlayıcılarından teknik dokümantasyonu sistem piyasaya sürüldükten sonra 10 yıl saklamalarını ister; size neyin uygulandığını kontrol edin) - Kanıt paketi: `forgelm --config job.yaml --compliance-export ./archive/` - Mümkünse immutable/append-only depolamada sakla diff --git a/docs/qms/sop_model_training.md b/docs/qms/sop_model_training.md index 564956a1..e8081889 100644 --- a/docs/qms/sop_model_training.md +++ b/docs/qms/sop_model_training.md @@ -62,7 +62,7 @@ Manual: ### 4.5 Record Retention -- Compliance artifacts retained for minimum **5 years** (or as required by regulation) +- Compliance artifacts retained for the period your retention policy sets (for example, EU AI Act Article 18 asks providers of high-risk systems to keep technical documentation for 10 years after the system is placed on the market; check what applies to you) - Evidence bundle: `forgelm --config job.yaml --compliance-export ./archive/` - Store in immutable/append-only storage if available diff --git a/docs/reference/configuration-tr.md b/docs/reference/configuration-tr.md index a073513e..1e79896a 100644 --- a/docs/reference/configuration-tr.md +++ b/docs/reference/configuration-tr.md @@ -232,7 +232,7 @@ training: | `track_categories` | bool | `false` | Llama Guard S1-S14 zarar kategorilerini ayrıştır | | `severity_thresholds` | dict | `null` | Ciddiyet bazlı sınırlar: `{"critical": 0, "high": 0.01}` | | `batch_size` | int | `8` | Güvenlik değerlendirmesi için batched generation boyutu. `1` batching'i devre dışı bırakır; geniş VRAM'de throughput için artırın, küçük VRAM'de OOM riskini azaltmak için düşürün. | -| `include_eval_samples` | bool | `false` | Ham `prompt` / `response` / `raw_verdict` dizgelerini `safety_results.json`'a yazar. GDPR / EU AI Act Madde 10 gizliliği için **varsayılan olarak kapalı** — adversarial prompt'lar ve yanıtlar hassas içerik açığa çıkarabilir; `raw_verdict` (`classifier_mode: generation` altında guard'ın kendi ürettiği metin) ise guard yanlış yapılandırıldığında probe'u yankılayabilir veya sürdürebilir. Yalnızca hata ayıklama için açın. | +| `include_eval_samples` | bool | `false` | Ham `prompt` / `response` / `raw_verdict` dizgelerini `safety_results.json`'a yazar. GDPR Madde 25(2) varsayılan olarak veri koruma ilkesi gereği **varsayılan olarak kapalı** — adversarial prompt'lar ve yanıtlar hassas içerik açığa çıkarabilir; `raw_verdict` (`classifier_mode: generation` altında guard'ın kendi ürettiği metin) ise guard yanlış yapılandırıldığında probe'u yankılayabilir veya sürdürebilir. Yalnızca hata ayıklama için açın. | #### `evaluation.llm_judge` (İsteğe bağlı) @@ -247,7 +247,7 @@ training: | `min_score` | float | `5.0` | Minimum ortalama puan (1-10) | | `min_valid_fraction` | float | `0.8` | Ortalamanın kanıt sayılabilmesi için parse edilebilir judge puanı üretmesi gereken eval prompt oranı. Ortalama **yalnızca** parse edilebilen puanlar üzerinden hesaplanır — başarısız bir judge çağrısı düşük sayılmaz, atılır — dolayısıyla bu taban olmadan iki yüz prompt içinden tek bir `9` puanı `min_score: 8` kapısını geçiyordu. Bu oranın altında kapı, `Average judge score … below minimum`'dan ayrı bir `Insufficient valid judge evidence` gerekçesiyle başarısız olur. `0.0`, 0.11.0'a kadarki davranışa döner. | | `batch_size` | int | `8` | LLM-hakim turunda puanlanan (prompt, completion) çift sayısı. `1` batching'i devre dışı bırakır. | -| `include_eval_samples` | bool | `false` | Ham eval `prompt`, `response` ve hakim `reason` dizgelerini `judge_results.json`'a yazar. GDPR / EU AI Act Madde 10 gizliliği için **varsayılan olarak kapalı** — hakim gerekçesi eval setinden PII alıntılayabilir. Yalnızca hata ayıklama için açın. | +| `include_eval_samples` | bool | `false` | Ham eval `prompt`, `response` ve hakim `reason` dizgelerini `judge_results.json`'a yazar. GDPR Madde 25(2) varsayılan olarak veri koruma ilkesi gereği **varsayılan olarak kapalı** — hakim gerekçesi eval setinden PII alıntılayabilir. Yalnızca hata ayıklama için açın. | > **Hakim girdisi kırpma:** her puanlama prompt'u oluşturulurken hakim, > eval prompt'unun en fazla ilk **500 karakterini** ve model yanıtının en fazla @@ -259,33 +259,34 @@ training: > için `min_score` ayarlarken bunu göz önünde bulundurun. > > **Kaldırıldı:** `evaluation.staging_ttl_days`, -> [`retention.staging_ttl_days`](#retention-isteğe-bağlı--gdpr-madde-17-silme-ufukları) +> [`retention.staging_ttl_days`](#retention-isteğe-bağlı--saklama-ufukları) > tarafından devralınmış ve v0.8.0'da kaldırılmıştır. `retention.staging_ttl_days` > kullanın; eski anahtarı hâlâ set eden YAML dosyaları `EXIT_CONFIG_ERROR` ile > config-load başarısızlığına yol açar. --- -## `retention` (İsteğe bağlı — GDPR Madde 17 silme ufukları) +## `retention` (İsteğe bağlı — saklama ufukları) -Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarını -belirler. Ufuklar, GDPR Madde 5(1)(e) "saklama sınırlaması"nın (sabit süre yoktur: -veri yalnızca gerektiği kadar tutulabilir) uygulanmasına yardımcı olan ve Madde 17 -kapsamındaki silme işlemlerini destekleyen, operatör tarafından belirlenen saklama -sınırlarıdır (talepler gereksiz gecikmeden kaçınılarak, en geç Madde 12(3)'teki bir -aylık süre içinde yanıtlanır; bu süre iki ay daha uzatılabilir). Değerleri kendi -gereklilik (necessity) değerlendirmenize göre belirleyin; varsayılanlar ForgeLM -tercihleridir, yasal süre değildir. Ufuklar **raporlanır**, zorlanmaz: -`forgelm purge --check-policy` ufku aşanları listeler ve kaç tane bulursa bulsun 0 ile çıkar (yüklenemeyen `--config` 1 ile çıkar); -hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `enforce` anahtarı -ayrılmıştır: henüz hiçbir komut onu okumaz. +Uyumluluk, eğitim ve değerlendirme artefaktları için saklama ufuklarıdır. Bunlar, +GDPR Madde 5(1)(e) saklama sınırlaması ilkesini (kişisel veri yalnızca gerektiği +kadar tutulabilir; Tüzük sabit bir süre belirlemez) uygulamanıza yardımcı olan, +sizin belirlediğiniz sınırlardır. Değerleri kendi gereklilik (necessity) +değerlendirmenize göre seçin: varsayılanlar ForgeLM tercihleridir, yasal süre +değildir. + +Ufuklar **raporlanır, zorlanmaz**: `forgelm purge --check-policy` ufku aşanları +listeler ve kaç tane bulursa bulsun 0 ile çıkar (yüklenemeyen `--config` 1 ile +çıkar); hiçbir şey otomatik silinmez, engellenmez ya da zamanlanmaz. `retention:` +bloğu yoksa hiçbir şey kontrol edilmez; `retention: {}` varsayılanları devreye alır. +`enforce` anahtarı ayrılmıştır: henüz hiçbir komut onu dikkate almaz. | Alan | Tip | Varsayılan | Açıklama | |------|-----|-----------|----------| | `audit_log_retention_days` | int | `1825` (~5 yıl) | `forgelm purge --check-policy`'nin `audit_log.jsonl` dosyasını "süresi geçmiş" olarak raporladığı gün sayısı (Madde 5(1)(e)). 5 yıllık varsayılan bir ForgeLM tercihidir, yasal süre değildir. `0` bu raporu kapatır (hiçbir şey silinmez); log'u yalnızca girdiler gerekli kaldığı sürece ya da adlandırabileceğiniz belirli bir yasal saklama yükümlülüğü gerektirdiği sürece tutun. | | `staging_ttl_days` | int | `7` | `forgelm purge --check-policy`'nin bir `final_model.staging./` dizinini (ve eski `final_model.staging/`'i) "süresi geçmiş" olarak raporladığı yaş (koşumun ilk audit event'inden, yoksa audit log'un ilk event'inden, o da yoksa dizinin mtime'ından); hiçbir şey otomatik silinmez ve `forgelm reject` kararının zamanı kullanılmaz. `0` bu raporu kapatır. v0.8.0'da kaldırılan `evaluation.staging_ttl_days` yerine geçer. | | `ephemeral_artefact_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/compliance/` ve `/data_audit_report.json`'ı "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` bu raporu kapatır. | -| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. `0` bu raporu kapatır. | +| `raw_documents_retention_days` | int | `90` | `forgelm purge --check-policy`'nin `/raw_documents/` veya `/ingestion_output/`'u (ingest edilmiş PDF / DOCX / EPUB / TXT / Markdown) "süresi geçmiş" olarak raporladığı yaş (gün; audit log'un ilk event'inden, yoksa mtime'dan); hiçbir şey silinmez. ForgeLM bu dizinleri oluşturmaz; ufuk yalnızca orada tuttuğunuz kaynak dokümanlara uygulanır. `0` bu raporu kapatır. | | `enforce` | string | `"log_only"` | Ayrılmış politika modu. Hiçbir komut onu kullanmaz: `"log_only"`, `"warn_on_excess"` ve `"block_on_excess"` kabul edilir ve aynı davranır; bir config son ikisinden birini ayarlarsa eğitim ve `--dry-run` uyarır. | **Paylaşılan çıktı dizinleri.** `compliance/`, `data_audit_report.json` ve ham-belge dizinlerinin yaşı, dosyanın en son yazıldığı andan değil, audit log'un *ilk* event'inden sayılır. Birçok koşum için yeniden kullanılan bir dizinde bugün yazılmış bir paket de ufku aşmış olarak raporlanabilir: raporun listelediği bir şeyi silmeden önce dosyanın kendi tarihine bakın. diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index cd1aee6a..00abe768 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -233,7 +233,7 @@ across retries. Each retry attempt is logged to the audit trail. | `track_categories` | bool | `false` | Parse Llama Guard S1-S14 harm categories | | `severity_thresholds` | dict | `null` | Per-severity limits: `{"critical": 0, "high": 0.01, "medium": 0.05}` | | `batch_size` | int | `8` | Batched generation size for safety evaluation. `1` disables batching; raise for throughput on large VRAM, lower to reduce OOM risk on small VRAM. | -| `include_eval_samples` | bool | `false` | Persist raw `prompt` / `response` / `raw_verdict` strings to `safety_results.json`. **Off by default** for GDPR / EU AI Act Art. 10 privacy — adversarial prompts and responses may surface sensitive content, and `raw_verdict` (the guard's own generated text under `classifier_mode: generation`) can echo or continue the probe when the guard is misconfigured. Opt in only for debugging. | +| `include_eval_samples` | bool | `false` | Persist raw `prompt` / `response` / `raw_verdict` strings to `safety_results.json`. **Off by default** for GDPR Art. 25(2) data protection by default — adversarial prompts and responses may surface sensitive content, and `raw_verdict` (the guard's own generated text under `classifier_mode: generation`) can echo or continue the probe when the guard is misconfigured. Opt in only for debugging. | #### `evaluation.llm_judge` (Optional) @@ -248,7 +248,7 @@ across retries. Each retry attempt is logged to the audit trail. | `min_score` | float | `5.0` | Minimum average score (1-10) | | `min_valid_fraction` | float | `0.8` | Fraction of eval prompts that must yield a parseable judge score before the average counts as evidence. The average is computed over parseable scores **only** — a failed judge call is dropped, not counted low — so without this floor a single score of `9` out of two hundred prompts cleared a `min_score: 8` gate. Below it the gate fails with an `Insufficient valid judge evidence` reason, distinct from `Average judge score … below minimum`. `0.0` restores the behaviour through 0.11.0. | | `batch_size` | int | `8` | Number of (prompt, completion) pairs scored per LLM-judge round. `1` disables batching. | -| `include_eval_samples` | bool | `false` | Persist raw eval `prompt`, `response`, and judge `reason` strings to `judge_results.json`. **Off by default** for GDPR / EU AI Act Art. 10 privacy — judge reasoning can quote PII from the eval set. Opt in only for debugging. | +| `include_eval_samples` | bool | `false` | Persist raw eval `prompt`, `response`, and judge `reason` strings to `judge_results.json`. **Off by default** for GDPR Art. 25(2) data protection by default — judge reasoning can quote PII from the eval set. Opt in only for debugging. | > **Judge input truncation:** when building each scoring prompt the judge > sees at most the first **500 characters of the eval prompt** and the first @@ -260,31 +260,33 @@ across retries. Each retry attempt is logged to the audit trail. > for long-form fine-tunes. > > **Removed:** `evaluation.staging_ttl_days` was superseded by -> [`retention.staging_ttl_days`](#retention-optional--gdpr-article-17-erasure-horizons) +> [`retention.staging_ttl_days`](#retention-optional--retention-horizons) > and was removed in v0.8.0. Use `retention.staging_ttl_days`; YAML files that > still set the legacy key will fail config-load with `EXIT_CONFIG_ERROR`. --- -## `retention` (Optional — GDPR Article 17 erasure horizons) +## `retention` (Optional — retention horizons) -Defines maximum retention horizons for compliance, training, and evaluation -artefacts. Horizons are operator-set retention limits that help implement GDPR -Article 5(1)(e) "storage limitation" (no fixed period: data may be kept only as -long as necessary) and support erasure handling under Article 17 (requests are -answered without undue delay, within the Article 12(3) one-month period, extendable by two further months). Choose -values from your own necessity assessment; the defaults are ForgeLM choices, not -legal periods. The horizons are **reported**, not enforced: `forgelm purge ---check-policy` lists what is past them and exits 0 however many it finds (an unloadable `--config` exits 1), and nothing is -deleted, blocked or scheduled automatically. The `enforce` knob is reserved: no -command reads it yet. +Retention horizons for compliance, training and evaluation artefacts. They are +limits you set to help apply the GDPR Article 5(1)(e) storage-limitation +principle (personal data may be kept only as long as necessary; the Regulation +sets no fixed period). Choose values from your own necessity assessment: the +defaults are ForgeLM choices, not legal periods. + +The horizons are **reported, not enforced**: `forgelm purge --check-policy` +lists what is past them and exits 0 however many it finds (an unloadable +`--config` exits 1), and nothing is deleted, blocked or scheduled +automatically. Without a `retention:` block nothing is checked at all; +`retention: {}` switches the defaults on. The `enforce` knob is reserved: no +command acts on it yet. | Field | Type | Default | Description | |-------|------|---------|-------------| | `audit_log_retention_days` | int | `1825` (~5 years) | Days after which `forgelm purge --check-policy` reports `audit_log.jsonl` as overdue (Article 5(1)(e)). The 5-year default is a ForgeLM choice, not a legal period. Set to `0` to stop reporting it (nothing is ever deleted); keep it only for as long as the entries stay necessary or a specific legal retention duty you can name requires it. | | `staging_ttl_days` | int | `7` | Age (from the run's first audit event, else the audit log's first event, else the directory's mtime) after which `forgelm purge --check-policy` reports a `final_model.staging./` directory (and the legacy `final_model.staging/`) as overdue; nothing is deleted automatically and the time of a `forgelm reject` decision is not used. Set to `0` to stop reporting it. Replaces the removed `evaluation.staging_ttl_days` (removed in v0.8.0). | | `ephemeral_artefact_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/compliance/` and `/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to stop reporting it. | -| `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to `0` to stop reporting it. | +| `raw_documents_retention_days` | int | `90` | Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or `/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. ForgeLM does not create these directories, so the horizon applies only to source documents you keep there. Set to `0` to stop reporting it. | | `enforce` | string | `"log_only"` | Reserved policy mode. No command acts on it: `"log_only"`, `"warn_on_excess"` and `"block_on_excess"` are all accepted and behave the same, and training and `--dry-run` warn when a config sets either of the last two. | **Shared output directories.** `compliance/`, `data_audit_report.json` and the raw-document directories are aged from the audit log's *first* event, not from when the file was last written. In a directory reused for many runs, a bundle written today can be reported as past its horizon: check the file's own date before deleting anything the report lists. diff --git a/docs/reference/purge_subcommand-tr.md b/docs/reference/purge_subcommand-tr.md index 4214a26b..b5774356 100644 --- a/docs/reference/purge_subcommand-tr.md +++ b/docs/reference/purge_subcommand-tr.md @@ -11,11 +11,11 @@ Deployer akışı (DSAR ticket → CLI çağrısı → doğrulama) için bkz. [` ```text forgelm purge --row-id ID --corpus PATH [--row-matches {one,all}] - [--output-dir DIR] [--justification TEXT] [--dry-run] - [--output-format {text,json}] + [--output-dir DIR] [--config PATH] [--justification TEXT] + [--dry-run] [--output-format {text,json}] forgelm purge --run-id RUN_ID --kind {staging,artefacts} - --output-dir DIR [--justification TEXT] [--dry-run] + [--output-dir DIR] [--justification TEXT] [--dry-run] [--output-format {text,json}] forgelm purge --check-policy --config PATH [--output-dir DIR] diff --git a/docs/reference/purge_subcommand.md b/docs/reference/purge_subcommand.md index dbe74ec6..aa209377 100644 --- a/docs/reference/purge_subcommand.md +++ b/docs/reference/purge_subcommand.md @@ -11,11 +11,11 @@ For the deployer-flow walkthrough (DSAR ticket → CLI invocation → verificati ```text forgelm purge --row-id ID --corpus PATH [--row-matches {one,all}] - [--output-dir DIR] [--justification TEXT] [--dry-run] - [--output-format {text,json}] + [--output-dir DIR] [--config PATH] [--justification TEXT] + [--dry-run] [--output-format {text,json}] forgelm purge --run-id RUN_ID --kind {staging,artefacts} - --output-dir DIR [--justification TEXT] [--dry-run] + [--output-dir DIR] [--justification TEXT] [--dry-run] [--output-format {text,json}] forgelm purge --check-policy --config PATH [--output-dir DIR] diff --git a/docs/standards/localization.md b/docs/standards/localization.md index bbac0955..53c565b9 100644 --- a/docs/standards/localization.md +++ b/docs/standards/localization.md @@ -137,6 +137,9 @@ For the parts we do translate, keep terminology consistent. A small glossary: | drift (model / data) | drift | Kept in English, as in the guides and the site. *Kayma* names a schema drift (*şema kayması*) or a threshold or preference shift | | metrics exporter | exporter | Kept in English, like *endpoint* and *webhook*; suffixes take an apostrophe (*exporter'ı*) | | alert / notification / warning | alarm / bildirim / uyarı | A monitoring or threshold alert (drift, cost) is *alarm*; a webhook's chat message is *bildirim*, even where English calls it an alert; a log warning is *uyarı* | +| data subject | veri sahibi | The GDPR term; where a page quotes KVKK (Law 6698) use the law's own *ilgili kişi* | +| controller / processor | veri sorumlusu / veri işleyen | Not *kontrolör* | +| salt (hash salt) / digest | salt / digest | Kept in English like *hash*; not *tuz* / *özet* | When in doubt, look at sibling TR docs and match the most frequent choice. Don't invent new terminology mid-series. diff --git a/docs/usermanuals/en/compliance/gdpr-erasure.md b/docs/usermanuals/en/compliance/gdpr-erasure.md index aa015fe1..903bf25c 100644 --- a/docs/usermanuals/en/compliance/gdpr-erasure.md +++ b/docs/usermanuals/en/compliance/gdpr-erasure.md @@ -96,7 +96,7 @@ The two primitives are intentionally separate: ## Exit codes -Both subcommands follow the project's `0/1/2/3/4` contract: +Both subcommands use the first three codes of the project's public exit-code contract (0-6): | Code | Meaning | |---|---| @@ -104,7 +104,7 @@ Both subcommands follow the project's `0/1/2/3/4` contract: | 1 | Config error: a missing `--kind`, `--row-id` without `--corpus`, malformed regex, no staging directory / compliance file matching `--run-id`, a `--config` that cannot be loaded. | | 2 | Runtime error: I/O failure, permission denied, unwritable `--audit-dir`, an argument-parser usage error (two modes together, `--kind logs`), ReDoS SIGALRM timeout, atomic-rename failure. | -Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of either subcommand's surface. +Codes 3 to 6 (`EXIT_EVAL_FAILURE`, `EXIT_AWAITING_APPROVAL`, `EXIT_WIZARD_CANCELLED`, `EXIT_INTEGRITY_FAILURE`) are not part of either subcommand's surface. ## Common pitfalls @@ -121,7 +121,7 @@ Codes 3 (`EXIT_EVAL_FAILURE`) and 4 (`EXIT_AWAITING_APPROVAL`) are not part of e ::: :::tip -**Wire `forgelm verify-audit` into your DSAR closeout.** Once the erasure / access event is in the chain, run `forgelm verify-audit /audit_log.jsonl --require-hmac` and attach the verification output to the DSAR ticket. The auditor will read it. +**Wire `forgelm verify-audit` into your DSAR closeout.** Once the erasure / access event is in the chain, run `forgelm verify-audit /audit_log.jsonl` and attach the verification output to the DSAR ticket. Add `--require-hmac` only if `FORGELM_AUDIT_SECRET` was set when the chain was written and is set now: the flag exits 1 without the secret and exits 6 if any line lacks an HMAC tag, so on a chain written without the secret it fails by design. The auditor will read it. ::: ## See also diff --git a/docs/usermanuals/en/compliance/gdpr.md b/docs/usermanuals/en/compliance/gdpr.md index b3ad4323..2ff22d6e 100644 --- a/docs/usermanuals/en/compliance/gdpr.md +++ b/docs/usermanuals/en/compliance/gdpr.md @@ -19,7 +19,7 @@ GDPR (Europe) and KVKK (Türkiye) impose data protection requirements on persona | **Integrity & confidentiality** | GDPR Art. 5(1)(f) | Protect against unauthorised access. | | **Accountability** | GDPR Art. 5(2) | You must be able to demonstrate compliance. | -KVKK (Türkiye, Law 6698) sets similar principles in its Article 4 and gives data subjects access and erasure rights in Articles 11 and 7; it has no separate accountability principle, and its security duties sit in Article 12. +KVKK (Türkiye, Law 6698) sets similar principles in its Article 4. Article 11 gives data subjects the right to learn whether their data are processed and to request erasure or destruction (Article 11(1)(e), under the conditions of Article 7, which also obliges the controller to erase, destroy or anonymise data once the reason for processing ends), and the controller must conclude a request within thirty days at the latest (Article 13(2)). KVKK has no separate accountability principle, and its security duties sit in Article 12. ## How ForgeLM addresses each @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml --output-dir ` lists the artefacts in that directory that are past their horizon and exits 0 however many it finds (without `--output-dir` it scans the current directory, not `training.output_dir`). Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) +`forgelm purge --check-policy --config config.yaml --output-dir ` lists the artefacts in that directory that are past their horizon and exits 0 however many it finds (without `--output-dir` it scans the current directory, not `training.output_dir`). Nothing is deleted automatically and nothing is written to the audit log for it; deleting what is overdue is your storage layer's job. (`retention.enforce` is reserved: no command acts on it yet.) ForgeLM does not create `raw_documents/` or `ingestion_output/`, so `raw_documents_retention_days` has an effect only if you keep source documents under `/raw_documents/` or `ingestion_output/`. ## Data subject requests @@ -70,7 +70,7 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, is a valid answer. The printed value is the first 16 hex digits of the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning; with a `data/*.jsonl` glob that puts `audit_log.jsonl` among your corpus files, so always pass `--output-dir`. It finds what is literally in the files you point it at: the match is case-sensitive and compares raw file text, so search spelling and case variants as well (`--type custom --query '(?i)…'`), and note that a corpus written with ASCII escapes such as `\u00e7` hides non-ASCII names from a literal query. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. +The scan reports the file and line where the identifier appears (the snippets are raw corpus text, so do not redirect them to a persistent log); zero matches, as above, means only that the literal text is in none of the files you scanned. It does not by itself show that you hold no data about the person, and an Article 15 response needs more than a search result: a copy of the data and the Article 15(1)(a)-(h) information (see [GDPR Erasure & Access](#/compliance/gdpr-erasure)). The printed value is the first 16 hex digits of the salted identifier hash that the audit event records, not the identifier. `--output-dir` is where ForgeLM keeps the per-directory salt (`.forgelm_audit_salt`) and appends the `data.access_request_query` event: give it the same directory you give `forgelm purge` so both events join one chain. Without it, both are written next to the first corpus file and ForgeLM logs a warning; with a `data/*.jsonl` glob that puts `audit_log.jsonl` among your corpus files, so always pass `--output-dir`. It finds what is literally in the files you point it at: the match is case-sensitive and compares raw file text, so search spelling and case variants as well (`--type custom --query '(?i)…'`), and note that a corpus written with ASCII escapes such as `\u00e7` hides non-ASCII names from a literal query. Masking only removes the identifier formats it detects, so a name or an undetected variant can remain, and the raw source documents are not covered by a scan of the masked JSONL: search those separately. ForgeLM cannot tell you what a trained model remembers. ### Right to erasure (Art. 17) @@ -107,7 +107,7 @@ finally: os.remove(tmp) ``` -Train from the file with ids. `forgelm reverse-pii` reports the file and line where the identifier occurs; read that row's `id` from the file and pass it to `forgelm purge --row-id`. Re-assigning ids after training breaks the link between the audit log and the corpus, so assign once. +Train from the file with ids. `forgelm reverse-pii` reports the file and line where the identifier occurs; read that row's `id` from the file and pass it to `forgelm purge --row-id`. Re-assigning ids after training breaks the link between the audit log and the corpus, so assign once. Once you have checked the new file, move or delete the original `data/train.jsonl`: it still holds every row without an id, so a `data/*.jsonl` glob matches both copies, `reverse-pii` reports the person in each, and `purge` edits only the file you name with `--corpus`. ### Model-level erasure diff --git a/docs/usermanuals/en/compliance/human-oversight.md b/docs/usermanuals/en/compliance/human-oversight.md index bdba2328..36ceb1ab 100644 --- a/docs/usermanuals/en/compliance/human-oversight.md +++ b/docs/usermanuals/en/compliance/human-oversight.md @@ -95,7 +95,7 @@ ForgeLM does **not** ship a built-in quorum gate. There is no `approval.quorum` ## Timeouts -ForgeLM does **not** time out staged runs. There is no `approval.timeout_hours`, no `human_approval.timeout` event, and no auto-fail clock. A staged run waits indefinitely; if the deployer wants to expire stale staging directories, configure `retention.staging_ttl_days` instead — that is a retention report, not an expiry: `forgelm purge --check-policy` lists staging directories older than the horizon, nothing is deleted automatically (remove one with `forgelm purge --run-id --kind staging`), and it never touches the approval gate. +ForgeLM does **not** time out staged runs. There is no `approval.timeout_hours`, no `human_approval.timeout` event, and no auto-fail clock. A staged run waits indefinitely. `retention.staging_ttl_days` does not change that: it only makes `forgelm purge --check-policy` list staging directories older than the horizon. Nothing is deleted or expired automatically, and the report never touches the approval gate. Removing a staging directory yourself (`forgelm purge --run-id --kind staging`) discards the pending model and leaves the approval undecided, so record a `forgelm reject` first if you want the decision on the audit log. ## Inspecting pending runs @@ -141,7 +141,7 @@ A `--show` against a granted / rejected run prints the full timeline (request ::: :::warn -**No quorum for shipping decisions.** For high-risk production deployments, single-reviewer approval is insufficient. Always require quorum >= 2. +**No quorum for shipping decisions.** ForgeLM records one approver per decision. If your risk assessment calls for more than one reviewer, enforce it outside ForgeLM as described above (Article 14(5) of the EU AI Act asks for two natural persons to verify an identification only for the remote biometric identification systems in Annex III point 1(a)). ::: :::tip diff --git a/docs/usermanuals/en/evaluation/safety.md b/docs/usermanuals/en/evaluation/safety.md index d0cbc39c..95d34ddb 100644 --- a/docs/usermanuals/en/evaluation/safety.md +++ b/docs/usermanuals/en/evaluation/safety.md @@ -187,7 +187,7 @@ The set is a *seed* — augment with your own per-domain probes before treating This example was produced under the default `classifier_mode: generation` (see the warning above): `safety_score` equals `safe_ratio` exactly because `confidence_weighted` degenerates to a safe-ratio average in that mode, and `details[].confidence` is `1.0` for a well-formed `unsafe` verdict — not a real probability. `failure_reason` comes from the always-active absolute gate in `forgelm/safety/_gates.py::_evaluate_safety_gates`: `unsafe_count=3` of `total_count=50` is a 6.00% unsafe ratio, which exceeds the default `max_safety_regression=0.05` (5.00%) ceiling — this gate fires regardless of `scoring_method`. `severity_distribution` always lists all four severity levels (`critical`/`high`/`medium`/`low`), zero-filled, when `track_categories: true`; here both unsafe, well-formed, category-tagged responses were `S5` (defamation), which `forgelm/safety/_types.py`'s `CATEGORY_SEVERITY` maps to `medium`, not `high`. The third unsafe response (counted in `low_confidence_count`) was a malformed guard verdict — scored fail-closed and excluded from the category/severity breakdown. That is what `unscored_count: 1` records, and why `scored_unsafe_count` is `2`: those two fields partition `unsafe_count` into pairs the classifier actually read as unsafe versus pairs it could not answer on at all. Read them before acting on a failure — a `failure_reason` quoting an unsafe ratio always appends the same decomposition in prose, because six malformed verdicts and six genuinely harmful completions otherwise produce an identical sentence. `evaluation_completed` is the field auto-revert keys on: `false` means the run is not usable evidence about the model (the classifier was unusable, or the failure was attributable entirely to unscored pairs), so the model is **failed but kept**, and `forgelm safety-eval` exits `2` rather than `3`. -`category_distribution` / `severity_distribution` are only present when `track_categories: true`. `details[].prompt`, `details[].response` and `details[].raw_verdict` are stripped by default for GDPR / EU AI Act Art. 10 privacy — set `include_eval_samples: true` to persist the raw text for debugging. Note what the third field is: under `classifier_mode: generation` (the default) `raw_verdict` is the guard's own generated output, truncated to 200 characters. It is the field to read when a run reports that the evaluation could not be performed — but a misconfigured guard echoes or continues the adversarial probe rather than answering it, so enabling this switch can write probe text to disk by a second route. `details[].label` stays in the artefact either way; it is rebuilt from a fixed vocabulary rather than sliced out of model output. That vocabulary is closed and space-separated: `safe`, `malformed`, or `unsafe` optionally followed by comma-joined S-codes — `"unsafe"`, `"unsafe S5"`, `"unsafe S1,S5"`. Parse on whitespace, not on a newline: earlier versions of this page showed `"unsafe\nS5"`, which was the pre-rewrite shape produced when the label was sliced out of raw model output. A consumer splitting on `\n` extracts no S-codes from any current artefact. +`category_distribution` / `severity_distribution` are only present when `track_categories: true`. `details[].prompt`, `details[].response` and `details[].raw_verdict` are stripped by default for GDPR Art. 25(2) data protection by default — set `include_eval_samples: true` to persist the raw text for debugging. Note what the third field is: under `classifier_mode: generation` (the default) `raw_verdict` is the guard's own generated output, truncated to 200 characters. It is the field to read when a run reports that the evaluation could not be performed — but a misconfigured guard echoes or continues the adversarial probe rather than answering it, so enabling this switch can write probe text to disk by a second route. `details[].label` stays in the artefact either way; it is rebuilt from a fixed vocabulary rather than sliced out of model output. That vocabulary is closed and space-separated: `safe`, `malformed`, or `unsafe` optionally followed by comma-joined S-codes — `"unsafe"`, `"unsafe S5"`, `"unsafe S1,S5"`. Parse on whitespace, not on a newline: earlier versions of this page showed `"unsafe\nS5"`, which was the pre-rewrite shape produced when the label was sliced out of raw model output. A consumer splitting on `\n` extracts no S-codes from any current artefact. `safety_trend.jsonl` appends one JSON object per run: @@ -211,7 +211,7 @@ This example was produced under the default `classifier_mode: generation` (see t | `track_categories` | bool | `false` | Parse Llama Guard S1-S14 categories per response and surface in the report. | | `severity_thresholds` | `Optional[Dict[str,float]]` | `null` | Per-severity unsafe-ratio ceilings — see Severity thresholds above. | | `batch_size` | int | `8` | Batched generation size for the fine-tuned model's probe responses; `1` disables batching. Does **not** apply to guard-verdict scoring, which is always sequential — see Common pitfalls below. | -| `include_eval_samples` | bool | `false` | Persist raw `prompt` / `response` / `raw_verdict` strings to `safety_results.json`. Off by default for GDPR / EU AI Act Art. 10 privacy — `raw_verdict` is the guard's own generated text under `classifier_mode: generation`. | +| `include_eval_samples` | bool | `false` | Persist raw `prompt` / `response` / `raw_verdict` strings to `safety_results.json`. Off by default for GDPR Art. 25(2) data protection by default — `raw_verdict` is the guard's own generated text under `classifier_mode: generation`. | ## Common pitfalls diff --git a/docs/usermanuals/tr/compliance/gdpr-erasure.md b/docs/usermanuals/tr/compliance/gdpr-erasure.md index 1c837d3a..f1970750 100644 --- a/docs/usermanuals/tr/compliance/gdpr-erasure.md +++ b/docs/usermanuals/tr/compliance/gdpr-erasure.md @@ -52,8 +52,8 @@ Audit zincirine altı event yazılır (bkz. [GitHub'daki Audit Event Kataloğu]( - **Audit log'unu silmez.** Tool, silmeyi geçmişi yeniden yazmak yerine yeni bir event olarak kaydeder ve audit log'dan hiçbir girdiyi silmez. Veri sahibini adlandıran girdilerin tutulup tutulamayacağı veri sorumlusunun kararıdır ve belgelenmelidir. Bir Madde 17(1) gerekçesi geçerliyse girdiler yalnızca bir Madde 17(3) istisnasının kapsadığı yerde kalır: 17(3)(b), yalnızca işlemenin veri sorumlusunun tabi olduğu AB veya üye devlet hukukundan doğan bir yasal yükümlülüğe uyum için (veya kamu yararına ya da kamu otoritesinin kullanılmasına ilişkin bir görev için) gerekli olduğu ölçüde uygulanır; 17(3)(e) ise hukuki talepleri kapsar. Hiçbir Madde 17(1) gerekçesi geçerli değilse (örneğin girdiler toplandıkları amaç için hâlâ gereklidir) silme gerekmez; ancak talebe yine de Madde 12 süresi içinde gerekçeli bir yanıt verilir ve şikâyet hakkı bildirilir (Madde 12(4)). - **İşaretsiz legacy `final_model.staging/` dizinini silmez.** `--kind staging`, `final_model.staging./` dizinini siler; legacy `final_model.staging/` dizinini yalnızca içinde run id'ye eşit bir `staging_run_id` dosyası varsa siler. ForgeLM bu işaretçiyi yazmaz, bu yüzden işaretsiz bir legacy dizini kendiniz silin. - **`--kind artefacts` ile standart compliance paketini bulmaz.** Trainer run id içermeyen sabit dosya adları yazar (`compliance_report.json`, `training_manifest.yaml`, ...); resolver ise yalnızca run id'yi embed eden dosya adlarını eşler. Normal bir çıktı dizininde komut hiçbir şey bulmaz, `data.erasure_failed` (`NoMatchingArtefacts`) yayar, 1 ile çıkar ve hiçbir şey silmez; paketi depolama katmanınız üzerinden kaldırın. -- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve ilgili kişi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. -- **Yanıtın tamamını vermez.** Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); ilgili kişiye verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). +- **Downstream tüketicilere bildirim göndermez.** Webhook receiver'ları, dataset mirror'ları ve deploy edilmiş model endpoint'leri operatörün runtime-katmanı sorumluluğudur (GDPR Madde 19, verinin ifşa edildiği her alıcıya silmeyi bildirmeyi ister; bu, imkânsız olmadıkça veya orantısız çaba gerektirmedikçe geçerlidir ve veri sahibi isterse bu alıcıları ona da bildirmek gerekir; Madde 17(2) ise veri sorumlusu veriyi kamuya açtıysa makul adımlar ekler). `data.erasure_warning_external_copies` event'ı açık hatırlatmadır. +- **Yanıtın tamamını vermez.** Eksiksiz bir yanıt, ForgeLM'in yapmadığı adımları da gerektirir: talep edenin kimliğini doğrulamak (Madde 12(6)); talebin alınmasından itibaren bir ay içinde yanıtlamak (Madde 12(3); karmaşık veya çok sayıda talep için iki ay daha uzatılabilir, uzatma ilk ay içinde bildirilir); veri sahibine verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini vermek (Madde 15(3)) ve bunu başkalarının hak ve özgürlüklerini olumsuz etkilemeden yapmak (Madde 15(4)); bir Madde 17(1) gerekçesinin geçerli olup olmadığına ve tuttuğunuz şeyi hangi Madde 17(3) istisnasının (varsa) kapsadığına karar vermek; alıcıları bilgilendirmek (Madde 19); satırların gittiğini doğrulamak için purge sonrasında erişim aramasını yeniden çalıştırmak; ve bir talebi reddederseniz aynı süre içinde gerekçeyi ve şikâyet hakkını bildirmek (Madde 12(4)). Eğitilmiş modelin kişisel veri sayılıp sayılmayacağı ayrı bir sorudur (EDPB Görüş 28/2024). ## Madde 15 — `forgelm reverse-pii` @@ -96,7 +96,7 @@ Chain satırı cleartext identifier'ı asla taşımaz, yalnızca salt'lı bir di ## Exit kodları -İki subcommand da projenin `0/1/2/3/4` sözleşmesine uyar: +İki subcommand da projenin herkese açık exit-code sözleşmesinin (0-6) ilk üç kodunu kullanır: | Kod | Anlamı | |---|---| @@ -104,7 +104,7 @@ Chain satırı cleartext identifier'ı asla taşımaz, yalnızca salt'lı bir di | 1 | Config hatası: eksik `--kind`, `--corpus`'suz `--row-id`, hatalı regex, `--run-id` ile eşleşen staging dizini / compliance dosyası yok, yüklenemeyen bir `--config`. | | 2 | Runtime hatası: I/O başarısızlığı, izin reddedildi, yazılamaz `--audit-dir`, argüman-parser kullanım hatası (birlikte verilen iki mod, `--kind logs`), ReDoS SIGALRM timeout, atomic-rename başarısızlığı. | -Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın da yüzeyinin parçası değildir. +Kod 3 ile 6 arası (`EXIT_EVAL_FAILURE`, `EXIT_AWAITING_APPROVAL`, `EXIT_WIZARD_CANCELLED`, `EXIT_INTEGRITY_FAILURE`) iki subcommand'ın da yüzeyinin parçası değildir. ## Sık hatalar @@ -121,7 +121,7 @@ Kod 3 (`EXIT_EVAL_FAILURE`) ve 4 (`EXIT_AWAITING_APPROVAL`) iki subcommand'ın d ::: :::tip -**`forgelm verify-audit`'i DSAR kapanışınıza wire'layın.** Erasure / access event zincire girdikten sonra `forgelm verify-audit /audit_log.jsonl --require-hmac` çalıştırın ve doğrulama çıktısını DSAR ticket'a iliştirin. Auditor okuyacak. +**`forgelm verify-audit`'i DSAR kapanışınıza wire'layın.** Erasure / access event zincire girdikten sonra `forgelm verify-audit /audit_log.jsonl` çalıştırın ve doğrulama çıktısını DSAR ticket'a iliştirin. `--require-hmac`'i yalnızca zincir yazılırken `FORGELM_AUDIT_SECRET` ayarlıysa ve şimdi de ayarlıysa ekleyin: bayrak secret olmadan 1 ile çıkar ve herhangi bir satırda HMAC etiketi yoksa 6 ile çıkar; yani secret olmadan yazılmış bir zincirde tasarım gereği başarısız olur. Auditor okuyacak. ::: ## Bkz. diff --git a/docs/usermanuals/tr/compliance/gdpr.md b/docs/usermanuals/tr/compliance/gdpr.md index e804fb4a..c9c8a426 100644 --- a/docs/usermanuals/tr/compliance/gdpr.md +++ b/docs/usermanuals/tr/compliance/gdpr.md @@ -19,7 +19,7 @@ GDPR (Avrupa) ve KVKK (Türkiye), eğitimde kullanılan kişisel verilere koruma | **Bütünlük & gizlilik** | GDPR Md. 5(1)(f) | Yetkisiz erişime karşı koruyun. | | **Hesap verebilirlik** | GDPR Md. 5(2) | Uyumu gösterebilmelisiniz. | -KVKK (Türkiye, 6698 sayılı Kanun) benzer ilkeleri 4. maddesinde sayar, ilgili kişilere erişim ve silme haklarını 11. ve 7. maddelerde tanır; ayrı bir hesap verebilirlik ilkesi yoktur ve güvenlik yükümlülükleri 12. maddededir. +KVKK (Türkiye, 6698 sayılı Kanun) benzer ilkeleri 4. maddesinde sayar. 11. madde ilgili kişiye verilerinin işlenip işlenmediğini öğrenme ve silinmesini ya da yok edilmesini isteme hakkını tanır (11/1-(e); 7. maddedeki koşullar çerçevesinde, 7. madde ayrıca işleme sebebi ortadan kalkınca veri sorumlusunu verileri silmeye, yok etmeye veya anonim hale getirmeye mecbur eder) ve veri sorumlusu başvuruyu en geç otuz gün içinde sonuçlandırır (13/2). KVKK'da ayrı bir hesap verebilirlik ilkesi yoktur ve güvenlik yükümlülükleri 12. maddededir. ## ForgeLM her birini nasıl ele alıyor @@ -53,7 +53,7 @@ retention: ephemeral_artefact_retention_days: 365 ``` -`forgelm purge --check-policy --config config.yaml --output-dir ` o dizinde ufku aşmış artefaktları listeler ve kaç tane bulursa bulsun 0 ile çıkar (`--output-dir` verilmezse `training.output_dir`'i değil, bulunulan dizini tarar). Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) +`forgelm purge --check-policy --config config.yaml --output-dir ` o dizinde ufku aşmış artefaktları listeler ve kaç tane bulursa bulsun 0 ile çıkar (`--output-dir` verilmezse `training.output_dir`'i değil, bulunulan dizini tarar). Hiçbir şey otomatik silinmez ve bunun için audit log'a bir şey yazılmaz; süresi geçeni silmek depolama katmanınızın işidir. (`retention.enforce` ayrılmıştır: henüz hiçbir komut onu kullanmaz.) ForgeLM `raw_documents/` veya `ingestion_output/` dizinlerini oluşturmaz; bu yüzden `raw_documents_retention_days`, yalnızca kaynak dokümanları `/raw_documents/` ya da `ingestion_output/` altında tutuyorsanız bir etki yapar. ## Veri sahibi talepleri @@ -70,7 +70,7 @@ $ forgelm reverse-pii --query "ali@example.com" --output-dir ./outputs data/*.js No matches for identifier hash 3f2a9c1e5b7d4a60… (literal, plaintext mode) in 3 file(s). ``` -Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme geçerli bir cevaptır. Basılan değer, audit event'inin kaydettiği tuzlu tanımlayıcı hash'inin ilk 16 hex hanesidir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar; `data/*.jsonl` gibi bir glob ile bu, `audit_log.jsonl`'i corpus dosyalarınızın arasına koyar, bu yüzden her zaman `--output-dir` verin. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur: eşleşme büyük/küçük harfe duyarlıdır ve ham dosya metnini karşılaştırır, bu yüzden yazım ve harf varyantlarını da arayın (`--type custom --query '(?i)…'`); `\u00e7` gibi ASCII kaçışlarıyla yazılmış bir corpus ASCII dışı isimleri literal sorgudan gizler. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. +Tarama, tanımlayıcının geçtiği dosyayı ve satırı raporlar (snippet'ler ham corpus metnidir; kalıcı bir log'a yönlendirmeyin); yukarıdaki gibi sıfır eşleşme yalnızca, taradığınız dosyaların hiçbirinde birebir metnin bulunmadığı anlamına gelir. Tek başına kişi hakkında hiç veri tutmadığınızı göstermez ve Madde 15 yanıtı bir arama sonucundan fazlasını ister: verinin bir kopyası ve Madde 15(1)(a)-(h) bilgileri (bkz. [GDPR Silme ve Erişim](#/compliance/gdpr-erasure)). Basılan değer, audit event'inin kaydettiği tuzlu tanımlayıcı hash'inin ilk 16 hex hanesidir, tanımlayıcının kendisi değildir. `--output-dir`, ForgeLM'in dizin başına tuzu (`.forgelm_audit_salt`) tuttuğu ve `data.access_request_query` event'ini eklediği yerdir: `forgelm purge`'e verdiğiniz dizini verin, böylece iki event aynı zincire katılır. Verilmezse ikisi de ilk corpus dosyasının yanına yazılır ve ForgeLM bir uyarı loglar; `data/*.jsonl` gibi bir glob ile bu, `audit_log.jsonl`'i corpus dosyalarınızın arasına koyar, bu yüzden her zaman `--output-dir` verin. Yalnızca işaret ettiğiniz dosyalarda gerçekten bulunanı bulur: eşleşme büyük/küçük harfe duyarlıdır ve ham dosya metnini karşılaştırır, bu yüzden yazım ve harf varyantlarını da arayın (`--type custom --query '(?i)…'`); `\u00e7` gibi ASCII kaçışlarıyla yazılmış bir corpus ASCII dışı isimleri literal sorgudan gizler. Maskeleme yalnızca tespit ettiği tanımlayıcı biçimlerini kaldırır; bir isim ya da tespit edilemeyen bir varyant kalabilir ve maskelenmiş JSONL taraması ham kaynak dokümanları kapsamaz: onları ayrıca arayın. ForgeLM eğitilmiş bir modelin neyi hatırladığını söyleyemez. ### Silme hakkı (Md. 17) @@ -107,7 +107,7 @@ finally: os.remove(tmp) ``` -Eğitimi id'li dosyadan yapın. `forgelm reverse-pii` tanımlayıcının geçtiği dosyayı ve satırı raporlar; o satırın `id` değerini dosyadan okuyup `forgelm purge --row-id`'ye verin. Eğitimden sonra id'leri yeniden atamak audit log ile corpus arasındaki bağı koparır; bu yüzden bir kez atayın. +Eğitimi id'li dosyadan yapın. `forgelm reverse-pii` tanımlayıcının geçtiği dosyayı ve satırı raporlar; o satırın `id` değerini dosyadan okuyup `forgelm purge --row-id`'ye verin. Eğitimden sonra id'leri yeniden atamak audit log ile corpus arasındaki bağı koparır; bu yüzden bir kez atayın. Yeni dosyayı kontrol ettikten sonra özgün `data/train.jsonl`'i taşıyın ya da silin: id'siz olarak hâlâ her satırı tutar; bu yüzden `data/*.jsonl` glob'u iki kopyayı da eşleştirir, `reverse-pii` kişiyi ikisinde de raporlar ve `purge` yalnızca `--corpus` ile adını verdiğiniz dosyayı düzenler. ### Model-seviyesi silme diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index 9344a58b..00b4d866 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -95,7 +95,7 @@ ForgeLM yerleşik bir quorum gate'i **göndermez**. `approval.quorum` alanı yok ## Timeout -ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler; deployer eski staging dizinlerinin sona ermesini istiyorsa bunun yerine `retention.staging_ttl_days`'i konfigüre edin — bu bir retention raporudur, süre sonu değildir: `forgelm purge --check-policy` horizon'dan eski staging dizinlerini listeler, hiçbir şey otomatik silinmez (birini `forgelm purge --run-id --kind staging` ile kaldırın) ve bu ayar approval gate'e hiçbir şekilde dokunmaz. +ForgeLM staged koşumları **timeout etmez**. `approval.timeout_hours`, `human_approval.timeout` event'i veya auto-fail saati yoktur. Staged koşum süresiz bekler. `retention.staging_ttl_days` bunu değiştirmez: yalnızca `forgelm purge --check-policy`'nin horizon'dan eski staging dizinlerini listelemesini sağlar. Hiçbir şey otomatik silinmez ya da sona erdirilmez ve rapor approval gate'e dokunmaz. Bir staging dizinini kendiniz kaldırmak (`forgelm purge --run-id --kind staging`) bekleyen modeli atar ve onayı karara bağlanmamış bırakır; kararın audit log'da görünmesini istiyorsanız önce bir `forgelm reject` kaydedin. ## Bekleyen koşuları inceleme @@ -141,7 +141,7 @@ Granted / rejected bir koşu üzerinde `--show` çalıştırıldığında tam za ::: :::warn -**Üretim kararları için quorum yok.** Yüksek-riskli üretim deployment'ları için tek-reviewer onayı yetersizdir. Her zaman quorum >= 2 isteyin. +**Üretim kararları için quorum yok.** ForgeLM her karar için tek bir onaylayıcı kaydeder. Risk değerlendirmeniz birden fazla reviewer gerektiriyorsa bunu yukarıda anlatıldığı gibi ForgeLM'in dışında zorlayın (EU AI Act Madde 14(5), iki gerçek kişinin doğrulamasını yalnızca Annex III madde 1(a) kapsamındaki uzaktan biyometrik tanımlama sistemleri için ister). ::: :::tip diff --git a/docs/usermanuals/tr/evaluation/safety.md b/docs/usermanuals/tr/evaluation/safety.md index 83ae7477..2874b3a3 100644 --- a/docs/usermanuals/tr/evaluation/safety.md +++ b/docs/usermanuals/tr/evaluation/safety.md @@ -187,7 +187,7 @@ Set bir *seed*'tir — safety skorunu release-gate olarak kullanmadan önce kend Bu örnek varsayılan `classifier_mode: generation` altında üretilmiştir (yukarıdaki uyarıya bakın): `safety_score`, `safe_ratio`'ya tam olarak eşittir çünkü `confidence_weighted` bu modda bir safe-ratio ortalamasına dejenere olur; `details[].confidence` da well-formed bir `unsafe` verdict için `1.0`'dır — gerçek bir probability değil. `failure_reason`, `forgelm/safety/_gates.py::_evaluate_safety_gates`'teki her zaman aktif mutlak kapıdan gelir: `total_count=50`'nin `unsafe_count=3`'ü %6.00'lık bir unsafe oranıdır, ki bu varsayılan `max_safety_regression=0.05` (%5.00) tavanını aşar — bu kapı `scoring_method`'dan bağımsız olarak ateşlenir. `severity_distribution`, `track_categories: true` iken her zaman dört severity seviyesinin tümünü (`critical`/`high`/`medium`/`low`) sıfır-doldurulmuş olarak listeler; burada unsafe, well-formed, kategori-etiketli iki yanıt da `S5` (iftira) idi, ki bu `forgelm/safety/_types.py`'nin `CATEGORY_SEVERITY`'sinde `high` değil `medium`'a eşlenir. Üçüncü unsafe yanıt (`low_confidence_count`'ta sayılan), malformed bir guard verdict'idir — fail-closed skorlanır ve kategori/severity dökümünden hariç tutulur. `unscored_count: 1` tam olarak bunu kaydeder; `scored_unsafe_count`'ın `2` olmasının nedeni de budur: bu iki alan `unsafe_count`'u, sınıflandırıcının gerçekten unsafe olarak *okuduğu* çiftler ile hiç cevap veremediği çiftler olarak ikiye ayırır. Bir başarısızlığa göre aksiyon almadan önce bunları okuyun — unsafe oranı alıntılayan bir `failure_reason` aynı ayrıştırmayı her zaman düz metin olarak da ekler, çünkü aksi hâlde altı malformed verdict ile gerçekten zararlı altı completion birebir aynı cümleyi üretir. `evaluation_completed`, auto-revert'in baktığı alandır: `false`, koşumun model hakkında kullanılabilir bir kanıt olmadığı anlamına gelir (sınıflandırıcı kullanılamazdı ya da başarısızlık tamamen unscored çiftlere atfedilebilirdi); bu durumda model **başarısız sayılır ama silinmez** ve `forgelm safety-eval` `3` yerine `2` ile çıkar. -`category_distribution` / `severity_distribution` yalnızca `track_categories: true` iken mevcuttur. `details[].prompt`, `details[].response` ve `details[].raw_verdict` GDPR / EU AI Act Madde 10 gizliliği için varsayılan olarak temizlenir — debug için ham metni saklamak üzere `include_eval_samples: true` ayarlayın. Üçüncü alanın ne olduğuna dikkat edin: varsayılan `classifier_mode: generation` altında `raw_verdict`, guard'ın kendi ürettiği çıktıdır ve 200 karaktere kırpılır. Bir koşum "değerlendirme yapılamadı" raporladığında okunacak alan odur — ancak yanlış yapılandırılmış bir guard probe'a cevap vermek yerine onu yankılar veya sürdürür; dolayısıyla bu anahtarı açmak probe metnini diske ikinci bir yoldan yazabilir. `details[].label` her iki durumda da artefaktta kalır; model çıktısından kesilerek değil sabit bir sözlükten yeniden kurulur. Bu sözlük kapalı ve boşlukla ayrılmıştır: `safe`, `malformed` ya da isteğe bağlı olarak virgülle birleştirilmiş S-kodlarının izlediği `unsafe` — `"unsafe"`, `"unsafe S5"`, `"unsafe S1,S5"`. Yeni satıra göre değil, boşluğa göre parse edin: bu sayfanın önceki sürümleri, etiketin ham model çıktısından kesildiği dönemdeki `"unsafe\nS5"` biçimini gösteriyordu. `\n` üzerinden bölen bir tüketici, güncel hiçbir artefakttan S-kodu çıkaramaz. +`category_distribution` / `severity_distribution` yalnızca `track_categories: true` iken mevcuttur. `details[].prompt`, `details[].response` ve `details[].raw_verdict` GDPR Madde 25(2) varsayılan olarak veri koruma ilkesi gereği varsayılan olarak temizlenir — debug için ham metni saklamak üzere `include_eval_samples: true` ayarlayın. Üçüncü alanın ne olduğuna dikkat edin: varsayılan `classifier_mode: generation` altında `raw_verdict`, guard'ın kendi ürettiği çıktıdır ve 200 karaktere kırpılır. Bir koşum "değerlendirme yapılamadı" raporladığında okunacak alan odur — ancak yanlış yapılandırılmış bir guard probe'a cevap vermek yerine onu yankılar veya sürdürür; dolayısıyla bu anahtarı açmak probe metnini diske ikinci bir yoldan yazabilir. `details[].label` her iki durumda da artefaktta kalır; model çıktısından kesilerek değil sabit bir sözlükten yeniden kurulur. Bu sözlük kapalı ve boşlukla ayrılmıştır: `safe`, `malformed` ya da isteğe bağlı olarak virgülle birleştirilmiş S-kodlarının izlediği `unsafe` — `"unsafe"`, `"unsafe S5"`, `"unsafe S1,S5"`. Yeni satıra göre değil, boşluğa göre parse edin: bu sayfanın önceki sürümleri, etiketin ham model çıktısından kesildiği dönemdeki `"unsafe\nS5"` biçimini gösteriyordu. `\n` üzerinden bölen bir tüketici, güncel hiçbir artefakttan S-kodu çıkaramaz. `safety_trend.jsonl` koşu başına bir JSON objesi ekler: @@ -211,7 +211,7 @@ Bu örnek varsayılan `classifier_mode: generation` altında üretilmiştir (yuk | `track_categories` | bool | `false` | Yanıt başı Llama Guard S1-S14 kategorilerini parse et ve raporda yüzeye çıkar. | | `severity_thresholds` | `Optional[Dict[str,float]]` | `null` | Severity-başı unsafe-ratio tavanları — yukarıdaki Severity eşikleri'ne bakın. | | `batch_size` | int | `8` | Fine-tuned modelin probe yanıtları için batched generation boyutu; `1` batching'i kapatır. Guard-verdict skorlamasına **uygulanmaz** — o her zaman sıralıdır, bkz. aşağıdaki Sık hatalar. | -| `include_eval_samples` | bool | `false` | Ham `prompt` / `response` / `raw_verdict` string'lerini `safety_results.json`'a kaydeder. GDPR / EU AI Act Madde 10 gizliliği için varsayılan kapalı — `raw_verdict`, `classifier_mode: generation` altında guard'ın kendi ürettiği metindir. | +| `include_eval_samples` | bool | `false` | Ham `prompt` / `response` / `raw_verdict` string'lerini `safety_results.json`'a kaydeder. GDPR Madde 25(2) varsayılan olarak veri koruma ilkesi gereği varsayılan kapalı — `raw_verdict`, `classifier_mode: generation` altında guard'ın kendi ürettiği metindir. | ## Sık hatalar diff --git a/docs/usermanuals/tr/reference/configuration.md b/docs/usermanuals/tr/reference/configuration.md index 5fad453f..8f79faa2 100644 --- a/docs/usermanuals/tr/reference/configuration.md +++ b/docs/usermanuals/tr/reference/configuration.md @@ -371,7 +371,7 @@ retention: staging_ttl_days: 7 # staging dizininin purge --check-policy tarafından süresi geçmiş sayıldığı yaş ephemeral_artefact_retention_days: 90 # uyumluluk paketleri, denetim raporları raw_documents_retention_days: 90 # ingest edilen PDF/DOCX/EPUB/TXT/Markdown - enforce: "log_only" # ayrılmış: henüz hiçbir komut onu kullanmaz (log_only | warn_on_excess | block_on_excess) + enforce: "log_only" # ayrılmış: henüz hiçbir komut onu dikkate almaz (log_only | warn_on_excess | block_on_excess) ``` ## `pipeline:` diff --git a/forgelm/cli/_config_load.py b/forgelm/cli/_config_load.py index 04c846e7..0338fb8e 100644 --- a/forgelm/cli/_config_load.py +++ b/forgelm/cli/_config_load.py @@ -15,7 +15,7 @@ def _warn_reserved_retention_enforce(config) -> None: - """``retention.enforce`` is validated and offered by both wizards, but no command reads it. + """``retention.enforce`` is validated and offered by both wizards, but no command acts on it. A config that sets ``warn_on_excess`` or ``block_on_excess`` is asking for a gate that does not exist; say so, rather than accept it silently. (This lives here, not in the schema, to keep ``config.py`` within its diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 5d7780b6..a9fb07c0 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -1143,7 +1143,7 @@ def _add_purge_subcommand(subparsers) -> None: type=str, default=None, metavar="DIR", - help="Output directory containing audit_log.jsonl + the per-output-dir salt file. Defaults to the corpus parent dir in row mode and to the current directory in run mode.", + help="Output directory containing audit_log.jsonl + the per-output-dir salt file. Defaults to the corpus parent dir in row mode and to the current directory in run mode and with --check-policy (where it names the directory to scan).", ) p.add_argument( "--config", diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index bf665181..1a37a483 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -251,7 +251,7 @@ def _validate_row_id_args(args, output_format: str) -> None: _output_error_and_exit( output_format, "`--row-id` requires `--corpus `. Bare row-id invocations are " - "rejected per Article 17 (per-row decision + per-row audit event).", + "rejected so that every erasure names the file it edits and gets its own audit event.", EXIT_CONFIG_ERROR, ) # ``--corpus`` is trusted operator self-service input (named directly on the diff --git a/forgelm/config.py b/forgelm/config.py index 52b030ec..970656c7 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -1001,7 +1001,7 @@ class SafetyConfig(BaseModel): default=False, description=( "Persist raw `prompt` / `response` strings to `safety_results.json`. " - "Default OFF for GDPR / EU AI Act Art. 10 privacy by default — adversarial " + "Default OFF for GDPR Art. 25(2) data protection by default — adversarial " "test prompts and model responses may surface sensitive content. Opt in " "for debugging." ), @@ -1140,8 +1140,8 @@ class JudgeConfig(BaseModel): default=False, description=( "Persist raw eval `prompt`, `response`, and judge `reason` strings to " - "`judge_results.json`. Default OFF for GDPR / EU AI Act Art. 10 " - "privacy by default — judge reasoning can quote PII from the eval set. " + "`judge_results.json`. Default OFF for GDPR Art. 25(2) " + "data protection by default — judge reasoning can quote PII from the eval set. " "Opt in for debugging." ), ) @@ -1592,9 +1592,9 @@ class RetentionConfig(BaseModel): ForgeLM's own choices, not legal periods (GDPR Article 5(1)(e) sets no fixed period): audit logs at 5 years, staging at 7 days (one work-week before `purge --check-policy` reports a staging dir), - ephemeral artefacts at 90 days (compliance bundle + audit reports - have a quarterly review cadence in most QMS), raw documents at 90 - days (typical ingestion-window before re-running data audit). + ephemeral artefacts at 90 days and raw documents at 90 days (round + numbers chosen for ForgeLM, not review cadences or ingestion windows + taken from any standard). Choose values from your own necessity assessment. Setting any horizon to ``0`` disables the policy for that artefact @@ -1630,7 +1630,7 @@ class RetentionConfig(BaseModel): ge=0, description=( "Age in days after which `forgelm purge --check-policy` reports `/compliance/` and " - "`/data_audit_report.json` as overdue; nothing is deleted. Set to 0 to stop reporting it." + "`/data_audit_report.json` as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to 0 to stop reporting it." ), ) raw_documents_retention_days: int = Field( @@ -1638,8 +1638,8 @@ class RetentionConfig(BaseModel): ge=0, description=( "Age in days after which `forgelm purge --check-policy` reports `/raw_documents/` or " - "`/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue; nothing is deleted. Set to 0 to stop reporting it. " - "Closes ghost-features GH-023 (was nested as `ingestion.retention.raw_documents.ttl_days`; now top-level)." + "`/ingestion_output/` (ingested PDF / DOCX / EPUB / TXT / Markdown) as overdue (aged from the audit log's first event, else mtime); nothing is deleted. Set to 0 to stop reporting it. " + "ForgeLM does not create these directories; the horizon applies only to source documents you keep there." ), ) enforce: Literal["log_only", "warn_on_excess", "block_on_excess"] = Field( @@ -1699,7 +1699,7 @@ class ForgeConfig(BaseModel): ) retention: Optional[RetentionConfig] = Field( default=None, - description="Phase 21 / GDPR Article 5(1)(e) storage limitation + Article 17 erasure horizons block.", + description="Retention horizons (GDPR Article 5(1)(e) storage limitation), reported by `forgelm purge --check-policy` and never enforced.", ) pipeline: Optional["PipelineConfig"] = Field( default=None, diff --git a/forgelm/data_audit/_types.py b/forgelm/data_audit/_types.py index 439d2cf7..0faf707b 100644 --- a/forgelm/data_audit/_types.py +++ b/forgelm/data_audit/_types.py @@ -24,7 +24,8 @@ # remember by hand that ``credit_card`` is materially worse than ``phone``. # The tiers below are ForgeLM's default weighting, not a regulatory # classification (PCI DSS protects card numbers; GDPR Art. 87 lets Member -# States restrict national identification numbers), so the audit surfaces a +# States set the specific conditions for processing national identification +# numbers), so the audit surfaces a # one-glance verdict via :data:`PII_SEVERITY`. Tweak the table when local law # disagrees — the audit honours whatever map is loaded at call time. PII_SEVERITY: Dict[str, str] = { diff --git a/notebooks/data_curation.ipynb b/notebooks/data_curation.ipynb index 00892a4d..a492f4db 100644 --- a/notebooks/data_curation.ipynb +++ b/notebooks/data_curation.ipynb @@ -373,7 +373,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "## What's next\n\n- **Train on the curated JSONL**: point any of the training notebooks at\n `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n set with public datasets\n- **Long-form guides**:\n - [`docs/guides/ingestion.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/ingestion.md)\n — every flag, every chunking strategy, every extra\n - [`docs/guides/data_audit.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/data_audit.md)\n — audit JSON schema, severity tiers, governance integration\n- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n when you point it at a directory of raw docs, and offers to run\n `forgelm audit` automatically once a JSONL is selected — closes the\n BYOD audit loop end-to-end.\n- **Other v0.5.0 audit add-ons**:\n - `--croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/)\n dataset card under the report's `croissant` key so the same JSON\n file doubles as both the EU AI Act Article 10 governance artifact\n and a Croissant-consumer dataset card.\n - `--pii-ml [--pii-ml-language LANG]` layers\n [Presidio](https://github.com/data-privacy-stack/presidio) NER on top of\n the regex detector — adds `person` / `organization` /\n `location` categories to `pii_summary` / `pii_severity`. **Two-step\n install:**\n ```bash\n pip install 'forgelm[ingestion-pii-ml]'\n python -m spacy download en_core_web_lg\n ```\n `presidio-analyzer` does not transitively ship a spaCy NER model;\n without the model `--pii-ml` raises an `ImportError` with the\n install recipe **before** any rows are scanned. Pass\n `--pii-ml-language tr` (and download the matching spaCy model) to\n audit Turkish corpora." + "source": "## What's next\n\n- **Train on the curated JSONL**: point any of the training notebooks at\n `./curated/policies.jsonl` via `data.dataset_name_or_path`.\n - [`quickstart_sft.ipynb`](./quickstart_sft.ipynb) — first SFT run\n - [`multi_dataset.ipynb`](./multi_dataset.ipynb) — mix the curated\n set with public datasets\n- **Long-form guides**:\n - [`docs/guides/ingestion.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/ingestion.md)\n — every flag, every chunking strategy, every extra\n - [`docs/guides/data_audit.md`](https://github.com/HodeTech/ForgeLM/blob/main/docs/guides/data_audit.md)\n — audit JSON schema, severity tiers, governance integration\n- **Wizard**: `forgelm --wizard` offers to run `forgelm ingest` inline\n when you point it at a directory of raw docs, and offers to run\n `forgelm audit` automatically once a JSONL is selected — closes the\n BYOD audit loop end-to-end.\n- **Other v0.5.0 audit add-ons**:\n - `--croissant` emits a [Google Croissant 1.0](http://mlcommons.org/croissant/)\n dataset card under the report's `croissant` key so the same JSON\n file doubles as both the EU AI Act Article 10 governance artifact\n and a Croissant-consumer dataset card.\n - `--pii-ml [--pii-ml-language LANG]` layers\n [Presidio](https://github.com/data-privacy-stack/presidio) NER on top of\n the regex detector — adds `person` / `organization` /\n `location` categories to `pii_summary` / `pii_severity`. **Two-step\n install:**\n ```bash\n pip install 'forgelm[ingestion-pii-ml]'\n python -m spacy download en_core_web_lg\n ```\n `presidio-analyzer` does not transitively ship a spaCy NER model;\n without the model `--pii-ml` raises an `ImportError` with the\n install recipe **before** any rows are scanned. For a\n non-English corpus pass `--pii-ml-language` with a supported code\n (for example `de`) and download the matching spaCy model; Turkish has\n no dedicated model, so use `--pii-ml-language xx` with\n `xx_ent_wiki_sm`." } ], "metadata": { diff --git a/site/js/translations.js b/site/js/translations.js index ba52dbb3..dba539a1 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -1140,7 +1140,7 @@ "features.gov.purge.title": "Satır silme (Madde 17 adımı)", "features.gov.purge.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve silmeyi append-only audit log'a kaydeder; Madde 17 kapsamındaki silmenin bir adımıdır (eğitilmiş ağırlıklar değişmez).", "features.gov.access.title": "Tanımlayıcı arama (Madde 15 girdisi)", - "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da tuzlanmış digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler: Madde 15 yanıtının bir girdisi.", + "features.gov.access.body": "forgelm reverse-pii JSONL corpus'larını tarayarak bir veri sahibi tanımlayıcısının düz metin ya da salt'lı digest eşleşmelerini arar ve taranan dosyalarda onu içeren satırları listeler: Madde 15 yanıtının bir girdisi.", "features.gov.annex.title": "Annex IV doğrulayıcı", "features.gov.annex.body": "forgelm verify-annex-iv Annex IV §1-9 şema tamlığını ve bundle'ın manifest_hash'ini tampering tespiti için doğrular. (Audit-chain bütünlüğü ayrı bir forgelm verify-audit komutudur.)", "features.gov.gguf.title": "GGUF bütünlük doğrulayıcı", @@ -1810,7 +1810,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Teacher API key env var", "wizard.synthetic.api_key_env.hint": "Environment variable name carrying the teacher API key. Empty = local teacher model.", - "wizard.retention.title": "Retention horizons (GDPR Art. 5+17)", + "wizard.retention.title": "Retention horizons (GDPR Art. 5(1)(e))", "wizard.retention.hint": "Ages at which forgelm purge --check-policy reports artefacts as overdue; nothing is deleted. Set a value to 0 to stop reporting that artefact. The retention block is written to the YAML only when a value differs from its default; without it, --check-policy checks nothing.", "wizard.retention.audit_log.label": "Audit log retention (days)", "wizard.retention.audit_log.hint": "How long before forgelm purge --check-policy reports audit_log.jsonl as overdue. Default 1825 (5 years) is a ForgeLM choice, not a legal period.", @@ -2208,7 +2208,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Teacher API key env değişkeni", "wizard.synthetic.api_key_env.hint": "Teacher API anahtarını taşıyan ortam değişkeni adı. Boş = yerel teacher modeli.", - "wizard.retention.title": "Saklama ufukları (GDPR Madde 5+17)", + "wizard.retention.title": "Saklama ufukları (GDPR Madde 5(1)(e))", "wizard.retention.hint": "forgelm purge --check-policy'nin artefact'ları süresi geçmiş olarak raporladığı yaşlar; hiçbir şey silinmez. Bir değeri 0 yapmak o artefact'ın raporlanmasını durdurur. retention bloğu YAML'a yalnızca bir değer varsayılandan farklıysa yazılır; blok yoksa --check-policy hiçbir şeyi kontrol etmez.", "wizard.retention.audit_log.label": "Audit log saklama (gün)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy audit_log.jsonl'i kaç gün sonra süresi geçmiş olarak raporlasın. Varsayılan 1825 (5 yıl) bir ForgeLM tercihidir, yasal süre değildir.", @@ -2602,7 +2602,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Teacher API-Key Env-Variable", "wizard.synthetic.api_key_env.hint": "Env-Variable mit dem Teacher-API-Key. Leer = lokales Teacher-Modell.", - "wizard.retention.title": "Aufbewahrungsfristen (DSGVO Art. 5+17)", + "wizard.retention.title": "Aufbewahrungshorizonte (DSGVO Art. 5 Abs. 1 Buchst. e)", "wizard.retention.hint": "Alter, ab dem forgelm purge --check-policy Artefakte als überfällig meldet; nichts wird gelöscht. Ein Wert von 0 beendet die Meldung für dieses Artefakt. Der retention-Block wird nur dann in die YAML geschrieben, wenn ein Wert vom Default abweicht; ohne ihn prüft --check-policy nichts.", "wizard.retention.audit_log.label": "Audit-Log Aufbewahrung (Tage)", "wizard.retention.audit_log.hint": "Nach wie vielen Tagen forgelm purge --check-policy audit_log.jsonl als überfällig meldet. Der Default 1825 (5 Jahre) ist eine Vorgabe von ForgeLM, keine gesetzliche Frist.", @@ -2996,7 +2996,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Variable env clé API teacher", "wizard.synthetic.api_key_env.hint": "Nom variable env portant clé API teacher. Vide = modèle teacher local.", - "wizard.retention.title": "Horizons de rétention (RGPD Art. 5+17)", + "wizard.retention.title": "Horizons de rétention (RGPD art. 5, paragraphe 1, point e)", "wizard.retention.hint": "Seuils d'âge à partir desquels forgelm purge --check-policy signale des artefacts comme ayant dépassé leur durée de conservation ; rien n'est supprimé. Une valeur de 0 arrête le signalement de cet artefact. Le bloc retention n'est écrit dans le YAML que si une valeur diffère de sa valeur par défaut ; sans lui, --check-policy ne vérifie rien.", "wizard.retention.audit_log.label": "Rétention audit log (jours)", "wizard.retention.audit_log.hint": "Nombre de jours au-delà duquel forgelm purge --check-policy signale audit_log.jsonl comme ayant dépassé sa durée de conservation. Par défaut : 1825 (5 ans), un choix de ForgeLM et non un délai légal.", @@ -3390,7 +3390,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Variable env clave API teacher", "wizard.synthetic.api_key_env.hint": "Nombre variable env con clave API teacher. Vacío = modelo teacher local.", - "wizard.retention.title": "Horizontes de retención (RGPD Art. 5+17)", + "wizard.retention.title": "Horizontes de retención (RGPD art. 5, apartado 1, letra e)", "wizard.retention.hint": "Antigüedad a partir de la cual forgelm purge --check-policy señala los artefactos como vencidos; no se elimina nada. Un valor de 0 deja de señalar ese artefacto. El bloque retention solo se escribe en el YAML si algún valor difiere de su valor por defecto; sin él, --check-policy no comprueba nada.", "wizard.retention.audit_log.label": "Retención audit log (días)", "wizard.retention.audit_log.hint": "Días tras los cuales forgelm purge --check-policy señala audit_log.jsonl como vencido. El valor por defecto, 1825 (5 años), es una elección de ForgeLM, no un plazo legal.", @@ -3784,7 +3784,7 @@ "wizard.synthetic.seed_file.placeholder": "./data/seed_prompts.jsonl", "wizard.synthetic.api_key_env.label": "Teacher API 密钥环境变量", "wizard.synthetic.api_key_env.hint": "承载 teacher API 密钥的环境变量名。空 = 本地 teacher 模型。", - "wizard.retention.title": "保留期限(GDPR 第 5+17 条)", + "wizard.retention.title": "保留期限(GDPR 第 5 条第 1 款第 (e) 项)", "wizard.retention.hint": "forgelm purge --check-policy 将 artefact 报告为已超期的期限(天数);不会删除任何内容。将某一项设为 0 即不再报告该 artefact。仅当某个值与默认值不同时,retention 块才会写入 YAML;没有该块时,--check-policy 不检查任何内容。", "wizard.retention.audit_log.label": "Audit log 保留(天)", "wizard.retention.audit_log.hint": "forgelm purge --check-policy 在多少天后将 audit_log.jsonl 报告为已超期。默认 1825(5 年)是 ForgeLM 的选择,不是法定期限。", @@ -3929,7 +3929,7 @@ "compliance.gdpr.title": "Ingest'te PII redaksiyonu", "compliance.gdpr.body": "--pii-mask chunk'lar JSONL'a düşmeden e-posta, telefon, kredi kartı (Luhn), IBAN, ulusal kimlik (TR/DE/FR/US-SSN) maskeler. Eğitime ulaşanı azaltır ve veri minimizasyonunu (Madde 5(1)(c)) destekler; isimler ve serbest metin tespit edilmez, bu yüzden bir yardımcıdır, garanti değildir.", "compliance.gdpr15.title": "Erişim hakkı", - "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz metin olarak geçtiği yerleri veya harici bir maskeleme pipeline'ının yazdığı tuzlanmış SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Eşleşme birebir ve büyük/küçük harfe duyarlıdır. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", + "compliance.gdpr15.body": "forgelm reverse-pii JSONL corpus'larında bir veri sahibi tanımlayıcısının düz metin olarak geçtiği yerleri veya harici bir maskeleme pipeline'ının yazdığı salt'lı SHA-256 digest'lerini arar ve taradığınız dosyalarda onu içeren her satırı raporlar. Eşleşme birebir ve büyük/küçük harfe duyarlıdır. Madde 15 yanıtınızın bir girdisidir; yanıt ayrıca verinin bir kopyasını ve Madde 15(1)(a)-(h) bilgilerini içermelidir. Ham kaynaklar ve eğitilmiş model aranmaz.", "compliance.gdpr17.title": "Silme hakkı", "compliance.gdpr17.body": "forgelm purge bir eğitim corpus'undan satırları id ile siler ve append-only audit log'a data.erasure_requested + data.erasure_completed olaylarını kaydeder. Madde 17 kapsamındaki silmenin bir adımıdır: eğitilmiş ağırlıklar, kaynak dokümanlar ve başka yerlerdeki kopyalar değişmez.", "compliance.iso.eyebrow": "ISO 27001 · SOC 2 Type II", diff --git a/site/js/wizard.js b/site/js/wizard.js index ca3dd6ab..08fe5b67 100644 --- a/site/js/wizard.js +++ b/site/js/wizard.js @@ -265,7 +265,7 @@ syntheticTeacherModel: 'gpt-4o', syntheticSeedFile: '', syntheticApiKeyEnv: 'OPENAI_API_KEY', - // GDPR Article 5+17 retention horizons. Defaults below match + // GDPR Article 5(1)(e) retention horizons. Defaults below match // forgelm/config.py RetentionConfig. retentionAuditLog: 1825, // 5 years retentionStaging: 7, @@ -3008,7 +3008,7 @@ } })); - // Retention horizons (GDPR Art. 5+17) accordion. + // Retention horizons (GDPR Art. 5(1)(e)) accordion. pane.appendChild(makeAccordion(state, persist, 'retention', 'wizard.retention.title', 'wizard.retention.hint', function (body) { function makeNumberRow(field, labelKey, hintKey) { From 71215ada3e1e74f79ff81eeae507a02963d429aa Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 22 Oct 2026 03:59:25 +0300 Subject: [PATCH 18/25] test: pin the GDPR wording against the code, per sentence and per locale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The retired-phrase tests could be defeated by rewording one hunk, and several "pinned" claims had no test at all: - The Article 17(3)(b)/(e) invariant is scoped to the mention's own sentence, covers the bare "(3)(b)" spelling and either case, accepts "insofar as", "only where", "ölçüsünde" and "kadarıyla" (and no longer treats any "only to" as a qualifier), and runs over every text file of the named public subtrees instead of a hand-picked list. Failures print the line and the matched text. - The retired lists gain the salt-rotation, memorisation-scope, Article 19, legal-floor, `--kind artefacts`, `--salt-source` correlation, "zero matches is a valid answer" and weak-`jq` phrases, the Turkish counterparts of four English-only pins, and narrower forms of three over-broad bans; the scanned set now includes the deployer guide, incident SOP, QMS, reverse-pii and catalog pages. - New positive pins: the CI-gate recipe, the digest-correlation rule, the exit-2 wording and the "finds nothing / deletes nothing" warning; gpu-cost, chat and pii-masking are checked against the code (every priced GPU, every chat flag, the detector count and its validated families) besides a list of retired fictions. - The `--kind artefacts` tripwire builds the bundle with the real exporter and writes the run id inside the files, so it also trips for an ownership-proving resolver (checked by mutation), and the AST test flags a marker name assembled from pieces. - The zero-match line shown on the GDPR page is matched against the command's real output. - Site tests pin the trained-weights negation, the Article 15 "input to" caveat, the 15(1)(a)-(h) information, literal case-sensitive matching and opt-in masking in all six locales, the five retention hints, and the static fallback of every GDPR string; the German privacy page no longer uses "Datensubjekt". Co-Authored-By: Claude Sonnet 5.5 --- docs/design/gdpr_erasure.md | 2 +- forgelm/cli/subcommands/_purge.py | 2 +- site/js/translations.js | 14 +- site/privacy.html | 2 +- site/terms.html | 2 +- tests/site/site_translations.test.cjs | 95 ++++- tests/test_gdpr_erasure.py | 55 ++- tests/test_reverse_pii.py | 18 + .../test_usermanual_reference_consistency.py | 393 +++++++++++++++--- 9 files changed, 478 insertions(+), 105 deletions(-) diff --git a/docs/design/gdpr_erasure.md b/docs/design/gdpr_erasure.md index 05564c2c..4ff90ef0 100644 --- a/docs/design/gdpr_erasure.md +++ b/docs/design/gdpr_erasure.md @@ -45,7 +45,7 @@ Phase 21 closes the gap with a real implementation. This Phase 20 document spec **Article 19** requires the controller to tell each recipient the data was disclosed to about the erasure, and **Article 17(2)** adds reasonable steps to inform other controllers where the controller made the data public. ForgeLM is not a publisher (we do not run inference); the deployer is. We document this boundary in the operator guide rather than implement push notification. -**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims. Each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to be able to name it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis; once the data subject withdraws consent or a successful objection removes that basis, it no longer carries them (Art. 17(1)(b)-(c)), and a refusal still needs reasons and a complaint route within the Art. 12 deadline (Art. 12(4)). +**Article 17(3)** lists exemptions where erasure does **not** apply — including (3)(b) compliance with a legal obligation under Union or Member State law (or a task in the public interest or in the exercise of official authority) and (3)(e) the establishment, exercise or defence of legal claims; each applies only to the extent the processing is necessary for that obligation, task or claim, and the controller has to be able to name it. Erasure is also owed only where an Art. 17(1) ground exists, so entries may equally stay because the processing is still necessary for its purpose on its own Art. 6 basis; once the data subject withdraws consent or a successful objection removes that basis, it no longer carries them (Art. 17(1)(b)-(c)), and a refusal still needs reasons and a complaint route within the Art. 12 deadline (Art. 12(4)). The take-away: erasure operates on **training data + derived artefacts**, not on the audit log. The audit log records *that an erasure happened* (Art. 5(2) accountability), and ForgeLM never edits or deletes an entry. Whether the entries that reference the now-erased subject may stay is the controller's decision, and the controller documents why: they are still necessary for their purpose, or an Art. 17(3) exception applies (for (3)(b) and (3)(e), only to the extent a specific obligation or claim requires it). diff --git a/forgelm/cli/subcommands/_purge.py b/forgelm/cli/subcommands/_purge.py index 1a37a483..8e8f9402 100644 --- a/forgelm/cli/subcommands/_purge.py +++ b/forgelm/cli/subcommands/_purge.py @@ -1117,7 +1117,7 @@ def _run_purge_check_policy(args, output_format: str) -> None: """Handle ``forgelm purge --check-policy [--config ]``. A successful report exits 0 (design §10 Q5: report-not-gate); an unloadable config exits 1. - Operators wiring a CI gate use ``--output-format json`` and pipe + Operators wiring a CI gate (under ``set -o pipefail``) use ``--output-format json`` and pipe to ``jq -e '.success == true and has("count") and .count == 0'``. Strict config loading: ``--check-policy`` is the one purge mode diff --git a/site/js/translations.js b/site/js/translations.js index dba539a1..989b1d77 100644 --- a/site/js/translations.js +++ b/site/js/translations.js @@ -43,7 +43,7 @@ "common.footer.issues": "Issues", "common.footer.discussions":"Discussions", "common.footer.releases": "Releases", - "common.legal.last_updated":"Last updated: 2026-05-09 · This is a plain-language summary, not legal advice." + "common.legal.last_updated":"Last updated: 2026-10-22 · This is a plain-language summary, not legal advice." }; T.tr = { @@ -71,7 +71,7 @@ "common.footer.issues": "Sorunlar", "common.footer.discussions":"Tartışmalar", "common.footer.releases": "Sürümler", - "common.legal.last_updated":"Son güncelleme: 09-05-2026 · Bu hukuki tavsiye değil, açık dilde özettir." + "common.legal.last_updated":"Son güncelleme: 22-10-2026 · Bu hukuki tavsiye değil, açık dilde özettir." }; T.de = { @@ -99,7 +99,7 @@ "common.footer.issues": "Issues", "common.footer.discussions":"Diskussionen", "common.footer.releases": "Releases", - "common.legal.last_updated":"Letzte Aktualisierung: 09.05.2026 · Eine verständliche Zusammenfassung, keine Rechtsberatung." + "common.legal.last_updated":"Letzte Aktualisierung: 22.10.2026 · Eine verständliche Zusammenfassung, keine Rechtsberatung." }; T.fr = { @@ -127,7 +127,7 @@ "common.footer.issues": "Issues", "common.footer.discussions":"Discussions", "common.footer.releases": "Versions", - "common.legal.last_updated":"Dernière mise à jour : 09/05/2026 · Résumé en langage clair, pas un conseil juridique." + "common.legal.last_updated":"Dernière mise à jour : 22/10/2026 · Résumé en langage clair, pas un conseil juridique." }; T.es = { @@ -155,7 +155,7 @@ "common.footer.issues": "Issues", "common.footer.discussions":"Discusiones", "common.footer.releases": "Versiones", - "common.legal.last_updated":"Última actualización: 09-05-2026 · Resumen en lenguaje sencillo; no es asesoramiento legal." + "common.legal.last_updated":"Última actualización: 22-10-2026 · Resumen en lenguaje sencillo; no es asesoramiento legal." }; T.zh = { @@ -183,7 +183,7 @@ "common.footer.issues": "Issues", "common.footer.discussions":"讨论", "common.footer.releases": "版本", - "common.legal.last_updated":"最后更新:2026-05-09 · 通俗语言总结,非法律建议。" + "common.legal.last_updated":"最后更新:2026-10-22 · 通俗语言总结,非法律建议。" }; /* ── Page-specific blocks below extend each language with Object.assign. ── */ @@ -4362,7 +4362,7 @@ "privacy.header.eyebrow": "Datenschutz", "privacy.header.title": "Datenschutzerklärung", "privacy.header.lead": "ForgeLM ist Open-Source-Software, die Sie auf Ihrer eigenen Infrastruktur betreiben. Diese Seite behandelt, was diese Website tut — nicht das Toolkit selbst.", - "privacy.body": "

Kurzversion

Diese Seite ist statisches HTML. Wir betreiben keine Analytik, keine Tracking-Pixel, keine Werbe-Tracker, keine A/B-Tests, kein Fingerprinting und keine Drittanbieter-Telemetrie. Hier können drei Drittanbieter-Anfragen erfolgen: Google Fonts wird bei jedem Seitenaufruf abgerufen (die Schriften Inter und JetBrains Mono); Mermaid wird ausschließlich auf der Guide-Seite von jsDelivr geladen (zum Rendern der Diagramme); und Formspree nur, wenn Sie das Kontaktformular absenden. Alle drei werden unten detailliert beschrieben; sonst nichts.

Was wir erfassen

Beim normalen Surfen

Nichts. Es werden keine Cookies gesetzt. Auf Ihrem Gerät werden zwei Werte in localStorage gespeichert:

  • forgelm-lang — Ihre gewählte Oberflächensprache (en, tr, de, fr, es, zh).
  • forgelm-theme — Ihr gewähltes Farbschema (dark oder light).

Beide bleiben auf Ihrem Gerät und werden nie übertragen. Das Leeren Ihres Browser-Speichers entfernt sie.

Vom Kontaktformular

Wenn Sie das Formular auf der Kontakt-Seite absenden, werden die eingegebenen Werte (Name, E-Mail, Organisation, Thema, Betreff, Nachricht) an Formspree gesendet, das sie an den Projekt-Maintainer weiterleitet. Die Datenschutzpraktiken von Formspree unterliegen ihrer eigenen Richtlinie unter formspree.io/legal/privacy-policy.

Ihre E-Mail wird ausschließlich zur Beantwortung Ihrer Nachricht verwendet. Wir nehmen sie nicht in eine Mailingliste auf, geben sie nicht an Dritte weiter und behalten sie nach Abschluss der Konversation nicht zurück.

Hosting

Die Seite wird als statische Dateien ausgeliefert. Ihr Browser tauscht Standard-Request-Header (User-Agent, Accept-Language, IP-Adresse) mit dem CDN / Webserver aus, der die Dateien hostet; dieser Austausch unterliegt den Datenschutzpraktiken des Hosting-Anbieters und liegt nicht in der Kontrolle von ForgeLM. Wir loggen diesen Verkehr selbst nicht.

Drittanbieter-Assets

Die Seite lädt:

  • Google Fonts — die Schriften Inter und JetBrains Mono. Google kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • jsDelivr — Mermaid 10.9.4 wird ausschließlich auf der Guide-Seite geladen, um die Architektur- / Flussdiagramme zu rendern. jsDelivr kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • Formspree — nur wenn Sie das Kontaktformular absenden, wie oben beschrieben.

Sonst wird nichts von Drittanbietern geladen. Kein Google Analytics, kein Facebook-Pixel, kein Hotjar, kein Mixpanel, kein wie-auch-immer-aktueller-Tracker.

Kinder

Diese Seite richtet sich an Entwickler, ML-Ingenieure und Compliance-Fachleute. Sie richtet sich nicht an Kinder unter 16.

Ihre Rechte (DSGVO / KVKK)

Wenn Sie der Meinung sind, dass wir personenbezogene Daten über Sie speichern, und Sie Auskunft, Berichtigung oder Löschung wünschen, schreiben Sie uns über das Kontaktformular (oder GitHub Issues); wir antworten innerhalb von 30 Tagen. Da die einzigen personenbezogenen Daten, die wir halten, das sind, was Sie im Kontaktformular gesendet haben, ist die Löschung einfach: Wir löschen den E-Mail-Thread.

Das Toolkit selbst

Das ForgeLM-Python-Paket — installiert über pip install forgelm — telefoniert nicht nach Hause. Es sammelt keine Telemetrie, sendet keine Nutzungsdaten irgendwohin und arbeitet offline (Air-Gap-fähig). Modelle, Datensätze und Tokenizer, die Sie in Ihrer YAML-Konfiguration referenzieren, werden nur auf Ihre ausdrückliche Anforderung und nur aus von Ihnen angegebenen Quellen heruntergeladen.

Datensubjekt-Rechte, die Sie in Ihrer eigenen Pipeline ausliefern können

Wenn Ihre Organisation ForgeLM einsetzt, liefert das Toolkit konkrete Primitive für die Datensubjekt-Rechte-Seite der DSGVO: forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Artikel 15), und forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log (ein Schritt der Löschung nach Artikel 17; trainierte Gewichte bleiben unverändert). Darüber hinaus ist ForgeLM mit ISO 27001 / SOC 2 Type II Kontrollen aligned — siehe die Compliance-Seite für die vollständige Nachweis-Karte.

Änderungen

Wenn wir diese Seite aktualisieren, ändert sich das Datum oben. Wesentliche Änderungen werden im CHANGELOG.md des Projekts vermerkt.

Fragen

Öffnen Sie das Kontaktformular oder eine GitHub Discussion.

" + "privacy.body": "

Kurzversion

Diese Seite ist statisches HTML. Wir betreiben keine Analytik, keine Tracking-Pixel, keine Werbe-Tracker, keine A/B-Tests, kein Fingerprinting und keine Drittanbieter-Telemetrie. Hier können drei Drittanbieter-Anfragen erfolgen: Google Fonts wird bei jedem Seitenaufruf abgerufen (die Schriften Inter und JetBrains Mono); Mermaid wird ausschließlich auf der Guide-Seite von jsDelivr geladen (zum Rendern der Diagramme); und Formspree nur, wenn Sie das Kontaktformular absenden. Alle drei werden unten detailliert beschrieben; sonst nichts.

Was wir erfassen

Beim normalen Surfen

Nichts. Es werden keine Cookies gesetzt. Auf Ihrem Gerät werden zwei Werte in localStorage gespeichert:

  • forgelm-lang — Ihre gewählte Oberflächensprache (en, tr, de, fr, es, zh).
  • forgelm-theme — Ihr gewähltes Farbschema (dark oder light).

Beide bleiben auf Ihrem Gerät und werden nie übertragen. Das Leeren Ihres Browser-Speichers entfernt sie.

Vom Kontaktformular

Wenn Sie das Formular auf der Kontakt-Seite absenden, werden die eingegebenen Werte (Name, E-Mail, Organisation, Thema, Betreff, Nachricht) an Formspree gesendet, das sie an den Projekt-Maintainer weiterleitet. Die Datenschutzpraktiken von Formspree unterliegen ihrer eigenen Richtlinie unter formspree.io/legal/privacy-policy.

Ihre E-Mail wird ausschließlich zur Beantwortung Ihrer Nachricht verwendet. Wir nehmen sie nicht in eine Mailingliste auf, geben sie nicht an Dritte weiter und behalten sie nach Abschluss der Konversation nicht zurück.

Hosting

Die Seite wird als statische Dateien ausgeliefert. Ihr Browser tauscht Standard-Request-Header (User-Agent, Accept-Language, IP-Adresse) mit dem CDN / Webserver aus, der die Dateien hostet; dieser Austausch unterliegt den Datenschutzpraktiken des Hosting-Anbieters und liegt nicht in der Kontrolle von ForgeLM. Wir loggen diesen Verkehr selbst nicht.

Drittanbieter-Assets

Die Seite lädt:

  • Google Fonts — die Schriften Inter und JetBrains Mono. Google kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • jsDelivr — Mermaid 10.9.4 wird ausschließlich auf der Guide-Seite geladen, um die Architektur- / Flussdiagramme zu rendern. jsDelivr kann Anfragen wie in seiner Datenschutzrichtlinie beschrieben loggen.
  • Formspree — nur wenn Sie das Kontaktformular absenden, wie oben beschrieben.

Sonst wird nichts von Drittanbietern geladen. Kein Google Analytics, kein Facebook-Pixel, kein Hotjar, kein Mixpanel, kein wie-auch-immer-aktueller-Tracker.

Kinder

Diese Seite richtet sich an Entwickler, ML-Ingenieure und Compliance-Fachleute. Sie richtet sich nicht an Kinder unter 16.

Ihre Rechte (DSGVO / KVKK)

Wenn Sie der Meinung sind, dass wir personenbezogene Daten über Sie speichern, und Sie Auskunft, Berichtigung oder Löschung wünschen, schreiben Sie uns über das Kontaktformular (oder GitHub Issues); wir antworten innerhalb von 30 Tagen. Da die einzigen personenbezogenen Daten, die wir halten, das sind, was Sie im Kontaktformular gesendet haben, ist die Löschung einfach: Wir löschen den E-Mail-Thread.

Das Toolkit selbst

Das ForgeLM-Python-Paket — installiert über pip install forgelm — telefoniert nicht nach Hause. Es sammelt keine Telemetrie, sendet keine Nutzungsdaten irgendwohin und arbeitet offline (Air-Gap-fähig). Modelle, Datensätze und Tokenizer, die Sie in Ihrer YAML-Konfiguration referenzieren, werden nur auf Ihre ausdrückliche Anforderung und nur aus von Ihnen angegebenen Quellen heruntergeladen.

Betroffenenrechte, die Sie in Ihrer eigenen Pipeline umsetzen können

Wenn Ihre Organisation ForgeLM einsetzt, liefert das Toolkit Bausteine für die Betroffenenrechte der DSGVO: forgelm reverse-pii findet die Zeilen, die eine von Ihnen angegebene Kennung enthalten (ein Baustein der Auskunft nach Art. 15 DSGVO), und forgelm purge löscht Zeilen anhand ihrer ID aus einem Trainingskorpus und protokolliert die Löschung im Append-only-Audit-Log (ein Schritt der Löschung nach Art. 17 DSGVO; trainierte Gewichte bleiben unverändert). Darüber hinaus ist ForgeLM mit ISO 27001 / SOC 2 Type II Kontrollen aligned — siehe die Compliance-Seite für die vollständige Nachweis-Karte.

Änderungen

Wenn wir diese Seite aktualisieren, ändert sich das Datum oben. Wesentliche Änderungen werden im CHANGELOG.md des Projekts vermerkt.

Fragen

Öffnen Sie das Kontaktformular oder eine GitHub Discussion.

" }); Object.assign(T.fr, { diff --git a/site/privacy.html b/site/privacy.html index 8712c99a..948a3d95 100644 --- a/site/privacy.html +++ b/site/privacy.html @@ -129,7 +129,7 @@

Privacy policy