From 93e0ddf2eeb9cee8afd9430d1bff8e026275bcb8 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Tue, 5 May 2026 23:23:18 +0300 Subject: [PATCH 01/12] docs(wave-4-faz-22): ISO 27001 / SOC 2 Type II alignment design (865 lines) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Faz 22 deliverable: docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md. Single design dosyası; no code or behaviour change. Scope: - §3 Full ISO 27001:2022 Annex A control mapping — all 93 controls scored across A.5 organisational (37) + A.6 people (8) + A.7 physical (14) + A.8 technological (34). Coverage tally: FL (full) 11 / FL-helps 50 / OOS 32. - §4 Full SOC 2 Trust Services Criteria mapping — Common Criteria CC1.x–CC9.x (Security mandatory) + Availability A1.x + Processing Integrity PI1.x + Confidentiality C1.x + Privacy P1.x–P8.x. - §5 Gap assessment: 15 items closed in Faz 23 (pip-audit, bandit, 4 new QMS docs, 1 deployer guide, 2 reference tables, README section, CHANGELOG line, SBOM determinism test). - §6 SBOM tooling — current state (Wave 2 era CycloneDX 1.5, pure-stdlib, integrated with publish.yml release tag) + Faz 23 determinism test contract. - §7 Vulnerability scanning — pip-audit nightly + bandit CI; both high-severity → fail, medium → warn, low → silent. - §8 New QMS document outlines (encryption_at_rest, access_control, risk_treatment_plan, statement_of_applicability) — Faz 23 ships. - §9–§10 Existing SOP expansion plans (incident_response, change_management) — adds ISO control mapping rows. - §11 Deployer-side audit cookbook outline. - §12 CI/CD integration plan (CI + nightly + publish + planned tools). - §13 Faz 23 work breakdown — used as the canonical task list. - §14 Acceptance criteria (length, control coverage, work breakdown, cited symbols). - §15 Out-of-scope items (PCI/HIPAA/NIST CSF mappings, SPDX SBOM, pen-test, bug-bounty, FedRAMP). - §16 Decision log — 10 material decisions captured (alignment-not- certified wording, CycloneDX-not-SPDX choice, security-extra packaging, bandit scope, EN-first-TR-in-Faz-26 sequencing, etc.). - §17 Deployer FAQ — 10 anticipated audit-team questions answered. Important framing decisions captured in §16 Decision Log: - **D-22-01: Customer-facing wording is "alignment", never "certified" / "compliant".** Software cannot be ISO certified; only organisations can. Mis-wording invites regulator pushback. - **D-22-02: SBOM stays CycloneDX 1.5 (Wave 2 emitter).** SPDX conversion is one cyclonedx-py call away if a deployer requires it. - **D-22-03: pip-audit + bandit ship as `[security]` optional extra**, not core dep — keeps install lean, CI installs explicitly. - **D-22-05: New QMS docs ship EN-only in Faz 23; TR mirrors land in Faz 26** with the existing-QMS bilingual sweep — avoids cross-wave doc-pair churn. Cited ForgeLM symbols verified against closure/wave4-integration HEAD b87c872. All 93 Annex A controls + 5 SOC 2 categories table- mapped to real ForgeLM features. Bilingual parity 9/9 OK (this file lives under docs/analysis/ which is not in `_PAIRS`; doesn't introduce a TR mirror obligation). Wave 4 phase plan: Faz 22 (this commit) → Faz 23 (impl, next) → Faz 26 (QMS TR mirrors) → Faz 30 (final docs + site). Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + .../iso27001-soc2-alignment-202605052315.md | 865 ++++++++++++++++++ 2 files changed, 866 insertions(+) create mode 100644 docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md diff --git a/.gitignore b/.gitignore index 351536e2..0711b9d8 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ docs/analysis/code_reviews/* !docs/analysis/code_reviews/wave2b-final-merge-review-prompt.md !docs/analysis/code_reviews/wave3-review-prompt.md !docs/analysis/code_reviews/wave3-followup-review-prompt.md +!docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md # Generated by tools/build_usermanuals.py — produced from # docs/usermanuals//**/*.md during the CI deploy. Not committed; diff --git a/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md b/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md new file mode 100644 index 00000000..704e208d --- /dev/null +++ b/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md @@ -0,0 +1,865 @@ +# ISO 27001 / SOC 2 Type II — ForgeLM Alignment Design (Faz 22) + +> Branch: `closure/wave4-integration` · Frozen: `2026-05-05 23:15` +> Closure-plan reference: [Faz 22](./closure-plan-202604300906.md#faz-22--iso-27001--soc-2-type-ii-alignment--analysis--gap-assessment-tek-pr-design-dosyası) +> Faz 22 acceptance bar: ≥800 satır design — see §13 acceptance criteria + +## Status + +**Wave 4 design pass.** Faz 22 ships a single deliverable: this document. Faz 23 +implements the gaps identified here (pip-audit + bandit CI, four new QMS docs, +deployer guide, two reference tables, README + CHANGELOG entries). Faz 26 picks +up QMS bilingual mirror obligations downstream of Faz 23. + +**Important framing.** ForgeLM is a Python library + CLI. *Software cannot be +ISO 27001 certified* — only **organisations** running an ISMS can. What +ForgeLM *can* do is make a deployer's ISO 27001 certification audit AND a SOC 2 +Type II audit demonstrably faster: the audit-trail, change-management, +data-governance, encryption, and incident-response evidence the auditor +requests is produced by ForgeLM as a byproduct of running the training +pipeline. This document is the deployer's evidence map. + +The phrasing in customer-facing material is therefore **"ISO 27001 / SOC 2 +Type II alignment"** — never "certified" / "compliant". The README ISO/SOC 2 +section in Faz 23 follows this rule explicitly. + +## 1. Background and scope + +### 1.1 Why this matters + +Most prospective deployers of an LLM fine-tuning toolkit operate inside an +organisation that already holds (or is pursuing) ISO 27001 + SOC 2 Type II. The +auditor will ask: + +- *"Show me the audit trail for every training run that produced model X."* +- *"Show me the access controls — who approved the deployment, when?"* +- *"Show me the data lineage: where did the training set come from, what's its + fingerprint?"* +- *"Show me the supply-chain inventory: what dependencies did this model + process?"* +- *"Show me the incident response runbook: what happens if the safety + classifier crashes during a run?"* +- *"Show me the encryption-at-rest policy for model weights and audit logs."* + +Every one of these questions has a precise mechanical answer in ForgeLM's +current code surface. This document maps each ISO 27001:2022 Annex A control +and each SOC 2 Trust Services Criterion to that answer. + +### 1.2 Audience + +Three readers: + +1. **The deployer's compliance team** — uses §3 (ISO Annex A mapping) and §4 + (SOC 2 TSC mapping) directly as audit evidence index. +2. **The deployer's engineering team** — uses §6–§8 (SBOM, vuln scanning, + QMS docs) to integrate ForgeLM evidence into their own controls. +3. **The ForgeLM maintainer (Cemil)** — uses §13 (Faz 23 implementation + plan) as the work breakdown for the next wave. + +### 1.3 What ForgeLM is NOT in this framing + +- **Not a substitute for the deployer's ISMS.** ForgeLM does not police + organisational policies, employment contracts, physical security, or + third-party vendor management. Annex A controls in those areas (A.5.x + organisational, A.6.1–A.6.6 people, A.7.x physical) are deployer-side full + stop. +- **Not a CA / certificate authority.** Salt + HMAC chain are tamper-evident, + not non-repudiation in a PKI sense. +- **Not a SIEM / log aggregator.** ForgeLM emits audit events to a JSONL file; + the deployer ships them to Splunk / Datadog / etc. via webhook or filebeat. +- **Not a key-management system.** ForgeLM consumes `FORGELM_AUDIT_SECRET` + from the env; the deployer rotates the secret in their KMS. + +This is the same "deployer-toolkit boundary" articulated in +`docs/marketing/strategy/05-yapmayacaklarimiz.md` and the root `CLAUDE.md` +"What ForgeLM is not" section. + +### 1.4 Bağımlılıklar (closure-plan) + +Faz 22 depends on every functional baseline that produces auditable evidence: + +| Dep | Wave | What it gives Faz 22 | +|-----|------|---------------------------------------------------| +| F-3 | W1 | Operator identity (`FORGELM_OPERATOR`, audit bind)| +| F-6 | W1 | `forgelm verify-audit` (chain integrity) | +| F-7 | W1 | `safe_post` HTTP discipline (SSRF / TLS / scheme) | +| F-8 | W1 | Webhook lifecycle vocabulary | +| F-9 | W1 | Article 14 staging + `forgelm approve`/`reject` | +| F-21| W2b | GDPR Article 17 erasure (`forgelm purge`) | +| F-38| W3 | GDPR Article 15 access (`forgelm reverse-pii`) | + +Wave 4 (this design + Faz 23 implementation) closes the deployer-cookbook +gap that previously existed. + +## 2. ForgeLM control surface inventory + +Cross-reference table of every ForgeLM feature that produces or enforces +auditable evidence. Cited symbols are real (verified against +`closure/wave4-integration` HEAD `b87c872`). + +| Capability | Symbol | Evidence artefact | +|-------------------------------------|------------------------------------------------------------|---------------------------------------------------------| +| Append-only audit log | `forgelm.compliance.AuditLogger` | `audit_log.jsonl` + `.manifest.json` | +| HMAC chain verification | `forgelm verify-audit` (+ `--require-hmac`) | exit 0/1/2/3, JSON envelope | +| Operator identity | `FORGELM_OPERATOR`, `getpass.getuser()` fallback | every audit entry's `operator` field | +| Genesis manifest sidecar | `_check_genesis_manifest`, manifest-truncation refusal | `/audit_log.manifest.json` | +| Salted identifier hashing | `_purge._resolve_salt`, `_purge._hash_target_id` | `data.erasure_*` `target_id`, `data.access_request_query` `query_hash` | +| Per-output-dir salt + env XOR | `/.forgelm_audit_salt` + `FORGELM_AUDIT_SECRET`| salt-source label in audit events | +| GDPR Article 17 erasure | `forgelm purge` (Phase 21) | `data.erasure_requested/completed/failed/warning_*` | +| GDPR Article 15 access | `forgelm reverse-pii` (Phase 38) | `data.access_request_query` | +| Article 14 staging gate | `forgelm approve` / `reject` (Phase 9) | `human_approval.required/granted/rejected` | +| Approval listing | `forgelm approvals` (Phase 37) | `--pending` + `--show` flows | +| Pre-flight env check | `forgelm doctor` (Phase 34) | check categories: python/cuda/extras/HF auth/disk/operator | +| Air-gap pre-cache | `forgelm cache-models` / `cache-tasks` (Phase 35) | `cache.populate_*_requested/completed/failed` | +| Standalone safety eval | `forgelm safety-eval` (Phase 36) | `safety_results.json`, exit-code gate | +| Annex IV bundle verification | `forgelm verify-annex-iv` (Phase 36) | manifest-hash check, missing-artefact detection | +| GGUF integrity | `forgelm verify-gguf` (Phase 36) | magic header + `.sha256` sidecar | +| Risk classification gate | `_warn_high_risk_compliance` (Faz 28 strict gate) | `ConfigError` for high-risk + safety-disabled | +| Auto-revert | `evaluation.auto_revert` + safety-eval | `pipeline.reverted` audit event | +| Webhook lifecycle | `notify_started/succeeded/failed/reverted/awaiting_approval` (Phase 8) | webhook event payload + audit trail | +| HTTP discipline | `safe_post` (Phase 7) | SSRF guard, TLS-only, no-redirect, header masking | +| Data audit pipeline | `forgelm audit` (Phase 11) | `data_audit_report.json` (PII / secrets / dedup) | +| Annex IV §8 governance | `_data_governance_block` (compliance.py) | `data_governance_report.json` Article 10 | +| Model integrity | `compute_artefact_sha256` | `model_integrity.json` | +| SBOM | `tools/generate_sbom.py` (Wave 2 era) | CycloneDX 1.5 JSON per (OS, py-version), uploaded as release artefact | +| Bilingual EN+TR docs | `tools/check_bilingual_parity.py` (Phase 24) | strict-mode diff exit code | + +Not in this table: features that exist in the codebase but do not produce +audit-grade evidence (e.g. CLI argument parsing, Pydantic models for non- +compliance fields, training metric snapshots without HMAC). + +## 3. ISO 27001:2022 Annex A control mapping + +ISO 27001:2022 Annex A organises 93 controls into four themes: +- **A.5 Organisational** (37 controls) +- **A.6 People** (8 controls) +- **A.7 Physical** (14 controls) +- **A.8 Technological** (34 controls) + +Each row classifies coverage as one of: + +- **`FL`** — *ForgeLM-supported*: ForgeLM directly produces audit evidence. +- **`FL-helps`** — *Deployer responsibility, ForgeLM helps*: ForgeLM provides + partial evidence the deployer combines with other sources. +- **`OOS`** — *Out of scope*: deployer-only, ForgeLM contributes nothing. + +### 3.1 A.5 Organisational controls + +| Control | Coverage | ForgeLM evidence | Deployer action | Out-of-scope note | +|---|---|---|---|---| +| A.5.1 Policies | FL-helps | Audit event vocabulary in `audit_event_catalog.md` documents what's logged | Ratify ISMS policy referencing ForgeLM audit log as authoritative source | ForgeLM does not enforce policy; only logs evidence | +| A.5.2 Roles & responsibilities | FL-helps | `FORGELM_OPERATOR` + `roles_responsibilities.md` QMS template | Appoint AI Officer / ML Lead / Data Steward / DPO with named owners | Org-chart specifics deployer-side | +| A.5.3 Segregation of duties | FL-helps | `human_approval.required` / `granted` / `rejected` events; `forgelm approve` requires ≠ trainer operator-id | Enforce CI-runner identity ≠ deployment-approver identity at IdP layer | ForgeLM records the IDs; doesn't enforce IdP-level separation | +| A.5.4 Management responsibilities | FL-helps | `forgelm doctor` env summary; `compliance_report.json`; `training_manifest.yaml` | Monthly review of audit-log + safety-eval outcomes; document escalation | Management process deployer-side | +| A.5.5 Contact with authorities | OOS | — | Establish regulator contact for serious-incident reporting (EU AI Act Art. 73) | ForgeLM cannot interface with regulators | +| A.5.6 Contact with special interest groups | OOS | — | ML safety / red-team community engagement | Out of scope | +| A.5.7 Threat intelligence | OOS | — | Subscribe to ML supply-chain CVE feeds, model-poisoning advisories | Out of scope | +| A.5.8 Information security in project management | FL-helps | `risk_assessment` config block; F-compliance-110 strict gate; Annex IV §9 metadata | Embed ForgeLM dry-run + safety-eval gates in project sign-off checklist | Project methodology deployer-side | +| A.5.9 Inventory of information & associated assets | FL-helps | `data_provenance.json`, `model_integrity.json`, SBOM | Maintain corporate asset register linking model SHA + data fingerprint | Org asset registry deployer-side | +| A.5.10 Acceptable use of information | OOS | — | Acceptable-use policy for trained-model outputs | Out of scope | +| A.5.11 Return of assets | OOS | — | Decommissioning of deployed models | Out of scope | +| A.5.12 Classification of information | FL-helps | `compliance.risk_classification` 5-tier (`unacceptable`/`high-risk`/`limited-risk`/`minimal-risk`/`out-of-scope`) | Map ForgeLM risk tiers to corporate confidentiality classes | Org classification scheme deployer-side | +| A.5.13 Labelling of information | FL-helps | `model_card.md` produced per run; risk classification stamped into manifest | Apply org-wide labelling on top of ForgeLM artefacts | Label taxonomy deployer-side | +| A.5.14 Information transfer | FL-helps | `safe_post` webhook discipline (TLS / SSRF / scheme); no plaintext PII in webhook payload | Sign data-transfer agreements with webhook recipients (Slack, Teams, etc.) | DTA legal deployer-side | +| A.5.15 Access control | FL-helps | Operator identity attribution; salted identifier hashing | IdP integration + per-runner credentials | Access-policy authoring deployer-side | +| A.5.16 Identity management | FL-helps | `FORGELM_OPERATOR` env contract; `getpass.getuser()` fallback | Configure CI runner to set FORGELM_OPERATOR from CI metadata | Identity directory deployer-side | +| A.5.17 Authentication information | FL-helps | `safe_post` rejects auth headers in non-HTTPS requests; `_mask` hides tokens in error logs | Vault-store webhook secrets, HF tokens, model-registry credentials | KMS deployer-side | +| A.5.18 Access rights | FL-helps | `human_approval` gate gives a recordable "deny" path | Manage reviewer-list per `evaluation.require_human_approval` | RBAC deployer-side | +| A.5.19 Information security in supplier relationships | FL-helps | `_fingerprint_hf_revision` pins HF Hub commit SHA; SBOM lists every transitive dep | Vendor risk assessments for HF, base-model providers | Vendor questionnaires deployer-side | +| A.5.20 Addressing information security within supplier agreements | OOS | — | Supplier MSA security clauses | Out of scope | +| A.5.21 Managing information security in the ICT supply chain | FL-helps | SBOM (CycloneDX 1.5 per OS/py-version) attached to every release; `pip-audit` (Faz 23) | Track upstream CVE advisories; subscribe to GitHub Dependabot for forgelm itself | Org-wide supply-chain program deployer-side | +| A.5.22 Monitoring, review and change management of supplier services | OOS | — | Supplier annual review | Out of scope | +| A.5.23 Information security for use of cloud services | OOS | — | Cloud-provider security configuration | ForgeLM is cloud-agnostic | +| A.5.24 Information security incident management planning and preparation | FL-helps | `sop_incident_response.md` QMS template; audit chain preserves incident state | Establish IR team, on-call rotation | Org structure deployer-side | +| A.5.25 Assessment and decision on information security events | FL-helps | `data.erasure_failed`, `pipeline.failed`, `audit.classifier_load_failed` events with `error_class`+`error_message` | Triage runbook tied to event class | Triage process deployer-side | +| A.5.26 Response to information security incidents | FL-helps | Audit chain preserves before/after state through HMAC chain | Document runbook actions per event class | Runbook deployer-side | +| A.5.27 Learning from information security incidents | FL-helps | `pipeline.reverted` events accumulate post-mortem evidence | Weekly post-mortem cadence | Process deployer-side | +| A.5.28 Collection of evidence | FL | `audit_log.jsonl` is forensic-grade (HMAC + hash chain + manifest sidecar); `forgelm verify-audit` validates | Ship to write-once storage (S3 Object Lock, Azure Immutable Blob) | Storage substrate deployer-side | +| A.5.29 Information security during disruption | FL-helps | `auto_revert` flips to baseline model on safety regression; `pipeline.reverted` event | Document base-model retention; multi-region replicas | DR infra deployer-side | +| A.5.30 ICT readiness for business continuity | OOS | — | DR planning | Out of scope | +| A.5.31 Identification of legal, statutory, regulatory and contractual requirements | FL-helps | EU AI Act mapping in `compliance.py`; GDPR Article 15/17 handling; Annex IV bundle | Track jurisdictional rule changes (e.g. UK Online Safety Act updates) | Legal monitoring deployer-side | +| A.5.32 Intellectual property rights | FL-helps | License extraction in SBOM; HF Hub model-card metadata recorded in manifest | Track per-model license terms | Legal review deployer-side | +| A.5.33 Protection of records | FL | `audit_log.jsonl` append-only with HMAC + manifest sidecar (`_check_genesis_manifest` refuses truncate-and-resume) | Off-site replicate the audit log | Storage substrate deployer-side | +| A.5.34 Privacy and protection of PII | FL | `forgelm reverse-pii` Article 15; `forgelm purge` Article 17; `forgelm audit` PII detection + masking; salted hashing in audit events | Subject access request workflow + DPIA | Workflow + DPIA template deployer-side | +| A.5.35 Independent review of information security | OOS | — | Annual external audit | Out of scope | +| A.5.36 Compliance with policies, rules and standards for information security | FL-helps | Pydantic config validation; `forgelm doctor`; CI gates (`forgelm --dry-run`) | Org-wide policy enforcement | Out of scope (org governance) | +| A.5.37 Documented operating procedures | FL-helps | QMS templates (5 SOPs in `docs/qms/` shipping; 4 more added Faz 23) | Adapt to org context | Org-specific deployer-side | + +### 3.2 A.6 People controls + +| Control | Coverage | ForgeLM evidence | Deployer action | Out-of-scope note | +|---|---|---|---|---| +| A.6.1 Screening | OOS | — | Background checks | HR domain | +| A.6.2 Terms and conditions of employment | OOS | — | NDAs / IP clauses | HR domain | +| A.6.3 Information security awareness, education and training | FL-helps | `audit_event_catalog.md` doubles as training material; `deployer_instructions.md` Article 13 output | Annual EU AI Act + GDPR training | Curriculum deployer-side | +| A.6.4 Disciplinary process | FL-helps | Operator attribution on every audit entry preserves accountability | Disciplinary process tied to `FORGELM_OPERATOR` audit | Out of scope (HR) | +| A.6.5 Responsibilities after termination or change of employment | FL-helps | Operator identity rotation simply changes `FORGELM_OPERATOR`; old IDs remain in audit history | Revoke departed staff's CI runner access; audit prior actions | Out of scope (HR) | +| A.6.6 Confidentiality or non-disclosure agreements | OOS | — | NDA on training data | Legal deployer-side | +| A.6.7 Remote working | FL-helps | `forgelm doctor --offline`; air-gap pre-cache (Phase 35) | VPN policy + endpoint hardening | Endpoint security deployer-side | +| A.6.8 Information security event reporting | FL-helps | `pipeline.failed`, `data.erasure_failed`, `audit.classifier_load_failed` events accumulate; `forgelm webhook` notifies external systems | SIEM ingestion + alerting | Out of scope (SIEM) | + +### 3.3 A.7 Physical controls + +ForgeLM is a software toolkit. All A.7 controls are **OOS**: + +| Control | Note | +|---|---| +| A.7.1 Physical perimeters | Datacenter / office security | +| A.7.2 Physical entry | Badge access | +| A.7.3 Securing offices, rooms, facilities | Clean-desk policy | +| A.7.4 Physical security monitoring | CCTV | +| A.7.5 Protecting against physical and environmental threats | Fire / flood / temp | +| A.7.6 Working in secure areas | SCIF compliance | +| A.7.7 Clear desk and clear screen | Endpoint policy | +| A.7.8 Equipment siting and protection | Hardware placement | +| A.7.9 Security of assets off-premises | Mobile-device management | +| A.7.10 Storage media | Disk-encryption policy on training nodes (deployer KMS) | +| A.7.11 Supporting utilities | UPS / generator | +| A.7.12 Cabling security | Datacenter cabling | +| A.7.13 Equipment maintenance | Hardware refresh | +| A.7.14 Secure disposal or re-use of equipment | E-waste / decommissioning | + +The deployer's existing IT operations procedures cover the entire A.7 theme. +ForgeLM contributes nothing here. + +### 3.4 A.8 Technological controls + +| Control | Coverage | ForgeLM evidence | Deployer action | +|---|---|---|---| +| A.8.1 User endpoint devices | FL-helps | `forgelm doctor` — Python / CUDA / GPU / extras / HF auth / disk / `FORGELM_OPERATOR` checks | Endpoint hardening (disk encryption, MDM) | +| A.8.2 Privileged access rights | FL-helps | Operator attribution on every audit entry; approval gate requires explicit non-trainer operator | RBAC at IdP layer | +| A.8.3 Information access restriction | FL | Salted identifier hashing in audit events; `forgelm reverse-pii` exposes Article 15 access path; `data_audit_report.json` PII registry | Subject access request workflow | +| A.8.4 Access to source code | FL-helps | `model.trust_remote_code` defaults `False`; `_fingerprint_hf_revision` pins commit SHA | VCS access control | +| A.8.5 Secure authentication | FL-helps | `safe_post` rejects auth headers on non-HTTPS; webhook secret discipline | MFA + token rotation | +| A.8.6 Capacity management | FL-helps | `forgelm doctor` reports VRAM / CPU / RAM; `resource_usage` block in manifest | Quota / autoscaling | +| A.8.7 Protection against malware | OOS | — | Antivirus on training hosts | +| A.8.8 Management of technical vulnerabilities | FL-helps | SBOM (CycloneDX 1.5); `pip-audit` nightly (Faz 23); `bandit` CI (Faz 23); Pydantic config validation | OS patch management | +| A.8.9 Configuration management | FL | YAML config validated via Pydantic; `forgelm --dry-run` gate; `compliance.config_hash` in audit events | Version-control configs | +| A.8.10 Information deletion | FL | `forgelm purge` Article 17; salted-hash audit; `data.erasure_warning_memorisation` flags model-weight retention risk | DSAR workflow | +| A.8.11 Data masking | FL | `forgelm audit` regex + Presidio ML-NER masking; `_SECRET_PATTERNS` credentials scan | Masking strategy decisions (replace vs delete) | +| A.8.12 Data leakage prevention | FL | `forgelm reverse-pii` plaintext residual scan; webhook never carries raw training rows | DLP at egress / endpoint | +| A.8.13 Information backup | FL-helps | `audit_log.jsonl` + `.manifest.json` form a backupable unit; Annex IV bundle (`forgelm --compliance-export`) | Off-site backup with encryption | +| A.8.14 Redundancy of information processing facilities | OOS | — | Multi-AZ infra | +| A.8.15 Logging | FL | `audit_log.jsonl` JSON Lines + HMAC chain + genesis manifest; `forgelm verify-audit` validates | Ship to SIEM | +| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle events; `safety_trend.jsonl` cross-run trend tracking | Define alert thresholds | +| A.8.17 Clock synchronisation | FL-helps | All audit entries ISO-8601 UTC | NTP on training hosts | +| A.8.18 Use of privileged utility programs | OOS | — | OS-level privilege management | +| A.8.19 Installation of software on operational systems | FL-helps | `forgelm doctor` reports installed packages + versions; `pyproject.toml` pins; `pip install forgelm==X.Y.Z` | Package allowlist | +| A.8.20 Networks security | FL-helps | `safe_post` HTTPS-only, SSRF guard, no redirect; `model.trust_remote_code=False` default | Egress firewall | +| A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement; cert rotation | +| A.8.22 Segregation of networks | OOS | — | VPC / subnet design | +| A.8.23 Web filtering | OOS | — | Egress proxy | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC-SHA-256 (audit chain); salted SHA-256 identifier hashing; per-output-dir salt + env XOR derivation | KMS for `FORGELM_AUDIT_SECRET` | +| A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md` + `docs/standards/release.md` + CI gates (ruff, pytest, dry-run, parity, SBOM) | SDLC framework | +| A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; Pydantic config validation; ReDoS guard in `_reverse_pii` | App-level threat modelling | +| A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture (`flock`+`fsync` per line); audit chain HMAC; lazy import discipline; `safe_post` SSRF guard | Defence-in-depth design | +| A.8.28 Secure coding | FL-helps | `docs/standards/coding.md` standards; type hints throughout; CommonMark escaping in `_sanitize_md_list` | Custom-extension review | +| A.8.29 Security testing in development and acceptance | FL-helps | `pytest` 1300+ tests (post-Wave 3); `bandit` static analysis (Faz 23); `forgelm safety-eval` standalone gate | E2E security tests | +| A.8.30 Outsourced development | OOS | — | Third-party-developer security | +| A.8.31 Separation of development, test and production environments | FL-helps | `forgelm --dry-run` flag; staging dir (Phase 9); `evaluation.require_human_approval` gate | Separate pipelines | +| A.8.32 Change management | FL | `human_approval.required/granted/rejected` chain; `compliance.config_hash`; staging dir snapshot | Change Advisory Board process | +| A.8.33 Test information | FL-helps | `forgelm audit` flags PII / secrets in test sets too | Test-data-handling policy | +| A.8.34 Protection of information systems during audit testing | OOS | — | Read-only audit access | + +**Coverage tally for §3:** 93 controls scored. **`FL` (full)**: 11 · +**`FL-helps`**: 50 · **`OOS`**: 32. The OOS bucket is dominated by +A.7 physical (14) + A.5 organisational governance (~10) + A.8 cloud / +network (~8) — all expected for a deployer-side toolkit boundary. + +## 4. SOC 2 Trust Services Criteria mapping + +The 2017 SOC 2 framework (revised 2022) defines 5 categories: + +- **Security** (Common Criteria CC1.x–CC9.x) — mandatory baseline. +- **Availability** (A1.x) — optional. +- **Processing Integrity** (PI1.x) — optional. +- **Confidentiality** (C1.x) — optional. +- **Privacy** (P1.x–P8.x) — optional. + +A SOC 2 *Type II* audit observes operating effectiveness over a 6–12 month +window. Every Common Criteria control here is expected; the optional +categories are scoped per-engagement. + +### 4.1 Security — Common Criteria + +| CC | Title | ForgeLM evidence | Deployer gap | +|---------|----------------------------------------|------------------------------------------------------------------|--------------| +| CC1.1 | Demonstrates commitment to integrity | `roles_responsibilities.md` QMS template; AI-Officer role definition | Sign tone-at-the-top doc | +| CC1.2 | Independence of governance | Approval gate operator ≠ trainer | Reporting structure | +| CC1.3 | Establishes structures, reporting lines| `roles_responsibilities.md` AI Officer / ML Lead / DPO / Compliance Officer | Adopt | +| CC1.4 | Demonstrates commitment to competence | Annual EU AI Act / GDPR training (deployer policy) | Curriculum + record | +| CC1.5 | Enforces accountability | `FORGELM_OPERATOR` attribution + audit chain | IdP integration | +| CC2.1 | Communicates internally about controls | `audit_event_catalog.md` + `deployer_instructions.md` | Internal comms plan | +| CC2.2 | Communicates externally | Article 13 `deployer_instructions.md`; Annex IV public summary | Customer-facing comms | +| CC2.3 | Communicates with regulators | Annex IV bundle ready to ship | Regulator contact list | +| CC3.1 | Specifies suitable objectives | `compliance.intended_purpose`; risk classification | Adopt | +| CC3.2 | Identifies and analyses risks | `risk_assessment` Pydantic block; safety eval | Risk register | +| CC3.3 | Considers fraud risks | Audit log tamper-evidence; HMAC chain; manifest sidecar | Fraud taxonomy | +| CC3.4 | Identifies and assesses changes | `human_approval.required` gate; `pipeline.config_hash` | Change review board | +| CC4.1 | Selects, develops, performs evaluations| `forgelm verify-audit`; `forgelm safety-eval` | Periodic schedule | +| CC4.2 | Communicates internal-control deficiencies | `pipeline.failed`/`reverted`/`erasure_failed` audit events | Deficiency reporting | +| CC5.1 | Selects, develops control activities | F-compliance-110 strict gate; auto-revert; staging | Control matrix | +| CC5.2 | Selects general IT controls | `safe_post` HTTP discipline; Pydantic config validation | Adopt | +| CC5.3 | Deploys policies and procedures | `docs/qms/` 5 SOPs (Wave 0); 4 new in Faz 23 | Adoption record | +| CC6.1 | Logical-access security software | Operator-id attribution; HMAC chain | IdP / RBAC | +| CC6.2 | Authorises new internal users | `human_approval.required`/`granted` chain | Onboarding workflow | +| CC6.3 | Removes access for terminated users | (Deployer revokes CI runner identity) | Offboarding workflow | +| CC6.4 | Restricts physical access | OOS | Datacenter security | +| CC6.5 | Protects against unauthorised disposal | `forgelm purge` Article 17; salted hashing | Off-site disposal | +| CC6.6 | Implements logical-access controls | Salted hashing in audit events; PII reverse scan | RBAC | +| CC6.7 | Restricts movement of information | `safe_post` egress discipline; webhook payload curation | Egress policy | +| CC6.8 | Detects/prevents unauthorised software | SBOM (Wave 2 era); `pip-audit` nightly (Faz 23); `bandit` CI (Faz 23) | Allowlist | +| CC7.1 | Detects vulnerabilities | `pip-audit` nightly (Faz 23); CVE feed | Vuln-mgmt cadence | +| CC7.2 | Monitors system components | `forgelm verify-audit`; `forgelm verify-gguf`; `safety_trend.jsonl` | SIEM dashboards | +| CC7.3 | Evaluates security events | `data.erasure_failed`, `pipeline.failed` events with `error_class`+`error_message` | Triage runbook | +| CC7.4 | Responds to security events | `auto_revert`, `pipeline.reverted` event; `data.erasure_failed` event | IR runbook | +| CC7.5 | Identifies, develops corrective actions| `human_approval.rejected` event; `sop_change_management.md` | CAPA cadence | +| CC8.1 | Authorises changes | `forgelm approve` Article 14 gate; staging dir | Change Advisory Board | +| CC9.1 | Identifies, manages risks | `risk_assessment` config + safety eval; `risk_treatment_plan.md` (Faz 23) | Risk register | +| CC9.2 | Manages vendor + business-partner risk | SBOM; HF Hub revision pin; license extraction | Vendor-risk programme | + +### 4.2 Availability (A1.x) + +ForgeLM is a single-node CLI; availability is dominantly deployer-side. + +| Control | ForgeLM contribution | Deployer responsibility | +|---|---|---| +| A1.1 Capacity planning | `forgelm doctor` resource report + `resource_usage` manifest block | Multi-AZ + autoscaling | +| A1.2 Recovery from incidents | `auto_revert` swap-back to baseline; audit chain continuity on resume | Runbook + drills | +| A1.3 Environmental protections | OOS | Datacenter UPS / generator | + +### 4.3 Processing Integrity (PI1.x) — strong ForgeLM contribution + +| Control | ForgeLM contribution | +|---|---| +| PI1.1 Quality of inputs | `compute_dataset_fingerprint` (SHA-256 + size + mtime); `data_governance_report` (collection_method, annotation_process, known_biases, personal_data_included, dpia_completed) | +| PI1.2 System processing | `forgelm verify-audit` validates HMAC chain end-to-end; `data_audit_report.json` flags PII / secrets / dedup before training | +| PI1.3 Outputs are accurate | `model_integrity.json` SHA-256 checksums per artefact; `model_card.md` with HF YAML front-matter | +| PI1.4 Inputs traceable | `_describe_adapter_method` canonicalisation; `pipeline.config_hash`; HF-revision pin | +| PI1.5 Outputs traceable | Annex IV bundle co-locates manifest + report + audit + integrity in one ZIP | + +### 4.4 Confidentiality (C1.x) + +| Control | ForgeLM contribution | +|---|---| +| C1.1 Protection of confidential information | `forgelm audit` regex + Presidio ML-NER PII detection; `_SECRET_PATTERNS` credentials scan; AWS / GitHub / Slack / OpenAI / Google / JWT key matchers | +| C1.2 Disposal of confidential information | `forgelm purge` Article 17 with salted-hash audit (no raw identifier in chain); `data.erasure_warning_memorisation` flags model-weight retention risk | + +### 4.5 Privacy (P1.x – P8.x) + +| Control | ForgeLM contribution | +|---|---| +| P1.1 Privacy notice | Article 13 `deployer_instructions.md` | +| P2.1 Choice and consent | `evaluation.require_human_approval` Article 14 gate | +| P3.1 Collection | `data.governance.personal_data_included` boolean; `dpia_completed` flag | +| P3.2 Quality of personal data | `data_audit_report.json` quality stats | +| P4.1 Use, retention, and disposal | `evaluation.approval_retention_days` + `forgelm purge --check-policy` retention audit | +| P5.1 Access | `forgelm reverse-pii` Article 15 scan; salted query-hash in audit | +| P5.2 Inquiries and complaints | (Deployer-side workflow) | +| P6.1 Disclosure to third parties | `safe_post` webhook discipline; HMAC payload signing | +| P6.2 Third-party agreements | (Deployer DPAs) | +| P7.1 Breach notification | `data.erasure_failed`, `audit.classifier_load_failed` events feed breach-detection | +| P7.2 Breach disclosure | (Deployer regulator-contact playbook) | +| P8.1 Inquiries, complaints, and disputes | `forgelm reverse-pii` + `forgelm purge` chain provides forensic evidence | + +## 5. Gap assessment — what ForgeLM does NOT yet ship + +Items that the closure plan + this design identify as gaps closed in +Faz 23, **not** as out-of-scope: + +1. **`pip-audit` nightly integration.** SBOM exists; supply-chain CVE + scanning does not. Faz 23 adds a nightly job that fails on + high-severity findings. +2. **`bandit` CI integration.** Static security scan; `[tool.bandit]` + config block exists in `pyproject.toml` but no workflow consumes it + yet. Faz 23 wires it into `ci.yml`. +3. **`[project.optional-dependencies] security`** extra. Bundles + `pip-audit` + `bandit` for installable security tooling. +4. **`docs/qms/encryption_at_rest.md`** + `-tr.md`. Operator-facing + guidance for model weights / audit logs / training data encryption. +5. **`docs/qms/access_control.md`** + `-tr.md`. Multi-user pipelines; + OS-level user isolation; `FORGELM_OPERATOR` rotation guidance. +6. **`docs/qms/risk_treatment_plan.md`** + `-tr.md`. Risk register + format with ForgeLM identified-risk rows pre-filled. +7. **`docs/qms/statement_of_applicability.md`** + `-tr.md`. ISO Annex A + × applicable / excluded × justification matrix (the table from §3 + reformatted as the QMS-style SoA). +8. **`docs/guides/iso_soc2_deployer_guide.md`** + `-tr.md`. Deployer- + facing audit cookbook ("how to use ForgeLM evidence in your ISO / + SOC 2 audit"). Step-by-step. +9. **`docs/reference/iso27001_control_mapping.md`** + `-tr.md`. The §3 + table as a reference doc (vs design-doc). +10. **`docs/reference/soc2_trust_criteria_mapping.md`** + `-tr.md`. The + §4 table as a reference doc. +11. **README ISO/SOC 2 alignment section.** 1 paragraph + link to the + deployer guide. +12. **`sop_incident_response.md` expansion.** ISO A.5.24–A.5.27 mapping; + security-incident playbook that runs alongside the existing safety- + incident text. +13. **`sop_change_management.md` expansion.** ISO A.8.32 mapping; CI + gates explicitly named as the change-control mechanism. +14. **CHANGELOG entry.** Wave 4 / Faz 22-23 line. +15. **`tests/test_sbom.py::test_deterministic`.** Same git tree → byte- + identical SBOM; pin determinism contract. + +## 6. SBOM tooling — current state and Faz 23 additions + +**What's already shipped (Wave 2 era):** + +`tools/generate_sbom.py` produces CycloneDX 1.5 JSON (177 lines, pure +stdlib emitter). `.github/workflows/publish.yml` runs it on every +release tag in the `cross-os-tests` job (line 94–101) and uploads +`sbom--py.json` per (OS, Python) pair as artefacts. + +**Why CycloneDX, not SPDX (closure-plan revision).** The closure plan +originally called for SPDX 2.3. The Wave 2 implementer chose CycloneDX +1.5 instead because: + +- Both formats are ISO/SOC 2 auditable; auditors accept either. +- CycloneDX 1.5 has stronger first-class supply-chain semantics + (vulnerability disclosures, BOM dependencies graph). +- `Dependency-Track` ≥ 4.10 — the most common open-source SBOM + consumer — natively ingests CycloneDX 1.5. +- The existing emitter has zero external dependencies (pure stdlib + + `importlib.metadata`). + +Faz 23 keeps CycloneDX 1.5; SPDX format addition is deferred to a +later wave if a deployer specifically requests SPDX. (The deployer +guide in Faz 23 documents this choice and offers the user a +"convert with `cyclonedx-py`" recipe if SPDX is required.) + +**What Faz 23 adds:** + +- `tests/test_sbom.py::test_deterministic` — same input must produce + byte-identical output. +- `tests/test_sbom.py::test_cyclonedx_schema_valid` — JSON validates + against CycloneDX 1.5 schema (use `jsonschema` from optional dev + extra, not at runtime). +- Documentation: `docs/reference/supply_chain_security.md` (+ TR) + documenting the SBOM artefact, where to download it, how to + consume it. + +## 7. Vulnerability scanning — Faz 23 implementation plan + +### 7.1 `pip-audit` (supply-chain CVE scan) + +- **Where:** `.github/workflows/nightly.yml` (existing daily 03:00 UTC). +- **Step:** + ```yaml + - name: pip-audit (supply-chain CVE scan) + if: always() # run even if previous step soft-failed + run: | + pip install 'pip-audit>=2.7.0,<3.0.0' + pip-audit --strict --format json --output /tmp/pip-audit.json || true + python tools/check_pip_audit.py /tmp/pip-audit.json + ``` +- **Behaviour:** New helper `tools/check_pip_audit.py` parses the + JSON; `high`-severity findings exit 1 (fail nightly); medium / low + emit warnings via `::warning::` GitHub annotation. +- **Suppression:** When a CVE is acknowledged + tracked, list it in + `tools/.pip-audit-ignore.json` with rationale + review date. + +### 7.2 `bandit` (static code security) + +- **Where:** `.github/workflows/ci.yml` (every PR + push). +- **Scope:** `forgelm/` only (`exclude_dirs = ["tests", ".venv", "build", "dist"]` + already in `pyproject.toml`). +- **Step:** + ```yaml + - name: bandit (static security scan) + run: | + pip install 'bandit[toml]>=1.7.0,<2.0.0' + bandit -c pyproject.toml -r forgelm/ -f json -o /tmp/bandit.json || true + python tools/check_bandit.py /tmp/bandit.json + ``` +- **Behaviour:** New helper `tools/check_bandit.py` parses the JSON; + `HIGH` severity exits 1, `MEDIUM` warns, `LOW` silent. +- **Suppression:** Inline `# nosec B101` (or whichever ID) with + justification in a trailing comment on the same line. + +### 7.3 Optional extra + +`pyproject.toml [project.optional-dependencies]`: + +```toml +security = [ + "pip-audit>=2.7.0,<3.0.0", + "bandit[toml]>=1.7.0,<2.0.0", +] +``` + +Operators install via `pip install forgelm[security]` if they want +to run the same checks locally. + +### 7.4 Determinism + +The output of `pip-audit` is non-deterministic (depends on CVE +database state); the goal is "no high-sev finding", not bit-stability. +The output of `bandit` IS deterministic over a fixed code tree; +`tests/test_bandit_clean.py` (post-Faz-23) pins the exit-code-0 +contract for a clean tree. + +## 8. New QMS document plans + +The four documents listed in §5 (#4–#7) are authored in Faz 23. Each +follows the existing QMS template style (see `docs/qms/sop_*.md` for +Wave 0 templates). Bilingual (`-tr.md` siblings). + +### 8.1 `encryption_at_rest.md` + +Outline (~150 lines): + +1. Purpose — operator-facing guidance for what ForgeLM does NOT + encrypt, and what the deployer must encrypt. +2. Scope — model weights (`final_model/`, `staging_model./`), + audit logs (`audit_log.jsonl` + `.manifest.json`), training data + (operator-supplied), config files (`config.yaml`). +3. Threat model — disk theft, backup theft, shared-tenancy disk + leak, log-shipping intercept. +4. Recommended controls per asset class — LUKS / dm-crypt for whole- + disk, S3 SSE-KMS for blob, GPG / age for individual files, + `gpg-agent` for `FORGELM_AUDIT_SECRET`. +5. ForgeLM contribution — none structurally (operator chooses + substrate); ForgeLM produces the auditable evidence on top. +6. Verification — `forgelm verify-audit` works on encrypted-at-rest + audit logs as long as the operator decrypts before invoking; the + audit chain itself is integrity-protected, encryption is + confidentiality-only. + +### 8.2 `access_control.md` + +Outline (~180 lines): + +1. Purpose — multi-user / multi-pipeline ForgeLM deployments. +2. `FORGELM_OPERATOR` env contract — required for every CI run; rotated + when staff change. +3. OS-level isolation — separate Unix user per pipeline; `chmod 0700` + on `output_dir`; `chown` on audit logs. +4. CI runner identity — best practice: GitHub Actions OIDC token → + `FORGELM_OPERATOR=ci//`. +5. Approval gate identity separation — trainer ≠ approver enforced + by `forgelm approve` (audits operator id at approve time). +6. Identity migration when staff change — old `FORGELM_OPERATOR` IDs + remain in the audit history immutably; new identity for new runs. + +### 8.3 `risk_treatment_plan.md` + +Outline (~250 lines): + +Pre-populated risk register following the ISO 27005 risk-treatment +template: + +| ID | Risk | Likelihood | Impact | Treatment | Residual | +|----|------|------------|--------|-----------|----------| +| R-01 | Training-data poisoning (adversarial corpus) | Med | High | `forgelm audit` PII / secrets / quality scan; data fingerprint pinned in manifest; deployer pre-flight review | Med → Low | +| R-02 | Supply-chain compromise (compromised PyPI dep) | Low | High | SBOM; `pip-audit` nightly; `forgelm doctor` pre-flight | Low | +| R-03 | Credential leak (HF token in config / log) | Med | Med | `safe_post` mask; `[ingestion-pii-ml]` secrets scan; `_sanitize_md_list` in deployer instructions | Low | +| R-04 | Audit-log tampering | Low | High | Append-only + HMAC + manifest sidecar; `forgelm verify-audit` | Low | +| R-05 | Memorisation of removed PII (Article 17) | High | Med | `data.erasure_warning_memorisation` flag; `forgelm safety-eval` post-erasure | Med | +| R-06 | Safety-classifier load failure | Med | High | F-compliance-110 strict gate raises `ConfigError`; `audit.classifier_load_failed` event | Low | +| R-07 | Webhook SSRF / data exfiltration | Low | Med | `safe_post` SSRF guard; HTTPS-only; HMAC sign | Low | +| R-08 | ReDoS via `--type custom` regex | Low | Low | POSIX SIGALRM 30s budget in `_scan_file_with_alarm` | Low | +| R-09 | Cross-tool digest mismatch (purge vs reverse-pii) | Low | Med | Salted-SHA-256 reuse via `_resolve_salt`; `salt_source` in audit | Low | +| R-10 | Unauthorised model deployment | Med | High | `evaluation.require_human_approval` Article 14 gate; staging dir | Low | + +Each row gets ~10 lines of detail in the published doc. + +### 8.4 `statement_of_applicability.md` + +Outline (~120 lines): + +The §3 table reformatted as a QMS-style Statement of Applicability: + +| Control | Applicable? | Justification | Implementation status (Wave 4) | +|---|---|---|---| +| A.5.1 Policies for information security | Applicable | EU AI Act Art. 17 mandates QMS | Implemented via `docs/qms/` SOPs | +| ... | ... | ... | ... | + +Plus the explicit "excluded" rows (A.7 physical controls etc.) with +"Excluded — ForgeLM is a software toolkit, deployer infra-side +domain" justification. + +## 9. Incident response runbook expansion (`sop_incident_response.md`) + +**Current state:** 76 lines, focused on AI-safety incidents (model +produces harmful output, accuracy drop, formatting errors). + +**Faz 23 additions:** ~80 new lines for security incidents: + +1. **Audit chain integrity violation** — `forgelm verify-audit` exits + non-zero. Runbook: isolate output_dir, capture `audit_log.jsonl` + + `.manifest.json` + `.sha256` sidecar, contact security team, + re-derive previous trusted state. +2. **Credential leak detected** — `forgelm audit` `_SECRET_PATTERNS` + match in training corpus or webhook log. Runbook: rotate the leaked + credential immediately, run `forgelm purge --row-id `, + flag the run as `data.erasure_warning_memorisation`. +3. **Supply-chain CVE flagged** — `pip-audit` nightly fails high- + severity. Runbook: pin to safe version, rebuild SBOM, regenerate + dependent artefacts, notify deployer if model already shipped. +4. **Webhook target compromised** — webhook recipient confirms breach. + Runbook: rotate `webhook.secret_env`; re-emit lifecycle events from + audit chain to confirm attacker did not splice events. +5. **PII subject access request (Article 15)** — `forgelm reverse-pii` + workflow + DSR response template. +6. **PII erasure request (Article 17)** — `forgelm purge` workflow + + memorisation-warning communication template. + +ISO mapping section gets added: A.5.24, A.5.25, A.5.26, A.5.27, +A.6.8, A.8.15, A.8.16. + +## 10. Change management runbook expansion (`sop_change_management.md`) + +**Current state:** 75 lines, focused on training-config changes +(major / minor / patch). + +**Faz 23 additions:** ~60 new lines: + +1. **CI gates as control** — explicit table of: ruff format / ruff + check / pytest / parity strict / SBOM determinism / dry-run all + green BEFORE merge. ISO A.8.32 mapping. +2. **Approval gate as Change Advisory Board substitute** — Article 14 + `forgelm approve` flow as the change-authorisation event. CC8.1 + mapping. +3. **Rollback procedure** — `auto_revert` for in-pipeline; manual + redeploy of previous model SHA for post-deployment. +4. **Configuration drift detection** — `tools/regenerate_config_doc.py` + diff-guard catches Pydantic-schema drift between code and docs. +5. **SBOM drift detection** — `tools/generate_sbom.py` deterministic + contract: a release's SBOM is reproducible from the corresponding + `git tag`. + +## 11. Deployer-side audit cookbook (`iso_soc2_deployer_guide.md`) + +**Outline (~400 lines):** + +1. **Audience** — your compliance team responding to an ISO 27001 + internal audit OR a SOC 2 Type II observation period. +2. **What ForgeLM gives you out-of-the-box** — bullet list mapping + to §3 + §4 of this design doc. +3. **Setup checklist before audit observation period:** + - Set `FORGELM_OPERATOR` per CI runner. + - Set `FORGELM_AUDIT_SECRET` from KMS (32+ random bytes). + - Enable `evaluation.require_human_approval` for every high / + unacceptable risk run. + - Configure webhook to ship lifecycle events to SIEM. + - Schedule weekly `forgelm verify-audit` cron. +4. **Walking the auditor through evidence:** + - "Show me the audit trail" → `audit_log.jsonl` + `forgelm verify-audit`. + - "Show me the change controls" → CI logs + `human_approval.granted` + events + `compliance.config_hash`. + - "Show me the data lineage" → `data_provenance.json`, + `compute_dataset_fingerprint`, HF Hub revision pin. + - "Show me the supply chain" → SBOM artefacts on every release tag. + - "Show me the access controls" → `FORGELM_OPERATOR` rotation log + + IdP MFA records. + - "Show me the encryption posture" → `docs/qms/encryption_at_rest.md` + deployer config + KMS audit log. + - "Show me the incident response" → `sop_incident_response.md` + (deployer-adapted) + `pipeline.failed`/`reverted` history. +5. **Common pitfalls and how to avoid them.** +6. **References.** + +## 12. CI/CD integration plan + +| Pipeline | Step | Phase added | Failure behaviour | +|---|---|---|---| +| ci.yml | `bandit -r forgelm/ -c pyproject.toml` | Faz 23 | High-sev → fail; med → warn | +| ci.yml | `pytest tests/test_sbom.py` | Faz 23 | SBOM determinism contract; fail on diff | +| ci.yml | `python3 tools/check_anchor_resolution.py` | Faz 26 | Markdown anchor resolution; strict mode in CI | +| ci.yml | `tools/check_cli_help_consistency.py` | Faz 30 | CLI help ↔ doc cross-reference; strict mode | +| nightly.yml | `pip-audit --strict` | Faz 23 | High-sev → fail; med → warn | +| nightly.yml | `bandit` repeat | Faz 23 | High-sev → fail; med → warn | +| publish.yml | `tools/generate_sbom.py` (per OS×py) | (existing Wave 2) | Already integrated | +| publish.yml | upload SBOM to release | Faz 23 (deferred) | `gh release upload` step (deferred to maintainer triage) | + +## 13. Faz 23 implementation work breakdown + +(Used by the Faz 23 PR description as the task list.) + +### 13.1 Tooling + +- [ ] `pyproject.toml` — add `[project.optional-dependencies] security`. +- [ ] `tools/check_pip_audit.py` — parse pip-audit JSON, exit 1 on + high-severity, warn on med/low. +- [ ] `tools/check_bandit.py` — parse bandit JSON, same severity + tiering. +- [ ] `tests/test_sbom.py` — determinism contract + schema validation. +- [ ] `.github/workflows/ci.yml` — bandit step. +- [ ] `.github/workflows/nightly.yml` — pip-audit + bandit steps. + +### 13.2 QMS docs (EN + TR pairs) + +- [ ] `docs/qms/encryption_at_rest.md` (+ `-tr.md`) — §8.1 outline. +- [ ] `docs/qms/access_control.md` (+ `-tr.md`) — §8.2 outline. +- [ ] `docs/qms/risk_treatment_plan.md` (+ `-tr.md`) — §8.3 outline. +- [ ] `docs/qms/statement_of_applicability.md` (+ `-tr.md`) — §8.4 outline. +- [ ] `docs/qms/sop_incident_response.md` — §9 expansion (+ TR mirror + lands in Faz 26). +- [ ] `docs/qms/sop_change_management.md` — §10 expansion (+ TR mirror + lands in Faz 26). + +### 13.3 Guides + references + +- [ ] `docs/guides/iso_soc2_deployer_guide.md` (+ `-tr.md`) — §11 outline. +- [ ] `docs/reference/iso27001_control_mapping.md` (+ `-tr.md`) — §3 + table reformatted as reference doc. +- [ ] `docs/reference/soc2_trust_criteria_mapping.md` (+ `-tr.md`) — §4 + table reformatted as reference doc. +- [ ] `docs/reference/supply_chain_security.md` (+ `-tr.md`) — SBOM / + pip-audit / bandit overview. + +### 13.4 Cross-cutting + +- [ ] `README.md` — 1 paragraph "ISO 27001 / SOC 2 Type II alignment" + section + link to deployer guide. +- [ ] `CHANGELOG.md` — Wave 4 / Faz 22-23 line. +- [ ] `tools/check_bilingual_parity.py` `_PAIRS` — register the new + EN ↔ TR pairs (≈8 new pairs). + +### 13.5 Tests + +- [ ] `tests/test_sbom.py::test_deterministic`. +- [ ] `tests/test_sbom.py::test_cyclonedx_1_5_schema_valid`. +- [ ] `tests/test_pip_audit_check.py` — `tools/check_pip_audit.py` + severity-tiering logic on synthetic JSON. +- [ ] `tests/test_bandit_check.py` — `tools/check_bandit.py` severity + tiering on synthetic JSON. + +## 14. Acceptance criteria + +Faz 22 (this design): + +- [ ] **Length ≥ 800 lines** (closure-plan acceptance bar). This file + will satisfy that on commit. +- [ ] **§3 covers all 93 ISO 27001:2022 Annex A controls.** ✓ (37 + 8 + + 14 + 34 = 93). +- [ ] **§4 covers all 5 SOC 2 categories with full Common Criteria.** ✓. +- [ ] **§13 work breakdown is the canonical Faz 23 task list.** ✓. +- [ ] **Cited symbols are real.** ✓ (all symbols verified against + `closure/wave4-integration` HEAD `b87c872`). +- [ ] **Bilingual parity unaffected.** ✓ (this file lives under + `docs/analysis/` which is not in `_PAIRS`). +- [ ] **Adds zero behaviour change.** ✓ (design doc only). + +Faz 23 (next phase, gated on this design landing): + +See §13 above for the full task list. Acceptance: §13 checkboxes all +ticked + tests green + bilingual parity strict. + +## 15. Out of scope for Faz 22 + 23 + +Items deferred to a later wave or explicitly out of plan: + +- **PCI DSS, HIPAA, NIST CSF mappings.** ISO + SOC 2 are the requested + baseline; other frameworks share most controls and can re-use this + evidence map but aren't authored as separate docs. +- **SPDX 2.3 SBOM generation.** CycloneDX 1.5 is the chosen format; + SPDX is convertible via `cyclonedx-py` if required by a specific + deployer. +- **External penetration test.** Operator responsibility; ForgeLM + cannot pen-test itself. +- **Bug bounty programme.** Operator-org choice. +- **FedRAMP / IL5 government clouds.** Specialised compliance regime + out of plan scope. + +## 16. Decision log + +Material decisions taken during Faz 22 authoring. Each row is a +contract that future authors should not silently reverse. + +| ID | Decision | Rationale | Reversible? | +|---|---|---|---| +| D-22-01 | Customer-facing wording is **"alignment"**, not **"compliance"** / **"certified"** | Software cannot be certified; only organisations can. Mis-wording invites regulator pushback | No (legal cost; would require customer-facing retraction) | +| D-22-02 | SBOM stays CycloneDX 1.5; SPDX deferred | Existing Wave 2 emitter is pure-stdlib + zero external deps; CycloneDX 1.5 is the open-source-tooling default consumed by `Dependency-Track` | Yes — operator can convert via `cyclonedx-py` if SPDX is required | +| D-22-03 | `pip-audit` + `bandit` shipped as **`security`** optional extra (not core dep) | Core install stays lean; ops teams installing for CI/SCA opt in | Yes — promotion to core dep is one-line edit | +| D-22-04 | `bandit` runs only on `forgelm/` (not `tests/`) | Test fixtures legitimately use insecure patterns (assert, dummy keys, pickle); scanning them produces signal-noise | Yes — wider scope possible if `# nosec` annotations sweep is done first | +| D-22-05 | New QMS docs ship in Faz 23 as EN-only first; TR mirrors land in Faz 26 with the existing-QMS TR sweep | Avoid cross-wave doc-pair churn; Faz 26 is the canonical QMS bilingual phase | Yes — TR can land in Faz 23 if maintainer prefers single-wave delivery | +| D-22-06 | Statement of Applicability is a SUMMARY of §3, not a third copy of the table | Single source of truth (this design doc); the QMS SoA cites + references rather than duplicates | Yes | +| D-22-07 | Risk Treatment Plan rows are **pre-filled** from ForgeLM's own threat model | Deployer customises; ForgeLM-introduced risks are owner-known and shouldn't require deployer rediscovery | Yes | +| D-22-08 | Faz 23 does NOT add SPDX SBOM, FedRAMP mapping, NIST CSF mapping, HIPAA mapping | Closure-plan scope is ISO + SOC 2; other frameworks share evidence and re-use this map | Yes — additional mappings are pure additive | +| D-22-09 | Audit chain encryption-at-rest is **deployer responsibility** | ForgeLM's tamper-evidence is integrity-only; confidentiality is substrate-side | No (architecturally) — adding crypto would conflate concerns | +| D-22-10 | `FORGELM_AUDIT_SECRET` rotation is **deployer responsibility** + documented in `access_control.md` | Secret-management is KMS domain; ForgeLM consumes, doesn't manage | No (architecturally) | + +## 17. FAQ for deployers + +Anticipated questions from the deployer's compliance team: + +**Q1.** *Is ForgeLM ISO 27001 certified?* +**A.** No. ISO 27001 certifies organisations, not software libraries. +ForgeLM is **aligned** with ISO 27001:2022 Annex A in that it produces +auditable evidence for ~61 of the 93 controls. The remaining 32 +controls (mainly A.7 physical, A.5 organisational governance, A.8 +network) are deployer-side. See §3 for the full mapping. + +**Q2.** *Is ForgeLM SOC 2 Type II compliant?* +**A.** Same answer as Q1. ForgeLM is **aligned** with the SOC 2 +Trust Services Criteria — the deployer's auditor uses ForgeLM +evidence (audit log, model integrity, data fingerprint, SBOM, +approval-gate events, etc.) when assessing the deployer's controls. + +**Q3.** *Where do I store `FORGELM_AUDIT_SECRET`?* +**A.** In your KMS / secrets vault (HashiCorp Vault, AWS Secrets +Manager, Azure Key Vault). Inject via env var at CI runner start. +See `docs/qms/access_control.md` (Faz 23) for the rotation cadence +recommendation (quarterly). + +**Q4.** *What encryption does ForgeLM apply at rest?* +**A.** None — ForgeLM is encryption-substrate-agnostic. The deployer +is responsible for disk / blob encryption. ForgeLM's audit chain is +**integrity-protected** (HMAC + manifest sidecar), which is +orthogonal to confidentiality. See `docs/qms/encryption_at_rest.md` +(Faz 23) for the recommended substrate per asset class. + +**Q5.** *How do I respond to a GDPR Article 15 access request using +ForgeLM?* +**A.** `forgelm reverse-pii --query data/*.jsonl` (after +running with `--salt-source per_dir` if your corpus is hash-masked). +The audit event `data.access_request_query` records the request in +the chain (with the identifier salted-and-hashed, never raw). + +**Q6.** *How do I respond to a GDPR Article 17 erasure request?* +**A.** `forgelm purge --row-id --corpus path/to/data.jsonl`. +The audit chain records the request + completion + any warnings +(`memorisation`, `synthetic_data_present`, `external_copies`). See +`docs/guides/gdpr_erasure.md` for the full workflow. + +**Q7.** *Can ForgeLM help me pass a SOC 2 Type II observation?* +**A.** Yes — ForgeLM's audit log is HMAC-chained and append-only; +running `forgelm verify-audit` periodically (weekly cron is the +deployer-guide recommendation) provides continuous-monitoring +evidence. See §11 deployer guide outline. + +**Q8.** *Where's the SBOM?* +**A.** Generated on every release tag (`v*` pattern) by +`tools/generate_sbom.py` and uploaded as a release artefact for each +(OS × Python-version) matrix cell. Format is CycloneDX 1.5 JSON. + +**Q9.** *Does ForgeLM scan its own dependencies for CVEs?* +**A.** Yes — Faz 23 adds `pip-audit` to the nightly workflow. +High-severity CVEs fail the nightly; medium / low warn. + +**Q10.** *Does ForgeLM perform static security analysis on its own +code?* +**A.** Yes — Faz 23 adds `bandit` to the CI workflow. Scope: +`forgelm/` (production code only). High severity fails CI. + +## 18. References + +- [ISO/IEC 27001:2022 Annex A](https://www.iso.org/standard/27001) — + controls source-of-truth. +- [AICPA SOC 2 Type II 2017 Trust Services Criteria (revised 2022)](https://www.aicpa-cima.com/) — + TSC source-of-truth. +- [CycloneDX 1.5 specification](https://cyclonedx.org/specification/overview/) — + SBOM format. +- [`docs/analysis/code_reviews/closure-plan-202604300906.md`](./closure-plan-202604300906.md) + — Faz 22 + Faz 23 source closure plan. +- [`docs/qms/`](../../qms/) — existing QMS templates (Wave 0). +- [`docs/reference/audit_event_catalog.md`](../../reference/audit_event_catalog.md) + — audit-event vocabulary. +- [`docs/standards/`](../../standards/) — engineering standards. + +--- + +*End of Faz 22 design document. Length verification: see commit +diff.* From a8308719a395f5bd8445dbd39b21708491bcd4bd Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 00:00:50 +0300 Subject: [PATCH 02/12] feat(wave-4-faz-23): ISO 27001 / SOC 2 alignment implementation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements Faz 23 of the closure plan per the Faz 22 design (commit 93e0ddf). Six work areas: Supply-chain security tooling ============================= * `pyproject.toml [project.optional-dependencies] security` — new opt-in extra with pip-audit>=2.7.0 + bandit[toml]>=1.7.0. Operators install via `pip install forgelm[security]` to run the same checks ForgeLM's nightly + CI workflows enforce. * `tools/check_pip_audit.py` (new, 116 lines) — pip-audit JSON severity gate. HIGH / CRITICAL → exit 1, MEDIUM → ::warning::, LOW / UNKNOWN silent. Tier-normalises MODERATE → MEDIUM (advisory-feed synonym). * `tools/check_bandit.py` (new, 88 lines) — bandit JSON severity gate. HIGH → exit 1, MEDIUM → ::warning::, LOW silent. * `tests/test_supply_chain_security.py` (new, 16 tests) — determinism contract for tools/generate_sbom.py (two invocations on the same env produce content-equal SBOMs modulo serialNumber + timestamp); CycloneDX 1.5 schema-required field presence; check_pip_audit + check_bandit severity-tiering on synthetic JSON fixtures. * `.github/workflows/ci.yml` — bandit step on forgelm/ in the validate job, after the License check. * `.github/workflows/nightly.yml` — new `supply-chain-security` job running pip-audit + bandit at 03:00 UTC; uploads pip-audit.json + bandit.json as artefacts; added to notify-failure deps so a failed scan opens an issue. QMS additions (EN-only ships; TR mirrors land in Faz 26 per design Decision D-22-05) =================================================== * `docs/qms/encryption_at_rest.md` (new, ~210 lines) — substrate-side encryption guidance for 7 ForgeLM artefact classes (final/staging weights, audit log + manifest, salt file, training corpus, configs, Annex IV bundle); threat model + per-substrate recommendations + verification checklist. * `docs/qms/access_control.md` (new, ~190 lines) — operator identity contract (`FORGELM_OPERATOR` resolution + namespaced ID form); FORGELM_AUDIT_SECRET rotation; CI runner identity binding for GitHub Actions / GitLab / Jenkins; OS-level isolation patterns; approval-gate identity separation. * `docs/qms/risk_treatment_plan.md` (new, ~210 lines) — pre-populated 12-row ISO 27005 risk register with rows for every ForgeLM-introduced risk: training-data poisoning, supply-chain compromise, credential leak, audit-log tampering, memorisation residual, safety-classifier load failure, auto-revert false positives, webhook SSRF, webhook target compromise, ReDoS, cross-tool digest mismatch, unauthorised deployment. * `docs/qms/statement_of_applicability.md` (new, ~150 lines) — 93-control SoA matrix in QMS-style format. * `docs/qms/sop_incident_response.md` — Wave 4 §4 expansion (security-incident playbook): audit-chain integrity violation, credential leak, supply-chain CVE, webhook compromise, GDPR Art. 15 / 17 DSAR procedures. ISO 27001:2022 + SOC 2 control mapping in header. * `docs/qms/sop_change_management.md` — Wave 4 §4 expansion (CI gates as formal change-control mechanism — 11-row table mapping each gate to ISO controls; Article 14 approval gate as Change Advisory Board substitute; configuration-drift detection via tools/regenerate_config_doc.py; SBOM-drift detection via tools/generate_sbom.py determinism contract). Operator guide + reference docs (EN+TR) ======================================== * `docs/guides/iso_soc2_deployer_guide.md` (+ `-tr.md`, ~280 lines each) — deployer audit cookbook with 8 common audit-floor questions answered with the exact ForgeLM artefact + grep / jq command. 6 common pitfalls section. * `docs/reference/iso27001_control_mapping.md` (+ `-tr.md`) — 93-control × ForgeLM evidence reference table. * `docs/reference/soc2_trust_criteria_mapping.md` (+ `-tr.md`) — 5-category × ForgeLM evidence reference table. * `docs/reference/supply_chain_security.md` (+ `-tr.md`) — SBOM + pip-audit + bandit overview for compliance teams. Cross-cutting ============= * `README.md` — new "ISO 27001 / SOC 2 Type II Alignment" bullet under Enterprise & MLOps with explicit "alignment, not certified" framing (closes Faz 22 design Decision D-22-01). * `tools/check_bilingual_parity.py::_PAIRS` — added 4 new pairs (deployer guide + 3 reference docs); registry now 13 pairs (was 9). * `CHANGELOG.md` — Wave 4 entry under [Unreleased] documenting Faz 22 + Faz 23 deliverables. Validation ========== - ruff format + ruff check clean. - pytest 1360 → 1376 (+16 supply-chain tests; net delta is the 16 new tests in test_supply_chain_security.py). - python3 tools/check_bilingual_parity.py --strict → 13/13 OK. - forgelm --config config_template.yaml --dry-run → green. - python3 tools/generate_sbom.py → emits valid CycloneDX 1.5 JSON; tools/check_pip_audit.py + check_bandit.py error paths exit 1 on missing/invalid input as designed. Closure-plan reference: Faz 23 task list at docs/analysis/code_reviews/closure-plan-202604300906.md:782-813. Faz 22 design rationale at docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/ci.yml | 14 + .github/workflows/nightly.yml | 59 ++- CHANGELOG.md | 78 ++++ README.md | 1 + docs/guides/iso_soc2_deployer_guide-tr.md | 279 ++++++++++++++ docs/guides/iso_soc2_deployer_guide.md | 281 +++++++++++++++ docs/qms/access_control.md | 231 ++++++++++++ docs/qms/encryption_at_rest.md | 221 ++++++++++++ docs/qms/risk_treatment_plan.md | 210 +++++++++++ docs/qms/sop_change_management.md | 73 +++- docs/qms/sop_incident_response.md | 158 +++++++- docs/qms/statement_of_applicability.md | 166 +++++++++ docs/reference/iso27001_control_mapping-tr.md | 140 ++++++++ docs/reference/iso27001_control_mapping.md | 140 ++++++++ .../soc2_trust_criteria_mapping-tr.md | 129 +++++++ docs/reference/soc2_trust_criteria_mapping.md | 130 +++++++ docs/reference/supply_chain_security-tr.md | 196 ++++++++++ docs/reference/supply_chain_security.md | 197 ++++++++++ pyproject.toml | 10 + tests/test_supply_chain_security.py | 340 ++++++++++++++++++ tools/check_bandit.py | 105 ++++++ tools/check_bilingual_parity.py | 4 + tools/check_pip_audit.py | 149 ++++++++ 23 files changed, 3305 insertions(+), 6 deletions(-) create mode 100644 docs/guides/iso_soc2_deployer_guide-tr.md create mode 100644 docs/guides/iso_soc2_deployer_guide.md create mode 100644 docs/qms/access_control.md create mode 100644 docs/qms/encryption_at_rest.md create mode 100644 docs/qms/risk_treatment_plan.md create mode 100644 docs/qms/statement_of_applicability.md create mode 100644 docs/reference/iso27001_control_mapping-tr.md create mode 100644 docs/reference/iso27001_control_mapping.md create mode 100644 docs/reference/soc2_trust_criteria_mapping-tr.md create mode 100644 docs/reference/soc2_trust_criteria_mapping.md create mode 100644 docs/reference/supply_chain_security-tr.md create mode 100644 docs/reference/supply_chain_security.md create mode 100644 tests/test_supply_chain_security.py create mode 100644 tools/check_bandit.py create mode 100644 tools/check_pip_audit.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 03e4dc96..066ebf88 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -210,3 +210,17 @@ jobs: test -f LICENSE || (echo "LICENSE file missing" && exit 1) head -1 LICENSE | grep -q "Apache" || (echo "Expected Apache License" && exit 1) echo "License: OK" + + # Wave 4 / Faz 23: bandit static-security analysis on forgelm/. + # tests/ is excluded via [tool.bandit] in pyproject.toml because + # test fixtures legitimately use insecure patterns (assert, dummy + # secrets). HIGH severity → fail; MEDIUM → warn; LOW → silent. + # See tools/check_bandit.py for the severity-tiering helper. + - name: bandit (static security analysis) + run: | + python -m pip install 'bandit[toml]>=1.7.0,<2.0.0' + # bandit returns exit 1 on ANY findings. We capture the JSON + # report and apply our own severity policy via the tiering + # helper (tools/check_bandit.py). + bandit -c pyproject.toml -r forgelm/ -f json -o /tmp/bandit.json || true + python3 tools/check_bandit.py /tmp/bandit.json diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index ab9d1195..180b1e13 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -215,9 +215,66 @@ jobs: - name: Run tests run: pytest tests/ -q --tb=short + supply-chain-security: + # ------------------------------------------------------------------ + # Wave 4 / Faz 23: ISO 27001 / SOC 2 Type II alignment supply-chain + # security gate. + # + # Two scans, both run on the latest published deps so we catch CVEs + # the moment a dependency advisory drops, even before the team + # re-bumps versions: + # + # - pip-audit (transitive CVEs against the OSV / GHSA databases) + # - bandit (static-analysis on forgelm/ production code only) + # + # Severity policy (per tools/check_pip_audit.py + tools/check_bandit.py): + # HIGH/CRITICAL → fail nightly; MEDIUM → ::warning::; LOW → silent. + # Operators install the same tooling locally via + # `pip install forgelm[security]`. + # ------------------------------------------------------------------ + name: Supply-chain security (pip-audit + bandit) + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-python@v6 + with: + python-version: "3.11" + cache: pip + + - name: Install ForgeLM with [security] extra + run: | + pip install --upgrade pip + pip install -e ".[security]" + + - name: pip-audit (transitive CVE scan) + run: | + # pip-audit returns non-zero on any finding; capture the JSON + # and apply our own severity policy via tools/check_pip_audit.py. + pip-audit --format json --output /tmp/pip-audit.json || true + python3 tools/check_pip_audit.py /tmp/pip-audit.json + + - name: bandit (static security analysis) + run: | + # Same scope/severity policy as ci.yml; nightly cadence catches + # newly-introduced HIGH issues even on long-lived branches. + bandit -c pyproject.toml -r forgelm/ -f json -o /tmp/bandit.json || true + python3 tools/check_bandit.py /tmp/bandit.json + + - name: Upload security scan artifacts + if: always() + uses: actions/upload-artifact@v5 + with: + name: supply-chain-scans + path: | + /tmp/pip-audit.json + /tmp/bandit.json + notify-failure: name: Notify on failure - needs: [test-latest-deps, test-min-deps, wheel-install-smoke] + needs: [test-latest-deps, test-min-deps, wheel-install-smoke, supply-chain-security] if: failure() runs-on: ubuntu-latest permissions: diff --git a/CHANGELOG.md b/CHANGELOG.md index b994349a..4201da72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,84 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Wave 4 — Faz 22 + 23 + 26 + 30 (`closure/wave4-integration`) + +ISO 27001 / SOC 2 Type II alignment + QMS bilingual sweep + final +documentation pass. + +**Faz 22 — ISO 27001 / SOC 2 alignment design** + +- New design document at + [`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) + (~865 lines). Maps every ISO 27001:2022 Annex A control (93 + controls across A.5 + A.6 + A.7 + A.8) and every SOC 2 Trust + Services Criterion (Common Criteria CC1.x–CC9.x mandatory + 4 + optional categories) to the ForgeLM feature that produces audit + evidence. Coverage tally: FL (full) 11 / FL-helps 50 / OOS 32. + 10-row Decision Log + 10-question deployer FAQ. + +**Faz 23 — ISO 27001 / SOC 2 alignment implementation** + +Supply-chain security tooling: + +- `pyproject.toml` — new `[security]` optional extra (`pip-audit`, + `bandit[toml]`). +- `tools/check_pip_audit.py` (new) — pip-audit JSON severity gate; + HIGH / CRITICAL → exit 1, MEDIUM → `::warning::`, LOW silent. +- `tools/check_bandit.py` (new) — bandit JSON severity gate; same + HIGH / MEDIUM / LOW tiering. +- `.github/workflows/ci.yml` — `bandit` step on `forgelm/` + (production code only; tests/ excluded). +- `.github/workflows/nightly.yml` — new `supply-chain-security` job + running `pip-audit` + `bandit` daily at 03:00 UTC. +- `tests/test_supply_chain_security.py` (new) — 16 tests pinning + SBOM determinism contract + check_pip_audit + check_bandit + severity-tiering logic. + +QMS additions (EN ships now; TR mirrors land in Faz 26): + +- `docs/qms/encryption_at_rest.md` (new) — substrate-side encryption + guidance per ForgeLM artefact class. +- `docs/qms/access_control.md` (new) — operator identity contract; + `FORGELM_OPERATOR` form recommendations; `FORGELM_AUDIT_SECRET` + rotation cadence; CI runner identity binding examples for GHA / + GitLab / Jenkins. +- `docs/qms/risk_treatment_plan.md` (new) — pre-populated 12-row + ISO 27005 risk register covering ForgeLM-introduced risks + (training-data poisoning, supply-chain compromise, audit-log + tampering, memorisation residual, ReDoS, etc.). +- `docs/qms/statement_of_applicability.md` (new) — 93-control SoA + matrix in QMS-style format. +- `docs/qms/sop_incident_response.md` — Wave 4 §4 expansion (security + incidents: audit-chain integrity violation, credential leak, + supply-chain CVE, webhook compromise, GDPR Art. 15/17 DSAR + playbooks). +- `docs/qms/sop_change_management.md` — Wave 4 §4 expansion (CI + gates as formal change-control mechanism — 11-row table mapping + each gate to ISO controls; Article 14 approval gate as CAB + substitute; config-drift + SBOM-drift detection). + +Operator guide + reference docs (EN+TR): + +- `docs/guides/iso_soc2_deployer_guide.md` (+ `-tr.md`) — deployer + audit cookbook covering 8 common audit-floor questions with the + exact ForgeLM artefact + grep / command answer. +- `docs/reference/iso27001_control_mapping.md` (+ `-tr.md`) — ISO + Annex A reference table (93 controls × ForgeLM evidence). +- `docs/reference/soc2_trust_criteria_mapping.md` (+ `-tr.md`) — SOC + 2 TSC reference table. +- `docs/reference/supply_chain_security.md` (+ `-tr.md`) — SBOM + + pip-audit + bandit overview for compliance teams. + +Cross-cutting: + +- `README.md` — new "ISO 27001 / SOC 2 Type II Alignment" bullet + under Enterprise & MLOps with explicit "alignment, not certified" + framing (closes the design doc's D-22-01 wording decision). +- `tools/check_bilingual_parity.py` `_PAIRS` registry — added 4 new + pairs (deployer guide + 3 reference docs); tally now 13 doc pairs + (was 9). + ### Wave 3 — Faz 24 + 28 + 38 (`closure/wave3-integration`) Single integration branch covering three closure-plan phases: diff --git a/README.md b/README.md index 4c2bbde5..58911ac9 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,7 @@ ### Enterprise & MLOps - **Config-Driven**: Declarative YAML — built for CI/CD pipelines, not notebooks - **EU AI Act Compliance**: Auto-generated audit trails, data provenance (SHA-256), training manifests +- **ISO 27001 / SOC 2 Type II Alignment**: Software cannot be ISO/SOC 2 *certified* — only organisations can — but ForgeLM produces the audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor explicitly asks for. CycloneDX 1.5 SBOM per release, `pip-audit` nightly, `bandit` CI, append-only HMAC-chained audit log, Article 14 staging gate, Article 15/17 GDPR tooling. See [`docs/guides/iso_soc2_deployer_guide.md`](docs/guides/iso_soc2_deployer_guide.md) for the deployer audit cookbook + 93-control coverage map. - **Docker**: Official Dockerfile and docker-compose for portable deployment - **Offline / Air-Gapped**: Full operation without internet for regulated industries - **JSON Output**: Machine-readable results with `--output-format json` for pipeline integration diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md new file mode 100644 index 00000000..1cf58ac0 --- /dev/null +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -0,0 +1,279 @@ +# ISO 27001 / SOC 2 Type II — Operatör (deployer) denetim cookbook'u + +> Hedef kitle: ISO 27001 iç denetim VEYA SOC 2 Type II gözlem dönemi +> için cevap hazırlayan compliance ekibiniz. +> +> **Kritik çerçeveleme**: yazılım ISO 27001 sertifikalı OLAMAZ — yalnız +> organizasyonlar sertifika alır. ForgeLM, eğitim pipeline'ınızda +> çalıştırıldığında denetçinin açıkça istediği auditable kanıtları +> ürettiği anlamda ISO 27001:2022 Annex A kontrolleri ve AICPA SOC 2 +> Trust Services Criteria ile **uyumludur (aligned)**. Bu rehber, +> denetim katındaki yaygın soruları teker teker yürütür ve hangi +> ForgeLM artefaktının cevap verdiğini gösterir. +> +> Çapraz referans: +> [`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) +> tam tasarım gerekçesi + 93 kontrollü kapsama haritası için. + +## ForgeLM'in size out-of-the-box sağladıkları + +ISO / SOC 2 denetçisinin en çok önem verdiği dört sütun, doğrudan +ForgeLM kanıtına sahip: + +1. **Audit trail** — eğitim koşumu başına `audit_log.jsonl`, + append-only HMAC + SHA-256 hash chain + genesis manifest sidecar. + `forgelm verify-audit` zinciri uçtan uca doğrular. +2. **Change control** — Madde 14 staging gate (`forgelm approve` / + `reject`) + `human_approval.required/granted/rejected` audit + olayları + koşum başına damgalanan `compliance.config_hash`. Her + model promotion çift kontrollü ve forensic olarak attribute + edilmiştir. +3. **Data lineage** — `data_provenance.json` (SHA-256 fingerprint + + size + mtime + HF Hub revision pin); `data_governance_report.json` + (collection_method, annotation_process, known_biases, + personal_data_included, dpia_completed). +4. **Supply chain** — yayın etiketi başına yayın matrisinin her (OS + × Python-version) hücresi için emit edilen CycloneDX 1.5 SBOM. + Wave 4 statik + dinamik güvenlik taraması için `pip-audit` + nightly + `bandit` CI ekler. + +## Denetim gözlem dönemi öncesi setup checklist'i + +Denetçi sormadan **önce** indirmeniz gereken on iki madde. + +### Identity + secrets + +- [ ] **`FORGELM_OPERATOR` ayarla** her CI runner'da — makine-okur + namespaced bir tanımlayıcı kullanın (örn. + `gha:Acme/repo:training:run-${{ github.run_id }}`). Önerilen + form için `docs/qms/access_control.md` §3'e bakın. +- [ ] **`FORGELM_AUDIT_SECRET` üretin** KMS / Vault'unuzda + (32+ rastgele bayt, AES-256-GCM-strength entropi). +- [ ] **Üç-aylık `FORGELM_AUDIT_SECRET` rotasyonunu zamanlayın** + KMS audit log'unuzda. +- [ ] **Approver identity ≠ trainer identity'yi yapılandırın** — + Madde 14 staging gate sizin in-pipeline Change Advisory + Board'unuzdur. + +### Pipeline yapılandırması + +- [ ] **`evaluation.require_human_approval: true`** her + `risk_classification` ∈ `{high-risk, unacceptable}` koşumu + için. Unutursanız F-compliance-110 `ConfigError` raise eder. +- [ ] **Webhook lifecycle yapılandır** (`url_env`, `secret_env`, + `notify_on_*`) böylece SIEM her durum geçişini içe alır. +- [ ] **`auto_revert: true` etkinleştir** üretim eğitimi için — + kalite regresyonu otomatik olarak baseline modele döner. + +### Audit + monitoring + +- [ ] **Haftalık `forgelm verify-audit` cron zamanla** her üretim + `` üzerinde. Sıfır-olmayan exit'e alarm bağla. +- [ ] **`audit_log.jsonl` gönder** write-once depolamaya (compliance + modunda S3 Object Lock retention, Azure Immutable Blob, + versioning'li MinIO). +- [ ] **Encrypt-at-rest substrate** model ağırlıkları, audit log, + eğitim verisi için — asset başı substrate önerileri için + `docs/qms/encryption_at_rest.md` belgesine bakın. + +### Supply-chain hijyeni + +- [ ] **ForgeLM'i pin'le** CI'da (`pip install forgelm==X.Y.Z`). +- [ ] **Nightly SBOM diff'lerine abone ol** bağımlı olduğunuz her + yayın için. + +## Denetçiyi kanıt üzerinde gezdirmek + +En sık sorulan sekiz denetim-katı sorusu, kanıtı üreten birebir +ForgeLM artefaktı + grep / komut ile. + +### S1: "Son 90 günde her model promotion için audit trail göster" + +```bash +# Önce zincir bütünlüğünü doğrula. +forgelm verify-audit --output-dir ./outputs --json + +# Sonra promotion olaylarını çıkar. +jq 'select(.event == "human_approval.granted")' ./outputs/audit_log.jsonl +``` + +Her `human_approval.granted` girişi şunları taşır: + +- `operator` — kim onayladı (eğiten DEĞİL **onaylayan** kimliği). +- `run_id` — modeli üreten eğitim koşumuna geri bağlanır. +- `prev_hash` + `_hmac` — zincir bütünlüğü. +- `compliance.config_hash` — hangi config kullanıldı; denetçi + `git log` içindeki YAML ile diff alabilir. + +### S2: "Change-control kanıtı göster — bu modeli kim onayladı?" + +Zincirin kendisi change-control kanıtıdır. Çapraz referans: + +```bash +# 1. Koşum X için training event'ı bul. +jq 'select(.run_id == "X" and .event == "training.started")' \ + ./outputs/audit_log.jsonl + +# 2. Aynı koşum için approval event'ı bul. +jq 'select(.run_id == "X" and .event == "human_approval.granted")' \ + ./outputs/audit_log.jsonl + +# 3. Operator ID'lerin farklı olduğunu teyit et. +jq -r 'select(.run_id == "X" and (.event == "training.started" or .event == "human_approval.granted")) | .operator' \ + ./outputs/audit_log.jsonl | sort -u +``` + +İki farklı operator ID görev ayrılığını kanıtlar (ISO A.5.3, SOC 2 +CC1.5). + +### S3: "Data lineage göster" + +```bash +cat ./outputs/data_provenance.json +# { +# "dataset_id": "Acme/customer-support-v3", +# "hf_revision": "9c7c8f3...", +# "fingerprint_sha256": "ab12...", +# "size_bytes": 14_982_011, +# "mtime": "2026-05-01T08:14:33Z", +# ... +# } +``` + +`fingerprint_sha256` + `hf_revision` birlikte korpusu deterministik +olarak pin'ler. Aynı girdi üzerinde `forgelm audit data/*.jsonl` +çalıştıran bir denetçi aynı fingerprint'i görmek zorundadır. + +### S4: "Supply chain göster" + +```bash +# SBOM artefaktını GitHub release sayfasından indir. +gh release download v0.5.5 --pattern 'sbom-*' + +# Veya istek üzerine yeniden üret. +python3 tools/generate_sbom.py > sbom.json + +# Son yayınla diff al. +diff <(jq -S . sbom-prev.json) <(jq -S . sbom.json) +``` + +CycloneDX 1.5 JSON her transitive bağımlılığı purl +(`pkg:pypi/...`) ve sürümle listeler. Dependency-Track CVE +korelasyonu için bunu doğal olarak içe alır. + +### S5: "Erişim kontrolleri göster — yalnız yetkili reviewer'ların model onayladığını nasıl ispatlıyorsunuz?" + +İki katman: + +1. **IdP katmanı** — CI runner kimlikleriniz ve insan reviewer + kimlikleriniz IdP'nizden çıkar. Denetçi issuance + revocation + cadence'ini teyit etmek için IdP audit log'unuzu yürür. +2. **ForgeLM katmanı** — her approval event onaylayanın + `FORGELM_OPERATOR`'unu kaydeder. Onaylayanın approval anında + yetkili olduğunu teyit etmek için IdP audit log'u ile çapraz + referans. + +```bash +# Tüm approval event'leri approver id + timestamp ile. +jq -r 'select(.event == "human_approval.granted") | + [.timestamp, .operator, .run_id] | @tsv' \ + ./outputs/audit_log.jsonl +``` + +### S6: "Encryption posture göster" + +Bu **deployer-side**'dır — ForgeLM artefaktları şifrelemez, +substrate şifreler. Referans: + +- `docs/qms/encryption_at_rest.md` — kurum içi politikanız ForgeLM + artefakt sınıflarına eşlenmiş. +- KMS audit log — encryption-in-use'un substrate-side kanıtı. +- ForgeLM `data_governance_report.json` — config block'unuz operatör + beyanı başına `encryption_at_rest: true|false` kaydeder. + +### S7: "Incident response göster — koşum ortasında safety classifier crash ederse ne olur?" + +Referans: + +- `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 + genişletme) güvenlik-incident playbook için. +- Zincirin kendisi: `audit.classifier_load_failed` event yangın + + `pipeline.failed` propagate + koşum `final_model/` dizini ÜRETMEZ. +- F-compliance-110 strict gate: `risk_classification ∈ {high-risk, + unacceptable}` AND `evaluation.safety.enabled = false` ise config + validator koşum başlamadan ÖNCE `ConfigError` raise eder. Denetçi + son 90 günde safety gate'siz hiçbir high-risk koşumu görmez. + +### S8: "GDPR Madde 15 + 17 talebine cevap verebildiğini göster" + +Madde 15 (erişim hakkı): + +```bash +forgelm reverse-pii --query "alice@example.com" --type email \ + --output-dir ./outputs data/*.jsonl +# JSON envelope { matches: [...], match_count: N } döner +# Audit chain QUERY HASH'LENMİŞ ŞEKİLDE bir +# `data.access_request_query` event damgalar (asla raw — salt audit +# log'un kendisine karşı wordlist saldırılarına karşı koruma sağlar). +``` + +Madde 17 (silme hakkı): + +```bash +forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ + --output-dir ./outputs +# Audit chain `target_id` HASH'LENMİŞ ŞEKİLDE +# `data.erasure_requested` → `data.erasure_completed` (veya +# `data.erasure_failed`) damgalar. Eğer satırı eğiten bir model +# `final_model/`e sahipse `data.erasure_warning_memorisation` DA +# yangın eder ve memorisation residual riskini özneye bildirmek +# zorundasınız. +``` + +Cross-tool digest correlation (purge `target_id` == reverse-pii +`query_hash` aynı `output_dir`'deki aynı identifier için) Wave 3 +follow-up testleriyle pin'lenmiştir. + +## Yaygın tuzaklar + +Operatörlerin ilk denetimlerinde yanlış yaptığı şeyler: + +1. **Pipeline'lar arasında `FORGELM_OPERATOR` paylaşımı.** "Sadece + `FORGELM_OPERATOR=ci` yaptık" — audit chain yorumlanamaz hale + gelir çünkü her giriş aynı string'e attribute olur. Pipeline + başına + koşum başına namespaced identifier kullanın. +2. **Webhook secret'larını YAML'de saklamak.** + `webhook.secret_env` → KMS'den runtime'da resolve. Bir secret + rotate edilse bile version control'deki plaintext-secret YAML + bir bulgudur (denetçi `git log`'da tarihsel exposure görür). +3. **CI'da `forgelm verify-audit` atlamak.** "Audit chain'e zımnen + güveniyoruz" savunulabilir bir pozisyon değildir. Haftalık cron + + sıfır-olmayan exit'e alarm zamanlayın; alarm geçmişini + denetçiye gösterin. +4. **Manifest sidecar'ı unutmak.** `audit_log.jsonl` tek başına + yetersizdir — `forgelm verify-audit` truncate-and-resume + tampering'ini tespit etmek için `audit_log.manifest.json` + sidecar'ı gerektirir. İKİSİNİ DE backup'layın. +5. **Üretim eğitiminde `auto_revert` yok.** Her zaman yeşil + eğitime bahis koyuyorsanız, regulator-reportable incident'a + bir safety-classifier degradation'ı uzaktasınız. `auto_revert: + true` etkinleştirin ve `pipeline.reverted` event'lerinin + çalışan safeguard kanıtı olarak birikmesine izin verin. +6. **Üretimde `FORGELM_ALLOW_ANONYMOUS_OPERATOR=1` ile ForgeLM + çalıştırmak.** O env var yalnız kısa-ömürlü test koşumları + içindir. Operator identity olmadan üretim koşumları ISO A.6.4 + + A.6.5 bulgusudur. + +## Bkz. + +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — tam tasarım gerekçesi + 93 kontrol haritası. +- [`../qms/encryption_at_rest.md`](../qms/encryption_at_rest.md) — substrate-side şifreleme rehberi. +- [`../qms/access_control.md`](../qms/access_control.md) — operator identity + secrets management. +- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. +- [`../qms/statement_of_applicability.md`](../qms/statement_of_applicability.md) — 93-kontrol SoA matrisi. +- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook (Wave 4 genişletme). +- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook (Wave 4 genişletme). +- [`../reference/iso27001_control_mapping.md`](../reference/iso27001_control_mapping.md) — ISO 27001:2022 Annex A kontrolleri × ForgeLM kanıtı. +- [`../reference/soc2_trust_criteria_mapping.md`](../reference/soc2_trust_criteria_mapping.md) — SOC 2 Trust Services Criteria × ForgeLM kanıtı. +- [`../reference/supply_chain_security.md`](../reference/supply_chain_security.md) — SBOM + pip-audit + bandit overview. +- [`../reference/audit_event_catalog.md`](../reference/audit_event_catalog.md) — audit-event vocabulary. diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md new file mode 100644 index 00000000..37d26317 --- /dev/null +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -0,0 +1,281 @@ +# ISO 27001 / SOC 2 Type II — Deployer's audit cookbook + +> Audience: your compliance team responding to an ISO 27001 internal +> audit OR a SOC 2 Type II observation period. +> +> **Critical framing**: software cannot be ISO 27001 certified — only +> organisations can. ForgeLM is **aligned** with the ISO 27001:2022 +> Annex A controls and the AICPA SOC 2 Trust Services Criteria, in +> the sense that running ForgeLM in your training pipeline produces +> auditable evidence the auditor explicitly asks for. This guide +> walks each common audit-floor question and shows you which +> ForgeLM artefact answers it. +> +> Cross-reference: +> [`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) +> for the full design rationale + 93-control coverage map. + +## What ForgeLM gives you out-of-the-box + +The four pillars an ISO / SOC 2 auditor cares about most all have +direct ForgeLM evidence: + +1. **Audit trail** — `audit_log.jsonl` per training run, append-only + with HMAC + SHA-256 hash chain + genesis manifest sidecar. + `forgelm verify-audit` validates the chain end-to-end. +2. **Change control** — Article 14 staging gate (`forgelm approve` / + `reject`) + `human_approval.required/granted/rejected` audit + events + `compliance.config_hash` stamped per run. Every + model promotion is dual-controlled and forensically attributed. +3. **Data lineage** — `data_provenance.json` (SHA-256 fingerprint + + size + mtime + HF Hub revision pin); `data_governance_report.json` + (collection_method, annotation_process, known_biases, + personal_data_included, dpia_completed). +4. **Supply chain** — CycloneDX 1.5 SBOM emitted per release tag for + every (OS × Python-version) cell of the publish matrix. Wave 4 + adds `pip-audit` nightly + `bandit` CI for static + dynamic + security scanning. + +## Setup checklist before audit observation period + +Twelve items to land **before** the auditor asks. + +### Identity + secrets + +- [ ] **Set `FORGELM_OPERATOR`** on every CI runner — use a + machine-readable namespaced identifier (e.g. + `gha:Acme/repo:training:run-${{ github.run_id }}`). See + `docs/qms/access_control.md` §3 for the recommended form. +- [ ] **Generate `FORGELM_AUDIT_SECRET`** in your KMS / Vault + (32+ random bytes, AES-256-GCM-strength entropy). +- [ ] **Schedule quarterly `FORGELM_AUDIT_SECRET` rotation** in + your KMS audit log. +- [ ] **Configure approver identity ≠ trainer identity** — the + Article 14 staging gate is your in-pipeline Change Advisory + Board. + +### Pipeline configuration + +- [ ] **`evaluation.require_human_approval: true`** for every + `risk_classification` ∈ `{high-risk, unacceptable}` run. + F-compliance-110 raises `ConfigError` if you forget. +- [ ] **Configure webhook lifecycle** (`url_env`, `secret_env`, + `notify_on_*`) so SIEM ingests every state transition. +- [ ] **Enable `auto_revert: true`** for production training so + a quality regression rolls back to the baseline model + automatically. + +### Audit + monitoring + +- [ ] **Schedule weekly `forgelm verify-audit` cron** on every + production ``. Wire alerts on non-zero exit. +- [ ] **Ship `audit_log.jsonl`** to write-once storage (S3 Object + Lock with compliance-mode retention, Azure Immutable Blob, + MinIO with versioning). +- [ ] **Encrypt-at-rest substrate** for model weights, audit logs, + training data — see `docs/qms/encryption_at_rest.md` for + per-asset substrate recommendations. + +### Supply-chain hygiene + +- [ ] **Pin ForgeLM** in CI (`pip install forgelm==X.Y.Z`). +- [ ] **Subscribe nightly SBOM diffs** for every release you + depend on. + +## Walking the auditor through evidence + +The eight most common audit-floor questions, with the exact ForgeLM +artefact + grep / command that produces the evidence. + +### Q1: "Show me the audit trail for every model promotion in the past 90 days" + +```bash +# Verify the chain integrity first. +forgelm verify-audit --output-dir ./outputs --json + +# Then extract the promotion events. +jq 'select(.event == "human_approval.granted")' ./outputs/audit_log.jsonl +``` + +Each `human_approval.granted` entry carries: + +- `operator` — who approved (the **approving** identity, not the + trainer). +- `run_id` — links back to the training run that produced the model. +- `prev_hash` + `_hmac` — chain integrity. +- `compliance.config_hash` — which config was used; an auditor can + diff against the YAML in `git log`. + +### Q2: "Show me the change-control evidence — who approved this model?" + +The chain itself **is** the change-control evidence. Cross-reference: + +```bash +# 1. Find the training event for run X. +jq 'select(.run_id == "X" and .event == "training.started")' \ + ./outputs/audit_log.jsonl + +# 2. Find the approval event for the same run. +jq 'select(.run_id == "X" and .event == "human_approval.granted")' \ + ./outputs/audit_log.jsonl + +# 3. Confirm the operator IDs differ. +jq -r 'select(.run_id == "X" and (.event == "training.started" or .event == "human_approval.granted")) | .operator' \ + ./outputs/audit_log.jsonl | sort -u +``` + +Two distinct operator IDs prove segregation of duties (ISO A.5.3, +SOC 2 CC1.5). + +### Q3: "Show me the data lineage" + +```bash +cat ./outputs/data_provenance.json +# { +# "dataset_id": "Acme/customer-support-v3", +# "hf_revision": "9c7c8f3...", +# "fingerprint_sha256": "ab12...", +# "size_bytes": 14_982_011, +# "mtime": "2026-05-01T08:14:33Z", +# ... +# } +``` + +The `fingerprint_sha256` + `hf_revision` together pin the corpus +deterministically. An auditor running `forgelm audit data/*.jsonl` +on the same input must see the same fingerprint. + +### Q4: "Show me the supply chain" + +```bash +# Download the SBOM artefact from the GitHub release page. +gh release download v0.5.5 --pattern 'sbom-*' + +# Or regenerate on demand. +python3 tools/generate_sbom.py > sbom.json + +# Diff against last release. +diff <(jq -S . sbom-prev.json) <(jq -S . sbom.json) +``` + +The CycloneDX 1.5 JSON lists every transitive dependency with its +purl (`pkg:pypi/...`) and version. Dependency-Track ingests this +natively for CVE correlation. + +### Q5: "Show me the access controls — how do you prove only authorised reviewers approve models?" + +Two layers: + +1. **IdP layer** — your CI runner identities and human reviewer + identities are issued by your IdP. The auditor walks your IdP + audit log to confirm the issuance + revocation cadence. +2. **ForgeLM layer** — every approval event records the approver's + `FORGELM_OPERATOR`. Cross-reference with the IdP audit log to + confirm the approver was authorised at approval time. + +```bash +# All approval events with approver id + timestamp. +jq -r 'select(.event == "human_approval.granted") | + [.timestamp, .operator, .run_id] | @tsv' \ + ./outputs/audit_log.jsonl +``` + +### Q6: "Show me the encryption posture" + +This is **deployer-side** — ForgeLM does not encrypt artefacts, +substrate does. Reference: + +- `docs/qms/encryption_at_rest.md` — your in-house policy mapped to + ForgeLM artefact classes. +- KMS audit log — substrate-side evidence of encryption-in-use. +- ForgeLM `data_governance_report.json` — your config block records + `encryption_at_rest: true|false` per the operator's declaration. + +### Q7: "Show me the incident response — what happens if the safety classifier crashes mid-run?" + +Reference: + +- `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 expansion) + for the security-incident playbook. +- The audit chain itself: `audit.classifier_load_failed` event fires + + `pipeline.failed` propagates + the run does NOT produce a + `final_model/` directory. +- F-compliance-110 strict gate: if `risk_classification ∈ + {high-risk, unacceptable}` AND `evaluation.safety.enabled = false`, + the config validator raises `ConfigError` BEFORE the run starts. + The auditor sees zero high-risk runs without a safety gate in + the past 90 days. + +### Q8: "Show me you can respond to a GDPR Article 15 + 17 request" + +Article 15 (right of access): + +```bash +forgelm reverse-pii --query "alice@example.com" --type email \ + --output-dir ./outputs data/*.jsonl +# JSON envelope returns { matches: [...], match_count: N } +# The audit chain stamps a `data.access_request_query` event with +# the QUERY HASHED (never raw — the salt protects against wordlist +# attacks against the audit log itself). +``` + +Article 17 (right to erasure): + +```bash +forgelm purge --row-id "alice@example.com" --corpus data/2026Q2.jsonl \ + --output-dir ./outputs +# The audit chain stamps `data.erasure_requested` → +# `data.erasure_completed` (or `data.erasure_failed`) with +# `target_id` HASHED. If a model trained on the row has +# `final_model/`, `data.erasure_warning_memorisation` ALSO fires +# and you must communicate the memorisation residual risk to the +# subject. +``` + +The cross-tool digest correlation (purge `target_id` == +reverse-pii `query_hash` for the same identifier in the same +`output_dir`) is pinned by Wave 3 follow-up tests. + +## Common pitfalls + +Things deployers get wrong on their first audit: + +1. **Sharing `FORGELM_OPERATOR` across pipelines.** "We just set + `FORGELM_OPERATOR=ci`" — the audit chain becomes uninterpretable + because every entry attributes to the same string. Use namespaced + identifiers per pipeline + per run. +2. **Storing webhook secrets in YAML.** Use `webhook.secret_env` → + resolve from KMS at runtime. A YAML in version control with a + plaintext secret is a finding even if the secret has been rotated + (the auditor sees historical exposure in `git log`). +3. **Skipping `forgelm verify-audit` in CI.** "We trust the audit + chain implicitly" is not a defensible position. Schedule the + weekly cron + alert on non-zero exit; have the alert history to + show the auditor. +4. **Forgetting the manifest sidecar.** `audit_log.jsonl` alone is + insufficient — `forgelm verify-audit` requires the + `audit_log.manifest.json` sidecar to detect truncate-and-resume + tampering. Back BOTH up. +5. **No `auto_revert` on production training.** If you're betting + on always-green training, you're a single safety-classifier + degradation away from a regulator-reportable incident. Enable + `auto_revert: true` and let `pipeline.reverted` events accumulate + as evidence of working safeguards. +6. **Running ForgeLM with `FORGELM_ALLOW_ANONYMOUS_OPERATOR=1` in + production.** That env var exists for short-lived test runs only. + Production runs without operator identity are an ISO A.6.4 + + A.6.5 finding. + +## References + +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — full design rationale + 93-control coverage map. +- [`../qms/encryption_at_rest.md`](../qms/encryption_at_rest.md) — substrate-side encryption guidance. +- [`../qms/access_control.md`](../qms/access_control.md) — operator identity + secrets management. +- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. +- [`../qms/statement_of_applicability.md`](../qms/statement_of_applicability.md) — 93-control SoA matrix. +- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook (Wave 4 expansion). +- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook (Wave 4 expansion). +- [`../reference/iso27001_control_mapping.md`](../reference/iso27001_control_mapping.md) — ISO 27001:2022 Annex A controls × ForgeLM evidence. +- [`../reference/soc2_trust_criteria_mapping.md`](../reference/soc2_trust_criteria_mapping.md) — SOC 2 Trust Services Criteria × ForgeLM evidence. +- [`../reference/supply_chain_security.md`](../reference/supply_chain_security.md) — SBOM + pip-audit + bandit overview. +- [`../reference/audit_event_catalog.md`](../reference/audit_event_catalog.md) — audit-event vocabulary. diff --git a/docs/qms/access_control.md b/docs/qms/access_control.md new file mode 100644 index 00000000..073aebb2 --- /dev/null +++ b/docs/qms/access_control.md @@ -0,0 +1,231 @@ +# QMS: Access Control + +> Quality Management System guidance — [YOUR ORGANIZATION] +> ISO 27001:2022 references: A.5.15, A.5.16, A.5.17, A.5.18, A.8.2, A.8.5 +> SOC 2 references: CC1.5, CC6.1, CC6.2, CC6.3, CC6.5, CC8.1 + +## 1. Purpose + +Define how the deployer organises identities, credentials, and access +rights around ForgeLM's audit-trail attribution model. + +**Core contract:** every event ForgeLM emits is stamped with a +`FORGELM_OPERATOR` identity. The audit chain is only as strong as +the identity-management substrate that issues the operator value. +Two operators sharing the same `FORGELM_OPERATOR` are +indistinguishable in the chain — that is a deployer-side identity +hygiene failure, not a ForgeLM defect. + +## 2. Scope + +The following identity-bearing surfaces: + +| Surface | What carries identity | +|---|---| +| `FORGELM_OPERATOR` env var | Stamped on every audit-log entry | +| Approval gate (`forgelm approve` / `reject`) | Operator id at approval time | +| Reverse-PII / purge subcommands | Operator id on each Article 15 / 17 event | +| Webhook lifecycle events | Operator id propagated to webhook payload | +| CI runner identity | The CI runner's service-account credential is the operator's "real" identity bound to `FORGELM_OPERATOR` | + +ForgeLM does NOT itself implement RBAC, MFA, or directory federation — +those live in the deployer's IdP (Okta, Azure AD, Google Workspace, +Keycloak). What ForgeLM does is bind every audit entry to whatever +string the deployer puts in `FORGELM_OPERATOR` so the IdP audit log +and ForgeLM audit log can be cross-referenced. + +## 3. Operator identity contract + +### 3.1 Resolution order + +ForgeLM resolves the operator identity at audit-event emit time: + +1. `FORGELM_OPERATOR` env var (preferred — explicit set). +2. `getpass.getuser()` (POSIX user name fallback). +3. `FORGELM_ALLOW_ANONYMOUS_OPERATOR=1` permits emission without + identity (intended for short-lived test runs only; ConfigError + otherwise). + +### 3.2 Required form + +Use a **machine-readable, namespaced identifier** rather than a free- +text human name: + +| Pattern | Example | Why | +|---|---|---| +| `//` | `prod/training/gh-runner-42` | Maps to CI runner pool | +| `` | `alice@acme.example` | OK for human-driven approval | +| `oidc::` | `oidc:gha:repo:Acme/forgelm-pipelines:ref:refs/heads/main` | OIDC-token-bound | + +Anti-patterns: + +- `root` / `cemil` / `admin` — ambiguous, unrotatable. +- Empty / `unknown` — defeats the audit attribution purpose. + +### 3.3 Rotation when staff change + +ForgeLM's audit chain is **immutable**. Old `FORGELM_OPERATOR` IDs +remain in the chain forever. When an operator leaves: + +1. **Revoke their CI runner identity** (rotate the OIDC trust / + delete the IAM role). +2. **Do NOT re-issue the same `FORGELM_OPERATOR` string** to a new + person — give them a new identifier so future entries are not + confused with departed-staff entries. +3. **Audit prior actions** by grepping the audit log for their + identifier; this is your termination-day forensic record. + +### 3.4 `FORGELM_AUDIT_SECRET` rotation + +The HMAC-chain key (per audit run; XOR'd with the per-output-dir +salt to derive the actual signing key — see +`forgelm/cli/subcommands/_purge.py:_resolve_salt`). Treat as a Tier-1 +secret: + +- **Length:** 32+ random bytes (256 bits of entropy). +- **Substrate:** KMS / Vault / equivalent. Never in `.env` checked + into VCS. +- **Rotation cadence:** quarterly recommended; immediately on + suspected compromise. +- **Rotation procedure:** new value lands in env BEFORE the next + pipeline run. Old runs' chains remain verifiable with the old + key (audit log entries are bound to the key live at emit time; + ForgeLM does NOT support key migration on the chain — by design). + +## 4. CI runner identity binding + +The recommended pattern when running ForgeLM from CI: + +### 4.1 GitHub Actions + +```yaml +jobs: + train: + runs-on: ubuntu-latest + permissions: + id-token: write # OIDC + contents: read + env: + FORGELM_OPERATOR: gha:${{ github.repository }}:${{ github.workflow }}:run-${{ github.run_id }} + FORGELM_AUDIT_SECRET: ${{ secrets.FORGELM_AUDIT_SECRET }} + steps: + - uses: actions/checkout@v5 + - run: pip install forgelm + - run: forgelm --config config.yaml +``` + +**Why this binding:** the run id is unique per workflow execution; an +auditor can correlate a single audit-log entry to the GitHub Actions +run page in seconds. + +### 4.2 GitLab CI + +```yaml +train: + script: + - export FORGELM_OPERATOR="gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" + - forgelm --config config.yaml +``` + +### 4.3 Jenkins + +```groovy +pipeline { + agent any + environment { + FORGELM_OPERATOR = "jenkins:${env.JOB_NAME}:build-${env.BUILD_ID}" + FORGELM_AUDIT_SECRET = credentials('forgelm-audit-secret') + } + stages { + stage('Train') { steps { sh 'forgelm --config config.yaml' } } + } +} +``` + +## 5. OS-level isolation + +Within a single training host, multiple pipelines should NOT share an +output directory: + +- One Unix user per pipeline (`forgelm-prod`, `forgelm-staging`, + `forgelm-research`). +- `chmod 0700 ` on each pipeline's directory. +- Audit logs in `` inherit the dir's perm; do not chmod + individually unless your KMS substrate requires read access. +- The `.forgelm_audit_salt` file is `0600` (owner-read only) per + ForgeLM's atomic O_EXCL creation in `_resolve_salt`. + +## 6. Approval gate identity separation + +Article 14 staging requires the approving operator to be different +from the training operator (segregation of duties — ISO A.5.3, SOC 2 +CC1.5): + +```bash +# Job 1 — training (CI runner identity) +FORGELM_OPERATOR="gha:Acme/pipelines:training:run-42" \ + forgelm --config config.yaml + +# Job 2 — approval (human reviewer identity) +FORGELM_OPERATOR="alice@acme.example" \ + forgelm approve --run-id +``` + +ForgeLM does not enforce that the two identities differ — that is a +deployer-side IdP control. The audit chain records both, so an +auditor can detect violations: + +```bash +# Find approvals where trainer == approver (segregation violation) +forgelm verify-audit --output-dir ./outputs --json | jq ' + .events[] | + select(.event == "human_approval.granted") | + .matching_training_event = ( + [.|.run_id == .. .run_id and .event == "training.started"] | first + ) | + select(.operator == .matching_training_event.operator) | + .' +``` + +## 7. Webhook secret separation + +Webhook URLs and HMAC-signing keys belong in env, never in YAML: + +```yaml +webhook: + url_env: SLACK_WEBHOOK_URL # resolved from env + secret_env: FORGELM_WEBHOOK_SECRET # HMAC body sign + timeout: 10 +``` + +The webhook lifecycle events (`notify_started`, `notify_succeeded`, +`notify_failed`, `notify_reverted`, `notify_awaiting_approval` — +Phase 8 vocabulary) carry the `FORGELM_OPERATOR` identity into the +webhook payload so the receiving system (Slack, Teams, custom +incident-management) can attribute the notification. + +## 8. Verification checklist + +For a deployer auditor walking access-control evidence: + +- [ ] Every CI runner pipeline sets `FORGELM_OPERATOR` from a + machine-readable namespace. +- [ ] No two active pipelines share an `FORGELM_OPERATOR` value. +- [ ] `FORGELM_AUDIT_SECRET` lives in a KMS / Vault substrate, never + in VCS or a plain `.env` file. +- [ ] Quarterly KMS audit log shows `FORGELM_AUDIT_SECRET` rotation. +- [ ] Approval-gate identity differs from training identity for every + `human_approval.granted` event in the past 90 days + (sample-audit, not exhaustive). +- [ ] `` permission masks: `0700` (dir), + `.forgelm_audit_salt` `0600`. +- [ ] No `FORGELM_OPERATOR=root` / `=admin` / `=unknown` events in + the chain. +- [ ] Webhook URLs and HMAC keys are env-resolved (`url_env`, + `secret_env`); no plaintext URL in any committed YAML. + +## 9. Review + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | Initial version (Wave 4 / Faz 23) | diff --git a/docs/qms/encryption_at_rest.md b/docs/qms/encryption_at_rest.md new file mode 100644 index 00000000..f550a298 --- /dev/null +++ b/docs/qms/encryption_at_rest.md @@ -0,0 +1,221 @@ +# QMS: Encryption at Rest + +> Quality Management System guidance — [YOUR ORGANIZATION] +> ISO 27001:2022 references: A.5.33, A.8.10, A.8.13, A.8.24 +> SOC 2 references: CC6.1, CC6.7 + +## 1. Purpose + +Define the deployer-side controls required to keep ForgeLM-produced +assets confidential at rest. ForgeLM emits **integrity-protected** +artefacts (HMAC-chained audit logs, SHA-256 model integrity files, +signed deployer instructions) but does NOT encrypt them — encryption +is a substrate concern owned by the deployer's storage layer. + +This document maps each ForgeLM artefact class to the recommended +encryption substrate and operator action. + +## 2. Scope + +The following ForgeLM-produced asset classes: + +| Asset class | Path pattern | Confidentiality concern | +|---|---|---| +| Model weights — final | `/final_model/` | Memorisation of training data; competitive moat | +| Model weights — staging | `/staging_model./` | Same as final, plus pre-approval state | +| Audit log | `/audit_log.jsonl` + `.manifest.json` + `.sha256` sidecar | Operator identity history; configuration hashes; chain-of-custody for compliance | +| Per-output-dir salt | `/.forgelm_audit_salt` | Unsalted SHA-256 of low-entropy identifiers (emails, phones) is brute-forcible — the salt is the secret that gives the audit hash its wordlist resistance | +| Training corpus | (operator-supplied; typically `data/*.jsonl`) | PII; trade secrets; client data | +| Quickstart-rendered config | (operator path) | HF tokens, webhook URLs, secret env-var names | +| Compliance bundle (Annex IV) | `/compliance/` ZIP | Aggregated form of every above artefact | + +Operator-controlled assets (HF auth tokens, webhook secrets, +`FORGELM_AUDIT_SECRET`) live in the deployer's secrets-management +substrate and are out of scope for this document — see +`access_control.md` for credential-handling guidance. + +## 3. Threat model + +The threats this guidance defends against: + +1. **Disk theft / physical loss** — laptop or training-host hard + drive falls into adversary hands. +2. **Backup compromise** — backup snapshot ends up on a misconfigured + bucket or third-party cloud; older snapshots may pre-date current + encryption policy. +3. **Shared-tenancy disk leak** — multi-tenant cloud storage where + another tenant gains read access via misconfiguration. +4. **Log-ship intercept** — audit log shipped to SIEM over an + unencrypted channel. +5. **Forensic recovery from decommissioned disks** — sectors not + wiped before disposal. + +Threats explicitly OUT of scope: + +- **Live-memory adversary** — process memory dumps; the deployer's + endpoint detection / sandboxing handles this. +- **Insider with legitimate decryption credentials** — encryption + cannot defend against authorised access; access control + audit + log is the answer. +- **Crypto-algorithm break** — assume SHA-256 / AES-256-GCM remain + intact for the QMS document's lifetime. + +## 4. Recommended controls per asset class + +### 4.1 Model weights (final + staging) + +| Substrate | Recommendation | +|---|---| +| Linux training host | LUKS / dm-crypt full-disk encryption (kernel-supported, transparent) | +| macOS training host | FileVault (transparent) | +| Windows training host | BitLocker (TPM-backed key) | +| Linux container | Encrypted block device passed in via Docker `--mount type=bind,...` from a LUKS-protected host volume | +| AWS S3 | SSE-KMS with a customer-managed key (CMK); enable bucket-level default encryption + bucket policy `aws:SecureTransport=true` | +| Azure Blob | Customer-managed key via Azure Key Vault | +| GCS | Customer-Supplied Encryption Keys (CSEK) or Cloud KMS | +| HDFS / Ceph | Native at-rest encryption with deployer-managed KMS | + +**Why full-disk and not file-level:** model weights are written by +HF Transformers / PEFT in many small files (sharded `.safetensors`, +adapter manifests). File-level GPG / age over each file produces +fragile file enumerations. Substrate-level encryption is transparent +and auditable. + +**Key rotation:** at least annually for SSE-KMS; rotate the CMK +itself, not just the data key. ForgeLM is unaffected by rotation — +the model files re-encrypt at the storage layer without the +training pipeline noticing. + +### 4.2 Audit log + manifest sidecar + +**Critical contract:** the audit chain is integrity-protected at +rest by ForgeLM (HMAC + SHA-256 chain + genesis manifest). Encryption +is purely confidentiality. **Do NOT encrypt at a layer that +re-orders or re-frames the bytes** — once `.manifest.json` is signed +over a specific byte range, any encryption that perturbs that range +breaks `forgelm verify-audit`. + +Safe substrates: + +- Same-disk encryption (LUKS, FileVault, BitLocker) — transparent. +- Cloud blob storage with SSE-S3 or SSE-KMS — transparent. + +Unsafe substrates: + +- Custom application-layer wrapping that base64s + GPG-encrypts the + JSONL — breaks `forgelm verify-audit` unless decryption happens + before the verify command runs. + +**Off-site replication:** ship to write-once storage (S3 Object Lock +in compliance mode, Azure Immutable Blob, MinIO with versioning) so +even an authorised operator cannot retroactively delete entries. +ForgeLM's append-only contract is on-host; durability is the +deployer's responsibility. + +### 4.3 Per-output-dir salt (`.forgelm_audit_salt`) + +This is the most operationally-sensitive single file ForgeLM emits. +Compromising it lets an attacker run an offline dictionary attack +against `data.access_request_query` and `data.erasure_*` audit events +to recover the cleartext identifiers ForgeLM intentionally hashed. + +**Recommended treatment:** + +- Mode `0600` on the file (`chmod 600 .forgelm_audit_salt`). +- Owned by the CI pipeline service account, not a human user. +- Backed up only to substrates that match the audit log's encryption + posture (NEVER plaintext to a debug bucket). +- Rotated only when the corresponding `` is decommissioned + — rotating mid-output-dir invalidates `forgelm verify-audit`'s + salted-hash assumptions for prior events. + +**Defence-in-depth:** export `FORGELM_AUDIT_SECRET` from your KMS so +the on-disk salt becomes one half of a two-component key. An attacker +who steals the disk but not the env secret cannot reverse the audit +hash. See `access_control.md` §3.4 for env-secret rotation. + +### 4.4 Training corpus + +The corpus is operator-supplied and operator-encrypted. ForgeLM +contributes: + +- `forgelm audit` PII detection + secrets scan to flag what NEEDS + encryption (e.g. corpus that turned out to contain credit-card + numbers). +- `data_audit_report.json` with severity tiers — operator decides + whether to re-encrypt, mask, or quarantine flagged corpora. +- `data.dataset_id` + `data_provenance.json` SHA-256 fingerprint — + detects substrate-level tampering after encryption. + +**Trade secret protection:** if the corpus is the deployer's +intellectual property (proprietary support tickets, internal +documentation), treat it as Top Secret and encrypt at the substrate +layer with a CMK whose key access is logged. Train within an isolated +VPC; do NOT let `forgelm` egress the corpus to webhook endpoints +(this is already enforced — `safe_post` never carries raw training +rows in webhook payloads). + +### 4.5 Quickstart-rendered config + operator YAML + +Configs may carry HF auth tokens (`HF_TOKEN`) or webhook URLs that +embed a host-internal hostname. Treat them as credential-bearing: + +- Store in a config-management substrate (Ansible Vault, Doppler, + Vault) NOT a flat file in the home directory. +- For ephemeral CI use, render the config from secrets-manager values + at job start; delete the rendered file in the job teardown. +- ForgeLM's `compliance.config_hash` in audit events is computed AFTER + any secret expansion, so two config files differing only in their + secret values produce different hashes — auditors can detect a + config swap mid-run. + +### 4.6 Compliance bundle (Annex IV ZIP) + +The `forgelm --compliance-export` ZIP aggregates every artefact above +into one file. Treat it as the union of all asset classes: + +- Encrypted at the substrate the most-sensitive asset requires + (typically the training corpus dictates). +- Shipped to the auditor over an authenticated channel (e.g. + password-protected ZIP via a regulator-supplied portal; signed S3 + pre-signed URL with 24-hour expiry). +- Receipt confirmation logged back into the audit chain via + `compliance.artifacts_exported` event. + +## 5. ForgeLM's contribution + +ForgeLM does not implement encryption itself but DOES: + +1. **Detect what to encrypt.** `forgelm audit` flags PII, secrets, + and credentials in training data so the operator can encrypt or + mask before training. +2. **Report what is encrypted.** `data_governance_report.json` records + `encryption_at_rest: true|false` per the operator's config block. +3. **Verify post-encryption integrity.** `forgelm verify-audit` + confirms the chain-after-decryption matches the chain-as-emitted; + any substrate-level corruption is detectable. +4. **Hash low-entropy identifiers.** Salted SHA-256 means the audit + chain itself does not need encryption to be confidential at rest + — only the salt does. + +## 6. Verification checklist + +For a deployer auditor walking your encryption-at-rest controls: + +- [ ] All training-host disks pass `dmsetup table` (Linux) or + `manage-bde -status` (Windows) showing encryption-in-use. +- [ ] Cloud storage buckets' default-encryption setting is enabled + with a customer-managed key. +- [ ] `/.forgelm_audit_salt` is mode `0600` and owned by + a service account. +- [ ] Audit log replication target is write-once (S3 Object Lock, + Azure Immutable Blob). +- [ ] Quarterly key-rotation evidence in KMS audit log. +- [ ] No ForgeLM artefact lives outside an encrypted substrate + (use `find` + storage-policy reports to confirm). + +## 7. Review + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | Initial version (Wave 4 / Faz 23) | diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md new file mode 100644 index 00000000..5a85dd00 --- /dev/null +++ b/docs/qms/risk_treatment_plan.md @@ -0,0 +1,210 @@ +# QMS: Risk Treatment Plan (RTP) + +> Quality Management System — [YOUR ORGANIZATION] +> ISO 27001:2022 references: A.5.7, A.5.8, A.5.9, A.5.31, A.6.8, A.8.8, A.8.30 +> SOC 2 references: CC3.2, CC3.3, CC9.1, CC9.2 + +## 1. Purpose + +Document the risks ForgeLM-introduced to the deployer's training +pipeline, the treatments ForgeLM ships out-of-the-box, and the +residual risk the deployer accepts or further mitigates. + +This is a living document — update it on every quarterly risk review +and after any significant incident (`pipeline.failed`, +`data.erasure_failed`, `audit.classifier_load_failed`, +serious-incident report). + +## 2. Methodology + +Following ISO 27005 risk-management principles: + +- **Likelihood** (L): how often the risk can materialise — `Low` + (yearly), `Med` (quarterly), `High` (monthly+). +- **Impact** (I): blast radius if it does — `Low` (operator nuisance), + `Med` (single run / single subject), `High` (regulator-reportable + incident, model recall). +- **Inherent risk** = L × I before any treatment. +- **Residual risk** = L × I after the treatment ForgeLM ships + + deployer-side controls. + +A residual `High` requires explicit **risk acceptance** by the AI +Officer (see `roles_responsibilities.md`). + +## 3. Risk register + +Pre-populated with risks ForgeLM's own threat model identifies. The +deployer's compliance team adds organisation-specific rows below +section §4. + +### 3.1 Training-pipeline risks + +#### R-01 Training-data poisoning (adversarial corpus) + +| Field | Value | +|---|---| +| Description | Adversarial actor injects malicious examples into the training corpus to bias model behaviour | +| L × I (inherent) | Med × High = HIGH | +| Treatment | `forgelm audit` PII / secrets / quality scan; `data_audit_report.json` flags severity tiers; deployer pre-flight review of flagged rows; `compute_dataset_fingerprint` SHA-256 stamps "what was trained on" into manifest | +| Residual L × I | Med × Low = MED | +| Owner | Data Steward | +| Review cadence | Per training run (pre-flight) | + +#### R-02 Supply-chain compromise (compromised PyPI dependency) + +| Field | Value | +|---|---| +| Description | Upstream package on PyPI gains a malicious release; ForgeLM consumes it via pip install | +| L × I (inherent) | Low × High = MED | +| Treatment | SBOM (CycloneDX 1.5) on every release tag; `pip-audit` nightly (Wave 4 / Faz 23); `forgelm doctor` pre-flight env check; pinned upper bounds in `pyproject.toml`; transitive-dep CVE feed | +| Residual L × I | Low × Med = LOW | +| Owner | ML Engineer + Compliance Officer | +| Review cadence | Continuous (nightly) + per-release (SBOM diff) | + +#### R-03 Credential leak (HF token / webhook secret in config or audit log) + +| Field | Value | +|---|---| +| Description | Operator commits a config with `HF_TOKEN: ghp_...` literal or a webhook URL with embedded credentials | +| L × I (inherent) | Med × Med = MED | +| Treatment | `safe_post` masks Authorization headers in error logs; `forgelm audit --secrets` regex scan flags credentials; `_sanitize_md_list` escapes operator-controlled strings in deployer instructions; `forgelm doctor` pre-flight HF-auth probe; webhook payload-format curation never carries config-derived secrets | +| Residual L × I | Low × Low = LOW | +| Owner | ML Engineer | +| Review cadence | Pre-merge config review | + +#### R-04 Audit-log tampering + +| Field | Value | +|---|---| +| Description | Adversary with write access to `audit_log.jsonl` rewrites entries to hide a deployment / approve a model that wasn't approved | +| L × I (inherent) | Low × High = MED | +| Treatment | Append-only `O_APPEND` + `flock` + `fsync` per line; HMAC chain (per-line `_hmac` field XOR'd with per-output-dir salt + env secret); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` refuses truncate-and-resume); `forgelm verify-audit` validates end-to-end | +| Residual L × I | Low × Low = LOW | +| Owner | AI Officer | +| Review cadence | Continuous (`forgelm verify-audit` cron) | + +#### R-05 Memorisation of removed PII (Article 17 erasure incomplete) + +| Field | Value | +|---|---| +| Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights | +| L × I (inherent) | High × Med = HIGH | +| Treatment | `data.erasure_warning_memorisation` audit event raised when the run that consumed the deleted row has a `final_model/` artefact; deployer notifies subject of the residual risk; for high-stakes deployments retrain from scratch | +| Residual L × I | Med × Med = MED | +| Owner | Data Protection Officer (DPO) | +| Review cadence | Per Article 17 request | + +### 3.2 Safety + alignment risks + +#### R-06 Safety-classifier load failure + +| Field | Value | +|---|---| +| Description | Llama Guard / configured safety classifier fails to load (HF Hub down, OOM, version mismatch); ForgeLM proceeds without the gate fired | +| L × I (inherent) | Med × High = HIGH | +| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` raises `ConfigError`; `audit.classifier_load_failed` event records the failure; deployer pipeline halts on non-zero exit | +| Residual L × I | Low × Low = LOW | +| Owner | ML Lead | +| Review cadence | Per training run | + +#### R-07 Auto-revert false positive + +| Field | Value | +|---|---| +| Description | `evaluation.auto_revert: true` triggers on a transient eval regression; the resulting `pipeline.reverted` looks like a safety failure to a downstream auditor | +| L × I (inherent) | Med × Low = LOW | +| Treatment | `pipeline.reverted` audit event carries the regression delta + threshold; `safety_trend.jsonl` provides cross-run context; deployer dashboard distinguishes "real" reverts from threshold-tuning issues | +| Residual L × I | Low × Low = LOW | +| Owner | ML Lead | +| Review cadence | Weekly trend review | + +### 3.3 Webhook + outbound-comms risks + +#### R-08 Webhook SSRF / data exfiltration + +| Field | Value | +|---|---| +| Description | Adversary configures `webhook.url_env=http://internal-metadata-server/...` to exfiltrate AWS instance credentials | +| L × I (inherent) | Low × Med = LOW | +| Treatment | `safe_post` (Phase 7) — HTTPS-only, SSRF guard rejects RFC 1918 / 169.254.x / loopback / link-local, no redirect-following, masked auth headers in error logs; webhook URL must come from `url_env`, never inline | +| Residual L × I | Low × Low = LOW | +| Owner | Security | +| Review cadence | Per config review | + +#### R-09 Webhook target compromised + +| Field | Value | +|---|---| +| Description | Operator's Slack / Teams workspace is compromised; attacker reads notify_started payloads and learns model-deployment cadence | +| L × I (inherent) | Low × Low = LOW | +| Treatment | Webhook payload curation never carries raw training data or unredacted PII; `FORGELM_AUDIT_SECRET`-signed payloads detect splicing; deployer rotates webhook secret on incident | +| Residual L × I | Low × Low = LOW | +| Owner | Security + ML Lead | +| Review cadence | Per webhook-target incident | + +### 3.4 ReDoS + ingestion risks + +#### R-10 ReDoS via `--type custom` regex (reverse-pii) + +| Field | Value | +|---|---| +| Description | Operator-supplied regex on `forgelm reverse-pii --type custom --query "(a+)+$"` triggers catastrophic backtracking | +| L × I (inherent) | Low × Low = LOW | +| Treatment | POSIX SIGALRM 30s per-file budget in `_scan_file_with_alarm` (Faz 38 / Wave 3 followup F-W3FU-T-01); thread-safety guard skips non-main thread; outer alarm preserved | +| Residual L × I | Low × Low = LOW | +| Owner | Security | +| Review cadence | Wave 3 closed; no further action | + +#### R-11 Cross-tool digest mismatch (purge ↔ reverse-pii) + +| Field | Value | +|---|---| +| Description | `forgelm purge` runs with one salt; `forgelm reverse-pii` for the same identifier runs with another (e.g. `FORGELM_AUDIT_SECRET` change between runs) — digest doesn't match, audit chain looks inconsistent | +| L × I (inherent) | Low × Med = LOW | +| Treatment | `salt_source` recorded in every audit event (Wave 3 followup F-W3-PS-07); explicit `--salt-source per_dir` honoured even when env secret set; cross-tool correlation test (`test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`) pinned in CI | +| Residual L × I | Low × Low = LOW | +| Owner | DPO | +| Review cadence | Per Article 15/17 request | + +### 3.5 Deployment risks + +#### R-12 Unauthorised model deployment + +| Field | Value | +|---|---| +| Description | Operator skips human approval; high-risk model reaches production without ML Lead / AI Officer sign-off | +| L × I (inherent) | Med × High = HIGH | +| Treatment | `evaluation.require_human_approval: true` for `risk_classification` ∈ `{high-risk, unacceptable}`; staging directory holds the model until `forgelm approve`; F-compliance-110 raises ConfigError if not configured; `human_approval.required/granted/rejected` chain forensically records every decision | +| Residual L × I | Low × Med = LOW | +| Owner | ML Lead + AI Officer | +| Review cadence | Per training run for high-risk | + +## 4. Deployer-specific rows + +[Add rows here for organisation-specific risks not anticipated by +ForgeLM's own threat model — e.g. industry-specific regulator +requirements, third-party legal commitments, integration-specific +risks. Use the same field structure as §3.] + +## 5. Risk acceptance log + +When a residual risk is `Med` or `High`, the AI Officer must sign +acceptance: + +| Risk ID | Inherent | Residual | Accepted by | Date | Justification | +|---|---|---|---|---|---| +| R-05 | HIGH | MED | [AI Officer] | [DATE] | Memorisation residual risk acknowledged; deployer policy: notify subject + retrain from scratch for high-stakes deployments | +| ... | ... | ... | ... | ... | ... | + +## 6. Review + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | Initial version (Wave 4 / Faz 23) — 12 ForgeLM-identified risks | + +Quarterly review cadence: + +- Re-score each row's L × I against the past quarter's incident + frequency. +- Add new rows for deployer-specific risks identified in retro. +- Confirm residual-risk acceptance log is current. diff --git a/docs/qms/sop_change_management.md b/docs/qms/sop_change_management.md index dde24338..42003c10 100644 --- a/docs/qms/sop_change_management.md +++ b/docs/qms/sop_change_management.md @@ -2,6 +2,8 @@ > Standard Operating Procedure — [YOUR ORGANIZATION] > EU AI Act Reference: Article 17(1)(b)(c) +> ISO 27001:2022: A.5.36, A.8.9, A.8.32 +> SOC 2: CC8.1, CC3.4, CC5.3 ## 1. Purpose @@ -59,7 +61,73 @@ If the new model is worse: 2. Document rollback in incident log 3. Investigate root cause -## 4. Version Tracking +## 4. CI gates as the formal change-control mechanism + +Wave 4 / Faz 23 expansion: this section formalises which CI gates +constitute the deployer-facing change-control evidence. ISO A.8.32 +demands "changes to information processing facilities and systems +shall be subject to change management procedures"; the table below +documents the procedure for ForgeLM-instrumented deployers. + +| Gate | Tool | Failure mode | ISO control | +|---|---|---|---| +| Lint clean | `ruff check .` + `ruff format --check .` | Style / syntax issues | A.8.28 | +| Pydantic field-description guard | `tools/check_field_descriptions.py --strict forgelm/config.py` | Operator-facing config drift | A.5.36 | +| HTTP discipline | `grep`-based regex guard in `ci.yml` | Undisciplined `requests.*` / `urllib.*` / `httpx.*` call | A.8.20 | +| Unit + integration tests | `pytest -q` | Regression on documented contract | A.8.29 | +| Coverage floor | `--cov-fail-under=40` (`pyproject.toml`) | Uncovered new code | A.8.29 | +| CLI dry-run | `forgelm --config config_template.yaml --dry-run` | Config schema break | A.8.9 | +| Site-as-tested-surface | `tools/check_site_claims.py --strict` | Marketing claim drift from code | A.5.31 | +| Bilingual H2/H3/H4 parity | `tools/check_bilingual_parity.py --strict` | EN/TR docs structural drift | A.5.36 | +| Bandit static-security (Wave 4) | `bandit -c pyproject.toml -r forgelm/` | High-severity finding | A.8.26, A.8.28 | +| pip-audit nightly (Wave 4) | `pip-audit --format json` | High-severity transitive CVE | A.8.8, A.8.21 | +| SBOM determinism (Wave 4) | `pytest tests/test_supply_chain_security.py` | SBOM content drift | A.5.21, A.8.8 | + +A change cannot merge to `main` (or `development` for Wave-style +flows) until every gate passes. The PR review template (`.github/ +PULL_REQUEST_TEMPLATE.md`) MUST require an explicit checkbox +confirming each gate green. + +### 4.1 Approval gate as Change Advisory Board (CAB) substitute + +For changes that produce a new training run (vs. configuration-only +changes), the **Article 14 staging gate** is the deployer's +in-pipeline CAB: + +1. CI training job lands the model in `/staging_model./`. +2. `human_approval.required` audit event fires. +3. A reviewer (NOT the trainer) runs `forgelm approve --run-id `. +4. `human_approval.granted` audit event fires; model promotes to + `/final_model/`. +5. If rejected, `human_approval.rejected` event fires; model stays + in staging until retention expires (`evaluation.approval_retention_days`). + +This gives a deployer SOC 2 CC8.1 evidence: every model promotion is +attributed, dual-controlled, and forensically recorded. + +### 4.2 Configuration drift detection + +`tools/regenerate_config_doc.py` (Phase 16) regenerates +`docs/reference/configuration.md` (and `-tr.md` mirror) from the +Pydantic schema. CI runs the diff guard; a config-schema change +without a corresponding doc update fails the build. + +This closes the "doc drift" failure mode where the schema evolves +but the operator-facing doc lags. ISO A.5.36 cites this as a +mandatory mechanism. + +### 4.3 SBOM drift detection + +`tools/generate_sbom.py` (Wave 2 era) produces a deterministic +CycloneDX 1.5 SBOM per (OS × Python-version) on every release tag +(`publish.yml`). Wave 4 adds a determinism contract test +(`tests/test_supply_chain_security.py::TestGenerateSbomDeterministic`). + +Drift detection: a release's SBOM is reproducible from its `git tag`. +An auditor can re-emit it on demand and diff against the artefact +attached to the GitHub release. + +## 5. Version Tracking Every training run produces: - `compliance_report.json` with ForgeLM version, config hash, timestamps @@ -68,8 +136,9 @@ Every training run produces: Use these to trace any model back to its exact training configuration and data. -## 5. Review +## 6. Review | Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | [DATE] | [AUTHOR] | Initial version | +| 1.1 | 2026-05-05 | Wave 4 / Faz 23 | Added §4 CI-gates-as-change-control table (11 gates × ISO controls); §4.1 Article 14 approval gate as CAB substitute; §4.2 config-drift detection via `regenerate_config_doc.py`; §4.3 SBOM drift detection via `generate_sbom.py` + determinism test | diff --git a/docs/qms/sop_incident_response.md b/docs/qms/sop_incident_response.md index b8ca8dbf..330eff1e 100644 --- a/docs/qms/sop_incident_response.md +++ b/docs/qms/sop_incident_response.md @@ -2,10 +2,17 @@ > Standard Operating Procedure — [YOUR ORGANIZATION] > EU AI Act Reference: Article 17(1)(h)(i) +> ISO 27001:2022: A.5.24, A.5.25, A.5.26, A.5.27, A.6.8, A.8.15, A.8.16 +> SOC 2: CC4.2, CC7.3, CC7.4, CC7.5, CC9.2 ## 1. Purpose -Define the procedure for handling safety incidents, model failures, and corrective actions for deployed fine-tuned models. +Define the procedure for handling safety incidents, model failures, +**security incidents**, and corrective actions for deployed +fine-tuned models. Wave 4 / Faz 23 expansion: §6 covers the security- +incident playbook (audit-chain integrity, credential leak, supply- +chain CVE, webhook target compromise, GDPR DSARs) alongside the +existing AI-safety incident flow. ## 2. Incident Classification @@ -59,7 +66,151 @@ Incidents may be detected by: 4. [ ] Review and update this SOP if needed 5. [ ] For EU AI Act: report serious incidents to relevant authority within **15 days** -## 4. Serious Incident Reporting (EU AI Act) +## 4. Security incidents — Wave 4 / Faz 23 expansion + +The Wave 4 ISO 27001 / SOC 2 alignment closure adds the security- +incident playbook below. AI-safety incidents (§§1–3) and security +incidents both flow through this SOP; the differentiator is the +detection event class. + +### 4.1 Audit-chain integrity violation + +**Trigger:** `forgelm verify-audit` exits non-zero (chain hash +mismatch, manifest sidecar truncation, HMAC signature mismatch). + +**Severity:** Critical. + +**Runbook:** + +1. [ ] **Isolate** the affected `` — `chmod 0500` on the + directory to prevent further writes. +2. [ ] **Preserve evidence** — copy `audit_log.jsonl`, + `audit_log.manifest.json`, the `.sha256` sidecar (if + present), and `/.forgelm_audit_salt` to a + write-once forensic substrate (S3 Object Lock, Azure + Immutable Blob). +3. [ ] **Identify the last trusted entry** — bisect with + `forgelm verify-audit --until-line N` to find the first bad + line; everything before that line is forensically trusted, + everything after must be considered tainted. +4. [ ] **Notify** the AI Officer + Security team + DPO (if any + PII-bearing event was after the bad line). +5. [ ] **Decide** whether to retain the tainted-tail entries as + evidence (recommended) or roll back. +6. [ ] **Audit** the IdP for unauthorised write access to the + `` substrate during the suspect time window. + +### 4.2 Credential leak detected + +**Trigger:** `forgelm audit` `_SECRET_PATTERNS` regex matches a +credential in the training corpus or a webhook log; OR an +external CVE / breach disclosure cites a token that was used. + +**Severity:** Critical. + +**Runbook:** + +1. [ ] **Rotate the leaked credential immediately** at the issuing + authority (HF Hub token, GitHub PAT, Slack webhook, OpenAI + API key, AWS access key, etc.). +2. [ ] **Run `forgelm purge --row-id `** against every + corpus that includes the leaked credential row. +3. [ ] **Flag the run as memorisation-tainted** — the + `data.erasure_warning_memorisation` event documents this. +4. [ ] **Document the rotation** in your KMS audit log; tie back to + the ForgeLM `data.erasure_completed` event timestamp. +5. [ ] **Re-train from scratch** for high-risk deployments. +6. [ ] **Update the training-data-onboarding checklist** to require + `forgelm audit --secrets` pre-flight. + +### 4.3 Supply-chain CVE flagged + +**Trigger:** `pip-audit` nightly fails high-severity (Wave 4 / Faz +23 introduced this gate); OR a CVE advisory drops on a dependency +ForgeLM uses. + +**Severity:** High. + +**Runbook:** + +1. [ ] **Pin to a safe version** in `pyproject.toml`. +2. [ ] **Rebuild the SBOM** for the new pinned set + (`tools/generate_sbom.py`). +3. [ ] **Regenerate downstream artefacts** — re-run dependent + training pipelines that consumed the affected dep. +4. [ ] **Notify deployers** if a model already shipped with the + affected dep in its training-time env (`compliance_report.json` + lists the env). +5. [ ] **File a tracking ticket** with the CVE id + the SBOM diff + + the affected runs (use the audit log's `compliance.config_hash` + to identify them). + +### 4.4 Webhook target compromised + +**Trigger:** Slack / Teams / custom-webhook recipient confirms a +breach; OR `safe_post` `_mask` shows redacted Authorization headers +that an attacker may have observed. + +**Severity:** High. + +**Runbook:** + +1. [ ] **Rotate `webhook.secret_env`** immediately. +2. [ ] **Re-emit lifecycle events** from the audit chain to confirm + the attacker did not splice events into the recipient. + (`forgelm verify-audit --replay-since ` is the + v0.6.0+ tool for this; for v0.5.5, manually walk the chain.) +3. [ ] **Check `safe_post` error logs** for masked Authorization + headers post-rotation — confirm the attacker no longer holds + a valid token. +4. [ ] **Audit the receiving system's** logs for unexpected actions + triggered by spliced events (Slack channel posts, Teams + cards, Jira tickets). + +### 4.5 GDPR Article 15 (right of access) request + +**Trigger:** Data subject submits an access request via deployer's +DSAR portal or contact form. + +**Severity:** Medium (regulatory deadline 30 days). + +**Runbook:** + +1. [ ] **Verify subject identity** per deployer's DSAR procedure. +2. [ ] **Run** `forgelm reverse-pii --query + --type data/*.jsonl --output-dir ` +3. [ ] **Hash-mask scan** for any corpus that ForgeLM masked + (`forgelm reverse-pii --query --salt-source per_dir + --output-dir `). +4. [ ] **Compose response letter** per DSAR template; cite the run's + `data.access_request_query` audit-event id. +5. [ ] **Retain** the response letter alongside the audit chain. + +### 4.6 GDPR Article 17 (right to erasure) request + +**Trigger:** Subject submits an erasure request. + +**Severity:** Medium (regulatory deadline 30 days; some EU member +states require shorter response). + +**Runbook:** + +1. [ ] **Verify identity** per deployer's DSAR procedure. +2. [ ] **Run** `forgelm purge --row-id --corpus + data/.jsonl --output-dir `. +3. [ ] **Check** for `data.erasure_warning_memorisation` event — + if a model trained on the row has a `final_model/` artefact, + memorisation residual risk applies. +4. [ ] **Communicate the memorisation caveat** to the subject in + the response letter (template: "we deleted the row but a + previously-trained model may have memorised it; we will + retrain or apply additional safeguards as appropriate"). +5. [ ] **For high-stakes deployments** (financial advice, medical + triage): retrain from scratch. +6. [ ] **Retain** the audit-event chain proving completion + warnings + fired. + +## 5. Serious Incident Reporting (EU AI Act) Under Article 73, providers must report serious incidents to market surveillance authorities. A "serious incident" includes: - Death or serious damage to health @@ -69,8 +220,9 @@ Under Article 73, providers must report serious incidents to market surveillance **Report to:** National market surveillance authority of the affected EU member state **Timeline:** Within 15 days of becoming aware -## 5. Review +## 6. Review | Version | Date | Author | Changes | |---------|------|--------|---------| | 1.0 | [DATE] | [AUTHOR] | Initial version | +| 1.1 | 2026-05-05 | Wave 4 / Faz 23 | Added §4 security-incident playbook (audit-chain integrity, credential leak, supply-chain CVE, webhook compromise, GDPR Art. 15/17 DSARs); ISO 27001:2022 + SOC 2 control mapping in header | diff --git a/docs/qms/statement_of_applicability.md b/docs/qms/statement_of_applicability.md new file mode 100644 index 00000000..c26ca8f1 --- /dev/null +++ b/docs/qms/statement_of_applicability.md @@ -0,0 +1,166 @@ +# QMS: Statement of Applicability (SoA) + +> Quality Management System — [YOUR ORGANIZATION] +> ISO 27001:2022 Annex A — required by clause 6.1.3 d) +> Cross-reference: [`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) +> for the full mapping rationale. + +## 1. Purpose + +The Statement of Applicability (SoA) is the deliverable an ISO 27001 +auditor opens FIRST. It states, for every Annex A control, whether +your ISMS treats it as **applicable** or **excluded**, and the +justification. + +This template is pre-populated from ForgeLM's own coverage map (see +the cross-referenced design doc). The deployer adapts each row to +their ISMS context — most rows stay applicable; a row that ForgeLM +covers natively (e.g. A.8.15 Logging) gives the deployer specific +ForgeLM-produced evidence to cite. + +## 2. SoA matrix + +Format: control ID → applicability → justification → ForgeLM +contribution → deployer-side action. + +### 2.1 A.5 Organisational controls (37) + +| Control | Applicable? | Justification / ForgeLM evidence | Implementation status | +|---|---|---|---| +| A.5.1 Policies for information security | YES | EU AI Act Art. 17 mandates QMS; ForgeLM audit-event vocabulary documents what's logged | Adopt org-wide ISMS policy referencing ForgeLM audit log | +| A.5.2 Information security roles and responsibilities | YES | `roles_responsibilities.md` QMS template defines AI Officer / ML Lead / Data Steward / DPO | Adopt role definitions | +| A.5.3 Segregation of duties | YES | `human_approval.required/granted` enforces trainer ≠ approver attribution | Configure CI runner identity ≠ human reviewer identity | +| A.5.4 Management responsibilities | YES | `forgelm doctor`, `compliance_report.json` provide management-review artefacts | Monthly review cadence | +| A.5.5 Contact with authorities | YES | EU AI Act Art. 73 serious-incident reporting | Maintain regulator contact list | +| A.5.6 Contact with special interest groups | YES | ML safety / red-team community | Subscribe to relevant threat intel | +| A.5.7 Threat intelligence | YES | ML supply-chain CVE feeds, model-poisoning advisories | Subscribe to relevant feeds | +| A.5.8 Information security in project management | YES | `risk_assessment` config + F-compliance-110 strict gate; Annex IV §9 metadata | Embed in project sign-off | +| A.5.9 Inventory of information and other associated assets | YES | `data_provenance.json`, `model_integrity.json`, SBOM | Maintain corporate asset register | +| A.5.10 Acceptable use of information | YES | Standard corporate AUP | Adopt | +| A.5.11 Return of assets | YES | Decommissioning of deployed models / training hosts | Adopt | +| A.5.12 Classification of information | YES | `compliance.risk_classification` 5-tier maps to org confidentiality classes | Map ForgeLM tiers to corp classes | +| A.5.13 Labelling of information | YES | `model_card.md`, manifest stamps risk class | Apply org labels on top | +| A.5.14 Information transfer | YES | `safe_post` webhook discipline, no plaintext PII in payload | Sign DTAs with webhook recipients | +| A.5.15 Access control | YES | Operator identity + salted hashing | IdP integration | +| A.5.16 Identity management | YES | `FORGELM_OPERATOR` env contract | Configure CI runner identity | +| A.5.17 Authentication information | YES | `safe_post` masks auth headers; `_mask` hides tokens | Vault-store webhook secrets, HF tokens | +| A.5.18 Access rights | YES | `human_approval` gate | RBAC at IdP | +| A.5.19 Information security in supplier relationships | YES | `_fingerprint_hf_revision`; SBOM | Vendor risk programme | +| A.5.20 Addressing information security within supplier agreements | YES | Standard supplier MSA security clauses | Adopt | +| A.5.21 Managing information security in the ICT supply chain | YES | SBOM (Wave 2 era); `pip-audit` nightly (Wave 4) | CVE monitoring | +| A.5.22 Monitoring, review and change management of supplier services | YES | Vendor annual review | Adopt | +| A.5.23 Information security for use of cloud services | YES | Cloud provider security configuration | Cloud-specific controls | +| A.5.24 Information security incident management planning and preparation | YES | `sop_incident_response.md`; audit chain preserves state | Establish IR team | +| A.5.25 Assessment and decision on information security events | YES | `data.erasure_failed`, `pipeline.failed`, `audit.classifier_load_failed` events with `error_class` + `error_message` | Triage runbook | +| A.5.26 Response to information security incidents | YES | Audit chain HMAC preserves before/after | Document runbook | +| A.5.27 Learning from information security incidents | YES | `pipeline.reverted` events accumulate post-mortem evidence | Weekly post-mortem cadence | +| A.5.28 Collection of evidence | YES | `audit_log.jsonl` forensic-grade; `forgelm verify-audit` validates | Ship to write-once storage | +| A.5.29 Information security during disruption | YES | `auto_revert` baseline-flip; `pipeline.reverted` event | Document base-model retention | +| A.5.30 ICT readiness for business continuity | YES | DR planning | Adopt | +| A.5.31 Identification of legal, statutory, regulatory and contractual requirements | YES | EU AI Act + GDPR mappings; Annex IV bundle | Track rule changes | +| A.5.32 Intellectual property rights | YES | License extraction in SBOM; HF model-card metadata | Per-model license review | +| A.5.33 Protection of records | YES | Append-only + HMAC + manifest; off-site replica | Off-site backup | +| A.5.34 Privacy and protection of PII | YES | `forgelm reverse-pii` Article 15; `forgelm purge` Article 17; `forgelm audit` PII detection + masking | DSAR workflow + DPIA | +| A.5.35 Independent review of information security | YES | Annual external audit | Adopt | +| A.5.36 Compliance with policies, rules and standards for information security | YES | Pydantic validation; CI gates; `forgelm doctor` | Org-wide enforcement | +| A.5.37 Documented operating procedures | YES | QMS templates (5 Wave 0 SOPs + 4 Wave 4 / Faz 23 additions) | Adapt + adopt | + +### 2.2 A.6 People controls (8) + +| Control | Applicable? | Justification | Implementation status | +|---|---|---|---| +| A.6.1 Screening | YES | HR background-check policy | Adopt | +| A.6.2 Terms and conditions of employment | YES | Standard NDA / IP clauses | Adopt | +| A.6.3 Information security awareness, education and training | YES | `audit_event_catalog.md` doubles as training material; `deployer_instructions.md` Article 13 output | Annual EU AI Act + GDPR training | +| A.6.4 Disciplinary process | YES | Operator attribution preserves accountability | Adopt | +| A.6.5 Responsibilities after termination or change of employment | YES | Operator id rotation; old IDs remain in audit history | Revoke departed CI runner; audit prior actions | +| A.6.6 Confidentiality or non-disclosure agreements | YES | Standard NDA | Adopt | +| A.6.7 Remote working | YES | `forgelm doctor --offline`; air-gap pre-cache | VPN policy | +| A.6.8 Information security event reporting | YES | `pipeline.failed`, `data.erasure_failed`, `audit.classifier_load_failed` events; webhook lifecycle | SIEM ingestion + alerting | + +### 2.3 A.7 Physical controls (14) — typically EXCLUDED from a software-toolkit ISMS + +These controls are excluded from ForgeLM's ISMS scope because ForgeLM +is a software toolkit. **The deployer's ISMS scope DOES include them +in their own SoA;** ForgeLM's exclusion here is purely about the +ForgeLM-specific control inventory. + +| Control | Applicable to deployer? | Excluded-from-ForgeLM justification | +|---|---|---| +| A.7.1 Physical perimeters | YES (deployer ISMS) | ForgeLM is software; substrate-side | +| A.7.2 Physical entry | YES | Datacenter-side | +| A.7.3 Securing offices, rooms and facilities | YES | Datacenter-side | +| A.7.4 Physical security monitoring | YES | CCTV — substrate-side | +| A.7.5 Protecting against physical and environmental threats | YES | Substrate-side | +| A.7.6 Working in secure areas | YES | SCIF policies | +| A.7.7 Clear desk and clear screen | YES | Endpoint policy | +| A.7.8 Equipment siting and protection | YES | Hardware placement | +| A.7.9 Security of assets off-premises | YES | MDM | +| A.7.10 Storage media | YES | LUKS / FileVault / BitLocker — substrate-side | +| A.7.11 Supporting utilities | YES | UPS / generator | +| A.7.12 Cabling security | YES | Datacenter cabling | +| A.7.13 Equipment maintenance | YES | Hardware refresh | +| A.7.14 Secure disposal or re-use of equipment | YES | E-waste policy | + +### 2.4 A.8 Technological controls (34) + +| Control | Applicable? | ForgeLM evidence | Deployer-side action | +|---|---|---|---| +| A.8.1 User endpoint devices | YES | `forgelm doctor` — Python / CUDA / GPU / extras / HF auth / disk / `FORGELM_OPERATOR` checks | Endpoint hardening | +| A.8.2 Privileged access rights | YES | Operator attribution; approval gate non-trainer operator | RBAC at IdP | +| A.8.3 Information access restriction | YES | Salted identifier hashing; `forgelm reverse-pii` Article 15 | DSAR workflow | +| A.8.4 Access to source code | YES | `model.trust_remote_code=False` default; `_fingerprint_hf_revision` pins commit SHA | VCS access control | +| A.8.5 Secure authentication | YES | `safe_post` rejects auth headers on non-HTTPS; webhook secret discipline | MFA + token rotation | +| A.8.6 Capacity management | YES | `forgelm doctor` resource report; `resource_usage` manifest block | Quota / autoscaling | +| A.8.7 Protection against malware | YES | Antivirus on training hosts | Adopt | +| A.8.8 Management of technical vulnerabilities | YES | SBOM; `pip-audit` nightly; `bandit` CI; Pydantic config validation | OS patch management | +| A.8.9 Configuration management | YES | YAML validated via Pydantic; `forgelm --dry-run` gate; `compliance.config_hash` in audit | Version-control configs | +| A.8.10 Information deletion | YES | `forgelm purge` Article 17; salted hash audit; `data.erasure_warning_memorisation` flag | DSAR workflow | +| A.8.11 Data masking | YES | `forgelm audit` regex + Presidio ML-NER; `_SECRET_PATTERNS` credentials scan | Masking policy | +| A.8.12 Data leakage prevention | YES | `forgelm reverse-pii` plaintext residual scan; webhook never carries raw rows | Egress DLP | +| A.8.13 Information backup | YES | `audit_log.jsonl` + `.manifest.json` backupable; `forgelm --compliance-export` ZIP | Off-site backup | +| A.8.14 Redundancy of information processing facilities | YES | Multi-AZ infra | Adopt | +| A.8.15 Logging | YES | `audit_log.jsonl` JSON Lines + HMAC + genesis manifest; `forgelm verify-audit` validates | Ship to SIEM | +| A.8.16 Monitoring activities | YES | Webhook lifecycle events; `safety_trend.jsonl` cross-run trend | Alert thresholds | +| A.8.17 Clock synchronisation | YES | All audit entries ISO-8601 UTC | NTP on training hosts | +| A.8.18 Use of privileged utility programs | YES | OS-level privilege management | Adopt | +| A.8.19 Installation of software on operational systems | YES | `forgelm doctor` reports installed packages; `pyproject.toml` pins; `pip install forgelm==X.Y.Z` | Package allowlist | +| A.8.20 Networks security | YES | `safe_post` HTTPS-only / SSRF guard / no-redirect; `model.trust_remote_code=False` | Egress firewall | +| A.8.21 Security of network services | YES | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement | +| A.8.22 Segregation of networks | YES | VPC / subnet design | Adopt | +| A.8.23 Web filtering | YES | Egress proxy | Adopt | +| A.8.24 Use of cryptography | YES | SHA-256 + HMAC-SHA-256 chain; salted SHA-256 identifier hashing; per-output-dir salt + env XOR | KMS for `FORGELM_AUDIT_SECRET` | +| A.8.25 Secure development life cycle | YES | `docs/standards/code-review.md`, `release.md`, CI gates | SDLC framework | +| A.8.26 Application security requirements | YES | F-compliance-110 strict gate; Pydantic validation; ReDoS guard in `_reverse_pii` | App-level threat modelling | +| A.8.27 Secure system architecture and engineering principles | YES | Append-only audit log architecture; HMAC chain; lazy import; SSRF guard | Defence-in-depth | +| A.8.28 Secure coding | YES | `docs/standards/coding.md`; type hints; CommonMark escaping in `_sanitize_md_list` | Custom-extension review | +| A.8.29 Security testing in development and acceptance | YES | `pytest` 1370+ tests; `bandit` static analysis; `forgelm safety-eval` standalone gate | E2E security tests | +| A.8.30 Outsourced development | YES | Third-party-developer security | Adopt | +| A.8.31 Separation of development, test and production environments | YES | `forgelm --dry-run`; staging dir; `evaluation.require_human_approval` | Separate pipelines | +| A.8.32 Change management | YES | `human_approval.required/granted/rejected`; `compliance.config_hash`; staging snapshot | CAB process | +| A.8.33 Test information | YES | `forgelm audit` flags PII / secrets in test sets too | Test-data-handling policy | +| A.8.34 Protection of information systems during audit testing | YES | Read-only audit access | Adopt | + +## 3. Coverage tally + +| Theme | Total | Applicable | Excluded (ForgeLM scope) | FL-supported | FL-helps | +|---|---|---|---|---|---| +| A.5 Organisational | 37 | 37 | 0 | 4 | 24 | +| A.6 People | 8 | 8 | 0 | 0 | 4 | +| A.7 Physical | 14 | 14 (deployer ISMS) | 14 (ForgeLM-specific) | 0 | 0 | +| A.8 Technological | 34 | 34 | 0 | 7 | 22 | +| **Total** | **93** | **93 (deployer ISMS)** | **14 (ForgeLM-specific)** | **11** | **50** | + +## 4. Review + +| Version | Date | Author | Changes | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | Initial version (Wave 4 / Faz 23) — 93 controls scored against ForgeLM v0.5.5 | + +Annual review cadence: + +- Re-confirm applicability (rare changes — most controls stay + applicable). +- Update ForgeLM-evidence column when a new ForgeLM phase ships. +- Update implementation-status column to reflect deployer-side + control posture. diff --git a/docs/reference/iso27001_control_mapping-tr.md b/docs/reference/iso27001_control_mapping-tr.md new file mode 100644 index 00000000..d8960a96 --- /dev/null +++ b/docs/reference/iso27001_control_mapping-tr.md @@ -0,0 +1,140 @@ +# ISO/IEC 27001:2022 Annex A — ForgeLM kontrol haritası + +> ForgeLM özelliklerinin ISO 27001:2022 Annex A kontrollerine nasıl +> eşlendiğini özetleyen referans tablo. Şu belgelere eşlik eder: +> [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) +> ve tasarım dökümanı +> [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md). +> +> **Coverage tier legend:** +> +> - **`FL`** — *ForgeLM-supported*: ForgeLM doğrudan audit kanıt üretir. +> - **`FL-helps`** — *Deployer responsibility, ForgeLM helps*: ForgeLM, +> operatörün diğer kaynaklarla birleştirdiği kısmi kanıt sağlar. +> - **`OOS`** — *Out of scope*: yalnız operatör; ForgeLM hiçbir katkı yapmaz. +> +> **Bu sürüm için kapsama özeti:** FL 11 / FL-helps 50 / OOS 32. + +## A.5 Organisational controls (37) + +| Kontrol | Tier | ForgeLM kanıtı | +|---|---|---| +| A.5.1 Bilgi güvenliği için politikalar | FL-helps | `audit_event_catalog.md` neyin loglandığını dokümante eder | +| A.5.2 Bilgi güvenliği rolleri ve sorumlulukları | FL-helps | `FORGELM_OPERATOR` + `roles_responsibilities.md` | +| A.5.3 Görev ayrılığı | FL-helps | `human_approval.required/granted` trainer ≠ approver'ı zorlar | +| A.5.4 Yönetim sorumlulukları | FL-helps | `forgelm doctor`; `compliance_report.json`; `training_manifest.yaml` | +| A.5.5 Yetkililerle iletişim | OOS | — | +| A.5.6 Özel ilgi gruplarıyla iletişim | OOS | — | +| A.5.7 Tehdit istihbaratı | OOS | — | +| A.5.8 Proje yönetiminde bilgi güvenliği | FL-helps | `risk_assessment` config; F-compliance-110 strict gate; Annex IV §9 | +| A.5.9 Bilgi ve ilişkili varlık envanteri | FL-helps | `data_provenance.json`; `model_integrity.json`; SBOM | +| A.5.10 Bilginin kabul edilebilir kullanımı | OOS | — | +| A.5.11 Varlıkların iadesi | OOS | — | +| A.5.12 Bilgi sınıflandırması | FL-helps | `compliance.risk_classification` 5-tier | +| A.5.13 Bilgi etiketleme | FL-helps | `model_card.md`; manifest'te risk class | +| A.5.14 Bilgi transferi | FL-helps | `safe_post` webhook discipline | +| A.5.15 Erişim kontrolü | FL-helps | Operator identity + salted hashing | +| A.5.16 Kimlik yönetimi | FL-helps | `FORGELM_OPERATOR` env contract | +| A.5.17 Kimlik doğrulama bilgileri | FL-helps | `safe_post` auth header'ları masklar; `_mask` token'ları gizler | +| A.5.18 Erişim hakları | FL-helps | `human_approval` gate | +| A.5.19 Tedarikçi ilişkilerinde bilgi güvenliği | FL-helps | `_fingerprint_hf_revision`; SBOM her bağımlılığı listeler | +| A.5.20 Tedarikçi anlaşmalarında bilgi güvenliği | OOS | — | +| A.5.21 ICT tedarik zincirinde bilgi güvenliği yönetimi | FL-helps | SBOM (CycloneDX 1.5); `pip-audit` nightly | +| A.5.22 Tedarikçi hizmetlerinin izlenmesi, gözden geçirilmesi ve değişim yönetimi | OOS | — | +| A.5.23 Bulut hizmetleri için bilgi güvenliği | OOS | — | +| A.5.24 Bilgi güvenliği olay yönetimi planlaması ve hazırlığı | FL-helps | `sop_incident_response.md`; audit chain durumu korur | +| A.5.25 Bilgi güvenliği olaylarının değerlendirilmesi ve karara bağlanması | FL-helps | `data.erasure_failed`, `pipeline.failed`, `audit.classifier_load_failed` | +| A.5.26 Bilgi güvenliği olaylarına yanıt | FL-helps | Audit chain HMAC öncesi/sonrası durumu korur | +| A.5.27 Bilgi güvenliği olaylarından öğrenme | FL-helps | `pipeline.reverted` olayları post-mortem kanıtı biriktirir | +| A.5.28 Kanıt toplama | FL | `audit_log.jsonl` forensic-grade; `forgelm verify-audit` doğrular | +| A.5.29 Aksaklık sırasında bilgi güvenliği | FL-helps | `auto_revert` baseline-flip; `pipeline.reverted` olayı | +| A.5.30 İş sürekliliği için ICT hazırlığı | OOS | — | +| A.5.31 Yasal, kanuni, düzenleyici ve sözleşmesel gerekliliklerin tanımlanması | FL-helps | EU AI Act + GDPR mappings; Annex IV bundle | +| A.5.32 Fikri mülkiyet hakları | FL-helps | SBOM'da lisans çıkarımı; HF model-card metadata | +| A.5.33 Kayıtların korunması | FL | Append-only + HMAC + manifest sidecar | +| A.5.34 Gizlilik ve PII korunması | FL | `forgelm reverse-pii` Madde 15; `forgelm purge` Madde 17 | +| A.5.35 Bilgi güvenliğinin bağımsız incelenmesi | OOS | — | +| A.5.36 Bilgi güvenliği için politikalar, kurallar ve standartlara uyum | FL-helps | Pydantic config validation; `forgelm doctor`; CI gates | +| A.5.37 Dokümante edilmiş işletim prosedürleri | FL-helps | `docs/qms/` SOPs | + +## A.6 People controls (8) + +| Kontrol | Tier | ForgeLM kanıtı | +|---|---|---| +| A.6.1 Tarama | OOS | — | +| A.6.2 İstihdam şartları ve koşulları | OOS | — | +| A.6.3 Bilgi güvenliği farkındalığı, eğitimi ve öğretimi | FL-helps | `audit_event_catalog.md` eğitim materyali işlevi de görür | +| A.6.4 Disiplin süreci | FL-helps | Operator attribution hesap verebilirliği korur | +| A.6.5 İstihdam değişimi veya sonlandırma sonrası sorumluluklar | FL-helps | Operator id rotasyonu; eski ID'ler audit history'de kalır | +| A.6.6 Gizlilik veya açıklamama anlaşmaları | OOS | — | +| A.6.7 Uzaktan çalışma | FL-helps | `forgelm doctor --offline`; air-gap pre-cache | +| A.6.8 Bilgi güvenliği olay raporlama | FL-helps | `pipeline.failed`, `data.erasure_failed`, `audit.classifier_load_failed`; webhook | + +## A.7 Physical controls (14) + +Tüm A.7 kontrolleri **OOS**'tur (ForgeLM yazılımdır). Operatörün +SoA'sının uçtan uca auditable olması için tamlık adına listelenmiştir. + +| Kontrol | Tier | +|---|---| +| A.7.1 Fiziksel çevreler | OOS | +| A.7.2 Fiziksel giriş | OOS | +| A.7.3 Ofislerin, odaların ve tesislerin güvenliğinin sağlanması | OOS | +| A.7.4 Fiziksel güvenlik izleme | OOS | +| A.7.5 Fiziksel ve çevresel tehditlere karşı koruma | OOS | +| A.7.6 Güvenli alanlarda çalışma | OOS | +| A.7.7 Açık masa ve açık ekran | OOS | +| A.7.8 Ekipman yerleşimi ve korunması | OOS | +| A.7.9 İşletme dışındaki varlıkların güvenliği | OOS | +| A.7.10 Depolama medyası | OOS | +| A.7.11 Destekleyici altyapılar | OOS | +| A.7.12 Kablolama güvenliği | OOS | +| A.7.13 Ekipman bakımı | OOS | +| A.7.14 Ekipmanın güvenli imhası veya yeniden kullanımı | OOS | + +## A.8 Technological controls (34) + +| Kontrol | Tier | ForgeLM kanıtı | +|---|---|---| +| A.8.1 Kullanıcı uç cihazları | FL-helps | `forgelm doctor` env özeti | +| A.8.2 Ayrıcalıklı erişim hakları | FL-helps | Operator attribution; approval-gate operator separation | +| A.8.3 Bilgi erişim kısıtlaması | FL | Salted identifier hashing; `forgelm reverse-pii` Madde 15 | +| A.8.4 Kaynak koduna erişim | FL-helps | `model.trust_remote_code=False` default; `_fingerprint_hf_revision` | +| A.8.5 Güvenli kimlik doğrulama | FL-helps | `safe_post` non-HTTPS'de auth header reddeder | +| A.8.6 Kapasite yönetimi | FL-helps | `forgelm doctor` resource report; `resource_usage` manifest block | +| A.8.7 Kötü amaçlı yazılıma karşı koruma | OOS | — | +| A.8.8 Teknik zafiyetlerin yönetimi | FL-helps | SBOM; `pip-audit` nightly; `bandit` CI | +| A.8.9 Yapılandırma yönetimi | FL | YAML Pydantic ile valide; `forgelm --dry-run`; `compliance.config_hash` | +| A.8.10 Bilgi silme | FL | `forgelm purge` Madde 17; salted-hash audit; `data.erasure_warning_memorisation` | +| A.8.11 Veri maskeleme | FL | `forgelm audit` regex + Presidio ML-NER | +| A.8.12 Veri sızıntısı önleme | FL | `forgelm reverse-pii` plaintext residual scan | +| A.8.13 Bilgi yedekleme | FL-helps | `audit_log.jsonl` + manifest yedeklenebilir | +| A.8.14 Bilgi işlem tesislerinin yedekliliği | OOS | — | +| A.8.15 Loglama | FL | `audit_log.jsonl` JSON Lines + HMAC + genesis manifest | +| A.8.16 İzleme aktiviteleri | FL-helps | Webhook lifecycle olayları; `safety_trend.jsonl` cross-run trend | +| A.8.17 Saat senkronizasyonu | FL-helps | Tüm audit girişleri ISO-8601 UTC | +| A.8.18 Ayrıcalıklı yardımcı program kullanımı | OOS | — | +| A.8.19 Operasyonel sistemlerde yazılım kurulumu | FL-helps | `forgelm doctor` paketler; `pyproject.toml` pin'ler | +| A.8.20 Ağ güvenliği | FL-helps | `safe_post` HTTPS-only / SSRF guard / no-redirect | +| A.8.21 Ağ hizmetlerinin güvenliği | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | +| A.8.22 Ağ ayrımı | OOS | — | +| A.8.23 Web filtreleme | OOS | — | +| A.8.24 Kriptografi kullanımı | FL | SHA-256 + HMAC chain; salted SHA-256 identifier hashing | +| A.8.25 Güvenli geliştirme yaşam döngüsü | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | +| A.8.26 Uygulama güvenliği gereksinimleri | FL-helps | F-compliance-110 strict gate; ReDoS guard | +| A.8.27 Güvenli sistem mimarisi ve mühendislik prensipleri | FL-helps | Append-only audit log mimarisi | +| A.8.28 Güvenli kodlama | FL-helps | `docs/standards/coding.md`; type hints; CommonMark escaping | +| A.8.29 Geliştirme ve kabul aşamasında güvenlik testi | FL-helps | `pytest` 1370+ test; `bandit` static analysis | +| A.8.30 Dış kaynaklı geliştirme | OOS | — | +| A.8.31 Geliştirme, test ve üretim ortamlarının ayrılması | FL-helps | `forgelm --dry-run`; staging dir | +| A.8.32 Değişim yönetimi | FL | `human_approval.*` chain; `compliance.config_hash`; staging snapshot | +| A.8.33 Test bilgisi | FL-helps | `forgelm audit` test setlerinde de PII / secrets'i flag eder | +| A.8.34 Denetim testi sırasında bilgi sistemlerinin korunması | OOS | — | + +## Bkz. + +- [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) — operatör denetim cookbook'u. +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — tam tasarım gerekçesi. +- [`soc2_trust_criteria_mapping-tr.md`](soc2_trust_criteria_mapping-tr.md) — SOC 2 mapping eşlikçisi. +- [`supply_chain_security-tr.md`](supply_chain_security-tr.md) — SBOM + pip-audit + bandit. +- [`audit_event_catalog-tr.md`](audit_event_catalog-tr.md) — audit-event vocabulary. diff --git a/docs/reference/iso27001_control_mapping.md b/docs/reference/iso27001_control_mapping.md new file mode 100644 index 00000000..05fa4af5 --- /dev/null +++ b/docs/reference/iso27001_control_mapping.md @@ -0,0 +1,140 @@ +# ISO/IEC 27001:2022 Annex A — ForgeLM control mapping + +> Reference table summarising how ForgeLM features map to ISO 27001:2022 +> Annex A controls. Companion to +> [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md) +> and the design doc +> [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md). +> +> **Coverage tier legend:** +> +> - **`FL`** — *ForgeLM-supported*: ForgeLM directly produces audit evidence. +> - **`FL-helps`** — *Deployer responsibility, ForgeLM helps*: ForgeLM +> provides partial evidence the deployer combines with other sources. +> - **`OOS`** — *Out of scope*: deployer-only, ForgeLM contributes nothing. +> +> **Coverage tally for this version:** FL 11 / FL-helps 50 / OOS 32. + +## A.5 Organisational controls (37) + +| Control | Tier | ForgeLM evidence | +|---|---|---| +| A.5.1 Policies for information security | FL-helps | `audit_event_catalog.md` documents what's logged | +| A.5.2 Information security roles and responsibilities | FL-helps | `FORGELM_OPERATOR` + `roles_responsibilities.md` | +| A.5.3 Segregation of duties | FL-helps | `human_approval.required/granted` enforces trainer ≠ approver | +| A.5.4 Management responsibilities | FL-helps | `forgelm doctor`; `compliance_report.json`; `training_manifest.yaml` | +| A.5.5 Contact with authorities | OOS | — | +| A.5.6 Contact with special interest groups | OOS | — | +| A.5.7 Threat intelligence | OOS | — | +| A.5.8 Information security in project management | FL-helps | `risk_assessment` config; F-compliance-110 strict gate; Annex IV §9 | +| A.5.9 Inventory of information and other associated assets | FL-helps | `data_provenance.json`; `model_integrity.json`; SBOM | +| A.5.10 Acceptable use of information | OOS | — | +| A.5.11 Return of assets | OOS | — | +| A.5.12 Classification of information | FL-helps | `compliance.risk_classification` 5-tier | +| A.5.13 Labelling of information | FL-helps | `model_card.md`; risk class in manifest | +| A.5.14 Information transfer | FL-helps | `safe_post` webhook discipline | +| A.5.15 Access control | FL-helps | Operator identity + salted hashing | +| A.5.16 Identity management | FL-helps | `FORGELM_OPERATOR` env contract | +| A.5.17 Authentication information | FL-helps | `safe_post` masks auth headers; `_mask` hides tokens | +| A.5.18 Access rights | FL-helps | `human_approval` gate | +| A.5.19 Information security in supplier relationships | FL-helps | `_fingerprint_hf_revision`; SBOM lists every dep | +| A.5.20 Addressing information security within supplier agreements | OOS | — | +| A.5.21 Managing information security in the ICT supply chain | FL-helps | SBOM (CycloneDX 1.5); `pip-audit` nightly | +| A.5.22 Monitoring, review and change management of supplier services | OOS | — | +| A.5.23 Information security for use of cloud services | OOS | — | +| A.5.24 Information security incident management planning and preparation | FL-helps | `sop_incident_response.md`; audit chain preserves state | +| A.5.25 Assessment and decision on information security events | FL-helps | `data.erasure_failed`, `pipeline.failed`, `audit.classifier_load_failed` | +| A.5.26 Response to information security incidents | FL-helps | Audit chain HMAC preserves before/after | +| A.5.27 Learning from information security incidents | FL-helps | `pipeline.reverted` events accumulate post-mortem evidence | +| A.5.28 Collection of evidence | FL | `audit_log.jsonl` forensic-grade; `forgelm verify-audit` validates | +| A.5.29 Information security during disruption | FL-helps | `auto_revert` baseline-flip; `pipeline.reverted` event | +| A.5.30 ICT readiness for business continuity | OOS | — | +| A.5.31 Identification of legal, statutory, regulatory and contractual requirements | FL-helps | EU AI Act + GDPR mappings; Annex IV bundle | +| A.5.32 Intellectual property rights | FL-helps | License extraction in SBOM; HF model-card metadata | +| A.5.33 Protection of records | FL | Append-only + HMAC + manifest sidecar | +| A.5.34 Privacy and protection of PII | FL | `forgelm reverse-pii` Article 15; `forgelm purge` Article 17 | +| A.5.35 Independent review of information security | OOS | — | +| A.5.36 Compliance with policies, rules and standards for information security | FL-helps | Pydantic config validation; `forgelm doctor`; CI gates | +| A.5.37 Documented operating procedures | FL-helps | `docs/qms/` SOPs | + +## A.6 People controls (8) + +| Control | Tier | ForgeLM evidence | +|---|---|---| +| A.6.1 Screening | OOS | — | +| A.6.2 Terms and conditions of employment | OOS | — | +| A.6.3 Information security awareness, education and training | FL-helps | `audit_event_catalog.md` doubles as training material | +| A.6.4 Disciplinary process | FL-helps | Operator attribution preserves accountability | +| A.6.5 Responsibilities after termination or change of employment | FL-helps | Operator id rotation; old IDs remain in audit history | +| A.6.6 Confidentiality or non-disclosure agreements | OOS | — | +| A.6.7 Remote working | FL-helps | `forgelm doctor --offline`; air-gap pre-cache | +| A.6.8 Information security event reporting | FL-helps | `pipeline.failed`, `data.erasure_failed`, `audit.classifier_load_failed`; webhook | + +## A.7 Physical controls (14) + +All A.7 controls are **OOS** (ForgeLM is software). Listed for +completeness so the deployer's SoA is auditable end-to-end. + +| Control | Tier | +|---|---| +| A.7.1 Physical perimeters | OOS | +| A.7.2 Physical entry | OOS | +| A.7.3 Securing offices, rooms and facilities | OOS | +| A.7.4 Physical security monitoring | OOS | +| A.7.5 Protecting against physical and environmental threats | OOS | +| A.7.6 Working in secure areas | OOS | +| A.7.7 Clear desk and clear screen | OOS | +| A.7.8 Equipment siting and protection | OOS | +| A.7.9 Security of assets off-premises | OOS | +| A.7.10 Storage media | OOS | +| A.7.11 Supporting utilities | OOS | +| A.7.12 Cabling security | OOS | +| A.7.13 Equipment maintenance | OOS | +| A.7.14 Secure disposal or re-use of equipment | OOS | + +## A.8 Technological controls (34) + +| Control | Tier | ForgeLM evidence | +|---|---|---| +| A.8.1 User endpoint devices | FL-helps | `forgelm doctor` env summary | +| A.8.2 Privileged access rights | FL-helps | Operator attribution; approval-gate operator separation | +| A.8.3 Information access restriction | FL | Salted identifier hashing; `forgelm reverse-pii` Article 15 | +| A.8.4 Access to source code | FL-helps | `model.trust_remote_code=False` default; `_fingerprint_hf_revision` | +| A.8.5 Secure authentication | FL-helps | `safe_post` rejects auth headers on non-HTTPS | +| A.8.6 Capacity management | FL-helps | `forgelm doctor` resource report; `resource_usage` manifest block | +| A.8.7 Protection against malware | OOS | — | +| A.8.8 Management of technical vulnerabilities | FL-helps | SBOM; `pip-audit` nightly; `bandit` CI | +| A.8.9 Configuration management | FL | YAML validated via Pydantic; `forgelm --dry-run`; `compliance.config_hash` | +| A.8.10 Information deletion | FL | `forgelm purge` Article 17; salted-hash audit; `data.erasure_warning_memorisation` | +| A.8.11 Data masking | FL | `forgelm audit` regex + Presidio ML-NER | +| A.8.12 Data leakage prevention | FL | `forgelm reverse-pii` plaintext residual scan | +| A.8.13 Information backup | FL-helps | `audit_log.jsonl` + manifest backupable | +| A.8.14 Redundancy of information processing facilities | OOS | — | +| A.8.15 Logging | FL | `audit_log.jsonl` JSON Lines + HMAC + genesis manifest | +| A.8.16 Monitoring activities | FL-helps | Webhook lifecycle events; `safety_trend.jsonl` cross-run trend | +| A.8.17 Clock synchronisation | FL-helps | All audit entries ISO-8601 UTC | +| A.8.18 Use of privileged utility programs | OOS | — | +| A.8.19 Installation of software on operational systems | FL-helps | `forgelm doctor` packages; `pyproject.toml` pins | +| A.8.20 Networks security | FL-helps | `safe_post` HTTPS-only / SSRF guard / no-redirect | +| A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | +| A.8.22 Segregation of networks | OOS | — | +| A.8.23 Web filtering | OOS | — | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC chain; salted SHA-256 identifier hashing | +| A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | +| A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; ReDoS guard | +| A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture | +| A.8.28 Secure coding | FL-helps | `docs/standards/coding.md`; type hints; CommonMark escaping | +| A.8.29 Security testing in development and acceptance | FL-helps | `pytest` 1370+ tests; `bandit` static analysis | +| A.8.30 Outsourced development | OOS | — | +| A.8.31 Separation of development, test and production environments | FL-helps | `forgelm --dry-run`; staging dir | +| A.8.32 Change management | FL | `human_approval.*` chain; `compliance.config_hash`; staging snapshot | +| A.8.33 Test information | FL-helps | `forgelm audit` flags PII / secrets in test sets too | +| A.8.34 Protection of information systems during audit testing | OOS | — | + +## See also + +- [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md) — deployer audit cookbook. +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — full design rationale. +- [`soc2_trust_criteria_mapping.md`](soc2_trust_criteria_mapping.md) — SOC 2 mapping companion. +- [`supply_chain_security.md`](supply_chain_security.md) — SBOM + pip-audit + bandit. +- [`audit_event_catalog.md`](audit_event_catalog.md) — audit-event vocabulary. diff --git a/docs/reference/soc2_trust_criteria_mapping-tr.md b/docs/reference/soc2_trust_criteria_mapping-tr.md new file mode 100644 index 00000000..6bef1e86 --- /dev/null +++ b/docs/reference/soc2_trust_criteria_mapping-tr.md @@ -0,0 +1,129 @@ +# AICPA SOC 2 Trust Services Criteria — ForgeLM haritası + +> ForgeLM özelliklerinin AICPA SOC 2 Trust Services Criteria (2017 +> framework, 2022 revize) ile nasıl eşlendiğini özetleyen referans +> tablo. Şu belgelere eşlik eder: +> [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) +> ve tasarım dökümanı +> [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md). + +## Kategoriler + +2017 SOC 2 framework'ü 5 kategori tanımlar: + +1. **Security** (Common Criteria CC1.x–CC9.x) — zorunlu baseline. +2. **Availability** (A1.x) — opsiyonel. +3. **Processing Integrity** (PI1.x) — opsiyonel. +4. **Confidentiality** (C1.x) — opsiyonel. +5. **Privacy** (P1.x–P8.x) — opsiyonel. + +SOC 2 *Type II* engagement'i 6–12 ay'lık bir pencerede operating +effectiveness'i gözlemler. Common Criteria zorunludur; opsiyonel +kategoriler engagement-bazında scoplanır. + +## Security — Common Criteria + +| CC | Başlık | ForgeLM kanıtı | +|---|---|---| +| CC1.1 | Bütünlüğe bağlılık göstergesi | `roles_responsibilities.md` QMS template; AI Officer rolü | +| CC1.2 | Yönetişim bağımsızlığı | Approval gate operator ≠ trainer | +| CC1.3 | Yapı, raporlama hatları kurma | `roles_responsibilities.md` AI Officer / ML Lead / DPO tanımlar | +| CC1.4 | Yetkinliğe bağlılık göstergesi | Yıllık EU AI Act / GDPR eğitimi (operatör politikası) | +| CC1.5 | Hesap verebilirliği uygular | `FORGELM_OPERATOR` attribution + audit chain | +| CC2.1 | İç kontrolleri içe iletişim | `audit_event_catalog.md` + `deployer_instructions.md` | +| CC2.2 | Dışa iletişim | Madde 13 `deployer_instructions.md`; Annex IV public summary | +| CC2.3 | Düzenleyicilerle iletişim | Annex IV bundle gönderime hazır | +| CC3.1 | Uygun hedefler belirler | `compliance.intended_purpose`; risk classification | +| CC3.2 | Riskleri tanımlar ve analiz eder | `risk_assessment` Pydantic block; safety eval; `risk_treatment_plan.md` | +| CC3.3 | Sahtekarlık risklerini değerlendirir | Audit log tamper-evidence; HMAC chain; manifest sidecar | +| CC3.4 | Değişimleri tanımlar ve değerlendirir | `human_approval.required` gate; `compliance.config_hash` | +| CC4.1 | Değerlendirmeleri seçer, geliştirir, gerçekleştirir | `forgelm verify-audit`; `forgelm safety-eval` | +| CC4.2 | İç kontrol eksikliklerini iletir | `pipeline.failed`/`reverted`/`erasure_failed` olayları | +| CC5.1 | Kontrol aktivitelerini seçer, geliştirir | F-compliance-110 strict gate; auto-revert; staging | +| CC5.2 | Genel BT kontrollerini seçer | `safe_post` HTTP discipline; Pydantic config validation | +| CC5.3 | Politikalar ve prosedürler dağıtır | `docs/qms/` 5 SOPs (Wave 0); 4 yeni Wave 4 / Faz 23'te | +| CC6.1 | Mantıksal-erişim güvenlik yazılımı | Operator-id attribution; HMAC chain | +| CC6.2 | Yeni iç kullanıcıları yetkilendirir | `human_approval.required`/`granted` chain | +| CC6.3 | Sonlandırılan kullanıcıların erişimini kaldırır | Operatör CI runner identity'sini iptal eder | +| CC6.4 | Fiziksel erişimi kısıtlar | OOS — datacenter güvenliği | +| CC6.5 | Yetkisiz imhaya karşı korur | `forgelm purge` Madde 17; salted hashing | +| CC6.6 | Mantıksal-erişim kontrolleri uygular | Salted hashing audit olaylarında; `forgelm reverse-pii` | +| CC6.7 | Bilgi hareketini kısıtlar | `safe_post` egress discipline; webhook payload curation | +| CC6.8 | Yetkisiz yazılımı tespit eder/önler | SBOM; `pip-audit` nightly; `bandit` CI | +| CC7.1 | Zafiyetleri tespit eder | `pip-audit` nightly; CVE feed | +| CC7.2 | Sistem bileşenlerini izler | `forgelm verify-audit`; `forgelm verify-gguf`; `safety_trend.jsonl` | +| CC7.3 | Güvenlik olaylarını değerlendirir | `data.erasure_failed`, `pipeline.failed` olayları `error_class` + `error_message` ile | +| CC7.4 | Güvenlik olaylarına yanıt verir | `auto_revert`; `pipeline.reverted` olayı | +| CC7.5 | Düzeltici eylemleri tanımlar, geliştirir | `human_approval.rejected`; `sop_change_management.md` | +| CC8.1 | Değişimleri yetkilendirir | `forgelm approve` Madde 14 gate; staging dir | +| CC9.1 | Riskleri tanımlar, yönetir | `risk_assessment` config + safety eval; `risk_treatment_plan.md` | +| CC9.2 | Tedarikçi + iş ortağı riskini yönetir | SBOM; HF Hub revision pin; lisans çıkarımı | + +## Availability (A1.x) + +ForgeLM tek-node CLI'dır; availability ağırlıklı olarak operatör-tarafıdır. + +| Kontrol | ForgeLM katkısı | +|---|---| +| A1.1 Kapasite planlama | `forgelm doctor` resource report + `resource_usage` manifest | +| A1.2 Olaylardan kurtarma | `auto_revert` swap-back; resume'da audit chain continuity | +| A1.3 Çevresel korumalar | OOS — substrate-side | + +## Processing Integrity (PI1.x) + +Güçlü ForgeLM katkısı. + +| Kontrol | ForgeLM katkısı | +|---|---| +| PI1.1 Girdi kalitesi | `compute_dataset_fingerprint`; `data_governance_report` | +| PI1.2 Sistem işleme | `forgelm verify-audit`; `data_audit_report.json` | +| PI1.3 Çıktıların doğruluğu | `model_integrity.json` SHA-256 checksums; `model_card.md` | +| PI1.4 Girdilerin izlenebilirliği | `_describe_adapter_method`; `pipeline.config_hash`; HF-revision pin | +| PI1.5 Çıktıların izlenebilirliği | Annex IV bundle manifest + report + audit + integrity'i co-locate eder | + +## Confidentiality (C1.x) + +| Kontrol | ForgeLM katkısı | +|---|---| +| C1.1 Gizli bilgilerin korunması | `forgelm audit` regex + Presidio ML-NER PII tespiti; `_SECRET_PATTERNS` credentials scan | +| C1.2 Gizli bilgilerin imhası | `forgelm purge` Madde 17; salted-hash audit | + +## Privacy (P1.x – P8.x) + +| Kontrol | ForgeLM katkısı | +|---|---| +| P1.1 Gizlilik bildirimi | Madde 13 `deployer_instructions.md` | +| P2.1 Seçim ve onay | `evaluation.require_human_approval` Madde 14 gate | +| P3.1 Toplama | `data.governance.personal_data_included`; `dpia_completed` | +| P3.2 Kişisel veri kalitesi | `data_audit_report.json` quality stats | +| P4.1 Kullanım, saklama ve imha | `evaluation.approval_retention_days`; `forgelm purge --check-policy` | +| P5.1 Erişim | `forgelm reverse-pii` Madde 15 scan; salted query-hash | +| P5.2 Sorular ve şikayetler | (Operatör-tarafı workflow) | +| P6.1 Üçüncü taraflara açıklama | `safe_post` webhook discipline; HMAC payload signing | +| P6.2 Üçüncü taraf anlaşmaları | (Operatör DPA'ları) | +| P7.1 İhlal bildirimi | `data.erasure_failed`, `audit.classifier_load_failed` olayları | +| P7.2 İhlal açıklaması | (Operatör regulator-contact playbook) | +| P8.1 Sorular, şikayetler ve anlaşmazlıklar | `forgelm reverse-pii` + `forgelm purge` chain | + +## Operatör eklentileri (ForgeLM'de değil) + +- **Credential management:** `FORGELM_AUDIT_SECRET`, webhook URL, + API anahtarları için Vault. +- **Evidence archive:** Write-once depolama (S3 Object Lock, Azure + Immutable Blob) + uzun vadeli retention politikası. +- **Access control:** Operator attribution için IdP entegrasyonu; + approval kararlarında MFA. +- **Incident response:** `data.erasure_failed`, safety classifier + crash'leri, audit-chain kırıklıkları için playbook. +- **Monitoring:** Audit log'ların SIEM ingestion'ı; high/unacceptable- + risk gate'lerde alarm; safety_eval için threshold tuning. +- **Documentation:** Risk Management dosyası, gizlilik bildirimi, + deployer instructions dağıtımı, Annex IV posting. + +## Bkz. + +- [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) — operatör denetim cookbook'u. +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — tam tasarım gerekçesi. +- [`iso27001_control_mapping-tr.md`](iso27001_control_mapping-tr.md) — ISO 27001 mapping eşlikçisi. +- [`supply_chain_security-tr.md`](supply_chain_security-tr.md) — SBOM + pip-audit + bandit. +- [`audit_event_catalog-tr.md`](audit_event_catalog-tr.md) — audit-event vocabulary. diff --git a/docs/reference/soc2_trust_criteria_mapping.md b/docs/reference/soc2_trust_criteria_mapping.md new file mode 100644 index 00000000..f0158cb1 --- /dev/null +++ b/docs/reference/soc2_trust_criteria_mapping.md @@ -0,0 +1,130 @@ +# AICPA SOC 2 Trust Services Criteria — ForgeLM mapping + +> Reference table summarising how ForgeLM features map to the AICPA +> SOC 2 Trust Services Criteria (2017 framework, revised 2022). +> Companion to +> [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md) +> and the design doc +> [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md). + +## Categories + +The 2017 SOC 2 framework defines 5 categories: + +1. **Security** (Common Criteria CC1.x–CC9.x) — mandatory baseline. +2. **Availability** (A1.x) — optional. +3. **Processing Integrity** (PI1.x) — optional. +4. **Confidentiality** (C1.x) — optional. +5. **Privacy** (P1.x–P8.x) — optional. + +A SOC 2 *Type II* engagement observes operating effectiveness over a +6–12 month window. Common Criteria are mandatory; the optional +categories are scoped per-engagement. + +## Security — Common Criteria + +| CC | Title | ForgeLM evidence | +|---|---|---| +| CC1.1 | Demonstrates commitment to integrity | `roles_responsibilities.md` QMS template; AI Officer role | +| CC1.2 | Independence of governance | Approval gate operator ≠ trainer | +| CC1.3 | Establishes structures, reporting lines | `roles_responsibilities.md` defines AI Officer / ML Lead / DPO | +| CC1.4 | Demonstrates commitment to competence | Annual EU AI Act / GDPR training (deployer policy) | +| CC1.5 | Enforces accountability | `FORGELM_OPERATOR` attribution + audit chain | +| CC2.1 | Communicates internally about controls | `audit_event_catalog.md` + `deployer_instructions.md` | +| CC2.2 | Communicates externally | Article 13 `deployer_instructions.md`; Annex IV public summary | +| CC2.3 | Communicates with regulators | Annex IV bundle ready to ship | +| CC3.1 | Specifies suitable objectives | `compliance.intended_purpose`; risk classification | +| CC3.2 | Identifies and analyses risks | `risk_assessment` Pydantic block; safety eval; `risk_treatment_plan.md` | +| CC3.3 | Considers fraud risks | Audit log tamper-evidence; HMAC chain; manifest sidecar | +| CC3.4 | Identifies and assesses changes | `human_approval.required` gate; `compliance.config_hash` | +| CC4.1 | Selects, develops, performs evaluations | `forgelm verify-audit`; `forgelm safety-eval` | +| CC4.2 | Communicates internal-control deficiencies | `pipeline.failed`/`reverted`/`erasure_failed` events | +| CC5.1 | Selects, develops control activities | F-compliance-110 strict gate; auto-revert; staging | +| CC5.2 | Selects general IT controls | `safe_post` HTTP discipline; Pydantic config validation | +| CC5.3 | Deploys policies and procedures | `docs/qms/` 5 SOPs (Wave 0); 4 new in Wave 4 / Faz 23 | +| CC6.1 | Logical-access security software | Operator-id attribution; HMAC chain | +| CC6.2 | Authorises new internal users | `human_approval.required`/`granted` chain | +| CC6.3 | Removes access for terminated users | Deployer revokes CI runner identity | +| CC6.4 | Restricts physical access | OOS — datacenter security | +| CC6.5 | Protects against unauthorised disposal | `forgelm purge` Article 17; salted hashing | +| CC6.6 | Implements logical-access controls | Salted hashing in audit events; `forgelm reverse-pii` | +| CC6.7 | Restricts movement of information | `safe_post` egress discipline; webhook payload curation | +| CC6.8 | Detects/prevents unauthorised software | SBOM; `pip-audit` nightly; `bandit` CI | +| CC7.1 | Detects vulnerabilities | `pip-audit` nightly; CVE feed | +| CC7.2 | Monitors system components | `forgelm verify-audit`; `forgelm verify-gguf`; `safety_trend.jsonl` | +| CC7.3 | Evaluates security events | `data.erasure_failed`, `pipeline.failed` events with `error_class` + `error_message` | +| CC7.4 | Responds to security events | `auto_revert`; `pipeline.reverted` event | +| CC7.5 | Identifies, develops corrective actions | `human_approval.rejected`; `sop_change_management.md` | +| CC8.1 | Authorises changes | `forgelm approve` Article 14 gate; staging dir | +| CC9.1 | Identifies, manages risks | `risk_assessment` config + safety eval; `risk_treatment_plan.md` | +| CC9.2 | Manages vendor + business-partner risk | SBOM; HF Hub revision pin; license extraction | + +## Availability (A1.x) + +ForgeLM is a single-node CLI; availability is dominantly +deployer-side. + +| Control | ForgeLM contribution | +|---|---| +| A1.1 Capacity planning | `forgelm doctor` resource report + `resource_usage` manifest | +| A1.2 Recovery from incidents | `auto_revert` swap-back; audit chain continuity on resume | +| A1.3 Environmental protections | OOS — substrate-side | + +## Processing Integrity (PI1.x) + +Strong ForgeLM contribution. + +| Control | ForgeLM contribution | +|---|---| +| PI1.1 Quality of inputs | `compute_dataset_fingerprint`; `data_governance_report` | +| PI1.2 System processing | `forgelm verify-audit`; `data_audit_report.json` | +| PI1.3 Outputs are accurate | `model_integrity.json` SHA-256 checksums; `model_card.md` | +| PI1.4 Inputs traceable | `_describe_adapter_method`; `pipeline.config_hash`; HF-revision pin | +| PI1.5 Outputs traceable | Annex IV bundle co-locates manifest + report + audit + integrity | + +## Confidentiality (C1.x) + +| Control | ForgeLM contribution | +|---|---| +| C1.1 Protection of confidential information | `forgelm audit` regex + Presidio ML-NER PII detection; `_SECRET_PATTERNS` credentials scan | +| C1.2 Disposal of confidential information | `forgelm purge` Article 17; salted-hash audit | + +## Privacy (P1.x – P8.x) + +| Control | ForgeLM contribution | +|---|---| +| P1.1 Privacy notice | Article 13 `deployer_instructions.md` | +| P2.1 Choice and consent | `evaluation.require_human_approval` Article 14 gate | +| P3.1 Collection | `data.governance.personal_data_included`; `dpia_completed` | +| P3.2 Quality of personal data | `data_audit_report.json` quality stats | +| P4.1 Use, retention, and disposal | `evaluation.approval_retention_days`; `forgelm purge --check-policy` | +| P5.1 Access | `forgelm reverse-pii` Article 15 scan; salted query-hash | +| P5.2 Inquiries and complaints | (Deployer-side workflow) | +| P6.1 Disclosure to third parties | `safe_post` webhook discipline; HMAC payload signing | +| P6.2 Third-party agreements | (Deployer DPAs) | +| P7.1 Breach notification | `data.erasure_failed`, `audit.classifier_load_failed` events | +| P7.2 Breach disclosure | (Deployer regulator-contact playbook) | +| P8.1 Inquiries, complaints, and disputes | `forgelm reverse-pii` + `forgelm purge` chain | + +## Deployer add-ons (not in ForgeLM) + +- **Credential management:** Vault for `FORGELM_AUDIT_SECRET`, + webhook URL, API keys. +- **Evidence archive:** Write-once storage (S3 Object Lock, Azure + Immutable Blob) + long-term retention policy. +- **Access control:** IdP integration for operator attribution; MFA + on approval decisions. +- **Incident response:** Playbook for `data.erasure_failed`, safety + classifier crashes, audit-chain breaks. +- **Monitoring:** SIEM ingestion of audit logs; alerting on + high/unacceptable-risk gates; threshold tuning for safety_eval. +- **Documentation:** Risk Management file, privacy notice, deployer + instructions distribution, Annex IV posting. + +## See also + +- [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md) — deployer audit cookbook. +- [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — full design rationale. +- [`iso27001_control_mapping.md`](iso27001_control_mapping.md) — ISO 27001 mapping companion. +- [`supply_chain_security.md`](supply_chain_security.md) — SBOM + pip-audit + bandit. +- [`audit_event_catalog.md`](audit_event_catalog.md) — audit-event vocabulary. diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md new file mode 100644 index 00000000..4b3098f0 --- /dev/null +++ b/docs/reference/supply_chain_security-tr.md @@ -0,0 +1,196 @@ +# Supply-chain güvenliği — SBOM + zafiyet taraması + +> ForgeLM'in supply-chain güvenlik tooling'i için referans dokuman: +> SBOM üretimi (Wave 2 dönemi), pip-audit nightly (Wave 4 / Faz 23), +> bandit static analysis (Wave 4 / Faz 23). Hedef kitle: operatör +> compliance ekibi + denetçi. + +## Software Bill of Materials (SBOM) + +### Format + +ForgeLM **CycloneDX 1.5 JSON** emit eder. Sebepler: + +- ISO 27001 / SOC 2 denetçileri CycloneDX veya SPDX'i kabul eder; + open-source dependency-scanning ekosistemi (Dependency-Track ≥ + 4.10, OWASP Dep-Check, Snyk Open Source) CycloneDX 1.5'i doğal + olarak içe alır. +- ForgeLM emitter'ı (`tools/generate_sbom.py`) saf-stdlib + sıfır + external bağımlılık — SBOM adımı aksi takdirde yeşil bir release + matrisini sessizce bozamaz. +- SPDX'e dönüşüm, belirli bir operatörün SPDX gerektirmesi durumunda + tek satırlık bir `cyclonedx-py` çağrısıdır. + +### Üretim + +Emitter her release tag'inde (`v*` pattern) publish workflow'u +içinde çalışır: + +```yaml +# .github/workflows/publish.yml (excerpt) +- name: Generate SBOM (CycloneDX 1.5) + shell: bash + run: python tools/generate_sbom.py > sbom-${{ matrix.os }}-py${{ matrix.python }}.json + +- uses: actions/upload-artifact@v5 + with: + name: sbom-${{ matrix.os }}-py${{ matrix.python }} + path: sbom-${{ matrix.os }}-py${{ matrix.python }}.json +``` + +Cross-OS test matrisinin her (OS × Python-version) hücresi için bir +SBOM. Artefaktlar GitHub release sayfasından indirilebilir. + +### Determinism contract + +Wave 4 / Faz 23 bir determinism testi ekler +(`tests/test_supply_chain_security.py`): + +- Aynı Python ortamında iki ardışık çağrı içerik-eşit SBOM'lar + üretmek zorundadır (CycloneDX semantiği gereği kasıtlı olarak + varyant olan `serialNumber` ve `metadata.timestamp` modulo). +- `tests/test_supply_chain_security.py::TestGenerateSbomDeterministic::test_two_invocations_produce_same_content` + CI'da bunu pin'ler. + +Bir denetçi karşılık gelen `git tag`'den SBOM'u yeniden emit edip +GitHub release'e ekli artefaktla diff alabilir; trivial olmayan +herhangi bir fark build ortamında dependency-resolution drift'ine +işaret eder. + +### SBOM'u tüketme + +```bash +# Bir release için tüm SBOM'ları indir. +gh release download v0.5.5 --pattern 'sbom-*' + +# İnsan inceleme için pretty-print. +jq . sbom-ubuntu-latest-py3.11.json | less + +# Her bağımlılık adı + sürümünü listele. +jq -r '.components[] | "\(.name) \(.version)"' sbom-ubuntu-latest-py3.11.json + +# İki release arasında diff. +diff <(jq -S '.components | sort_by(.purl)' sbom-prev.json) \ + <(jq -S '.components | sort_by(.purl)' sbom.json) + +# Dependency-Track'e ingest. +curl -X POST -H "X-Api-Key: $DT_KEY" -H "Content-Type: application/octet-stream" \ + --data-binary @sbom-ubuntu-latest-py3.11.json \ + https://depTrack.example.com/api/v1/bom +``` + +## pip-audit (transitive CVE taraması) + +Wave 4 / Faz 23 nightly workflow'a `pip-audit` ekler. Davranış: + +- `closure/wave3-integration`-türetilmiş development branch + main'de + her gün 03:00 UTC'de çalışır. +- `tools/check_pip_audit.py` üzerinden severity politikası: + - **HIGH / CRITICAL** → exit 1 (nightly fail eder; mevcut + notify-failure job'u üzerinden GitHub issue açar). + - **MEDIUM / MODERATE** → `::warning::` annotation; nightly yeşil + kalır. + - **LOW / UNKNOWN** → sessiz. +- OSV / GHSA veritabanlarını kullanır (pip-audit varsayılanı). + +Operatörler aynı tooling'i lokalde kurar: + +```bash +pip install forgelm[security] +pip-audit --strict --format json --output /tmp/pip-audit.json +python3 tools/check_pip_audit.py /tmp/pip-audit.json +``` + +### Suppression (kasıtlı CVE kabulü) + +Bir CVE kabul edildiyse ama henüz düzeltilemiyorsa (upstream release +beklemede, vb.), operatörün risk acceptance log'unda dokümante edip +nightly run için bastırmak amacıyla `pip-audit --ignore-vuln ` +kullanın. ForgeLM proje-seviyesinde bir ignore listesi göndermez — +her suppression operatör-tarafında ve quarterly-reviewed olmalıdır. + +## bandit (static security analysis) + +Wave 4 / Faz 23 `bandit`'i şuralara ekler: + +- `.github/workflows/ci.yml` — main'e her PR + push. +- `.github/workflows/nightly.yml` — günlük 03:00 UTC. + +Scope: `forgelm/` (yalnız üretim kodu). `tests/` hariç tutulur çünkü +test fixture'ları meşru olarak güvensiz pattern kullanır (`assert`, +dummy secrets, test girdilerinde `pickle.loads`). + +`tools/check_bandit.py` üzerinden severity politikası: + +- **HIGH** → exit 1 (CI / nightly fail eder). +- **MEDIUM** → `::warning::` annotation; CI / nightly yeşil kalır. +- **LOW / UNDEFINED** → sessiz. + +`pyproject.toml`'daki yapılandırma: + +```toml +[tool.bandit] +exclude_dirs = ["tests", ".venv", "build", "dist"] +skips = ["B101"] # assert_used — tests/ hariç tutulduğunda anlamsız +``` + +### Suppression syntax + +Üretim kodunda dokümante edilmiş bir gerekçe ile meşru olarak yangın +eden belirli bir bandit kuralı için: + +```python +import subprocess +# nosec B603 B607 — args literal, operatör-kontrollü değil +result = subprocess.run([sys.executable, "-m", "pip", "list"], ...) +``` + +Format: `# nosec [ ]` ardından açıklama. Bandit satırı +yok sayar; `tools/check_bandit.py` suppression'ın trailing prose ile +gerekçelendirildiğini teyit eder. + +## Opsiyonel `[security]` extra + +`pyproject.toml` security tooling'i opt-in extra olarak sunar: + +```toml +[project.optional-dependencies] +security = [ + "pip-audit>=2.7.0,<3.0.0", + "bandit[toml]>=1.7.0,<2.0.0", +] +``` + +Operatörler şöyle kurar: + +```bash +pip install forgelm[security] +``` + +Bu, ForgeLM'in nightly + CI workflow'larının uyguladığı aynı +kontrolleri PR push'undan önce lokalde çalıştırılabilir hale +getirmek için `pip-audit` ve `bandit`'i çeker. + +## İlgili kontroller + +| Standart | Kontrol | Eşleşme | +|---|---|---| +| ISO 27001:2022 | A.5.21 | ICT tedarik zincirinde bilgi güvenliği yönetimi | +| ISO 27001:2022 | A.8.8 | Teknik zafiyetlerin yönetimi | +| ISO 27001:2022 | A.8.30 | Dış kaynaklı geliştirme (transitive dep'ler outsourced kod olarak) | +| SOC 2 | CC7.1 | Zafiyetleri tespit eder | +| SOC 2 | CC8.1 | Değişimleri yetkilendirir (son release ile SBOM diff) | +| SOC 2 | CC9.2 | Tedarikçi + iş ortağı riskini yönetir | +| EU AI Act | Md. 9 risk yönetimi | Supply-chain riski genel risk register'ın parçası | + +## Bkz. + +- [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) — operatör denetim cookbook'u. +- [`iso27001_control_mapping-tr.md`](iso27001_control_mapping-tr.md) — ISO 27001 Annex A kontrolleri × ForgeLM kanıtı. +- [`soc2_trust_criteria_mapping-tr.md`](soc2_trust_criteria_mapping-tr.md) — SOC 2 Trust Services Criteria × ForgeLM kanıtı. +- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. +- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook. +- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook. +- `tools/generate_sbom.py` — CycloneDX 1.5 emitter. +- `tools/check_pip_audit.py` — pip-audit severity gate. +- `tools/check_bandit.py` — bandit severity gate. diff --git a/docs/reference/supply_chain_security.md b/docs/reference/supply_chain_security.md new file mode 100644 index 00000000..b71b165a --- /dev/null +++ b/docs/reference/supply_chain_security.md @@ -0,0 +1,197 @@ +# Supply-chain security — SBOM + vulnerability scanning + +> Reference doc for ForgeLM's supply-chain security tooling: SBOM +> generation (Wave 2 era), pip-audit nightly (Wave 4 / Faz 23), +> bandit static analysis (Wave 4 / Faz 23). Audience: deployer +> compliance team + auditor. + +## Software Bill of Materials (SBOM) + +### Format + +ForgeLM emits **CycloneDX 1.5 JSON**. Reasons: + +- ISO 27001 / SOC 2 auditors accept either CycloneDX or SPDX; the + open-source dependency-scanning ecosystem (Dependency-Track ≥ 4.10, + OWASP Dep-Check, Snyk Open Source) consumes CycloneDX 1.5 + natively. +- The ForgeLM emitter (`tools/generate_sbom.py`) is pure-stdlib + zero + external dependencies — the SBOM step cannot silently degrade an + otherwise-green release matrix. +- SPDX conversion is a one-line `cyclonedx-py` invocation if a + specific deployer requires SPDX. + +### Generation + +The emitter runs on every release tag (`v*` pattern) inside the +publish workflow: + +```yaml +# .github/workflows/publish.yml (excerpt) +- name: Generate SBOM (CycloneDX 1.5) + shell: bash + run: python tools/generate_sbom.py > sbom-${{ matrix.os }}-py${{ matrix.python }}.json + +- uses: actions/upload-artifact@v5 + with: + name: sbom-${{ matrix.os }}-py${{ matrix.python }} + path: sbom-${{ matrix.os }}-py${{ matrix.python }}.json +``` + +One SBOM per (OS × Python-version) cell of the cross-OS test matrix. +The artefacts are downloadable from the GitHub release page. + +### Determinism contract + +Wave 4 / Faz 23 adds a determinism test (`tests/test_supply_chain_security.py`): + +- Two consecutive invocations on the same Python environment must + produce content-identical SBOMs (modulo `serialNumber` and + `metadata.timestamp`, both intentionally varying per CycloneDX + semantics). +- `tests/test_supply_chain_security.py::TestGenerateSbomDeterministic::test_two_invocations_produce_same_content` + pins this in CI. + +An auditor can re-emit the SBOM from the corresponding `git tag` +and diff against the artefact attached to the GitHub release; any +non-trivial difference indicates dependency-resolution drift in the +build environment. + +### Consuming the SBOM + +```bash +# Download all SBOMs for a release. +gh release download v0.5.5 --pattern 'sbom-*' + +# Pretty-print for human review. +jq . sbom-ubuntu-latest-py3.11.json | less + +# List every dependency name + version. +jq -r '.components[] | "\(.name) \(.version)"' sbom-ubuntu-latest-py3.11.json + +# Diff between two releases. +diff <(jq -S '.components | sort_by(.purl)' sbom-prev.json) \ + <(jq -S '.components | sort_by(.purl)' sbom.json) + +# Ingest into Dependency-Track. +curl -X POST -H "X-Api-Key: $DT_KEY" -H "Content-Type: application/octet-stream" \ + --data-binary @sbom-ubuntu-latest-py3.11.json \ + https://depTrack.example.com/api/v1/bom +``` + +## pip-audit (transitive CVE scan) + +Wave 4 / Faz 23 adds `pip-audit` to the nightly workflow. Behaviour: + +- Runs daily at 03:00 UTC on the `closure/wave3-integration`-derived + development branch + main. +- Severity policy via `tools/check_pip_audit.py`: + - **HIGH / CRITICAL** → exit 1 (fails nightly; opens a GitHub issue + via the existing notify-failure job). + - **MEDIUM / MODERATE** → `::warning::` annotation; nightly stays + green. + - **LOW / UNKNOWN** → silent. +- Uses the OSV / GHSA databases (pip-audit's default). + +Operators install the same tooling locally: + +```bash +pip install forgelm[security] +pip-audit --strict --format json --output /tmp/pip-audit.json +python3 tools/check_pip_audit.py /tmp/pip-audit.json +``` + +### Suppression (intentional CVE acceptance) + +If a CVE is acknowledged but not yet fixable (upstream release +pending, etc.), document it in the deployer's risk acceptance log +and use `pip-audit --ignore-vuln ` to suppress it for the +nightly run. ForgeLM does NOT ship a project-level +ignore list — every suppression should be deployer-side and +quarterly-reviewed. + +## bandit (static security analysis) + +Wave 4 / Faz 23 adds `bandit` to: + +- `.github/workflows/ci.yml` — every PR + push to main. +- `.github/workflows/nightly.yml` — daily at 03:00 UTC. + +Scope: `forgelm/` (production code only). `tests/` is excluded +because test fixtures legitimately use insecure patterns (`assert`, +dummy secrets, `pickle.loads` on test inputs). + +Severity policy via `tools/check_bandit.py`: + +- **HIGH** → exit 1 (fails CI / nightly). +- **MEDIUM** → `::warning::` annotation; CI / nightly stays green. +- **LOW / UNDEFINED** → silent. + +Configuration in `pyproject.toml`: + +```toml +[tool.bandit] +exclude_dirs = ["tests", ".venv", "build", "dist"] +skips = ["B101"] # assert_used — irrelevant once tests/ is excluded +``` + +### Suppression syntax + +For a specific bandit rule that legitimately fires on production +code with a documented justification: + +```python +import subprocess +# nosec B603 B607 — args are literal, not operator-controlled +result = subprocess.run([sys.executable, "-m", "pip", "list"], ...) +``` + +Format: `# nosec [ ]` followed by an explanation. +Bandit ignores the line; `tools/check_bandit.py` confirms the +suppression is justified by the trailing prose. + +## Optional `[security]` extra + +`pyproject.toml` exposes the security tooling as an opt-in extra: + +```toml +[project.optional-dependencies] +security = [ + "pip-audit>=2.7.0,<3.0.0", + "bandit[toml]>=1.7.0,<2.0.0", +] +``` + +Operators install via: + +```bash +pip install forgelm[security] +``` + +This pulls in `pip-audit` and `bandit` so the same checks ForgeLM's +nightly + CI workflows enforce can be run locally before pushing a +PR. + +## Related controls + +| Standard | Control | Mapping | +|---|---|---| +| ISO 27001:2022 | A.5.21 | Managing information security in the ICT supply chain | +| ISO 27001:2022 | A.8.8 | Management of technical vulnerabilities | +| ISO 27001:2022 | A.8.30 | Outsourced development (transitive deps as outsourced code) | +| SOC 2 | CC7.1 | Detects vulnerabilities | +| SOC 2 | CC8.1 | Authorises changes (SBOM diff against last release) | +| SOC 2 | CC9.2 | Manages vendor + business-partner risk | +| EU AI Act | Art. 9 risk management | Supply-chain risk part of overall risk register | + +## See also + +- [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md) — deployer audit cookbook. +- [`iso27001_control_mapping.md`](iso27001_control_mapping.md) — ISO 27001 Annex A controls × ForgeLM evidence. +- [`soc2_trust_criteria_mapping.md`](soc2_trust_criteria_mapping.md) — SOC 2 Trust Services Criteria × ForgeLM evidence. +- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. +- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook. +- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook. +- `tools/generate_sbom.py` — CycloneDX 1.5 emitter. +- `tools/check_pip_audit.py` — pip-audit severity gate. +- `tools/check_bandit.py` — bandit severity gate. diff --git a/pyproject.toml b/pyproject.toml index 06b041e5..29503214 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -126,6 +126,16 @@ dev = [ "ruff>=0.4.0", ] +# Wave 4 / Faz 23: ISO 27001 / SOC 2 alignment supply-chain security tooling. +# Operators install via `pip install forgelm[security]` to run the same +# pip-audit (CVE) + bandit (static-analysis) checks ForgeLM's nightly + +# CI workflows enforce. Both are dev/CI tooling, never imported by +# the runtime path. +security = [ + "pip-audit>=2.7.0,<3.0.0", + "bandit[toml]>=1.7.0,<2.0.0", +] + [project.urls] Homepage = "https://github.com/cemililik/ForgeLM" Documentation = "https://github.com/cemililik/ForgeLM/tree/main/docs/guides" diff --git a/tests/test_supply_chain_security.py b/tests/test_supply_chain_security.py new file mode 100644 index 00000000..e21d827b --- /dev/null +++ b/tests/test_supply_chain_security.py @@ -0,0 +1,340 @@ +"""Wave 4 / Faz 23 — supply-chain security tooling tests. + +Three contracts pinned here: + +1. ``tools/generate_sbom.py`` produces deterministic content for the + same Python environment (modulo timestamp + UUID serial number, + both intentionally non-deterministic per CycloneDX 1.5 semantics). +2. ``tools/check_pip_audit.py`` severity-tiering: high/critical → exit 1, + medium → warning, low/unknown → silent. +3. ``tools/check_bandit.py`` severity-tiering: high → exit 1, medium → + warning, low → silent. + +The two helper-script tests exercise synthetic JSON fixtures so the +suite stays offline — no actual ``pip-audit`` or ``bandit`` run. +""" + +from __future__ import annotations + +import importlib.util +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +_REPO_ROOT = Path(__file__).resolve().parent.parent +_TOOLS = _REPO_ROOT / "tools" +_SBOM_TOOL = _TOOLS / "generate_sbom.py" +_PIP_AUDIT_TOOL = _TOOLS / "check_pip_audit.py" +_BANDIT_TOOL = _TOOLS / "check_bandit.py" + + +def _load_tool_module(path: Path, name: str): + """Import a ``tools/*.py`` script as a module without polluting ``sys.path``. + + Same pattern as ``tests/test_check_bilingual_parity.py`` and + ``tests/test_check_field_descriptions.py``. + """ + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise RuntimeError(f"could not load {path!r}") + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +# --------------------------------------------------------------------------- +# §1 — generate_sbom.py determinism contract +# --------------------------------------------------------------------------- + + +class TestGenerateSbomDeterministic: + """The SBOM emitter must produce content-stable output for a fixed env. + + CycloneDX 1.5 semantics intentionally vary two top-level fields + across runs: + + - ``serialNumber`` — a fresh UUID per run (CycloneDX 1.5 spec + treats each emit as a distinct BOM document). + - ``metadata.timestamp`` — wall-clock at emit time. + + Stripping those two fields, the SBOM body (``components``, all + metadata other than ``timestamp``, BOM format/version) MUST be + byte-identical for two consecutive invocations on the same Python + environment. That contract is what auditors verify when asking + "is the dependency list reproducible from the git tag?". + """ + + @staticmethod + def _strip_volatile(sbom_json: str) -> dict: + sbom = json.loads(sbom_json) + sbom.pop("serialNumber", None) + meta = sbom.get("metadata") or {} + meta.pop("timestamp", None) + sbom["metadata"] = meta + return sbom + + def test_two_invocations_produce_same_content(self) -> None: + first = subprocess.run( + [sys.executable, str(_SBOM_TOOL)], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + second = subprocess.run( + [sys.executable, str(_SBOM_TOOL)], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + # Modulo ``serialNumber`` + ``metadata.timestamp`` (both volatile + # by CycloneDX 1.5 design), the two SBOMs must be content-equal. + normalised_a = self._strip_volatile(first.stdout) + normalised_b = self._strip_volatile(second.stdout) + assert normalised_a == normalised_b, ( + "SBOM emitter is non-deterministic on the same environment; " + "the content (excluding serialNumber + timestamp) must be stable " + "so an auditor can reproduce it from the git tag" + ) + + def test_required_cyclonedx_1_5_fields_present(self) -> None: + result = subprocess.run( + [sys.executable, str(_SBOM_TOOL)], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + sbom = json.loads(result.stdout) + assert sbom.get("bomFormat") == "CycloneDX" + assert sbom.get("specVersion") == "1.5" + assert "serialNumber" in sbom + assert isinstance(sbom.get("components"), list) + meta = sbom.get("metadata") or {} + assert "timestamp" in meta + assert (meta.get("component") or {}).get("name") == "forgelm" + # Every component must have purl + bom-ref + name + version + # so the SBOM is consumable by Dependency-Track / Snyk / etc. + for comp in sbom["components"]: + assert comp.get("type") == "library", comp + assert comp.get("name"), comp + assert comp.get("version"), comp + assert comp.get("purl", "").startswith("pkg:pypi/"), comp + + +# --------------------------------------------------------------------------- +# §2 — check_pip_audit.py severity-tiering +# --------------------------------------------------------------------------- + + +class TestCheckPipAudit: + """Synthetic JSON fixtures exercise each severity tier without + needing pip-audit installed. Real pip-audit shape captured from + the v2.7 schema.""" + + @pytest.fixture + def tool(self): + return _load_tool_module(_PIP_AUDIT_TOOL, "check_pip_audit") + + def _write_report(self, tmp_path: Path, payload: dict) -> Path: + path = tmp_path / "pip-audit.json" + path.write_text(json.dumps(payload), encoding="utf-8") + return path + + def test_clean_report_exits_zero(self, tmp_path: Path, tool) -> None: + path = self._write_report(tmp_path, {"dependencies": []}) + rc = tool.main(["check_pip_audit", str(path)]) + assert rc == 0 + + def test_low_severity_silent(self, tmp_path: Path, tool, capsys) -> None: + report = { + "dependencies": [ + { + "name": "example", + "version": "1.0.0", + "vulns": [ + {"id": "GHSA-aaaa-bbbb-cccc", "severity": "LOW", "fix_versions": ["1.0.1"]}, + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::" not in captured + assert "::error::" not in captured + + def test_medium_severity_warns_but_does_not_fail(self, tmp_path: Path, tool, capsys) -> None: + report = { + "dependencies": [ + { + "name": "midpkg", + "version": "2.0.0", + "vulns": [ + {"id": "GHSA-mid", "severity": "MEDIUM", "fix_versions": ["2.0.1"]}, + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::pip-audit" in captured + assert "midpkg" in captured + + def test_high_severity_fails(self, tmp_path: Path, tool, capsys) -> None: + report = { + "dependencies": [ + { + "name": "scarypkg", + "version": "0.1.0", + "vulns": [ + {"id": "CVE-2026-XXXX", "severity": "HIGH", "fix_versions": []}, + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 1 + assert "::error::pip-audit" in captured + assert "scarypkg" in captured + + def test_critical_severity_also_fails(self, tmp_path: Path, tool) -> None: + report = { + "dependencies": [ + { + "name": "scarypkg", + "version": "0.1.0", + "vulns": [ + {"id": "CVE-2026-XXXX", "severity": "CRITICAL", "fix_versions": []}, + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + assert rc == 1 + + def test_moderate_synonym_treated_as_medium(self, tmp_path: Path, tool, capsys) -> None: + # Some advisory feeds use "MODERATE" instead of "MEDIUM"; the + # tier normaliser must collapse them. + report = { + "dependencies": [ + { + "name": "syn", + "version": "1.0.0", + "vulns": [{"id": "GHSA-syn", "severity": "MODERATE"}], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + assert rc == 0 + assert "::warning::" in capsys.readouterr().out + + def test_missing_file_fails_loudly(self, tmp_path: Path, tool, capsys) -> None: + rc = tool.main(["check_pip_audit", str(tmp_path / "nope.json")]) + assert rc == 1 + assert "::error::pip-audit report not readable" in capsys.readouterr().err + + def test_invalid_json_fails_loudly(self, tmp_path: Path, tool, capsys) -> None: + bad = tmp_path / "bad.json" + bad.write_text("{not json", encoding="utf-8") + rc = tool.main(["check_pip_audit", str(bad)]) + assert rc == 1 + assert "::error::pip-audit report" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# §3 — check_bandit.py severity-tiering +# --------------------------------------------------------------------------- + + +class TestCheckBandit: + @pytest.fixture + def tool(self): + return _load_tool_module(_BANDIT_TOOL, "check_bandit") + + def _write_report(self, tmp_path: Path, payload: dict) -> Path: + path = tmp_path / "bandit.json" + path.write_text(json.dumps(payload), encoding="utf-8") + return path + + def test_clean_report_exits_zero(self, tmp_path: Path, tool) -> None: + rc = tool.main(["check_bandit", str(self._write_report(tmp_path, {"results": []}))]) + assert rc == 0 + + def test_low_severity_silent(self, tmp_path: Path, tool, capsys) -> None: + report = { + "results": [ + { + "test_id": "B101", + "test_name": "assert_used", + "issue_severity": "LOW", + "issue_confidence": "HIGH", + "filename": "forgelm/x.py", + "line_number": 1, + "issue_text": "assert used", + } + ] + } + rc = tool.main(["check_bandit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::" not in captured + assert "::error::" not in captured + + def test_medium_severity_warns_but_does_not_fail(self, tmp_path: Path, tool, capsys) -> None: + report = { + "results": [ + { + "test_id": "B603", + "test_name": "subprocess_without_shell_equals_true", + "issue_severity": "MEDIUM", + "issue_confidence": "MEDIUM", + "filename": "forgelm/x.py", + "line_number": 42, + "issue_text": "subprocess call", + } + ] + } + rc = tool.main(["check_bandit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::bandit" in captured + assert "B603" in captured + + def test_high_severity_fails(self, tmp_path: Path, tool, capsys) -> None: + report = { + "results": [ + { + "test_id": "B301", + "test_name": "pickle", + "issue_severity": "HIGH", + "issue_confidence": "HIGH", + "filename": "forgelm/x.py", + "line_number": 10, + "issue_text": "pickle.loads on untrusted input", + } + ] + } + rc = tool.main(["check_bandit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 1 + assert "::error::bandit" in captured + assert "B301" in captured + + def test_missing_file_fails_loudly(self, tmp_path: Path, tool, capsys) -> None: + rc = tool.main(["check_bandit", str(tmp_path / "nope.json")]) + assert rc == 1 + assert "::error::bandit report not readable" in capsys.readouterr().err + + def test_results_not_a_list_fails(self, tmp_path: Path, tool, capsys) -> None: + bad = self._write_report(tmp_path, {"results": "not-a-list"}) + rc = tool.main(["check_bandit", str(bad)]) + assert rc == 1 + assert "results" in capsys.readouterr().err diff --git a/tools/check_bandit.py b/tools/check_bandit.py new file mode 100644 index 00000000..6a884ecc --- /dev/null +++ b/tools/check_bandit.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Wave 4 / Faz 23 — `bandit` JSON output severity gate. + +Reads the JSON report produced by ``bandit -f json`` and applies +ForgeLM's severity policy: + +- ``HIGH`` issues exit 1 (fail CI / nightly). +- ``MEDIUM`` issues emit a ``::warning::`` GitHub annotation but do + not fail. +- ``LOW`` issues are silent. + +Used in ``.github/workflows/ci.yml`` (every PR) and +``.github/workflows/nightly.yml`` (daily) after the ``bandit`` step. + +Exit codes (per ``tools/`` contract — NOT the public 0/1/2/3/4 surface +that ``forgelm/`` honours): + +- ``0`` — no high-severity issues (medium/low may be present + warned). +- ``1`` — at least one high-severity issue OR the input file is + missing / unparseable. + +Usage:: + + bandit -c pyproject.toml -r forgelm/ -f json -o /tmp/bandit.json || true + python3 tools/check_bandit.py /tmp/bandit.json +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any + +# bandit's severity vocabulary is HIGH / MEDIUM / LOW (UNDEFINED for +# rules that don't carry severity metadata). Confidence is reported +# separately and not used for gating — operators reviewing warnings +# can read it from the artefact. +_HIGH = "HIGH" +_MED = "MEDIUM" + + +def _format_issue(issue: dict[str, Any]) -> str: + test_id = issue.get("test_id") or "B???" + test_name = issue.get("test_name") or "" + filename = issue.get("filename") or "" + line = issue.get("line_number") or "?" + severity = (issue.get("issue_severity") or "UNDEFINED").upper() + confidence = (issue.get("issue_confidence") or "UNDEFINED").upper() + text = (issue.get("issue_text") or "").splitlines()[0] if issue.get("issue_text") else "" + return f"[{severity}/{confidence}] {filename}:{line} {test_id} {test_name} — {text}" + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print(f"usage: {argv[0]} ", file=sys.stderr) + return 1 + + report_path = Path(argv[1]) + try: + raw = report_path.read_text(encoding="utf-8") + except OSError as exc: + print(f"::error::bandit report not readable at {report_path}: {exc}", file=sys.stderr) + return 1 + + try: + report = json.loads(raw) + except json.JSONDecodeError as exc: + print(f"::error::bandit report at {report_path} is not valid JSON: {exc}", file=sys.stderr) + return 1 + + results = report.get("results") or [] + if not isinstance(results, list): + print("::error::bandit report 'results' field is not a list", file=sys.stderr) + return 1 + + high: list[str] = [] + medium: list[str] = [] + for issue in results: + if not isinstance(issue, dict): + continue + severity = (issue.get("issue_severity") or "UNDEFINED").upper() + line = _format_issue(issue) + if severity == _HIGH: + high.append(line) + elif severity == _MED: + medium.append(line) + # LOW / UNDEFINED are silent; the raw JSON remains in artefacts. + + for line in medium: + print(f"::warning::bandit {line}") + + if high: + for line in high: + print(f"::error::bandit {line}") + print(f"::error::bandit found {len(high)} high-severity issue(s); failing the run.") + return 1 + + if medium: + print(f"bandit: {len(medium)} medium-severity issue(s) (warning only).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) diff --git a/tools/check_bilingual_parity.py b/tools/check_bilingual_parity.py index 0217db5f..4aed75f2 100644 --- a/tools/check_bilingual_parity.py +++ b/tools/check_bilingual_parity.py @@ -66,12 +66,16 @@ ("docs/guides/data_audit.md", "docs/guides/data_audit-tr.md"), ("docs/guides/gdpr_erasure.md", "docs/guides/gdpr_erasure-tr.md"), ("docs/guides/ingestion.md", "docs/guides/ingestion-tr.md"), + ("docs/guides/iso_soc2_deployer_guide.md", "docs/guides/iso_soc2_deployer_guide-tr.md"), # docs/reference/ ("docs/reference/architecture.md", "docs/reference/architecture-tr.md"), ("docs/reference/audit_event_catalog.md", "docs/reference/audit_event_catalog-tr.md"), ("docs/reference/configuration.md", "docs/reference/configuration-tr.md"), ("docs/reference/data_preparation.md", "docs/reference/data_preparation-tr.md"), ("docs/reference/distributed_training.md", "docs/reference/distributed_training-tr.md"), + ("docs/reference/iso27001_control_mapping.md", "docs/reference/iso27001_control_mapping-tr.md"), + ("docs/reference/soc2_trust_criteria_mapping.md", "docs/reference/soc2_trust_criteria_mapping-tr.md"), + ("docs/reference/supply_chain_security.md", "docs/reference/supply_chain_security-tr.md"), ("docs/reference/usage.md", "docs/reference/usage-tr.md"), ) diff --git a/tools/check_pip_audit.py b/tools/check_pip_audit.py new file mode 100644 index 00000000..f0539442 --- /dev/null +++ b/tools/check_pip_audit.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +"""Wave 4 / Faz 23 — `pip-audit` JSON output severity gate. + +Reads the JSON report produced by ``pip-audit --format json`` and +applies ForgeLM's severity policy: + +- ``HIGH`` / ``CRITICAL`` findings exit 1 (fail nightly). +- ``MEDIUM`` findings emit a ``::warning::`` GitHub annotation but do + not fail. +- ``LOW`` findings are silent. + +Used in ``.github/workflows/nightly.yml`` after the ``pip-audit`` step. + +Exit codes (per ``tools/`` contract — NOT the public 0/1/2/3/4 surface +that ``forgelm/`` honours): + +- ``0`` — no high/critical CVEs (medium/low may be present and warned). +- ``1`` — at least one high or critical CVE OR the input file is + missing / unparseable. + +Usage:: + + pip-audit --format json --output /tmp/pip-audit.json || true + python3 tools/check_pip_audit.py /tmp/pip-audit.json +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path +from typing import Any, Iterable, Optional + +# pip-audit's JSON shape (≥2.7) puts findings under ``dependencies[].vulns``, +# each vuln carrying ``id``, ``aliases``, ``description``, ``fix_versions``. +# Severity is sourced from the OSV / GHSA aliases via ``severity`` (a list +# of {type, score} pairs) when present; pip-audit normalises into +# ``severity`` strings on the top-level vuln dict from 2.7.x onwards. +_HIGH_TIERS: frozenset[str] = frozenset({"HIGH", "CRITICAL"}) +_MED_TIERS: frozenset[str] = frozenset({"MEDIUM", "MODERATE"}) + + +def _normalise_severity(raw: Optional[str]) -> str: + """Upper-case + collapse synonyms; unknown/missing → ``UNKNOWN``.""" + if not raw: + return "UNKNOWN" + upper = raw.upper().strip() + if upper in {"MODERATE"}: + return "MEDIUM" + return upper + + +def _vuln_severity(vuln: dict[str, Any]) -> str: + """Extract a single severity tier from a pip-audit vuln entry. + + Falls back through several fields because pip-audit's JSON shape + has shifted across point releases: + - 2.7.x: top-level ``severity`` string + - 2.6.x and earlier: nested under ``aliases[].severity[]`` + - GHSA imports often carry only ``severity_score`` (CVSS) + + When no field is parseable we return ``"UNKNOWN"`` and let the + operator review the raw report manually. + """ + direct = vuln.get("severity") + if isinstance(direct, str): + return _normalise_severity(direct) + if isinstance(direct, list) and direct: + first = direct[0] + if isinstance(first, dict): + return _normalise_severity(first.get("type") or first.get("severity")) + if isinstance(first, str): + return _normalise_severity(first) + return "UNKNOWN" + + +def _iter_findings(report: dict[str, Any]) -> Iterable[tuple[str, dict[str, Any]]]: + """Yield (package_name, vuln_dict) pairs from a pip-audit report.""" + deps = report.get("dependencies") or [] + if not isinstance(deps, list): + return + for dep in deps: + if not isinstance(dep, dict): + continue + name = dep.get("name") or "" + for vuln in dep.get("vulns") or []: + if isinstance(vuln, dict): + yield name, vuln + + +def _format_finding(name: str, vuln: dict[str, Any], severity: str) -> str: + vid = vuln.get("id") or "" + fix_versions = vuln.get("fix_versions") or vuln.get("fix_version") or [] + if isinstance(fix_versions, str): + fix_text = fix_versions + elif isinstance(fix_versions, list) and fix_versions: + fix_text = ", ".join(str(v) for v in fix_versions) + else: + fix_text = "(no fix available)" + return f"[{severity}] {name} {vid} — fix: {fix_text}" + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print(f"usage: {argv[0]} ", file=sys.stderr) + return 1 + + report_path = Path(argv[1]) + try: + raw = report_path.read_text(encoding="utf-8") + except OSError as exc: + print(f"::error::pip-audit report not readable at {report_path}: {exc}", file=sys.stderr) + return 1 + + try: + report = json.loads(raw) + except json.JSONDecodeError as exc: + print(f"::error::pip-audit report at {report_path} is not valid JSON: {exc}", file=sys.stderr) + return 1 + + high: list[str] = [] + medium: list[str] = [] + for name, vuln in _iter_findings(report): + severity = _vuln_severity(vuln) + line = _format_finding(name, vuln, severity) + if severity in _HIGH_TIERS: + high.append(line) + elif severity in _MED_TIERS: + medium.append(line) + # LOW + UNKNOWN are silent; the raw JSON remains in artefacts. + + for line in medium: + # GitHub Actions annotation; surfaces in the run summary without + # failing the build. + print(f"::warning::pip-audit {line}") + + if high: + for line in high: + print(f"::error::pip-audit {line}") + print(f"::error::pip-audit found {len(high)} high/critical-severity finding(s); failing the run.") + return 1 + + if medium: + print(f"pip-audit: {len(medium)} medium-severity finding(s) (warning only).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv)) From 6f526c736905c25c88f1158f5ddce72857911342 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 00:09:53 +0300 Subject: [PATCH 03/12] docs(wave-4-faz-26): TR mirrors for QMS docs (part 1: 6 existing + 1 new) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Faz 26 part 1 — bilingual mirrors land for the QMS templates. Closure-plan reference: Faz 26 task #1 (six existing QMS docs need TR mirrors). Per Faz 22 design Decision D-22-05, the four Faz 23 new QMS docs (encryption_at_rest, access_control, risk_treatment_plan, statement_of_applicability) ship EN-only in Faz 23 and pair their TR mirrors here in Faz 26. This commit ships: Existing-QMS TR mirrors (6 files, closure-plan task #1): - README-tr.md - roles_responsibilities-tr.md - sop_model_training-tr.md - sop_data_management-tr.md - sop_incident_response-tr.md (Wave 4 §4 security-incident playbook included) - sop_change_management-tr.md (Wave 4 §4 CI-gates-as-change-control table included) New-Faz-23 QMS TR mirrors (1 of 4 — rest in part 2): - encryption_at_rest-tr.md EN-side delta: - docs/qms/README.md — table extended with the 4 Wave-4 / Faz-23 rows (Access Control, Encryption at Rest, Risk Treatment Plan, Statement of Applicability) so the index reflects the actual doc inventory. Faz 26 part 2 follow-ups (next commit on this branch): - access_control-tr.md - risk_treatment_plan-tr.md - statement_of_applicability-tr.md - docs/reference/compliance_summary.md cleanup (H1 + scope blockquote, fix broken paths, anchor refresh) - tools/check_anchor_resolution.py + tests + CI step Bilingual parity unaffected (QMS docs are not in tools/check_bilingual_parity.py::_PAIRS today; QMS-pair registration will land alongside the anchor-resolution checker so the parity tool guards QMS drift too). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/qms/README-tr.md | 37 +++++ docs/qms/README.md | 4 + docs/qms/encryption_at_rest-tr.md | 231 ++++++++++++++++++++++++++ docs/qms/roles_responsibilities-tr.md | 98 +++++++++++ docs/qms/sop_change_management-tr.md | 150 +++++++++++++++++ docs/qms/sop_data_management-tr.md | 122 ++++++++++++++ docs/qms/sop_incident_response-tr.md | 229 +++++++++++++++++++++++++ docs/qms/sop_model_training-tr.md | 80 +++++++++ 8 files changed, 951 insertions(+) create mode 100644 docs/qms/README-tr.md create mode 100644 docs/qms/encryption_at_rest-tr.md create mode 100644 docs/qms/roles_responsibilities-tr.md create mode 100644 docs/qms/sop_change_management-tr.md create mode 100644 docs/qms/sop_data_management-tr.md create mode 100644 docs/qms/sop_incident_response-tr.md create mode 100644 docs/qms/sop_model_training-tr.md diff --git a/docs/qms/README-tr.md b/docs/qms/README-tr.md new file mode 100644 index 00000000..4c7579aa --- /dev/null +++ b/docs/qms/README-tr.md @@ -0,0 +1,37 @@ +# Kalite Yönetim Sistemi (QMS) Şablonları + +> **EU AI Act Madde 17** yüksek-riskli AI sistem sağlayıcılarının bir Kalite Yönetim Sistemi kurmasını gerektirir. Bu şablonlar bir başlangıç noktası sağlar — kurumunuza uyarlayın. + +## Şablonlar + +| Şablon | EU AI Act Referansı | Amaç | +|----------|-------------------|---------| +| [Model Eğitimi SOP](sop_model_training.md) | Md. 17(1)(b)(c)(d) | Fine-tuning onay, yürütme ve doğrulama için standart prosedür | +| [Veri Yönetimi SOP](sop_data_management.md) | Md. 17(1)(f), Md. 10 | Veri toplama, annotation, kalite güvencesi ve yönetişim | +| [Olay Yanıtı SOP](sop_incident_response.md) | Md. 17(1)(h)(i) | Model arızalarını, güvenlik olaylarını ve düzeltici eylemleri yönetme | +| [Değişim Yönetimi SOP](sop_change_management.md) | Md. 17(1)(b)(c) | Versiyonlama, inceleme, onay ve geri alma prosedürleri | +| [Roller & Sorumluluklar](roles_responsibilities.md) | Md. 17(1)(m) | AI Officer, Data Steward, ML Engineer, Compliance Officer rolleri | +| [Erişim Kontrolü](access_control.md) | Md. 17(1)(c) + ISO A.5.15-A.8.5 | Operatör kimliği + secret rotasyonu (Wave 4 / Faz 23) | +| [Encryption at Rest](encryption_at_rest.md) | Md. 17(1)(c) + ISO A.5.33 + A.8.10 + A.8.24 | Substrate-side şifreleme rehberi (Wave 4 / Faz 23) | +| [Risk Treatment Plan](risk_treatment_plan.md) | Md. 17(1)(c) + ISO A.5.7-A.8.30 | ISO 27005 risk register şablonu (Wave 4 / Faz 23) | +| [Statement of Applicability](statement_of_applicability.md) | Md. 17(1)(c) + ISO 6.1.3 d) | 93-kontrol uygulanabilirlik matrisi (Wave 4 / Faz 23) | + +## Nasıl Kullanılır + +1. Bu şablonları kurumunuzun iç dokümantasyon sistemine kopyalayın +2. `[YOUR ORGANIZATION]` ifadesini şirket adınızla değiştirin +3. Her role gerçek isimler atayın +4. Yasal/uyumluluk ekibinizle inceleyin ve onaylayın +5. ForgeLM'in otomatik artefaktlarını kanıt olarak referans verin + +## ForgeLM Otomatik Artefaktları (QMS'ye Eşleşir) + +| QMS Gereksinimi | ForgeLM Artefaktı | Üreten | +|----------------|-----------------|-------------| +| Eğitim kayıtları | `compliance_report.json` | `forgelm --config job.yaml` | +| Veri provenance | `data_provenance.json` | Koşum başına otomatik | +| Değerlendirme kanıtı | `benchmark_results.json`, `safety_results.json` | Koşum başına otomatik | +| Model kimliği | `model_integrity.json` | Koşum başına otomatik | +| Audit trail | `audit_log.jsonl` | Koşum başına otomatik | +| Risk değerlendirmesi | `risk_assessment.json` | Config `risk_assessment:` bölümünden | +| Operatör (deployer) talimatları | `deployer_instructions.md` | Koşum başına otomatik | diff --git a/docs/qms/README.md b/docs/qms/README.md index 2d2909b3..e7e21b17 100644 --- a/docs/qms/README.md +++ b/docs/qms/README.md @@ -11,6 +11,10 @@ | [Incident Response SOP](sop_incident_response.md) | Art. 17(1)(h)(i) | Handling model failures, safety incidents, and corrective actions | | [Change Management SOP](sop_change_management.md) | Art. 17(1)(b)(c) | Versioning, review, approval, and rollback procedures | | [Roles & Responsibilities](roles_responsibilities.md) | Art. 17(1)(m) | AI Officer, Data Steward, ML Engineer, Compliance Officer roles | +| [Access Control](access_control.md) | Art. 17(1)(c) + ISO A.5.15-A.8.5 | Operator identity + secret rotation (Wave 4 / Faz 23) | +| [Encryption at Rest](encryption_at_rest.md) | Art. 17(1)(c) + ISO A.5.33 + A.8.10 + A.8.24 | Substrate-side encryption guidance (Wave 4 / Faz 23) | +| [Risk Treatment Plan](risk_treatment_plan.md) | Art. 17(1)(c) + ISO A.5.7-A.8.30 | ISO 27005 risk register template (Wave 4 / Faz 23) | +| [Statement of Applicability](statement_of_applicability.md) | Art. 17(1)(c) + ISO 6.1.3 d) | 93-control applicability matrix (Wave 4 / Faz 23) | ## How to Use diff --git a/docs/qms/encryption_at_rest-tr.md b/docs/qms/encryption_at_rest-tr.md new file mode 100644 index 00000000..966cf981 --- /dev/null +++ b/docs/qms/encryption_at_rest-tr.md @@ -0,0 +1,231 @@ +# QMS: Atıl-durumda Şifreleme + +> Kalite Yönetim Sistemi rehberi — [YOUR ORGANIZATION] +> ISO 27001:2022 referansları: A.5.33, A.8.10, A.8.13, A.8.24 +> SOC 2 referansları: CC6.1, CC6.7 + +## 1. Amaç + +ForgeLM-ürettiği artefaktların atıl-durumda gizli kalması için +operatör-tarafı kontrolleri tanımla. ForgeLM **integrity-protected** +artefaktlar emit eder (HMAC-zincirli audit log'ları, SHA-256 model +integrity dosyaları, imzalı operatör talimatları) ama bunları +şifrelemez — şifreleme operatörün depolama katmanı tarafından +sahiplenilen substrate sorunudur. + +Bu doküman her ForgeLM artefakt sınıfını önerilen şifreleme substrate'i +ve operatör eylemine eşler. + +## 2. Kapsam + +Aşağıdaki ForgeLM-ürettiği asset sınıfları: + +| Asset sınıfı | Path pattern | Gizlilik endişesi | +|---|---|---| +| Model ağırlıkları — final | `/final_model/` | Eğitim verisi memorisation; rekabetçi moat | +| Model ağırlıkları — staging | `/staging_model./` | Final ile aynı, artı pre-approval state | +| Audit log | `/audit_log.jsonl` + `.manifest.json` + `.sha256` sidecar | Operatör kimliği geçmişi; yapılandırma hashleri; uyumluluk için chain-of-custody | +| Per-output-dir salt | `/.forgelm_audit_salt` | Düşük-entropi identifier'ların salt'sız SHA-256'sı brute-force edilebilir — salt audit hash'ine wordlist direnci veren secret'tir | +| Eğitim corpusu | (operatör-tedarikli; tipik olarak `data/*.jsonl`) | PII; ticari sırlar; müşteri verisi | +| Quickstart-rendered config | (operatör path'i) | HF token'ları, webhook URL'leri, secret env-var isimleri | +| Uyumluluk paketi (Annex IV) | `/compliance/` ZIP | Yukarıdaki her artefaktın aggregate formu | + +Operatör-kontrollü asset'ler (HF auth token'ları, webhook secret'ları, +`FORGELM_AUDIT_SECRET`) operatörün secrets-management substrate'inde +yaşar ve bu doküman için kapsam dışıdır — credential-handling rehberi +için `access_control.md`'a bakın. + +## 3. Tehdit modeli + +Bu rehberin savunduğu tehditler: + +1. **Disk hırsızlığı / fiziksel kayıp** — laptop veya eğitim-host hard + drive'ı düşman eline düşer. +2. **Yedek compromise** — yedek snapshot yanlış-yapılandırılmış bir + bucket'a veya üçüncü-taraf bir cloud'a düşer; eski snapshot'lar + mevcut şifreleme politikasından önce gelmiş olabilir. +3. **Shared-tenancy disk leak** — başka bir tenant'ın + yanlış-yapılandırma yoluyla okuma erişimi kazandığı multi-tenant + cloud depolaması. +4. **Log-ship intercept** — audit log'un SIEM'e şifrelenmemiş kanal + üzerinden gönderimi. +5. **Decommissioned disklerden forensic recovery** — imha öncesi + silinmemiş sektörler. + +Açıkça kapsam DIŞI tehditler: + +- **Live-memory düşman** — process memory dump'ları; operatörün + endpoint detection / sandboxing bunu yönetir. +- **Meşru decryption credential'lı insider** — şifreleme yetkili + erişime karşı koruyamaz; cevap erişim kontrolü + audit log'dur. +- **Kripto-algoritma kırılması** — SHA-256 / AES-256-GCM'in QMS + doküman ömrü boyunca sağlam kalacağı varsayılır. + +## 4. Asset sınıfı başına önerilen kontroller + +### 4.1 Model ağırlıkları (final + staging) + +| Substrate | Öneri | +|---|---| +| Linux eğitim hostu | LUKS / dm-crypt full-disk encryption (kernel-supported, transparent) | +| macOS eğitim hostu | FileVault (transparent) | +| Windows eğitim hostu | BitLocker (TPM-backed key) | +| Linux container | LUKS-protected host volume'dan Docker `--mount type=bind,...` üzerinden geçirilen şifrelenmiş block device | +| AWS S3 | Customer-managed key (CMK) ile SSE-KMS; bucket-level default encryption + bucket policy `aws:SecureTransport=true` etkinleştir | +| Azure Blob | Azure Key Vault üzerinden customer-managed key | +| GCS | Customer-Supplied Encryption Keys (CSEK) veya Cloud KMS | +| HDFS / Ceph | Operatör-managed KMS ile native at-rest encryption | + +**Neden full-disk dosya-seviyesi değil:** model ağırlıkları HF +Transformers / PEFT tarafından birçok küçük dosyaya yazılır (sharded +`.safetensors`, adapter manifestleri). Her dosya üzerinde dosya- +seviyesi GPG / age kırılgan dosya enumeration'ları üretir. Substrate- +seviyesi şifreleme transparent ve denetlenebilirdir. + +**Anahtar rotasyonu:** SSE-KMS için en az yıllık; sadece data key'i +değil CMK'nın kendisini rotate et. ForgeLM rotasyondan etkilenmez — +model dosyaları storage katmanında eğitim pipeline'ı fark etmeden +yeniden şifrelenir. + +### 4.2 Audit log + manifest sidecar + +**Kritik contract:** audit chain ForgeLM tarafından atıl-durumda +integrity-protected'tır (HMAC + SHA-256 chain + genesis manifest). +Şifreleme yalnız gizliliktir. **Bayt'ları yeniden sıralayan veya +yeniden çerçeveleyen bir katmanda şifreleme YAPMA** — `.manifest.json` +belirli bir bayt aralığı üzerinde imzalandıktan sonra, o aralığı +bozan herhangi bir şifreleme `forgelm verify-audit`'i bozar. + +Güvenli substrate'ler: + +- Aynı-disk şifreleme (LUKS, FileVault, BitLocker) — transparent. +- SSE-S3 veya SSE-KMS ile cloud blob storage — transparent. + +Güvensiz substrate'ler: + +- JSONL'i base64 + GPG-encrypt eden custom uygulama-katmanı sarma — + decryption verify komutu çalışmadan önce gerçekleşmediği sürece + `forgelm verify-audit`'i bozar. + +**Off-site replikasyon:** write-once depolamaya gönder (uyumluluk +modunda S3 Object Lock, Azure Immutable Blob, versioning'li MinIO) +böylece yetkili operatör bile geriye dönük girişleri silemez. +ForgeLM'in append-only contract'ı host-üzerindedir; durabilite +operatörün sorumluluğudur. + +### 4.3 Per-output-dir salt (`.forgelm_audit_salt`) + +Bu, ForgeLM'in emit ettiği operasyonel olarak en hassas tek dosyadır. +Onu compromise etmek bir saldırgana ForgeLM'in kasıtlı olarak +hashlediği cleartext identifier'ları geri kazanmak için +`data.access_request_query` ve `data.erasure_*` audit olaylarına +karşı offline dictionary saldırısı çalıştırmasına izin verir. + +**Önerilen tedavi:** + +- Dosyada mod `0600` (`chmod 600 .forgelm_audit_salt`). +- İnsan kullanıcı değil CI pipeline service hesabı tarafından + sahip edilmiş. +- Sadece audit log'un şifreleme posture'ı ile eşleşen substrate'lere + yedeklenir (HİÇBİR ZAMAN debug bucket'a plaintext). +- Sadece karşılık gelen `` decommissioned olduğunda + rotate edilir — output-dir-ortasında rotate etmek `forgelm + verify-audit`'in önceki olaylar için salted-hash varsayımlarını + geçersiz kılar. + +**Defence-in-depth:** `FORGELM_AUDIT_SECRET`'i KMS'inizden export et +böylece on-disk salt iki-bileşenli anahtarın yarısı haline gelir. +Diski çalan ama env secret'ini çalmayan bir saldırgan audit hash'ini +tersine çeviremez. Env-secret rotasyonu için `access_control.md` +§3.4'e bakın. + +### 4.4 Eğitim corpusu + +Corpus operatör-tedarikli ve operatör-şifrelenmiş'tir. ForgeLM +katkı yapar: + +- Şifreleme GEREKEN şeyleri flag'lemek için `forgelm audit` PII + tespiti + secrets scan (örn. credit-card numarası içerdiği ortaya + çıkan corpus). +- Severity tier'ları ile `data_audit_report.json` — operatör + flag'lenen corpus'ları yeniden şifreleyip şifrelemeyeceğine, + maskeleyeceğine veya karantinaya alacağına karar verir. +- `data.dataset_id` + `data_provenance.json` SHA-256 fingerprint — + şifreleme sonrası substrate-seviyesi tampering'i tespit eder. + +**Ticari sır koruması:** corpus operatörün fikri mülkiyetiyse +(proprietary destek bilet'leri, dahili dokümantasyon), Top Secret +olarak ele al ve key access'i loglanan bir CMK ile substrate +katmanında şifrele. İzole bir VPC içinde eğit; `forgelm`'in corpusu +webhook endpoint'lerine egress etmesine İZİN VERME (bu zaten +zorunludur — `safe_post` webhook payload'larında asla raw eğitim +satırları taşımaz). + +### 4.5 Quickstart-rendered config + operatör YAML + +Configler HF auth token'ları (`HF_TOKEN`) veya host-internal +hostname gömen webhook URL'leri taşıyabilir. Credential-bearing +olarak ele al: + +- Home dizinindeki düz dosya değil bir config-management substrate'inde + sakla (Ansible Vault, Doppler, Vault). +- Geçici CI kullanımı için, configi job başlangıcında secrets-manager + değerlerinden render et; rendered dosyayı job teardown'da sil. +- ForgeLM'in audit olaylarındaki `compliance.config_hash` herhangi bir + secret expansion'dan SONRA hesaplanır, yani sadece secret değerlerinde + farklı olan iki config dosyası farklı hash'ler üretir — denetçiler + koşum-ortası bir config swap'i tespit edebilir. + +### 4.6 Uyumluluk paketi (Annex IV ZIP) + +`forgelm --compliance-export` ZIP yukarıdaki her artefaktı tek bir +dosyada aggregate eder. Yukarıdaki tüm asset sınıflarının birleşimi +olarak ele al: + +- En-hassas asset'in gerektirdiği substrate'de şifrelenmiş (tipik + olarak eğitim corpusu dikte eder). +- Authenticated kanal üzerinden denetçiye gönderildi (örn. düzenleyici- + tedarikli portal üzerinden parola-korumalı ZIP; 24-saat süre dolmalı + imzalı S3 pre-signed URL). +- `compliance.artifacts_exported` event üzerinden audit chain'e geri + receipt confirmation log'u. + +## 5. ForgeLM'in katkısı + +ForgeLM kendisi şifreleme uygulamaz ama ŞUNLARI YAPAR: + +1. **Neyi şifreleyeceğini tespit et.** `forgelm audit` eğitim + verisindeki PII, secrets ve credential'ları flag'ler ki operatör + eğitim öncesi şifreleyebilir veya maskeleyebilir. +2. **Neyin şifrelendiğini rapor et.** `data_governance_report.json` + operatörün config block'u başına `encryption_at_rest: true|false` + kaydeder. +3. **Şifreleme-sonrası bütünlüğü doğrula.** `forgelm verify-audit` + chain-after-decryption'ın chain-as-emitted ile eşleştiğini teyit + eder; substrate-seviyesi herhangi bir bozulma tespit edilebilirdir. +4. **Düşük-entropi identifier'ları hash'le.** Salted SHA-256, audit + chain'in kendisinin atıl-durumda gizli olması için şifrelenmesine + ihtiyaç olmadığı anlamına gelir — yalnız salt'ın. + +## 6. Doğrulama checklist'i + +Şifreleme-at-rest kontrollerinizi yürüyen operatör denetçi için: + +- [ ] Tüm eğitim-host diskleri `dmsetup table` (Linux) veya + `manage-bde -status` (Windows) encryption-in-use gösteriyor. +- [ ] Cloud depolama bucket'larının default-encryption ayarı + customer-managed key ile etkin. +- [ ] `/.forgelm_audit_salt` mod `0600` ve service + hesabı tarafından sahipli. +- [ ] Audit log replikasyon hedefi write-once (S3 Object Lock, + Azure Immutable Blob). +- [ ] KMS audit log'unda quarterly anahtar-rotasyon kanıtı. +- [ ] Hiçbir ForgeLM artefaktı şifrelenmiş substrate dışında + yaşamıyor (teyit etmek için `find` + storage-policy raporları + kullan). + +## 7. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon (Wave 4 / Faz 23) | diff --git a/docs/qms/roles_responsibilities-tr.md b/docs/qms/roles_responsibilities-tr.md new file mode 100644 index 00000000..ba3eb18d --- /dev/null +++ b/docs/qms/roles_responsibilities-tr.md @@ -0,0 +1,98 @@ +# AI Model Yönetimi için Roller & Sorumluluklar + +> EU AI Act Referansı: Madde 17(1)(m) +> Kurumunuzun yapısına uyarlayın. + +## Rol Matrisi + +```mermaid +graph TD + AIO["AI Officer
(Hesap Verebilirlik)"] --> MLL["ML Lead
(Teknik Gözetim)"] + AIO --> CO["Compliance Officer
(Düzenleyici)"] + MLL --> MLE["ML Engineer
(Yürütme)"] + MLL --> DS["Data Steward
(Veri Kalitesi)"] + CO --> DPO["Data Protection Officer
(Gizlilik)"] + + style AIO fill:#ef4444,color:#fff + style MLL fill:#3b82f6,color:#fff + style CO fill:#22c55e,color:#fff +``` + +## Rol Tanımları + +### AI Officer + +| | | +|---|---| +| **Rapor Verir** | CTO / Yönetim Kurulu | +| **Sorumluluk** | AI sistemleri uyumluluğunun genel hesap verebilirliği | +| **Onaylar** | Yüksek-riskli model dağıtımları, risk değerlendirmeleri, olay yükseltmeleri | +| **ForgeLM eşleşmesi** | `risk_assessment.json` inceler, `require_human_approval` gate'ini onaylar | + +### ML Lead / Reviewer + +| | | +|---|---| +| **Rapor Verir** | AI Officer | +| **Sorumluluk** | Eğitim pipeline'ı ve model kalitesinin teknik gözetimi | +| **Onaylar** | Eğitim configleri (PR review), değerlendirme sonuçları, küçük değişimler | +| **ForgeLM eşleşmesi** | `compliance_report.json`, `benchmark_results.json`, `safety_results.json` inceler | + +### ML Engineer + +| | | +|---|---| +| **Rapor Verir** | ML Lead | +| **Sorumluluk** | Config hazırlama, eğitim yürütme, model değerlendirme | +| **Yürütür** | `forgelm --config`, veri hazırlama, debug | +| **ForgeLM eşleşmesi** | YAML configleri oluşturur, eğitim çalıştırır, webhook'ları izler | + +### Data Steward + +| | | +|---|---| +| **Rapor Verir** | ML Lead | +| **Sorumluluk** | Veri kalitesi, yönetişim uyumluluğu, bias değerlendirmesi | +| **Doğrular** | Veri kümesi kalitesi, annotation süreci, temsil edicilik | +| **ForgeLM eşleşmesi** | `data.governance:` config bölümünü doldurur, `data_provenance.json` inceler | + +### Compliance Officer + +| | | +|---|---| +| **Rapor Verir** | AI Officer / Yasal | +| **Sorumluluk** | EU AI Act uyumluluğu, denetim hazırlığı, düzenleyici raporlama | +| **İnceler** | Uyumluluk artefaktları, risk değerlendirmeleri, olay raporları | +| **ForgeLM eşleşmesi** | Evidence bundle (`--compliance-export`) inceler, QMS dokümantasyonunu yürütür | + +### Data Protection Officer (DPO) + +| | | +|---|---| +| **Rapor Verir** | Yasal / Yönetim Kurulu | +| **Sorumluluk** | Kişisel veri koruması, DPIA gözetimi | +| **İnceler** | Veri yönetişim configi (`personal_data_included`, `dpia_completed`) | +| **ForgeLM eşleşmesi** | `data.governance:` kişisel veri alanlarını doğrular | + +## Onay Matrisi + +| Eylem | ML Engineer | ML Lead | AI Officer | Compliance | +|--------|:-----------:|:-------:|:----------:|:----------:| +| Eğitim configi oluştur | ✅ | İncele | — | — | +| Eğitim yürüt (minimal-risk) | ✅ | — | — | — | +| Eğitim yürüt (high-risk) | ✅ | Onayla | Onayla | İncele | +| Model dağıt (minimal-risk) | — | ✅ | — | — | +| Model dağıt (high-risk) | — | ✅ | ✅ | ✅ | +| Olay yönet (Critical) | Soruştur | Koordine et | Karar ver | Rapor et | +| Risk değerlendirmesini güncelle | Taslak | İncele | Onayla | Doğrula | +| Güvenlik eşiklerini değiştir | — | ✅ | Bildir | — | + +## Eğitim Gereksinimleri + +| Rol | Gerekli Eğitim | +|------|------------------| +| Tüm roller | EU AI Act farkındalığı (yıllık) | +| ML Engineer | ForgeLM tooling, güvenlik değerlendirmesi | +| Data Steward | Veri yönetişimi, bias değerlendirmesi | +| AI Officer | Risk yönetimi, düzenleyici uyumluluk | +| Compliance Officer | EU AI Act detaylı gereksinimleri, denetim prosedürleri | diff --git a/docs/qms/sop_change_management-tr.md b/docs/qms/sop_change_management-tr.md new file mode 100644 index 00000000..16c1b237 --- /dev/null +++ b/docs/qms/sop_change_management-tr.md @@ -0,0 +1,150 @@ +# SOP: AI Modelleri için Değişim Yönetimi + +> Standart İşletim Prosedürü — [YOUR ORGANIZATION] +> EU AI Act Referansı: Madde 17(1)(b)(c) +> ISO 27001:2022: A.5.36, A.8.9, A.8.32 +> SOC 2: CC8.1, CC3.4, CC5.3 + +## 1. Amaç + +Fine-tuned modellere, eğitim yapılandırmalarına ve değerlendirme +kriterlerine yapılan değişiklikleri yönetme prosedürünü tanımla. + +## 2. Değişim Kategorileri + +| Kategori | Örnekler | Onay Gerekli | +|----------|---------|------------------| +| **Major** | Yeni base model, yeni trainer tipi, risk kategorisi değişimi | AI Officer + ML Lead | +| **Minor** | Hyperparameter tuning, LoRA rank değişimi, veri kümesi güncellemesi | ML Lead | +| **Patch** | Bug fix, config formatlama, dokümantasyon | ML Engineer (self-approve) | + +## 3. Değişim Prosedürü + +### 3.1 Önerme + +1. Branch oluştur: `git checkout -b change/description` +2. Eğitim configini (YAML) değiştir +3. PR açıklamasında değişim gerekçesini dokümante et +4. Dry-run çalıştır: `forgelm --config new_config.yaml --dry-run` + +### 3.2 İnceleme + +1. `main`'e Pull Request aç +2. PR şunları içermelidir: + - [ ] Neyin değiştiğini gösteren config diff + - [ ] Değişim gerekçesi + - [ ] Model kalitesi/güvenliği üzerinde beklenen etki + - [ ] **Major** değişiklikler için: güncellenmiş risk değerlendirmesi +3. İnceleyici kontroller: + - [ ] Config doğrulanır (`--dry-run` geçer) + - [ ] Güvenlik değerlendirmesi etkin (yüksek-riskli modeller için) + - [ ] Config'de secret veya token yok + +### 3.3 Yürütme + +1. PR'ı merge et (branch protection kuralları başına onay gerektirir) +2. CI/CD pipeline eğitimi tetikler: `forgelm --config job.yaml` +3. ForgeLM yeni koşum için uyumluluk artefaktları otomatik üretir +4. Yeni `audit_log.jsonl` girişleri değişimi izler + +### 3.4 Doğrulama + +1. Yeni model metriklerini önceki sürümle karşılaştır +2. Benchmark skorlarını, güvenlik skorlarını, judge skorlarını incele +3. `require_human_approval: true` ise: dağıtım öncesi açık onay + +### 3.5 Geri Alma + +Yeni model daha kötüyse: +- ForgeLM `auto_revert` eğitim sırasında otomatik geri alma yapar +- Dağıtım sonrası geri alma için: + 1. Önceki model sürümünü yeniden dağıt + 2. Geri almayı olay log'una dokümante et + 3. Kök nedeni soruştur + +## 4. Formal değişim-kontrol mekanizması olarak CI gate'leri + +Wave 4 / Faz 23 genişletmesi: bu bölüm, hangi CI gate'lerinin +operatöre dönük değişim-kontrol kanıtı oluşturduğunu formalize +eder. ISO A.8.32 "bilgi işleme tesisleri ve sistemlere yapılan +değişimler değişim yönetim prosedürlerine tabi olmalıdır" gerektirir; +aşağıdaki tablo ForgeLM-instrumented operatörler için prosedürü +dokümante eder. + +| Gate | Araç | Arıza modu | ISO kontrolü | +|---|---|---|---| +| Lint clean | `ruff check .` + `ruff format --check .` | Stil / sözdizimi sorunları | A.8.28 | +| Pydantic field-description guard | `tools/check_field_descriptions.py --strict forgelm/config.py` | Operatöre dönük config drift | A.5.36 | +| HTTP discipline | `ci.yml`'da `grep`-tabanlı regex guard | Disiplinsiz `requests.*` / `urllib.*` / `httpx.*` çağrısı | A.8.20 | +| Birim + entegrasyon testleri | `pytest -q` | Dokümante edilmiş contract'a regresyon | A.8.29 | +| Coverage zemini | `--cov-fail-under=40` (`pyproject.toml`) | Yeni kod kapsamsız | A.8.29 | +| CLI dry-run | `forgelm --config config_template.yaml --dry-run` | Config schema break | A.8.9 | +| Site-as-tested-surface | `tools/check_site_claims.py --strict` | Pazarlama iddiası koddan drift | A.5.31 | +| Bilingual H2/H3/H4 parity | `tools/check_bilingual_parity.py --strict` | EN/TR docs yapısal drift | A.5.36 | +| Bandit static-security (Wave 4) | `bandit -c pyproject.toml -r forgelm/` | High-severity bulgu | A.8.26, A.8.28 | +| pip-audit nightly (Wave 4) | `pip-audit --format json` | High-severity transitive CVE | A.8.8, A.8.21 | +| SBOM determinism (Wave 4) | `pytest tests/test_supply_chain_security.py` | SBOM içerik drift | A.5.21, A.8.8 | + +Bir değişim her gate geçene kadar `main`'e (veya Wave-style flow'lar +için `development`'a) merge edilemez. PR review template'i +(`.github/PULL_REQUEST_TEMPLATE.md`) her gate'in yeşil olduğunu +açıkça teyit eden bir checkbox gerektirmelidir. + +### 4.1 Change Advisory Board (CAB) yedek olarak Approval Gate + +Yeni eğitim koşumu üreten değişimler için (yapılandırma-yalnız +değişimler vs.), **Madde 14 staging gate** operatörün in-pipeline +CAB'sıdır: + +1. CI eğitim job'u modeli `/staging_model./`'e + yerleştirir. +2. `human_approval.required` audit event yangın eder. +3. Bir reviewer (eğiten DEĞİL) `forgelm approve --run-id ` + çalıştırır. +4. `human_approval.granted` audit event yangın eder; model + `/final_model/`'e promote olur. +5. Reddedilirse `human_approval.rejected` event yangın eder; model + retention süresi dolana kadar staging'de kalır + (`evaluation.approval_retention_days`). + +Bu, operatöre SOC 2 CC8.1 kanıtı verir: her model promotion +attribute, çift kontrollü ve forensic olarak kayıtlıdır. + +### 4.2 Yapılandırma drift tespiti + +`tools/regenerate_config_doc.py` (Phase 16) +`docs/reference/configuration.md`'yi (ve `-tr.md` mirror'ını) +Pydantic schema'dan yeniden üretir. CI diff guard çalıştırır; +karşılık gelen doc güncellemesi olmadan bir config-schema değişimi +build'i fail eder. + +Bu, schema'nın evrildiği ama operatöre dönük doc'un geri kaldığı +"doc drift" arıza modunu kapatır. ISO A.5.36 bunu zorunlu mekanizma +olarak atfeder. + +### 4.3 SBOM drift tespiti + +`tools/generate_sbom.py` (Wave 2 dönemi) her release tag'inde +(`publish.yml`) (OS × Python-version) başına deterministic CycloneDX +1.5 SBOM üretir. Wave 4 bir determinism contract testi ekler +(`tests/test_supply_chain_security.py::TestGenerateSbomDeterministic`). + +Drift tespiti: bir release'in SBOM'u `git tag`'inden tekrarlanabilir. +Bir denetçi bunu istek üzerine yeniden emit edip GitHub release'e +ekli artefakta diff alabilir. + +## 5. Versiyon İzleme + +Her eğitim koşumu üretir: +- ForgeLM versiyonu, config hash, timestamps ile `compliance_report.json` +- Artefakt SHA-256 checksumları ile `model_integrity.json` +- Tam olay geçmişi ile `audit_log.jsonl` + +Herhangi bir modeli tam eğitim yapılandırması ve verisine geri izlemek için bunları kullan. + +## 6. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon | +| 1.1 | 2026-05-05 | Wave 4 / Faz 23 | §4 CI-gates-as-change-control tablosu eklendi (11 gate × ISO kontrolleri); §4.1 Madde 14 approval gate CAB substitute olarak; §4.2 `regenerate_config_doc.py` üzerinden config-drift tespiti; §4.3 `generate_sbom.py` + determinism testi üzerinden SBOM drift tespiti | diff --git a/docs/qms/sop_data_management-tr.md b/docs/qms/sop_data_management-tr.md new file mode 100644 index 00000000..58b04f28 --- /dev/null +++ b/docs/qms/sop_data_management-tr.md @@ -0,0 +1,122 @@ +# SOP: Veri Yönetimi ve Yönetişim + +> Standart İşletim Prosedürü — [YOUR ORGANIZATION] +> EU AI Act Referansı: Madde 17(1)(f), Madde 10 + +## 1. Amaç + +LLM fine-tuning için eğitim verisinin toplanması, annotation'ı, +saklanması ve yönetişimi için standartları tanımla. + +## 2. Kapsam + +Model eğitimi, validation ve değerlendirme için kullanılan tüm veri kümeleri. + +## 3. Roller + +| Rol | Sorumluluk | +|------|---------------| +| **Data Steward** | Veri kalitesi gözetimi, yönetişim uyumluluğu | +| **ML Engineer** | Veri hazırlama, ön işleme, format doğrulama | +| **DPO (Data Protection Officer)** | Kişisel veri değerlendirmesi, DPIA inceleme | + +## 4. Veri Toplama + +### 4.1 Gereksinimler + +- [ ] Veri kaynağını ve toplama yöntemini config'de dokümante et: + ```yaml + data: + governance: + collection_method: "Dahili bilgi tabanından manuel curasyon" + ``` +- [ ] Temsil ediciliği değerlendir: veri hedeflenen dağıtım bağlamını yansıtıyor mu? +- [ ] Coğrafi, demografik ve bağlamsal denge kontrolü +- [ ] Bilinen biasları config'de dokümante et: + ```yaml + data: + governance: + known_biases: "Veri kümesi EU bölgesindeki İngilizce konuşan müşterilere meyilli" + ``` + +### 4.2 Kişisel Veri + +- [ ] Veri kümesinin kişisel veri içerip içermediğini belirle +- [ ] İçeriyorsa: Veri Koruma Etki Değerlendirmesi (DPIA) tamamla +- [ ] Config'de dokümante et: + ```yaml + data: + governance: + personal_data_included: true + dpia_completed: true + ``` +- [ ] Veri minimizasyonu uygula — yalnız gerekli kişisel veriyi dahil et +- [ ] Mümkün olduğunda anonimleştirme/pseudonymization uygula + +## 5. Veri Hazırlama + +### 5.1 Annotation + +- [ ] Annotation sürecini dokümante et: + ```yaml + data: + governance: + annotation_process: "Örnek başına iki annotator, kıdemli annotator tarafından arabuluculuk" + ``` +- [ ] Annotator-arası anlaşma kayıtlarını koru +- [ ] Annotation rehberlerini sürüm kontrol et + +### 5.2 Kalite Kontrolleri + +ForgeLM otomatik kontroller: +- Veri kümesi fingerprinting (SHA-256 hash, boyut, timestamp) +- Trainer tipine göre format doğrulama (SFT, DPO, KTO, GRPO) +- Metin temizleme (`clean_text: true`) +- **ForgeLM audit pipeline (v0.5.0+, `forgelm audit `; legacy + `forgelm --data-audit` deprecated, v0.7.0'da kaldırılma planlandı)** — + per-split sample sayıları, uzunluk dağılımı, top-3 dil tespiti, + near-duplicate oranı (varsayılan 64-bit simhash, veya >50K-row + corpora için `--dedup-method minhash` üzerinden **MinHash LSH**), + cross-split leakage check, severity tier'lı PII flag sayıları + (email / phone / Luhn-validated credit card / IBAN / TR–DE–FR–US + national IDs) ve always-on **secrets/credentials scan** (AWS / + GitHub / Slack / OpenAI / Google / JWT / private-key headers) ile + `data_audit_report.json` üretir; opsiyonel **heuristic quality + filter** (`--quality-filter`) `quality_summary` block ekler. + `forgelm audit --croissant` raporun yanına Google Croissant 1.0 + veri kümesi kartı emit eder. Trainer'ın `output_dir`'inde mevcut + olduğunda rapor EU AI Act Madde 10 yönetişim artefaktına otomatik + inline edilir — operatörler bundle'ı kendine yeterli tutmak için + audit'i eğitim **öncesi** çalıştırmalıdır. Phase'ler 11 + 11.5 + + 12 + 12.5 birlikte `v0.5.0` consolidation release'inde gönderildi + (PyPI 2026-04-30). +- **ForgeLM ingestion pipeline (v0.5.0+, `forgelm ingest`)** — ham + PDF / DOCX / EPUB / TXT / Markdown'u SFT-ready JSONL'e çevirir, + paragraph / sliding / **markdown-aware** chunking, DOCX→markdown + tablo koruması, opsiyonel **Presidio ML-NER** adapter (`--pii-ml`, + `[ingestion-pii-ml]` extra) ve chunk'lar depolamaya ulaşmadan + tespit edilen PII / credential span'larını redact etmek için + `--pii-mask` / `--secrets-mask` / `--all-mask` shorthand'leri. + +Manuel kontroller: +- [ ] `forgelm audit` çalıştır ve `data_audit_report.json` incele: + cross-split leakage > %0, near-duplicate oranı, beklenmedik dil + karışımı, PII flag sayıları, **`secrets_summary` (sıfır olmayan + herhangi bir sayı stop-the-line bir olaydır; eğitim öncesi + remediate et)** ve `--quality-filter` kullanıldıysa + `quality_summary`. +- [ ] Sample inceleme: 50+ rastgele örnek incele. +- [ ] Etiket doğruluğunu doğrula (preference/KTO verisi için). + +## 6. Veri Saklama ve Retention + +- Eğitim verisi sürüm kontrollü veya immutable depolamada saklanır +- SHA-256 fingerprint her eğitim koşumu için `data_provenance.json`'da kaydedilir +- Eğitim verisi en az **5 yıl** model artefaktlarıyla birlikte saklanır +- Erişim yetkili ML ekibi üyeleriyle sınırlıdır + +## 7. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon | diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md new file mode 100644 index 00000000..b017fad3 --- /dev/null +++ b/docs/qms/sop_incident_response-tr.md @@ -0,0 +1,229 @@ +# SOP: AI Modelleri için Olay Yanıtı + +> Standart İşletim Prosedürü — [YOUR ORGANIZATION] +> EU AI Act Referansı: Madde 17(1)(h)(i) +> ISO 27001:2022: A.5.24, A.5.25, A.5.26, A.5.27, A.6.8, A.8.15, A.8.16 +> SOC 2: CC4.2, CC7.3, CC7.4, CC7.5, CC9.2 + +## 1. Amaç + +Dağıtılan fine-tuned modeller için güvenlik olaylarını, model +arızalarını, **güvenlik olaylarını** ve düzeltici eylemleri +yönetme prosedürünü tanımla. Wave 4 / Faz 23 genişletmesi: §6 +güvenlik-olay playbook'unu (audit-chain integrity, credential leak, +supply-chain CVE, webhook target compromise, GDPR DSAR'lar) mevcut +AI-safety olay akışı yanında kapsar. + +## 2. Olay Sınıflandırması + +| Önem | Tanım | Yanıt Süresi | Örnek | +|----------|-----------|--------------|---------| +| **Critical** | Model zararlı, ayrımcı veya tehlikeli çıktı üretir | Anında (< 1 saat) | Güvenlik classifier arızası, zararlı içerik üretimi | +| **High** | Model iş kararlarını etkileyen yanlış çıktı üretir | < 4 saat | Yanlış politika bilgisi, yanlış finansal veri | +| **Medium** | Model kalitesinde düşüş tespit edildi | < 24 saat | Eşik altı doğruluk düşüşü, hallucination artışı | +| **Low** | Küçük kalite sorunu, kozmetik | < 1 hafta | Format hataları, ara sıra alakasız yanıtlar | + +## 3. Olay Yanıt Prosedürü + +### 3.1 Tespit + +Olaylar şunlardan tespit edilebilir: +- Runtime izleme alarmları (`monitoring.alert_on_drift: true` ise) +- Kullanıcı/operatör raporları +- Periyodik kalite denetimleri +- ForgeLM webhook arıza bildirimleri + +### 3.2 Anında Eylemler + +**Critical/High için:** +1. [ ] **Durdur**: Modeli üretimden kaldır veya fallback'e geç +2. [ ] **Dokümante et**: Olay detaylarını kaydet (girdi, çıktı, timestamp, etki) +3. [ ] **Bildir**: AI Officer ve etkilenen paydaşları uyar +4. [ ] **Koru**: İnceleme için model artefaktlarını ve logları sakla + +### 3.3 Soruşturma + +1. [ ] Bildirilen girdi ile sorunu tekrarla +2. [ ] Eğitim koşumu detayları için `audit_log.jsonl` kontrol et +3. [ ] Orijinal eğitimden `safety_results.json` incele +4. [ ] Model davranışını baseline ile karşılaştır +5. [ ] Kök nedeni tanımla (veri sorunu, eğitim sorunu, dağıtım sorunu) + +### 3.4 Düzeltici Eylem + +| Kök Neden | Eylem | +|-----------|--------| +| Eğitim verisi sorunu | Veriyi düzelt → yeniden eğit → yeniden değerlendir → yeniden dağıt | +| Güvenlik regresyonu | Önceki model versiyonuna geri dön | +| Yapılandırma hatası | Configi düzelt → düzeltilmiş parametrelerle yeniden eğit | +| Dağıtım hatası | Dağıtımı düzelt, model iyi | + +### 3.5 Olay Sonrası + +1. [ ] Kök nedeni ve çözümü dokümante et +2. [ ] Yeni risk tespit edildiyse risk değerlendirmesini güncelle +3. [ ] Olay senaryosunu kapsamak için güvenlik test promptlarını güncelle +4. [ ] Bu SOP'yu gerekirse incele ve güncelle +5. [ ] EU AI Act için: ciddi olayları **15 gün** içinde ilgili otoriteye rapor et + +## 4. Güvenlik olayları — Wave 4 / Faz 23 genişletmesi + +Wave 4 ISO 27001 / SOC 2 alignment kapanışı aşağıdaki güvenlik-olay +playbook'unu ekler. AI-safety olayları (§§1–3) ve güvenlik olayları +ikisi de bu SOP'tan akar; ayırt edici tespit olay sınıfıdır. + +### 4.1 Audit-chain integrity ihlali + +**Tetik:** `forgelm verify-audit` sıfır-olmayan çıkar (chain hash +mismatch, manifest sidecar truncation, HMAC signature mismatch). + +**Önem:** Critical. + +**Runbook:** + +1. [ ] **İzole et** etkilenen `` — daha fazla yazımı + önlemek için dizin üzerinde `chmod 0500`. +2. [ ] **Kanıtı koru** — `audit_log.jsonl`, + `audit_log.manifest.json`, `.sha256` sidecar'ı (varsa) ve + `/.forgelm_audit_salt`'ı write-once forensic + substrate'e (S3 Object Lock, Azure Immutable Blob) kopyala. +3. [ ] **Son güvenilen girişi tanımla** — `forgelm verify-audit + --until-line N` ile bisect ederek ilk kötü satırı bul; o + satırdan önce her şey forensic olarak güvenilirdir, sonra + her şey tainted sayılmalıdır. +4. [ ] **Bildir** AI Officer + Güvenlik ekibi + DPO (kötü satırdan + sonra herhangi bir PII-bearing event varsa). +5. [ ] **Karar ver** tainted-tail girişlerini kanıt olarak (önerilen) + saklamak veya geri sarmak hakkında. +6. [ ] **Denetle** şüpheli zaman penceresi sırasında `` + substrate'ine yetkisiz yazma erişimi için IdP'yi. + +### 4.2 Credential leak tespit edildi + +**Tetik:** `forgelm audit` `_SECRET_PATTERNS` regex'i eğitim +korpusunda veya webhook log'unda bir credential ile eşleşir; VEYA +external CVE / breach disclosure kullanılan bir token'a atıf yapar. + +**Önem:** Critical. + +**Runbook:** + +1. [ ] **Sızdırılan credential'ı hemen rotate et** issuing + authority'de (HF Hub token, GitHub PAT, Slack webhook, OpenAI + API key, AWS access key, vb.). +2. [ ] **`forgelm purge --row-id ` çalıştır** sızdırılan + credential satırını içeren her korpus karşı. +3. [ ] **Koşumu memorisation-tainted olarak flag'le** — + `data.erasure_warning_memorisation` olayı bunu dokümante eder. +4. [ ] **Rotasyonu dokümante et** KMS audit log'unda; ForgeLM + `data.erasure_completed` olayı timestamp'ine geri bağla. +5. [ ] **Sıfırdan yeniden eğit** yüksek-riskli dağıtımlar için. +6. [ ] **Eğitim-veri-onboarding checklist'ini güncelle** `forgelm + audit --secrets` pre-flight gerektirmek için. + +### 4.3 Supply-chain CVE flag'lendi + +**Tetik:** `pip-audit` nightly high-severity fail eder (Wave 4 / Faz +23 bu gate'i tanıttı); VEYA ForgeLM'in kullandığı bir bağımlılık +üzerinde CVE advisory düşer. + +**Önem:** High. + +**Runbook:** + +1. [ ] **Güvenli sürüme pin'le** `pyproject.toml`'da. +2. [ ] **SBOM'u yeniden inşa et** yeni pinlenen küme için + (`tools/generate_sbom.py`). +3. [ ] **Bağımlı artefaktları yeniden üret** — etkilenen dep'i + tüketen bağımlı eğitim pipeline'larını yeniden çalıştır. +4. [ ] **Operatörleri bilgilendir** etkilenen dep ile bir model + eğitim-anı env'inde zaten gönderilmişse + (`compliance_report.json` env'i listeler). +5. [ ] **Tracking ticket aç** CVE id + SBOM diff + etkilenen + koşumlar ile (etkilenenleri tanımlamak için audit log'un + `compliance.config_hash`'ini kullan). + +### 4.4 Webhook target ele geçirildi + +**Tetik:** Slack / Teams / custom-webhook recipient bir ihlali +teyit eder; VEYA `safe_post` `_mask` redact'lı Authorization +header'larını gösterir ki bir saldırgan bunları gözlemlemiş +olabilir. + +**Önem:** High. + +**Runbook:** + +1. [ ] **`webhook.secret_env`'i hemen rotate et**. +2. [ ] **Lifecycle olaylarını yeniden emit et** audit chain'den + saldırganın olayları recipient'a splice etmediğini teyit + etmek için. (`forgelm verify-audit --replay-since + ` v0.6.0+ aracıdır; v0.5.5 için chain'i manuel + yürü.) +3. [ ] **`safe_post` error log'larını kontrol et** rotation sonrası + redact'lı Authorization header'lar için — saldırganın artık + geçerli token tutmadığını teyit et. +4. [ ] **Receiving system log'larını denetle** splice'lı olaylar + tarafından tetiklenen beklenmedik eylemler için (Slack + channel postaları, Teams kartları, Jira ticket'lar). + +### 4.5 GDPR Madde 15 (erişim hakkı) talebi + +**Tetik:** Veri sahibi operatörün DSAR portalı veya iletişim formu +üzerinden erişim talebi gönderir. + +**Önem:** Medium (regülasyon deadline 30 gün). + +**Runbook:** + +1. [ ] **Sahip kimliğini doğrula** operatörün DSAR prosedürü başına. +2. [ ] **`forgelm reverse-pii --query + --type data/*.jsonl --output-dir ` çalıştır**. +3. [ ] **Hash-mask scan** ForgeLM'in maskelediği herhangi bir + korpus için (`forgelm reverse-pii --query + --salt-source per_dir --output-dir `). +4. [ ] **Yanıt mektubu oluştur** DSAR şablonu başına; koşumun + `data.access_request_query` audit-event id'sine atıf yap. +5. [ ] **Yanıt mektubunu sakla** audit chain yanında. + +### 4.6 GDPR Madde 17 (silme hakkı) talebi + +**Tetik:** Sahip silme talebi gönderir. + +**Önem:** Medium (regülasyon deadline 30 gün; bazı EU üye +ülkeleri daha kısa yanıt gerektirir). + +**Runbook:** + +1. [ ] **Kimliği doğrula** operatörün DSAR prosedürü başına. +2. [ ] **`forgelm purge --row-id --corpus + data/.jsonl --output-dir ` çalıştır**. +3. [ ] **`data.erasure_warning_memorisation` olayını kontrol et** — + satırı eğiten bir model `final_model/` artefaktına sahipse, + memorisation residual riski uygulanır. +4. [ ] **Memorisation caveat'ı sahibe bildir** yanıt mektubunda + (şablon: "satırı sildik ama önceden eğitilmiş bir model bunu + memorise etmiş olabilir; uygun şekilde yeniden eğiteceğiz veya + ek safeguard'lar uygulayacağız"). +5. [ ] **Yüksek-riskli dağıtımlar için** (finansal danışmanlık, tıbbi + triaj): sıfırdan yeniden eğit. +6. [ ] **Audit-event chain'ini sakla** completion + warning'lerin + fired olduğunu kanıtlayan. + +## 5. Ciddi Olay Raporlama (EU AI Act) + +Madde 73 altında, sağlayıcılar ciddi olayları piyasa gözetim +otoritelerine raporlamalıdır. "Ciddi bir olay" şunları içerir: +- Ölüm veya sağlığa ciddi zarar +- Temel haklara ciddi ihlal +- Kritik altyapıya ciddi aksaklık + +**Rapor:** Etkilenen EU üye ülkesinin ulusal piyasa gözetim otoritesi +**Süre:** Farkına varıldıktan sonraki 15 gün içinde + +## 6. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon | +| 1.1 | 2026-05-05 | Wave 4 / Faz 23 | §4 güvenlik-olay playbook'u eklendi (audit-chain integrity, credential leak, supply-chain CVE, webhook compromise, GDPR Md. 15/17 DSAR'ları); başlıkta ISO 27001:2022 + SOC 2 kontrol haritalaması | diff --git a/docs/qms/sop_model_training-tr.md b/docs/qms/sop_model_training-tr.md new file mode 100644 index 00000000..58031c32 --- /dev/null +++ b/docs/qms/sop_model_training-tr.md @@ -0,0 +1,80 @@ +# SOP: Model Eğitimi ve Fine-Tuning + +> Standart İşletim Prosedürü — [YOUR ORGANIZATION] +> EU AI Act Referansı: Madde 17(1)(b)(c)(d) + +## 1. Amaç + +Her adımda kalite, güvenlik ve uyumluluğu sağlayarak dil modellerinin +fine-tuning'i için standart prosedürü tanımla. + +## 2. Kapsam + +ForgeLM veya eşdeğer araçlar kullanan tüm LLM fine-tuning aktivitelerine uygulanır. + +## 3. Roller + +| Rol | Sorumluluk | +|------|---------------| +| **ML Engineer** | Config hazırlar, veri hazırlar, eğitim yürütür | +| **ML Lead / Reviewer** | Config ve değerlendirme sonuçlarını inceler, dağıtımı onaylar | +| **Data Steward** | Veri kalitesi ve yönetişim uyumluluğunu doğrular | +| **AI Officer** | Yüksek-riskli modeller için final onay | + +## 4. Prosedür + +### 4.1 Eğitim Öncesi + +- [ ] Hedeflenen amacı tanımla ve config'de dokümante et (`compliance.intended_purpose`) +- [ ] Risk değerlendirmesini tamamla (YAML'de `risk_assessment:` bölümü) +- [ ] Veri kümesi kalitesini doğrula (data governance raporu çalıştır) +- [ ] Eğitim configini incele ve onayla (ML Lead PR review) +- [ ] Dry-run doğrulaması: `forgelm --config job.yaml --dry-run` + +### 4.2 Eğitim Yürütme + +- [ ] Eğitimi çalıştır: `forgelm --config job.yaml --output-format json` +- [ ] Webhook bildirimleri veya TensorBoard ile izle +- [ ] Eğitim otomatik artefaktlar üretir: + - `audit_log.jsonl` — olay izi + - `compliance_report.json` — tam audit kaydı + - `data_provenance.json` — veri kümesi fingerprintleri + +### 4.3 Eğitim Sonrası Değerlendirme + +ForgeLM tarafından otomatik: +- [ ] Loss tabanlı değerlendirme (eval_loss vs baseline) +- [ ] Benchmark değerlendirmesi (lm-eval-harness görevleri) +- [ ] Güvenlik değerlendirmesi (Llama Guard classifier) +- [ ] LLM-as-Judge kalite skorlama + +Manuel: +- [ ] ML Lead değerlendirme sonuçlarını inceler +- [ ] Eğer `require_human_approval: true` ise, ML Lead `checkpoints/compliance/` inceler ve dağıtımı onaylar +- [ ] 10+ temsili prompt üzerinde model çıktılarını spot-check et + +### 4.4 Dağıtım Onayı + +- [ ] **minimal-risk** için: ML Lead onayı yeterli +- [ ] **limited-risk** için: ML Lead + AI Officer onayı +- [ ] **high-risk** için: ML Lead + AI Officer + Yasal/Compliance review +- [ ] Operatör (deployer) talimatları (`deployer_instructions.md`) dağıtım ekibiyle paylaşılır +- [ ] Model bütünlüğü doğrulanır: `model_integrity.json` checksumları eşleşir + +### 4.5 Kayıt Tutma + +- Uyumluluk artefaktları en az **5 yıl** saklanır (veya regülasyon gerektirdiği kadar) +- Kanıt paketi: `forgelm --config job.yaml --compliance-export ./archive/` +- Mümkünse immutable/append-only depolamada sakla + +## 5. İstisnalar + +Bu SOP'tan herhangi bir sapma dokümante edilmeli ve AI Officer tarafından onaylanmalıdır. + +## 6. İnceleme + +Bu SOP yıllık olarak veya eğitim pipeline'ında önemli değişiklikler olduğunda incelenir. + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon | From 32149f191fb4bfa31e8e83a2c6204d1a2137e115 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 00:24:56 +0300 Subject: [PATCH 04/12] feat(wave-4-faz-26): QMS bilingual mirrors + compliance_summary cleanup + anchor checker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements Faz 26 of the closure plan. Three work areas: QMS bilingual mirrors (part 2: remaining 3 new-Faz-23 TR mirrors) ================================================================= * docs/qms/access_control-tr.md (new) — Wave 4 / Faz 23 QMS doc TR mirror. * docs/qms/risk_treatment_plan-tr.md (new) — TR mirror. * docs/qms/statement_of_applicability-tr.md (new) — TR mirror. Combined with part 1 (commit 8be6dab on this branch), all 10 QMS docs now ship EN+TR pairs. compliance_summary.md cleanup (Faz 26 task #3 + #4) ==================================================== * docs/reference/compliance_summary.md — full rewrite: - Added H1 + scope blockquote. - Removed all 11 stale ``forgelm/x.py#L33``-style line anchors; replaced with module-path + symbol-name references that survive refactors. - Updated "50 prompts × 3 categories" → "140 × 6" to match the Wave 1 v0.5.0 safety-prompt expansion. - Linked from docs/qms/README.md + docs/qms/README-tr.md (See also section). - Restructured around the EU AI Act + ISO + SOC 2 trio so the summary stays a useful index even after Faz 30's broader user-facing doc sweep. Anchor resolution checker (Faz 26 task #5) ========================================== * tools/check_anchor_resolution.py (new) — markdown link resolution guard. Walks every *.md under docs/ (excluding the gitignored docs/analysis/ research tree by default) and validates: - repo-relative paths resolve to existing files; - cross-file anchors resolve against GFM-style header slugs; - self-anchors (#section) resolve in the same file; - line-number anchors on code files (#L42) are flagged as brittle (drift on every refactor); - external URLs / mailto / SPA hash-router fragments / tel / javascript skipped as out-of-scope. * tests/test_check_anchor_resolution.py (new) — 21 tests pinning slugify behaviour, resolution paths (relative + anchor + skip patterns), and CLI surface (--strict, --quiet, --exclude). CI integration policy ===================== The closure plan calls for a CI gate, but the live tree currently has 36 pre-existing broken links (mostly forgelm/data_audit.py and forgelm/cli.py references that drifted when Faz 14 + Faz 15 split those modules into packages). Strict-mode wire-up would turn the build red on day one. Decision: ship the tool with **advisory mode by default** (WARN + exit 0) and **--strict opt-in** (FAIL + exit 1). CI wire-up to use --strict lands as part of Faz 30 after the broken-link cleanup completes. Operators / contributors can still run the tool locally: python3 tools/check_anchor_resolution.py # advisory python3 tools/check_anchor_resolution.py --strict # CI form Validation ========== - ruff format + ruff check clean. - pytest 1376 → 1397 (+21 anchor-checker tests). - python3 tools/check_bilingual_parity.py --strict → 13/13 OK. - forgelm --config config_template.yaml --dry-run → green. Closure-plan reference: Faz 26 task list at docs/analysis/code_reviews/closure-plan-202604300906.md:879-902. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/qms/README-tr.md | 6 + docs/qms/README.md | 6 + docs/qms/access_control-tr.md | 234 ++++++++++++++++ docs/qms/risk_treatment_plan-tr.md | 212 ++++++++++++++ docs/qms/statement_of_applicability-tr.md | 168 ++++++++++++ docs/reference/compliance_summary.md | 320 +++++++++++++--------- tests/test_check_anchor_resolution.py | 230 ++++++++++++++++ tools/check_anchor_resolution.py | 306 +++++++++++++++++++++ 8 files changed, 1360 insertions(+), 122 deletions(-) create mode 100644 docs/qms/access_control-tr.md create mode 100644 docs/qms/risk_treatment_plan-tr.md create mode 100644 docs/qms/statement_of_applicability-tr.md create mode 100644 tests/test_check_anchor_resolution.py create mode 100644 tools/check_anchor_resolution.py diff --git a/docs/qms/README-tr.md b/docs/qms/README-tr.md index 4c7579aa..71e2ed0f 100644 --- a/docs/qms/README-tr.md +++ b/docs/qms/README-tr.md @@ -24,6 +24,12 @@ 4. Yasal/uyumluluk ekibinizle inceleyin ve onaylayın 5. ForgeLM'in otomatik artefaktlarını kanıt olarak referans verin +## Bkz. + +- [Uyumluluk özeti](../reference/compliance_summary.md) — özlü EU AI Act + ISO + SOC 2 kanıt haritası. +- [ISO / SOC 2 operatör (deployer) rehberi](../guides/iso_soc2_deployer_guide-tr.md) — Wave 4 denetim cookbook'u. +- [Audit event kataloğu](../reference/audit_event_catalog-tr.md) — tam event vocabulary. + ## ForgeLM Otomatik Artefaktları (QMS'ye Eşleşir) | QMS Gereksinimi | ForgeLM Artefaktı | Üreten | diff --git a/docs/qms/README.md b/docs/qms/README.md index e7e21b17..d2c7726b 100644 --- a/docs/qms/README.md +++ b/docs/qms/README.md @@ -24,6 +24,12 @@ 4. Review and approve with your legal/compliance team 5. Reference ForgeLM's automated artifacts as evidence +## See also + +- [Compliance summary](../reference/compliance_summary.md) — concise EU AI Act + ISO + SOC 2 evidence map. +- [ISO / SOC 2 deployer guide](../guides/iso_soc2_deployer_guide.md) — Wave 4 audit cookbook. +- [Audit event catalog](../reference/audit_event_catalog.md) — full event vocabulary. + ## ForgeLM Automated Artifacts (Maps to QMS) | QMS Requirement | ForgeLM Artifact | Generated By | diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md new file mode 100644 index 00000000..22a57924 --- /dev/null +++ b/docs/qms/access_control-tr.md @@ -0,0 +1,234 @@ +# QMS: Erişim Kontrolü + +> Kalite Yönetim Sistemi rehberi — [YOUR ORGANIZATION] +> ISO 27001:2022 referansları: A.5.15, A.5.16, A.5.17, A.5.18, A.8.2, A.8.5 +> SOC 2 referansları: CC1.5, CC6.1, CC6.2, CC6.3, CC6.5, CC8.1 + +## 1. Amaç + +Operatörün kimlikleri, credential'ları ve erişim haklarını ForgeLM'in +audit-trail attribution modeli etrafında nasıl organize ettiğini +tanımla. + +**Çekirdek contract:** ForgeLM'in emit ettiği her olay bir +`FORGELM_OPERATOR` kimliğiyle damgalanır. Audit chain operatör +değerini issue eden identity-management substrate'i kadar güçlüdür. +Aynı `FORGELM_OPERATOR`'ı paylaşan iki operatör chain'de ayırt +edilemez — bu operatör-tarafı kimlik hijyen başarısızlığıdır, +ForgeLM defect değildir. + +## 2. Kapsam + +Aşağıdaki kimlik-taşıyan yüzeyler: + +| Yüzey | Kimliği taşıyan | +|---|---| +| `FORGELM_OPERATOR` env var | Her audit-log girişine damgalanır | +| Approval gate (`forgelm approve` / `reject`) | Onay anında operatör id | +| Reverse-PII / purge subcommand'ları | Her Madde 15 / 17 olayında operatör id | +| Webhook lifecycle olayları | Operatör id webhook payload'a yayılır | +| CI runner identity | CI runner'ın service-account credential'ı `FORGELM_OPERATOR`'a bound operatörün "gerçek" kimliğidir | + +ForgeLM kendisi RBAC, MFA veya directory federation uygulamaz — +bunlar operatörün IdP'sinde (Okta, Azure AD, Google Workspace, +Keycloak) yaşar. ForgeLM'in yaptığı her audit girişini operatörün +`FORGELM_OPERATOR`'a koyduğu string'e bağlamaktır, böylece IdP +audit log'u ve ForgeLM audit log'u çapraz-referanslanabilir. + +## 3. Operatör kimliği contract'ı + +### 3.1 Resolution sırası + +ForgeLM operatör kimliğini audit-event emit anında resolve eder: + +1. `FORGELM_OPERATOR` env var (tercih edilen — açık set). +2. `getpass.getuser()` (POSIX kullanıcı adı fallback). +3. `FORGELM_ALLOW_ANONYMOUS_OPERATOR=1` kimlik olmadan emit'e izin + verir (yalnız kısa-ömürlü test koşumları içindir; aksi takdirde + ConfigError). + +### 3.2 Gerekli form + +Free-text human name yerine **machine-readable, namespaced +identifier** kullanın: + +| Pattern | Örnek | Neden | +|---|---|---| +| `//` | `prod/training/gh-runner-42` | CI runner pool'a haritalanır | +| `` | `alice@acme.example` | İnsan-driven approval için OK | +| `oidc::` | `oidc:gha:repo:Acme/forgelm-pipelines:ref:refs/heads/main` | OIDC-token-bound | + +Anti-patterns: + +- `root` / `cemil` / `admin` — belirsiz, rotate edilemez. +- Boş / `unknown` — audit attribution amacını bozar. + +### 3.3 Personel değiştiğinde rotasyon + +ForgeLM'in audit chain'i **immutable**'dir. Eski `FORGELM_OPERATOR` +ID'leri sonsuza kadar chain'de kalır. Bir operatör ayrıldığında: + +1. **CI runner kimliklerini iptal et** (OIDC trust'ı rotate et / + IAM rolünü sil). +2. **Aynı `FORGELM_OPERATOR` string'ini yeniden issue ETME** yeni bir + kişiye — onlara yeni bir tanımlayıcı verin böylece gelecek + girişler ayrılan-personel girişleriyle karıştırılmaz. +3. **Önceki eylemleri denetle** tanımlayıcılarına grep ederek; bu + sizin sonlandırma-günü forensic kaydınızdır. + +### 3.4 `FORGELM_AUDIT_SECRET` rotasyonu + +HMAC-chain anahtarı (audit koşumu başına; gerçek imzalama anahtarını +türetmek için per-output-dir salt ile XOR'lanır — bkz. +`forgelm/cli/subcommands/_purge.py:_resolve_salt`). Tier-1 secret +olarak ele al: + +- **Uzunluk:** 32+ rastgele bayt (256 bit entropi). +- **Substrate:** KMS / Vault / eşdeğer. VCS'e checked-in `.env` asla. +- **Rotasyon cadence'i:** quarterly önerilen; şüpheli compromise'da + hemen. +- **Rotasyon prosedürü:** yeni değer sonraki pipeline koşumundan + ÖNCE env'e iner. Eski koşumların chain'leri eski anahtarla + doğrulanabilir kalır (audit log girişleri emit anındaki anahtarla + bound; ForgeLM chain üzerinde anahtar migration'u DESTEKLEMEZ — + tasarımdan). + +## 4. CI runner identity binding + +ForgeLM'i CI'dan çalıştırırken önerilen pattern: + +### 4.1 GitHub Actions + +```yaml +jobs: + train: + runs-on: ubuntu-latest + permissions: + id-token: write # OIDC + contents: read + env: + FORGELM_OPERATOR: gha:${{ github.repository }}:${{ github.workflow }}:run-${{ github.run_id }} + FORGELM_AUDIT_SECRET: ${{ secrets.FORGELM_AUDIT_SECRET }} + steps: + - uses: actions/checkout@v5 + - run: pip install forgelm + - run: forgelm --config config.yaml +``` + +**Bu binding'in nedeni:** run id workflow yürütümü başına benzersizdir; +bir denetçi tek bir audit-log girişini saniyeler içinde GitHub Actions +run sayfasına korelate edebilir. + +### 4.2 GitLab CI + +```yaml +train: + script: + - export FORGELM_OPERATOR="gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" + - forgelm --config config.yaml +``` + +### 4.3 Jenkins + +```groovy +pipeline { + agent any + environment { + FORGELM_OPERATOR = "jenkins:${env.JOB_NAME}:build-${env.BUILD_ID}" + FORGELM_AUDIT_SECRET = credentials('forgelm-audit-secret') + } + stages { + stage('Train') { steps { sh 'forgelm --config config.yaml' } } + } +} +``` + +## 5. OS-seviyesi izolasyon + +Tek bir eğitim-host içinde, birden fazla pipeline output dizinini +PAYLAŞMAMALIDIR: + +- Pipeline başına bir Unix kullanıcı (`forgelm-prod`, + `forgelm-staging`, `forgelm-research`). +- Her pipeline'ın dizininde `chmod 0700 `. +- ``'deki audit log'ları dizinin perm'ini devralır; + KMS substrate'iniz read access gerektirmiyorsa bireysel + chmod yapma. +- `.forgelm_audit_salt` dosyası ForgeLM'in `_resolve_salt`'taki + atomic O_EXCL creation'ı başına `0600` (sadece owner-read). + +## 6. Approval gate identity ayrılığı + +Madde 14 staging onaylayan operatörün eğiten operatörden farklı +olmasını gerektirir (görev ayrılığı — ISO A.5.3, SOC 2 CC1.5): + +```bash +# Job 1 — eğitim (CI runner identity) +FORGELM_OPERATOR="gha:Acme/pipelines:training:run-42" \ + forgelm --config config.yaml + +# Job 2 — onay (insan reviewer identity) +FORGELM_OPERATOR="alice@acme.example" \ + forgelm approve --run-id +``` + +ForgeLM iki kimliğin farklı olduğunu zorlamaz — bu operatör-tarafı +IdP kontrolüdür. Audit chain her ikisini kaydeder, böylece bir +denetçi ihlalleri tespit edebilir: + +```bash +# Trainer == approver olduğu approval'ları bul (görev ayrılığı ihlali) +forgelm verify-audit --output-dir ./outputs --json | jq ' + .events[] | + select(.event == "human_approval.granted") | + .matching_training_event = ( + [.|.run_id == .. .run_id and .event == "training.started"] | first + ) | + select(.operator == .matching_training_event.operator) | + .' +``` + +## 7. Webhook secret ayrılığı + +Webhook URL'leri ve HMAC-imzalama anahtarları env'e ait, asla +YAML'a değil: + +```yaml +webhook: + url_env: SLACK_WEBHOOK_URL # env'den resolve + secret_env: FORGELM_WEBHOOK_SECRET # HMAC body sign + timeout: 10 +``` + +Webhook lifecycle olayları (`notify_started`, `notify_succeeded`, +`notify_failed`, `notify_reverted`, `notify_awaiting_approval` — +Phase 8 vocabulary) `FORGELM_OPERATOR` kimliğini webhook payload'a +taşır böylece alıcı sistem (Slack, Teams, custom incident-management) +bildirimi attribute edebilir. + +## 8. Doğrulama checklist'i + +Erişim-kontrolü kanıtını yürüyen operatör denetçi için: + +- [ ] Her CI runner pipeline'ı `FORGELM_OPERATOR`'ı machine-readable + bir namespace'den set ediyor. +- [ ] İki aktif pipeline aynı `FORGELM_OPERATOR` değerini paylaşmıyor. +- [ ] `FORGELM_AUDIT_SECRET` KMS / Vault substrate'inde yaşıyor, + VCS'de veya düz `.env` dosyasında değil. +- [ ] Quarterly KMS audit log'u `FORGELM_AUDIT_SECRET` rotasyonu + gösteriyor. +- [ ] Son 90 gündeki her `human_approval.granted` event'i için + approval-gate kimliği training kimliğinden farklı (sample-audit, + exhaustive değil). +- [ ] `` izin maskeleri: `0700` (dizin), + `.forgelm_audit_salt` `0600`. +- [ ] Chain'de `FORGELM_OPERATOR=root` / `=admin` / `=unknown` + eventi yok. +- [ ] Webhook URL'leri ve HMAC anahtarları env-resolved (`url_env`, + `secret_env`); committed YAML'da plaintext URL yok. + +## 9. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon (Wave 4 / Faz 23) | diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md new file mode 100644 index 00000000..79a23a8e --- /dev/null +++ b/docs/qms/risk_treatment_plan-tr.md @@ -0,0 +1,212 @@ +# QMS: Risk Treatment Plan (RTP) + +> Kalite Yönetim Sistemi — [YOUR ORGANIZATION] +> ISO 27001:2022 referansları: A.5.7, A.5.8, A.5.9, A.5.31, A.6.8, A.8.8, A.8.30 +> SOC 2 referansları: CC3.2, CC3.3, CC9.1, CC9.2 + +## 1. Amaç + +ForgeLM-introduced'ın operatörün eğitim pipeline'ına getirdiği +riskleri, ForgeLM'in out-of-the-box gönderdiği treatment'ları ve +operatörün kabul ettiği veya daha fazla mitigate ettiği residual +riski dokümante et. + +Bu yaşayan bir dokümandır — her quarterly risk review'da ve +herhangi bir önemli olaydan sonra (`pipeline.failed`, +`data.erasure_failed`, `audit.classifier_load_failed`, +serious-incident raporu) güncelle. + +## 2. Metodoloji + +ISO 27005 risk-management prensiplerini takip ediyor: + +- **Likelihood** (L): riskin ne sıklıkta materialize olabileceği — + `Low` (yıllık), `Med` (quarterly), `High` (aylık+). +- **Impact** (I): materialize olursa blast radius — `Low` + (operatör nuisance), `Med` (tek koşum / tek özne), `High` + (regulator-reportable olay, model recall). +- **Inherent risk** = L × I, herhangi bir treatment'tan önce. +- **Residual risk** = L × I, ForgeLM'in gönderdiği treatment + + operatör-tarafı kontrollerden sonra. + +Residual `High`, AI Officer tarafından açık **risk acceptance** +gerektirir (bkz. `roles_responsibilities.md`). + +## 3. Risk register + +ForgeLM'in kendi tehdit modelinin tanımladığı risklerle pre-populated. +Operatörün compliance ekibi §4 altına organizasyon-spesifik satırlar +ekler. + +### 3.1 Eğitim-pipeline riskleri + +#### R-01 Eğitim-veri zehirlenmesi (adversarial corpus) + +| Alan | Değer | +|---|---| +| Açıklama | Adversarial actor model davranışını biaslamak için eğitim corpusuna kötü amaçlı örnekler enjekte eder | +| L × I (inherent) | Med × High = HIGH | +| Treatment | `forgelm audit` PII / secrets / quality scan; `data_audit_report.json` severity tier'ları flag'ler; operatör flag'lenen satırların pre-flight review'u; `compute_dataset_fingerprint` SHA-256 manifest'e "ne üzerinde eğitildi"yi damgalar | +| Residual L × I | Med × Low = MED | +| Sahip | Data Steward | +| Review cadence | Eğitim koşumu başına (pre-flight) | + +#### R-02 Supply-chain compromise (compromised PyPI bağımlılığı) + +| Alan | Değer | +|---|---| +| Açıklama | PyPI'daki upstream package kötü amaçlı release alır; ForgeLM pip install üzerinden tüketir | +| L × I (inherent) | Low × High = MED | +| Treatment | Her release tag'inde SBOM (CycloneDX 1.5); `pip-audit` nightly (Wave 4 / Faz 23); `forgelm doctor` pre-flight env check; `pyproject.toml`'da pinned upper bound'lar; transitive-dep CVE feed | +| Residual L × I | Low × Med = LOW | +| Sahip | ML Engineer + Compliance Officer | +| Review cadence | Continuous (nightly) + per-release (SBOM diff) | + +#### R-03 Credential leak (HF token / webhook secret config'de veya audit log'da) + +| Alan | Değer | +|---|---| +| Açıklama | Operatör `HF_TOKEN: ghp_...` literal'ıyla bir config commit eder veya gömülü credential'lı bir webhook URL'i | +| L × I (inherent) | Med × Med = MED | +| Treatment | `safe_post` error log'larında Authorization header'ları masklar; `forgelm audit --secrets` regex scan credential'ları flag'ler; `_sanitize_md_list` deployer instructions'ta operatör-kontrollü string'leri escape eder; `forgelm doctor` pre-flight HF-auth probe; webhook payload-format curation asla config-derived secret taşımaz | +| Residual L × I | Low × Low = LOW | +| Sahip | ML Engineer | +| Review cadence | Pre-merge config review | + +#### R-04 Audit-log tampering + +| Alan | Değer | +|---|---| +| Açıklama | `audit_log.jsonl`'a write erişimi olan adversary girişleri yeniden yazıp bir dağıtımı gizler / onaylanmamış bir modeli onaylar | +| L × I (inherent) | Low × High = MED | +| Treatment | Append-only `O_APPEND` + `flock` + per-line `fsync`; HMAC chain (per-line `_hmac` field per-output-dir salt + env secret ile XOR); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` truncate-and-resume reddeder); `forgelm verify-audit` uçtan uca doğrular | +| Residual L × I | Low × Low = LOW | +| Sahip | AI Officer | +| Review cadence | Continuous (`forgelm verify-audit` cron) | + +#### R-05 Kaldırılan PII'nin memorisation'ı (Madde 17 silme tamamlanmamış) + +| Alan | Değer | +|---|---| +| Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir | +| L × I (inherent) | High × Med = HIGH | +| Treatment | Silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` audit event raise edilir; operatör residual riski özneye bildirir; yüksek-stakes dağıtımlar için sıfırdan yeniden eğit | +| Residual L × I | Med × Med = MED | +| Sahip | Data Protection Officer (DPO) | +| Review cadence | Madde 17 talebi başına | + +### 3.2 Güvenlik + alignment riskleri + +#### R-06 Safety-classifier load arızası + +| Alan | Değer | +|---|---| +| Açıklama | Llama Guard / yapılandırılmış safety classifier yüklenemiyor (HF Hub down, OOM, version mismatch); ForgeLM gate yangın etmeden devam eder | +| L × I (inherent) | Med × High = HIGH | +| Treatment | F-compliance-110 strict gate: `risk_classification` ∈ `{high-risk, unacceptable}` + `evaluation.safety.enabled: false` `ConfigError` raise eder; `audit.classifier_load_failed` event arızayı kaydeder; operatör pipeline'ı non-zero exit'te durur | +| Residual L × I | Low × Low = LOW | +| Sahip | ML Lead | +| Review cadence | Eğitim koşumu başına | + +#### R-07 Auto-revert false positive + +| Alan | Değer | +|---|---| +| Açıklama | `evaluation.auto_revert: true` geçici bir eval regression'ında tetiklenir; sonuç `pipeline.reverted` downstream denetçiye safety failure gibi görünür | +| L × I (inherent) | Med × Low = LOW | +| Treatment | `pipeline.reverted` audit event regression delta + threshold taşır; `safety_trend.jsonl` cross-run context sağlar; operatör dashboard "gerçek" revert'leri threshold-tuning sorunlarından ayırır | +| Residual L × I | Low × Low = LOW | +| Sahip | ML Lead | +| Review cadence | Haftalık trend review | + +### 3.3 Webhook + outbound-comms riskleri + +#### R-08 Webhook SSRF / data exfiltration + +| Alan | Değer | +|---|---| +| Açıklama | Adversary AWS instance credential'ları exfiltre etmek için `webhook.url_env=http://internal-metadata-server/...` yapılandırır | +| L × I (inherent) | Low × Med = LOW | +| Treatment | `safe_post` (Phase 7) — HTTPS-only, SSRF guard RFC 1918 / 169.254.x / loopback / link-local reddeder, no redirect-following, error log'larda masked auth header'lar; webhook URL `url_env`'den gelmeli, asla inline değil | +| Residual L × I | Low × Low = LOW | +| Sahip | Güvenlik | +| Review cadence | Config review başına | + +#### R-09 Webhook target ele geçirildi + +| Alan | Değer | +|---|---| +| Açıklama | Operatörün Slack / Teams workspace'i ele geçirilir; saldırgan notify_started payload'larını okur ve model-deployment cadence'ini öğrenir | +| L × I (inherent) | Low × Low = LOW | +| Treatment | Webhook payload curation asla raw eğitim verisi veya unredacted PII taşımaz; `FORGELM_AUDIT_SECRET`-imzalı payload'lar splicing'i tespit eder; operatör olayda webhook secret'ı rotate eder | +| Residual L × I | Low × Low = LOW | +| Sahip | Güvenlik + ML Lead | +| Review cadence | Webhook-target olayı başına | + +### 3.4 ReDoS + ingestion riskleri + +#### R-10 `--type custom` regex (reverse-pii) üzerinden ReDoS + +| Alan | Değer | +|---|---| +| Açıklama | `forgelm reverse-pii --type custom --query "(a+)+$"` üzerindeki operatör-tedarikli regex catastrophic backtracking tetikler | +| L × I (inherent) | Low × Low = LOW | +| Treatment | `_scan_file_with_alarm`'da POSIX SIGALRM 30s per-file budget (Faz 38 / Wave 3 followup F-W3FU-T-01); thread-safety guard non-main thread'de skip eder; outer alarm korunur | +| Residual L × I | Low × Low = LOW | +| Sahip | Güvenlik | +| Review cadence | Wave 3 kapatıldı; başka eylem yok | + +#### R-11 Cross-tool digest mismatch (purge ↔ reverse-pii) + +| Alan | Değer | +|---|---| +| Açıklama | `forgelm purge` bir salt ile çalışır; aynı identifier için `forgelm reverse-pii` başka bir salt ile çalışır (örn. koşumlar arası `FORGELM_AUDIT_SECRET` değişimi) — digest eşleşmez, audit chain inconsistent görünür | +| L × I (inherent) | Low × Med = LOW | +| Treatment | Her audit event'te `salt_source` kaydedilir (Wave 3 followup F-W3-PS-07); env secret set olsa bile explicit `--salt-source per_dir` honor edilir; cross-tool correlation testi (`test_purge_target_id_matches_reverse_pii_query_hash_on_same_output_dir`) CI'da pin'lenmiş | +| Residual L × I | Low × Low = LOW | +| Sahip | DPO | +| Review cadence | Madde 15/17 talebi başına | + +### 3.5 Dağıtım riskleri + +#### R-12 Yetkisiz model dağıtımı + +| Alan | Değer | +|---|---| +| Açıklama | Operatör human approval'ı atlar; high-risk model ML Lead / AI Officer sign-off olmadan üretime ulaşır | +| L × I (inherent) | Med × High = HIGH | +| Treatment | `risk_classification` ∈ `{high-risk, unacceptable}` için `evaluation.require_human_approval: true`; staging dizini `forgelm approve` olana kadar modeli tutar; F-compliance-110 yapılandırılmamışsa ConfigError raise eder; `human_approval.required/granted/rejected` chain her kararı forensic olarak kaydeder | +| Residual L × I | Low × Med = LOW | +| Sahip | ML Lead + AI Officer | +| Review cadence | High-risk için eğitim koşumu başına | + +## 4. Operatöre özgü satırlar + +[ForgeLM'in kendi tehdit modelinin öngörmediği organizasyon-spesifik +riskler için buraya satır ekle — örn. endüstri-spesifik düzenleyici +gereksinimleri, üçüncü-taraf yasal taahhütler, entegrasyon-spesifik +riskler. Aynı alan yapısını §3 olarak kullan.] + +## 5. Risk acceptance log'u + +Bir residual risk `Med` veya `High` olduğunda, AI Officer kabulü +imzalamalıdır: + +| Risk ID | Inherent | Residual | Kabul eden | Tarih | Gerekçe | +|---|---|---|---|---|---| +| R-05 | HIGH | MED | [AI Officer] | [TARİH] | Memorisation residual riski kabul edildi; operatör politikası: özneyi bilgilendir + yüksek-stakes dağıtımlar için sıfırdan yeniden eğit | +| ... | ... | ... | ... | ... | ... | + +## 6. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon (Wave 4 / Faz 23) — 12 ForgeLM-tanımlı risk | + +Quarterly review cadence: + +- Her satırın L × I'sını geçen quarter'ın olay sıklığına karşı + yeniden skorla. +- Retro'da tanımlanan operatör-spesifik riskler için yeni satır + ekle. +- Residual-risk acceptance log'unun güncel olduğunu teyit et. diff --git a/docs/qms/statement_of_applicability-tr.md b/docs/qms/statement_of_applicability-tr.md new file mode 100644 index 00000000..09f6e746 --- /dev/null +++ b/docs/qms/statement_of_applicability-tr.md @@ -0,0 +1,168 @@ +# QMS: Statement of Applicability (SoA) + +> Kalite Yönetim Sistemi — [YOUR ORGANIZATION] +> ISO 27001:2022 Annex A — clause 6.1.3 d) tarafından gerekli +> Çapraz referans: [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) +> tam mapping gerekçesi için. + +## 1. Amaç + +Statement of Applicability (SoA) bir ISO 27001 denetçisinin İLK +açtığı deliverable'dır. Her Annex A kontrolü için ISMS'inizin +**applicable** veya **excluded** olarak ele aldığını ve gerekçeyi +belirtir. + +Bu şablon ForgeLM'in kendi kapsama haritasından pre-populated'dır +(çapraz referans verilen design dökümanına bakın). Operatör her +satırı ISMS context'lerine uyarlar — çoğu satır applicable kalır; +ForgeLM'in natively kapsadığı bir satır (örn. A.8.15 Logging) +operatöre atıfta bulunmak için spesifik ForgeLM-üretilmiş kanıt +verir. + +## 2. SoA matrisi + +Format: kontrol ID → applicability → gerekçe / ForgeLM kanıtı → +operatör-tarafı eylem. + +### 2.1 A.5 Organisational controls (37) + +| Kontrol | Applicable? | Gerekçe / ForgeLM kanıtı | Implementation status | +|---|---|---|---| +| A.5.1 Bilgi güvenliği için politikalar | YES | EU AI Act Md. 17 QMS gerektirir; ForgeLM audit-event vocabulary neyin loglandığını dokümante eder | ForgeLM audit log'u referans veren kurum-genelinde ISMS politikası benimse | +| A.5.2 Bilgi güvenliği rolleri ve sorumlulukları | YES | `roles_responsibilities.md` QMS şablonu AI Officer / ML Lead / Data Steward / DPO tanımlar | Rol tanımlarını benimse | +| A.5.3 Görev ayrılığı | YES | `human_approval.required/granted` trainer ≠ approver attribution'ı zorlar | CI runner identity ≠ insan reviewer identity'yi yapılandır | +| A.5.4 Yönetim sorumlulukları | YES | `forgelm doctor`, `compliance_report.json` yönetim-review artefaktları sağlar | Aylık review cadence | +| A.5.5 Yetkililerle iletişim | YES | EU AI Act Md. 73 ciddi-olay raporlama | Düzenleyici contact listesi tut | +| A.5.6 Özel ilgi gruplarıyla iletişim | YES | ML safety / red-team topluluğu | İlgili threat intel'e abone ol | +| A.5.7 Tehdit istihbaratı | YES | ML supply-chain CVE feed'leri, model-poisoning advisory'leri | İlgili feed'lere abone ol | +| A.5.8 Proje yönetiminde bilgi güvenliği | YES | `risk_assessment` config + F-compliance-110 strict gate; Annex IV §9 metadata | Proje sign-off'unda göm | +| A.5.9 Bilgi ve diğer ilişkili varlıkların envanteri | YES | `data_provenance.json`, `model_integrity.json`, SBOM | Kurumsal asset register tut | +| A.5.10 Bilginin kabul edilebilir kullanımı | YES | Standart kurumsal AUP | Benimse | +| A.5.11 Varlıkların iadesi | YES | Dağıtılan modellerin / eğitim host'larının decommissioning'i | Benimse | +| A.5.12 Bilgi sınıflandırması | YES | `compliance.risk_classification` 5-tier kurumsal gizlilik sınıflarına haritalanır | ForgeLM tier'larını corp class'larına haritala | +| A.5.13 Bilgi etiketleme | YES | `model_card.md`, manifest risk class'ı damgalar | Üzerine kurumsal etiket uygula | +| A.5.14 Bilgi transferi | YES | `safe_post` webhook discipline, payload'da plaintext PII yok | Webhook recipient'larıyla DTA imzala | +| A.5.15 Erişim kontrolü | YES | Operatör kimliği + salted hashing | IdP entegrasyonu | +| A.5.16 Kimlik yönetimi | YES | `FORGELM_OPERATOR` env contract | CI runner identity'yi yapılandır | +| A.5.17 Kimlik doğrulama bilgileri | YES | `safe_post` auth header'larını masklar; `_mask` token'ları gizler | Vault-store webhook secret'ları, HF token'ları | +| A.5.18 Erişim hakları | YES | `human_approval` gate | IdP'de RBAC | +| A.5.19 Tedarikçi ilişkilerinde bilgi güvenliği | YES | `_fingerprint_hf_revision`; SBOM | Vendor risk programı | +| A.5.20 Tedarikçi anlaşmalarında bilgi güvenliği | YES | Standart tedarikçi MSA güvenlik clause'ları | Benimse | +| A.5.21 ICT tedarik zincirinde bilgi güvenliği yönetimi | YES | SBOM (Wave 2 dönemi); `pip-audit` nightly (Wave 4) | CVE izleme | +| A.5.22 Tedarikçi hizmetlerinin izlenmesi, gözden geçirilmesi ve değişim yönetimi | YES | Vendor yıllık review | Benimse | +| A.5.23 Bulut hizmetleri için bilgi güvenliği | YES | Cloud sağlayıcı güvenlik yapılandırması | Cloud-spesifik kontroller | +| A.5.24 Bilgi güvenliği olay yönetimi planlaması ve hazırlığı | YES | `sop_incident_response.md`; audit chain durumu korur | IR ekibi kur | +| A.5.25 Bilgi güvenliği olaylarının değerlendirilmesi ve karara bağlanması | YES | `data.erasure_failed`, `pipeline.failed`, `audit.classifier_load_failed` event'ler `error_class` + `error_message` ile | Triage runbook | +| A.5.26 Bilgi güvenliği olaylarına yanıt | YES | Audit chain HMAC öncesi/sonrası korur | Runbook dokümante et | +| A.5.27 Bilgi güvenliği olaylarından öğrenme | YES | `pipeline.reverted` event'leri post-mortem kanıtı biriktirir | Haftalık post-mortem cadence | +| A.5.28 Kanıt toplama | YES | `audit_log.jsonl` forensic-grade; `forgelm verify-audit` doğrular | Write-once depolamaya gönder | +| A.5.29 Aksaklık sırasında bilgi güvenliği | YES | `auto_revert` baseline-flip; `pipeline.reverted` event | Base-model retention dokümante et | +| A.5.30 İş sürekliliği için ICT hazırlığı | YES | DR planlama | Benimse | +| A.5.31 Yasal, kanuni, düzenleyici ve sözleşmesel gerekliliklerin tanımlanması | YES | EU AI Act + GDPR mapping'ler; Annex IV bundle | Kural değişikliklerini izle | +| A.5.32 Fikri mülkiyet hakları | YES | SBOM'da lisans çıkarımı; HF model-card metadata | Per-model lisans review | +| A.5.33 Kayıtların korunması | YES | Append-only + HMAC + manifest sidecar; off-site replica | Off-site backup | +| A.5.34 Gizlilik ve PII korunması | YES | `forgelm reverse-pii` Md. 15; `forgelm purge` Md. 17; `forgelm audit` PII tespiti + maskeleme | DSAR workflow + DPIA | +| A.5.35 Bilgi güvenliğinin bağımsız incelenmesi | YES | Yıllık external denetim | Benimse | +| A.5.36 Bilgi güvenliği için politikalara, kurallara ve standartlara uyum | YES | Pydantic validation; CI gate'ler; `forgelm doctor` | Kurum-genelinde uygulama | +| A.5.37 Dokümante edilmiş işletim prosedürleri | YES | QMS şablonları (5 Wave 0 SOPs + 4 Wave 4 / Faz 23 eklemeleri) | Uyarlama + benimseme | + +### 2.2 A.6 People controls (8) + +| Kontrol | Applicable? | Gerekçe | Implementation status | +|---|---|---|---| +| A.6.1 Tarama | YES | HR background-check politikası | Benimse | +| A.6.2 İstihdam şartları ve koşulları | YES | Standart NDA / IP clause'ları | Benimse | +| A.6.3 Bilgi güvenliği farkındalığı, eğitimi ve öğretimi | YES | `audit_event_catalog.md` eğitim materyali işlevi de görür; `deployer_instructions.md` Md. 13 çıktısı | Yıllık EU AI Act + GDPR eğitimi | +| A.6.4 Disiplin süreci | YES | Operatör attribution hesap verebilirliği korur | Benimse | +| A.6.5 İstihdam değişimi veya sonlandırma sonrası sorumluluklar | YES | Operatör id rotasyonu; eski ID'ler audit history'de kalır | Ayrılan CI runner'ı iptal et; önceki eylemleri denetle | +| A.6.6 Gizlilik veya açıklamama anlaşmaları | YES | Standart NDA | Benimse | +| A.6.7 Uzaktan çalışma | YES | `forgelm doctor --offline`; air-gap pre-cache | VPN politikası | +| A.6.8 Bilgi güvenliği olay raporlama | YES | `pipeline.failed`, `data.erasure_failed`, `audit.classifier_load_failed` event'ler; webhook lifecycle | SIEM ingestion + alarm | + +### 2.3 A.7 Physical controls (14) — bir software-toolkit ISMS'sinden tipik olarak EXCLUDED + +Bu kontroller ForgeLM'in ISMS scope'undan hariç tutulur çünkü ForgeLM +bir software toolkit'tir. **Operatörün ISMS scope'u kendi SoA'sında +bunları DAHİL EDER;** ForgeLM'in burada exclusion'ı yalnız +ForgeLM-spesifik kontrol envanteriyle ilgilidir. + +| Kontrol | Operatöre Applicable? | Excluded-from-ForgeLM gerekçesi | +|---|---|---| +| A.7.1 Fiziksel çevreler | YES (operatör ISMS) | ForgeLM yazılımdır; substrate-side | +| A.7.2 Fiziksel giriş | YES | Datacenter-side | +| A.7.3 Ofislerin, odaların ve tesislerin güvenliğinin sağlanması | YES | Datacenter-side | +| A.7.4 Fiziksel güvenlik izleme | YES | CCTV — substrate-side | +| A.7.5 Fiziksel ve çevresel tehditlere karşı koruma | YES | Substrate-side | +| A.7.6 Güvenli alanlarda çalışma | YES | SCIF politikaları | +| A.7.7 Açık masa ve açık ekran | YES | Endpoint politikası | +| A.7.8 Ekipman yerleşimi ve korunması | YES | Hardware placement | +| A.7.9 İşletme dışındaki varlıkların güvenliği | YES | MDM | +| A.7.10 Depolama medyası | YES | LUKS / FileVault / BitLocker — substrate-side | +| A.7.11 Destekleyici altyapılar | YES | UPS / generator | +| A.7.12 Kablolama güvenliği | YES | Datacenter cabling | +| A.7.13 Ekipman bakımı | YES | Hardware refresh | +| A.7.14 Ekipmanın güvenli imhası veya yeniden kullanımı | YES | E-waste politikası | + +### 2.4 A.8 Technological controls (34) + +| Kontrol | Applicable? | ForgeLM kanıtı | Operatör-tarafı eylem | +|---|---|---|---| +| A.8.1 Kullanıcı uç cihazları | YES | `forgelm doctor` — Python / CUDA / GPU / extras / HF auth / disk / `FORGELM_OPERATOR` checks | Endpoint hardening | +| A.8.2 Ayrıcalıklı erişim hakları | YES | Operatör attribution; approval gate non-trainer operator | IdP'de RBAC | +| A.8.3 Bilgi erişim kısıtlaması | YES | Salted identifier hashing; `forgelm reverse-pii` Md. 15 | DSAR workflow | +| A.8.4 Kaynak koduna erişim | YES | `model.trust_remote_code=False` default; `_fingerprint_hf_revision` commit SHA pin'ler | VCS erişim kontrolü | +| A.8.5 Güvenli kimlik doğrulama | YES | `safe_post` non-HTTPS'de auth header reddeder; webhook secret discipline | MFA + token rotasyonu | +| A.8.6 Kapasite yönetimi | YES | `forgelm doctor` resource report; `resource_usage` manifest block | Quota / autoscaling | +| A.8.7 Kötü amaçlı yazılıma karşı koruma | YES | Eğitim host'larında antivirus | Benimse | +| A.8.8 Teknik zafiyetlerin yönetimi | YES | SBOM; `pip-audit` nightly; `bandit` CI; Pydantic config validation | OS patch yönetimi | +| A.8.9 Yapılandırma yönetimi | YES | YAML Pydantic ile valide; `forgelm --dry-run` gate; `compliance.config_hash` audit'te | Config'leri sürüm-kontrol | +| A.8.10 Bilgi silme | YES | `forgelm purge` Md. 17; salted hash audit; `data.erasure_warning_memorisation` flag | DSAR workflow | +| A.8.11 Veri maskeleme | YES | `forgelm audit` regex + Presidio ML-NER; `_SECRET_PATTERNS` credentials scan | Maskeleme politikası | +| A.8.12 Veri sızıntısı önleme | YES | `forgelm reverse-pii` plaintext residual scan; webhook asla raw rows taşımaz | Egress DLP | +| A.8.13 Bilgi yedekleme | YES | `audit_log.jsonl` + `.manifest.json` yedeklenebilir; `forgelm --compliance-export` ZIP | Off-site yedek | +| A.8.14 Bilgi işlem tesislerinin yedekliliği | YES | Multi-AZ infra | Benimse | +| A.8.15 Loglama | YES | `audit_log.jsonl` JSON Lines + HMAC + genesis manifest; `forgelm verify-audit` doğrular | SIEM'e gönder | +| A.8.16 İzleme aktiviteleri | YES | Webhook lifecycle event'leri; `safety_trend.jsonl` cross-run trend | Alarm eşikleri | +| A.8.17 Saat senkronizasyonu | YES | Tüm audit girişleri ISO-8601 UTC | Eğitim host'larında NTP | +| A.8.18 Ayrıcalıklı yardımcı program kullanımı | YES | OS-seviyesi privilege management | Benimse | +| A.8.19 Operasyonel sistemlerde yazılım kurulumu | YES | `forgelm doctor` paketleri rapor eder; `pyproject.toml` pin'ler; `pip install forgelm==X.Y.Z` | Package allowlist | +| A.8.20 Ağ güvenliği | YES | `safe_post` HTTPS-only / SSRF guard / no-redirect; `model.trust_remote_code=False` | Egress firewall | +| A.8.21 Ağ hizmetlerinin güvenliği | YES | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement | +| A.8.22 Ağ ayrımı | YES | VPC / subnet design | Benimse | +| A.8.23 Web filtreleme | YES | Egress proxy | Benimse | +| A.8.24 Kriptografi kullanımı | YES | SHA-256 + HMAC-SHA-256 chain; salted SHA-256 identifier hashing; per-output-dir salt + env XOR | `FORGELM_AUDIT_SECRET` için KMS | +| A.8.25 Güvenli geliştirme yaşam döngüsü | YES | `docs/standards/code-review.md`, `release.md`, CI gate'leri | SDLC framework | +| A.8.26 Uygulama güvenliği gereksinimleri | YES | F-compliance-110 strict gate; Pydantic validation; `_reverse_pii`'da ReDoS guard | App-seviyesi tehdit modelleme | +| A.8.27 Güvenli sistem mimarisi ve mühendislik prensipleri | YES | Append-only audit log mimarisi; HMAC chain; lazy import; SSRF guard | Defence-in-depth | +| A.8.28 Güvenli kodlama | YES | `docs/standards/coding.md`; type hints; `_sanitize_md_list`'te CommonMark escaping | Custom-extension review | +| A.8.29 Geliştirme ve kabul aşamasında güvenlik testi | YES | `pytest` 1370+ test; `bandit` static analysis; `forgelm safety-eval` standalone gate | E2E güvenlik testleri | +| A.8.30 Dış kaynaklı geliştirme | YES | Üçüncü-taraf-developer güvenliği | Benimse | +| A.8.31 Geliştirme, test ve üretim ortamlarının ayrılması | YES | `forgelm --dry-run`; staging dir; `evaluation.require_human_approval` | Ayrı pipeline'lar | +| A.8.32 Değişim yönetimi | YES | `human_approval.required/granted/rejected`; `compliance.config_hash`; staging snapshot | CAB süreci | +| A.8.33 Test bilgisi | YES | `forgelm audit` test setlerinde de PII / secrets'i flag eder | Test-data-handling politikası | +| A.8.34 Denetim testi sırasında bilgi sistemlerinin korunması | YES | Read-only audit erişimi | Benimse | + +## 3. Kapsama özeti + +| Tema | Toplam | Applicable | Excluded (ForgeLM scope) | FL-supported | FL-helps | +|---|---|---|---|---|---| +| A.5 Organisational | 37 | 37 | 0 | 4 | 24 | +| A.6 People | 8 | 8 | 0 | 0 | 4 | +| A.7 Physical | 14 | 14 (operatör ISMS) | 14 (ForgeLM-spesifik) | 0 | 0 | +| A.8 Technological | 34 | 34 | 0 | 7 | 22 | +| **Toplam** | **93** | **93 (operatör ISMS)** | **14 (ForgeLM-spesifik)** | **11** | **50** | + +## 4. İnceleme + +| Versiyon | Tarih | Yazar | Değişiklikler | +|---------|------|--------|---------| +| 1.0 | [DATE] | [AUTHOR] | İlk versiyon (Wave 4 / Faz 23) — 93 kontrol ForgeLM v0.5.5'e karşı skorlandı | + +Yıllık review cadence: + +- Applicability'yi yeniden teyit et (nadir değişimler — çoğu kontrol + applicable kalır). +- Yeni bir ForgeLM phase göndertiğinde ForgeLM-evidence column'unu + güncelle. +- Implementation-status column'unu operatör-tarafı kontrol postürünü + yansıtmak için güncelle. diff --git a/docs/reference/compliance_summary.md b/docs/reference/compliance_summary.md index 4cb2bca3..74d8480e 100644 --- a/docs/reference/compliance_summary.md +++ b/docs/reference/compliance_summary.md @@ -1,122 +1,198 @@ -## Compliance Summary: EU AI Act (Article 17 + related requirements) - -Purpose -- Provide a concise, machine‑friendly summary of how ForgeLM implements evidence, controls and artifacts that support compliance with the EU AI Act (not legal advice). - -Scope -- This document maps repo components, evidence files, and operational gaps to common EU AI Act expectations for high‑risk systems (Article 17 QMS, provenance, audit trails, monitoring). - -Quick conclusion -- ForgeLM already implements many technical controls relevant to Article 17: post‑training safety evaluation with auto‑revert, dataset and checkpoint SHA‑256 fingerprinting, and machine‑readable compliance artifacts. Additional organizational, signing, and immutable‑logging steps are recommended to harden auditability for formal QMS/third‑party review. - -Required / relevant EU AI Act items (high level) -- Risk assessment and governance (who is responsible, risk classification). -- Quality Management System (QMS) processes and records (Article 17): documented procedures, versioning, roles, and training records. -- Data provenance and documentation: dataset source, preprocessing, schema, fingerprints, timestamps. -- Technical documentation & conformity evidence: model lineage, evaluation results, safety tests, validation reports. -- Monitoring & post‑market surveillance: runtime monitoring, incident logging, corrective actions. - -Where ForgeLM meets these (technical evidence) -- Safety evaluation & auto‑revert - - Implementation: core logic in `forgelm/trainer.py` (auto‑revert triggers and `_revert_model`). See [forgelm/trainer.py](forgelm/trainer.py#L37) and `_revert_model` at [forgelm/trainer.py](forgelm/trainer.py#L278). - - Behavior: configurable `max_safety_regression` and `auto_revert` flags; safety checks executed post‑training. - -- Safety classifier and thresholds - - Implementation: `forgelm/safety.py` performs classification on adversarial prompts and enforces thresholds. See [forgelm/safety.py](forgelm/safety.py#L33) and threshold logic at [forgelm/safety.py](forgelm/safety.py#L129). - -- Data provenance (SHA‑256) and compliance export - - Implementation: `forgelm/compliance.py` computes SHA‑256 fingerprints, writes `data_provenance.json`, and exports compliance artifacts. See [forgelm/compliance.py](forgelm/compliance.py#L1) and hash calculation at [forgelm/compliance.py](forgelm/compliance.py#L33) and file output at [forgelm/compliance.py](forgelm/compliance.py#L167). - -- Documentation & examples - - Usage and safety compliance guidance: [docs/guides/safety_compliance.md](docs/guides/safety_compliance.md#L23). - - Template config shows `auto_revert` and safety settings: [config_template.yaml](config_template.yaml#L101). - - Marketing/README highlight features: [README.md](README.md#L21). - -- Tests - - Unit tests validate provenance and revert logic: `tests/test_compliance.py` and `tests/test_trainer.py`. Example: `sha256` checks in [tests/test_compliance.py](tests/test_compliance.py#L134). - -Gaps, residual risks and operational considerations -- Organizational QMS (Article 17) — gap - - Code and generated artifacts exist, but formal QMS documents (SOPs, roles, training records, approvals) live outside repo. Recommendation: create `docs/qms/` with SOP templates, roles and review signoffs. - -- Immutable evidence & signing — risk - - SHA‑256 fingerprints are good, but unauthenticated files can be tampered with. Recommendation: add timestamping and digital signatures (GPG/PKI) for `data_provenance.json` and `compliance_report.json`, or integrate with an append‑only log (e.g., remote ledger, WORM storage). - -- Revert safety & retention — operational risk - - Current `_revert_model` deletes artifacts. For audits and incident analysis, prefer archiving reverted artifacts to a secure, read‑only archive before deletion and keep an audit entry. Add configurable retention / require human approval for production revert. - -- Access controls & secrets — risk - - Ensure provenance and model artifacts are protected by RBAC and encrypted at rest. Document expected IAM roles for auditors vs engineers. - -- Post‑market monitoring — gap - - Add runtime safety telemetry and incident reporting hooks (webhooks/metrics) to track deployed model behavior and feed into QMS corrective actions. - -Firms' opportunities -- Competitive advantage: highlight existing technical controls (auto‑revert + SHA‑256) and market as “EU AI Act–ready” packages. -- Productization: compliance export + audit‑bundle, GitHub Actions workflows, and audit checklist templates. -- Enterprise add‑ons: QMS integration, signed/timestamped provenance, RBAC, and admin dashboard. - -Recommended additions (prioritized) -1. Compliance summary document (this file) + `docs/qms/` skeleton with SOP templates and roles (low effort). -2. Evidence bundle export: implement `forge compliance bundle` to produce `compliance_bundle.zip` (model card, `data_provenance.json`, `compliance_report.json`, evaluation logs). -3. Signed provenance: timestamp + GPG or PKI signing of provenance file; optionally publish signatures to a public bulletin (or store in S3 with immutable versioning). -4. CI workflow example: add `.github/workflows/auto_revert.yml` demonstrating post‑training safety checks, revert flow, and notification (Slack/Issue). -5. Revert safe‑guards: archive reverted artifacts and add a configurable human approval window for production pipelines. -6. Runtime monitoring & incident reporting hooks for post‑market surveillance. - -Implementation notes and quick examples -- Evidence bundle command (example): - -```bash -# Export compliance artifacts from a completed training run -forgelm --config job.yaml --compliance-export ./compliance_output/ - -# Package all artifacts into a single auditor-ready ZIP -# (export_evidence_bundle() in forgelm/compliance.py) -``` - -- CI workflow skeleton (concept): create `.github/workflows/auto_revert.yml` that runs train → evaluate → safety checks → if fail: archive artifacts, create issue, optionally revert. Add Slack or webhook notifications. - -- Signing provenance (concept): - -```bash -gpg --armor --detach-sign data_provenance.json -sha256sum data_provenance.json > data_provenance.sha256 -``` - -Evidence locations (quick links) -- Auto‑revert logic: [forgelm/trainer.py](../../forgelm/trainer.py) — `_revert_model` -- Safety evaluation: [forgelm/safety.py](../../forgelm/safety.py) -- Provenance + AuditLogger + hash chain: [forgelm/compliance.py](../../forgelm/compliance.py) -- Config template: [config_template.yaml](../../config_template.yaml) -- Safety guidance: [docs/guides/safety_compliance.md](../guides/safety_compliance.md) -- Tests: [tests/test_compliance.py](../../tests/test_compliance.py), [tests/test_trainer.py](../../tests/test_trainer.py) - -Phase 8 implemented features (all complete) -- ComplianceMetadataConfig: provider_name, intended_purpose, risk_classification, system_name (Annex IV) -- RiskAssessmentConfig: intended_use, foreseeable_misuse, risk_category, mitigation_measures (Art. 9) -- DataGovernanceConfig: collection_method, annotation_process, known_biases, DPIA tracking (Art. 10) -- AuditLogger: append-only JSON Lines event log with run_id, operator, timestamps, SHA-256 hash chain, per-line HMAC, flock concurrent-write guard, genesis manifest sidecar (Art. 12) -- generate_deployer_instructions(): auto-generated markdown for non-ML deployers (Art. 13) -- require_human_approval: pipeline pauses for review, exit code 4 (Art. 14) -- generate_model_integrity(): SHA-256 checksums on all output artifacts (Art. 15) -- MonitoringConfig: post-market monitoring hooks — Prometheus/Datadog (Art. 17) -- export_evidence_bundle(): ZIP archive of all compliance artifacts -- QMS templates: 5 SOP documents in `docs/qms/` (training, data, incident, change, roles) -- `--compliance-export` CLI flag: standalone compliance artifact generation without GPU - -Phase 9 implemented features (advanced safety scoring) -- Confidence score extraction: captures classifier probability per response -- Confidence-weighted safety score: `scoring: "confidence_weighted"`, `min_safety_score` threshold -- Low-confidence alerts: flags responses where classifier confidence < `min_classifier_confidence` -- Llama Guard S1-S14 harm category parsing: 14 categories mapped to severity levels -- Severity levels: critical/high/medium/low with per-severity threshold enforcement -- Cross-run trend tracking: `safety_trend.jsonl` (append-only, longitudinal analysis) -- Built-in adversarial prompt library: `configs/safety_prompts/` (50 prompts in 3 categories) -- 3-layer safety gate: binary ratio → confidence score → severity thresholds - -Remaining considerations -- Signed/timestamped provenance (GPG/PKI) for tamper-proof artifacts -- Append-only evidence store integration (WORM storage, remote ledger) -- Revert artifact archival (archive before deletion for audit trail) -- RBAC documentation for auditor vs engineer access +# Compliance Summary — EU AI Act + ISO 27001 + SOC 2 + +> **Scope.** Concise, machine-friendly summary of how ForgeLM +> implements evidence, controls and artefacts that support +> compliance with the EU AI Act (high-risk systems, Article 17 QMS +> + related provisions) AND the deployer's ISO 27001 / SOC 2 Type II +> alignment. Not legal advice. +> +> **Audience.** Compliance officer / auditor / deployer engineering +> lead. +> +> Wave 4 / Faz 26 cleanup: this document used to anchor at literal +> source-code line numbers (e.g. `compliance.py#L33`) that drifted +> as the codebase evolved. References below now use symbol-name +> + module-path form so they survive refactors. + +## Quick conclusion + +ForgeLM ships out-of-the-box: + +- **EU AI Act Article 9** risk-management evidence via the strict + gate (`_warn_high_risk_compliance`) + safety-eval auto-revert. +- **EU AI Act Article 10** data-governance evidence via + `data_governance_report.json` + `forgelm audit` PII / secrets / + quality scan. +- **EU AI Act Article 11 + Annex IV** technical documentation via + `compliance.export_compliance_artifacts` + ZIP bundle. +- **EU AI Act Article 12** record-keeping via append-only + `AuditLogger` (HMAC chain + manifest sidecar). +- **EU AI Act Article 13** deployer instructions via + `generate_deployer_instructions`. +- **EU AI Act Article 14** human-oversight gate via + `forgelm approve` / `reject` Article 14 staging. +- **EU AI Act Article 15** model-integrity via + `compute_artefact_sha256` + `model_integrity.json`. +- **EU AI Act Article 17** QMS templates in `docs/qms/` (Wave 0 + baseline + Wave 4 ISO additions). +- **GDPR Article 15** right-of-access via `forgelm reverse-pii`. +- **GDPR Article 17** right-to-erasure via `forgelm purge`. +- **ISO 27001 / SOC 2 alignment** — see Wave 4 design doc + deployer + guide. + +## EU AI Act high-level checklist + +What the regulator asks vs. how ForgeLM answers: + +| Regulator question | ForgeLM evidence | +|---|---| +| Risk classification + governance | `compliance.risk_classification` 5-tier; F-compliance-110 strict gate | +| QMS processes + records | `docs/qms/` 9 SOPs (5 Wave 0 + 4 Wave 4); audit chain | +| Data provenance | `data_provenance.json`; `compute_dataset_fingerprint` (SHA-256 + size + mtime); HF-revision pin | +| Technical documentation | `annex_iv_metadata.json`; Annex IV §§1-9 canonical layout | +| Conformity evidence | `compliance_report.json`; `model_card.md`; `model_integrity.json` | +| Monitoring + post-market surveillance | Webhook lifecycle (`notify_*`); `safety_trend.jsonl` cross-run trend | +| Human oversight | Article 14 staging gate; `human_approval.required/granted/rejected` | + +## Where ForgeLM meets each requirement + +### Safety evaluation + auto-revert + +- Implementation: `forgelm.trainer` post-training evaluation chain; + `auto_revert` flag triggers fall-back to baseline on regression. +- Evidence: `safety_results.json` (per-prompt classification); + `pipeline.reverted` audit event with regression delta. +- Configuration: `evaluation.safety.enabled`, + `evaluation.auto_revert`, `evaluation.safety.scoring`, + `evaluation.safety.min_safety_score`. + +### Safety classifier + 3-layer gate + +- Implementation: `forgelm.safety` runs Llama Guard 3 (or operator- + configured classifier) on a 140-prompt corpus across 6 categories + (was 50 × 3 in the v0.4 series; expanded in v0.5.0). +- 3-layer gate: binary safe-ratio → confidence-weighted score → + severity threshold. Each layer fails the run with a distinct + `audit.classifier_*` event so the operator can attribute the + rejection. + +### Data provenance (SHA-256) + compliance export + +- Implementation: `forgelm.compliance` computes per-corpus + fingerprints (`_fingerprint_local_file`, `_fingerprint_hf_revision`) + and writes `data_provenance.json`; `export_compliance_artifacts` + ZIPs the bundle. +- CLI: `forgelm --config job.yaml --compliance-export ./out/`. + +### Audit chain (Article 12) + +- Implementation: `forgelm.compliance.AuditLogger` — JSON Lines + append-only log at `/audit_log.jsonl`, HMAC-chained + with `FORGELM_AUDIT_SECRET` XOR'd against the per-output-dir salt + in `/.forgelm_audit_salt`. Genesis manifest sidecar + (`audit_log.manifest.json`) refuses truncate-and-resume tampering. +- Verification: `forgelm verify-audit [--require-hmac]` validates + the chain end-to-end; exits 0/1/2/3. + +### Article 14 staging gate + +- Implementation: when `evaluation.require_human_approval: true` + the trained model lands in `/staging_model./` + awaiting `forgelm approve --run-id ` from a non-trainer + operator. +- Listing: `forgelm approvals --pending` (Phase 37). +- Audit: `human_approval.required/granted/rejected` events. + +### GDPR Article 15 + 17 (Wave 2b + Wave 3) + +- Article 17 erasure: `forgelm purge --row-id --corpus + data/file.jsonl` with salted-hash audit (`data.erasure_*` events). +- Article 15 access: `forgelm reverse-pii --query --type + email|phone|... data/*.jsonl` with salted-hash audit + (`data.access_request_query` event). + +### ISO 27001 / SOC 2 Type II alignment (Wave 4) + +- Design doc: [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) + (~865 lines, full 93-control coverage map). +- Deployer cookbook: [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md). +- Reference tables: [`iso27001_control_mapping.md`](iso27001_control_mapping.md), + [`soc2_trust_criteria_mapping.md`](soc2_trust_criteria_mapping.md). +- Supply chain: [`supply_chain_security.md`](supply_chain_security.md) + — CycloneDX 1.5 SBOM, `pip-audit` nightly, `bandit` CI. + +## Gaps + residual operator-side considerations + +ForgeLM ships ~61 of the 93 ISO 27001 Annex A controls' technical +evidence; the remaining ~32 are deployer-side (physical security, +HR processes, network segregation, etc.). For the deployer's +ISMS posture: + +- **Encryption at rest** — ForgeLM is encryption-substrate-agnostic; + see [`../qms/encryption_at_rest.md`](../qms/encryption_at_rest.md) + for substrate recommendations per artefact class. +- **Access control** — operator identity contract + + `FORGELM_AUDIT_SECRET` rotation cadence in + [`../qms/access_control.md`](../qms/access_control.md). +- **Risk treatment** — pre-populated 12-row register in + [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md). +- **Statement of Applicability** — 93-control matrix in + [`../qms/statement_of_applicability.md`](../qms/statement_of_applicability.md). + +## Recommended adoption sequence + +1. Adopt the `docs/qms/` SOPs ([Model Training](../qms/sop_model_training.md), + [Data Management](../qms/sop_data_management.md), + [Incident Response](../qms/sop_incident_response.md), + [Change Management](../qms/sop_change_management.md), + [Roles & Responsibilities](../qms/roles_responsibilities.md)). +2. Set `FORGELM_OPERATOR` + `FORGELM_AUDIT_SECRET` per + [`../qms/access_control.md`](../qms/access_control.md). +3. Configure `evaluation.require_human_approval: true` for every + high-risk run. +4. Schedule weekly `forgelm verify-audit` cron. +5. Enable `auto_revert: true` in production training. +6. Ship `audit_log.jsonl` to write-once storage. +7. For full ISO / SOC 2 alignment: walk + [`../guides/iso_soc2_deployer_guide.md`](../guides/iso_soc2_deployer_guide.md). + +## Evidence locations (symbol references — line-stable) + +Wave 4 / Faz 26 cleanup: each link points at the file root, not a +line anchor. The auditor opens the file and greps the cited symbol +name; this survives refactors that the prior `#L33` form did not. + +- **Auto-revert + safety-eval gate**: `forgelm.trainer` (search for + `_revert_model`, `auto_revert`, `_run_safety_eval`). +- **Safety classifier + 3-layer gate**: `forgelm.safety` (search for + `LlamaGuardClassifier`, `_evaluate_3_layer_gate`). +- **Audit chain + HMAC + manifest**: `forgelm.compliance` (search + for `AuditLogger`, `_check_genesis_manifest`, + `compute_artefact_sha256`). +- **Salted identifier hashing**: `forgelm.cli.subcommands._purge` + (search for `_resolve_salt`, `_read_persistent_salt`, + `_hash_target_id`). +- **GDPR Article 15 reverse-pii**: `forgelm.cli.subcommands._reverse_pii`. +- **Article 14 staging + approve / reject**: + `forgelm.cli.subcommands._approve`, + `forgelm.cli.subcommands._reject`, + `forgelm.cli.subcommands._approvals`. +- **Webhook lifecycle**: `forgelm.webhook` (search for + `notify_started`, `notify_succeeded`, `notify_failed`, + `notify_reverted`, `notify_awaiting_approval`). +- **HTTP discipline**: `forgelm._http` (search for `safe_post`, + `safe_get`). +- **Config validation**: `forgelm.config` + (`_warn_high_risk_compliance`, `_validate_galore`, + `_validate_distributed`). + +## See also + +- [Audit event catalog](audit_event_catalog.md) — full event vocabulary. +- [ISO 27001 control mapping](iso27001_control_mapping.md) — Annex A × ForgeLM evidence. +- [SOC 2 Trust Services Criteria mapping](soc2_trust_criteria_mapping.md) — TSC × ForgeLM evidence. +- [Supply chain security](supply_chain_security.md) — SBOM + pip-audit + bandit. +- [QMS index](../qms/README.md) — SOP templates. +- [GDPR erasure guide](../guides/gdpr_erasure.md) — Article 15 + 17 workflows. +- [Safety + compliance guide](../guides/safety_compliance.md) — operator-facing how-to. +- [ISO / SOC 2 deployer guide](../guides/iso_soc2_deployer_guide.md) — audit cookbook (Wave 4). diff --git a/tests/test_check_anchor_resolution.py b/tests/test_check_anchor_resolution.py new file mode 100644 index 00000000..d7cb316a --- /dev/null +++ b/tests/test_check_anchor_resolution.py @@ -0,0 +1,230 @@ +"""Wave 4 / Faz 26 — tools/check_anchor_resolution.py regression tests. + +The anchor-resolution checker is a CI-bound gate (once Faz 30 +broken-link cleanup completes); a regression that silently +under-reports drift would let docs anchor decay go unnoticed. + +Pinned contracts: + +1. Repo-relative paths resolve against the source file's parent. +2. Markdown anchor links resolve against the target file's headers + using GitHub-flavoured slugify. +3. External URLs (https://, mailto:, javascript:) are skipped. +4. SPA hash-router fragments (`#/path`) are skipped. +5. Pure anchors (`#section`) resolve against the SAME file. +6. Line-anchor references on code files (`forgelm/x.py#L42`) are + flagged as brittle. +7. `--strict` flips exit code on broken; advisory mode reports but + exits 0. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parent.parent +_TOOLS = _REPO_ROOT / "tools" +_TOOL_PATH = _TOOLS / "check_anchor_resolution.py" + + +def _load_tool() -> object: + spec = importlib.util.spec_from_file_location("check_anchor_resolution", _TOOL_PATH) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules["check_anchor_resolution"] = module + spec.loader.exec_module(module) + return module + + +def _write(path: Path, body: str) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body, encoding="utf-8") + return path + + +# --------------------------------------------------------------------------- +# §1 — slugify_heading: GFM-style header anchor algorithm +# --------------------------------------------------------------------------- + + +class TestSlugifyHeading: + def test_lowercases_and_dashes_spaces(self): + tool = _load_tool() + assert tool._slugify_heading("Hello World") == "hello-world" + + def test_strips_punctuation(self): + tool = _load_tool() + assert tool._slugify_heading("Section 1.2: Why?") == "section-12-why" + + def test_collapses_multiple_dashes(self): + tool = _load_tool() + assert tool._slugify_heading("a -- b c") == "a-b-c" + + def test_unicode_word_chars_preserved(self): + # Turkish letters, etc. — \w with re.UNICODE matches them. + tool = _load_tool() + assert tool._slugify_heading("Dil Seçimi") == "dil-seçimi" + + def test_trims_leading_trailing_dashes(self): + tool = _load_tool() + assert tool._slugify_heading(" a ") == "a" + assert tool._slugify_heading("--a--") == "a" + + +# --------------------------------------------------------------------------- +# §2 — Resolution: relative paths + anchors +# --------------------------------------------------------------------------- + + +class TestResolveRelativePath: + def test_existing_relative_path_resolves(self, tmp_path: Path): + tool = _load_tool() + _write(tmp_path / "subdir" / "target.md", "# Target\n") + source = _write(tmp_path / "src.md", "[link](subdir/target.md)\n") + results = list(tool._extract_links(source)) + assert len(results) == 1 + assert tool._resolve_link(results[0], tmp_path) is None + + def test_missing_relative_path_flagged(self, tmp_path: Path): + tool = _load_tool() + source = _write(tmp_path / "src.md", "[link](missing.md)\n") + results = list(tool._extract_links(source)) + broken = tool._resolve_link(results[0], tmp_path) + assert broken is not None + assert "target file not found" in broken.reason + + def test_external_url_skipped(self, tmp_path: Path): + tool = _load_tool() + source = _write(tmp_path / "src.md", "[link](https://example.com)\n") + results = list(tool._extract_links(source)) + assert tool._resolve_link(results[0], tmp_path) is None + + def test_mailto_skipped(self, tmp_path: Path): + tool = _load_tool() + source = _write(tmp_path / "src.md", "[email](mailto:me@example.com)\n") + assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + + def test_spa_hash_router_skipped(self, tmp_path: Path): + tool = _load_tool() + source = _write(tmp_path / "src.md", "[ref](#/reference/usage)\n") + assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + + +class TestResolveAnchor: + def test_self_anchor_resolves(self, tmp_path: Path): + tool = _load_tool() + source = _write( + tmp_path / "src.md", + "# Top\n\n## My Section\n\n[link](#my-section)\n", + ) + results = list(tool._extract_links(source)) + assert tool._resolve_link(results[0], tmp_path) is None + + def test_self_anchor_missing_flagged(self, tmp_path: Path): + tool = _load_tool() + source = _write( + tmp_path / "src.md", + "# Top\n\n[link](#nonexistent)\n", + ) + results = list(tool._extract_links(source)) + broken = tool._resolve_link(results[0], tmp_path) + assert broken is not None + assert "anchor" in broken.reason + + def test_cross_file_anchor_resolves(self, tmp_path: Path): + tool = _load_tool() + _write(tmp_path / "target.md", "## Section A\n") + source = _write(tmp_path / "src.md", "[link](target.md#section-a)\n") + results = list(tool._extract_links(source)) + assert tool._resolve_link(results[0], tmp_path) is None + + def test_line_anchor_on_code_file_flagged_as_brittle(self, tmp_path: Path): + tool = _load_tool() + # Plant a real .py file so target-not-found doesn't fire first. + _write(tmp_path / "x.py", "def foo(): pass\n") + source = _write(tmp_path / "src.md", "[ref](x.py#L42)\n") + results = list(tool._extract_links(source)) + broken = tool._resolve_link(results[0], tmp_path) + assert broken is not None + assert "brittle" in broken.reason + + def test_non_line_anchor_on_code_file_silent(self, tmp_path: Path): + tool = _load_tool() + _write(tmp_path / "x.py", "def foo(): pass\n") + source = _write(tmp_path / "src.md", "[ref](x.py#some-symbol-anchor)\n") + results = list(tool._extract_links(source)) + # Code-file anchors that don't match #L are accepted + # (they may be e.g. function-name fragments rendered by GitHub). + assert tool._resolve_link(results[0], tmp_path) is None + + +# --------------------------------------------------------------------------- +# §3 — CLI surface: --strict, --quiet, --exclude +# --------------------------------------------------------------------------- + + +class TestCLI: + def test_clean_tree_exits_zero(self, tmp_path: Path, capsys): + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "a.md", "# A\n[link](b.md)\n") + _write(scope / "b.md", "# B\n") + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs"]) + assert rc == 0 + captured = capsys.readouterr().out + assert "OK" in captured + + def test_advisory_mode_reports_but_exits_zero(self, tmp_path: Path, capsys): + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "a.md", "# A\n[broken](missing.md)\n") + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs"]) + assert rc == 0 + captured = capsys.readouterr().out + assert "WARN" in captured + assert "missing.md" in captured + + def test_strict_mode_exits_one_on_broken(self, tmp_path: Path, capsys): + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "a.md", "# A\n[broken](missing.md)\n") + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs", "--strict"]) + assert rc == 1 + captured = capsys.readouterr().out + assert "FAIL" in captured + + def test_quiet_suppresses_ok_summary(self, tmp_path: Path, capsys): + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "a.md", "# A\n") + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs", "--quiet"]) + assert rc == 0 + assert "OK" not in capsys.readouterr().out + + def test_exclude_strips_subtree(self, tmp_path: Path, capsys): + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "public.md", "# A\n[ok](b.md)\n") + _write(scope / "b.md", "# B\n") + # The "internal" subdirectory has a broken link but is excluded. + _write(scope / "internal" / "research.md", "[broken](../../tmp/foo.md)\n") + rc = tool.main( + [ + "--repo-root", + str(tmp_path), + "--scope", + "docs", + "--exclude", + "internal", + "--strict", + ] + ) + assert rc == 0 + + def test_missing_scope_exits_one(self, tmp_path: Path, capsys): + tool = _load_tool() + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "nonexistent"]) + assert rc == 1 + assert "scope directory not found" in capsys.readouterr().err diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py new file mode 100644 index 00000000..2a3f23b8 --- /dev/null +++ b/tools/check_anchor_resolution.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python3 +"""Wave 4 / Faz 26 — markdown anchor + relative-link resolution guard. + +Walks every Markdown file under ``docs/`` and validates that every +inline link of the form ``[text](path)`` resolves: + +- **Repo-relative paths** (``../qms/foo.md``, ``forgelm/x.py``) must + point at an existing file in the working tree. +- **Markdown anchor links** (``other.md#section-title``) must point + at a header that exists in the target file. +- **External URLs** (``https://...``, ``http://...``) are skipped — + out of scope for offline checks. +- **SPA hash-router fragments** (``#/reference/json-output``) are + skipped — those resolve at site-render time, not in source. +- **Pure anchors** (``#section-title`` with no path) must resolve to + a header in the SAME file. + +Exit codes (per ``tools/`` contract — NOT the public 0/1/2/3/4 +surface that ``forgelm/`` honours): + +- ``0`` — every link resolves cleanly. +- ``1`` — at least one broken link or invalid argument. + +Usage:: + + python3 tools/check_anchor_resolution.py + python3 tools/check_anchor_resolution.py --strict # alias of default; exits 1 on drift + python3 tools/check_anchor_resolution.py --quiet # silent on success + +Closure-plan reference: Faz 26 task #5 +(``tools/check_anchor_resolution.py`` — markdown anchor'ları +resolve edilebilir mi diff). +""" + +from __future__ import annotations + +import argparse +import re +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable + +# Match a Markdown inline link: ``[text](href)``. The href may +# contain spaces (escaped as ``%20`` typically, but we accept raw +# here too). We deliberately do NOT match reference-style links +# (`[text][ref]` + `[ref]: href`) — they are rare in this project +# and add parsing complexity for marginal coverage. +# +# Backtracking-free: ``\[([^\]]*)\]\(([^)]*)\)`` uses negated +# character classes only — no nested quantifiers. +_LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]*)\)") + +# Match an ATX heading: 1-6 leading hashes + space + body. Setext +# headings are out of scope for the anchor target check (the project +# uses ATX exclusively per docs/standards/documentation.md). +_HEADING_RE = re.compile(r"^(#{1,6}) +(.+?)\s*#*\s*$") + +# Markdown image syntax (``![alt](src)``) reuses the link form but is +# checked the same way — every ``src`` must resolve. + +# Skip patterns: anything matching these is treated as out-of-scope. +_SKIP_HREF_PATTERNS = ( + re.compile(r"^https?://"), # external URL + re.compile(r"^mailto:"), # email + re.compile(r"^#/"), # SPA hash-router (user-manual viewer) + re.compile(r"^tel:"), # phone + re.compile(r"^javascript:"), +) + + +@dataclass(frozen=True) +class Link: + source: Path + line: int + text: str + href: str + + +@dataclass(frozen=True) +class BrokenLink: + link: Link + reason: str + + +def _slugify_heading(text: str) -> str: + """GitHub-flavoured Markdown heading-anchor algorithm. + + GFM lowercases, strips most punctuation, replaces spaces with + dashes, and preserves alphanumerics + dashes + underscores. + Approximation good enough for the link-resolution check; matches + the slug GitHub renders for ``[text](#anchor)`` links. + """ + slug = text.lower().strip() + # Strip everything except alphanumeric, space, dash, underscore. + slug = re.sub(r"[^\w\s-]", "", slug, flags=re.UNICODE) + # Collapse whitespace to dashes. + slug = re.sub(r"\s+", "-", slug) + # Collapse multiple dashes. + slug = re.sub(r"-+", "-", slug) + # Trim leading/trailing dashes. + return slug.strip("-") + + +def _extract_anchors(target: Path) -> set[str]: + """Return the set of GFM-style header anchors defined in ``target``.""" + if not target.is_file(): + return set() + anchors: set[str] = set() + try: + text = target.read_text(encoding="utf-8") + except OSError: + return set() + for line in text.splitlines(): + match = _HEADING_RE.match(line) + if match is None: + continue + anchors.add(_slugify_heading(match.group(2))) + return anchors + + +def _walk_markdown_files(root: Path, excluded_dirs: tuple[Path, ...]) -> Iterable[Path]: + """Yield every ``*.md`` under ``root``, sorted for determinism. + + Files under any directory in ``excluded_dirs`` (resolved-form + comparison) are skipped — the default exclude list strips the + gitignored ``docs/analysis/`` research tree (which references + local-only paths from the maintainer's research workflow) so + the checker validates only public docs. + """ + excluded = tuple(p.resolve() for p in excluded_dirs) + for path in sorted(p for p in root.rglob("*.md") if p.is_file()): + resolved = path.resolve() + if any(_is_under(resolved, ex) for ex in excluded): + continue + yield path + + +def _is_under(child: Path, ancestor: Path) -> bool: + """Return True iff ``child`` is contained in ``ancestor`` (inclusive).""" + try: + child.relative_to(ancestor) + except ValueError: + return False + return True + + +def _extract_links(source: Path) -> Iterable[Link]: + try: + text = source.read_text(encoding="utf-8") + except OSError: + return + for line_no, line in enumerate(text.splitlines(), start=1): + for match in _LINK_RE.finditer(line): + text_part = match.group(1) + href_part = match.group(2).strip() + if not href_part: + continue + yield Link(source=source, line=line_no, text=text_part, href=href_part) + + +def _is_skipped(href: str) -> bool: + return any(pat.match(href) for pat in _SKIP_HREF_PATTERNS) + + +def _split_path_anchor(href: str) -> tuple[str, str]: + """Split ``path#anchor`` into ``(path, anchor)``. Either may be empty.""" + if "#" not in href: + return href, "" + path, _, anchor = href.partition("#") + return path, anchor + + +def _resolve_link(link: Link, repo_root: Path) -> BrokenLink | None: + """Return ``None`` if the link resolves; otherwise a ``BrokenLink``.""" + href = link.href + if _is_skipped(href): + return None + path_part, anchor_part = _split_path_anchor(href) + # Pure anchor (``#section``) — target is the same file. + if not path_part: + if not anchor_part: + return BrokenLink(link, "empty href") + anchors = _extract_anchors(link.source) + if anchor_part not in anchors: + return BrokenLink(link, f"anchor #{anchor_part!r} not found in {link.source.name}") + return None + + # Repo-relative path. Resolve against the source file's parent. + target = (link.source.parent / path_part).resolve() + if not target.exists(): + # Fall back: maybe the path is repo-root-relative (legacy + # references). Try that before declaring broken. + alt = (repo_root / path_part).resolve() + if not alt.exists(): + return BrokenLink(link, f"target file not found: {path_part!r}") + target = alt + + if anchor_part: + # Anchors only meaningful for Markdown targets. + if target.suffix.lower() != ".md": + # Code-file anchors (e.g. forgelm/x.py#L42) are stale-line + # references — flag them since they drift on refactor. + # Fail-loud for ``#L`` form, ignore other code-anchor + # forms (e.g. Sphinx `?` queries). + if re.fullmatch(r"L\d+(?:-L\d+)?", anchor_part): + return BrokenLink( + link, f"line-number anchor #{anchor_part!r} on code file is brittle (use symbol references)" + ) + return None + anchors = _extract_anchors(target) + if anchor_part not in anchors: + return BrokenLink(link, f"anchor #{anchor_part!r} not found in {path_part!r}") + return None + + +def _format_broken(broken: BrokenLink) -> str: + rel = broken.link.source + return f"{rel}:{broken.link.line} [{broken.link.text}]({broken.link.href}) → {broken.reason}" + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Validate markdown anchor + relative-path links under docs/.", + ) + parser.add_argument( + "--repo-root", + type=Path, + default=Path(__file__).resolve().parent.parent, + help="Repository root (default: parent of tools/).", + ) + parser.add_argument( + "--scope", + type=str, + default="docs", + help="Subdirectory under repo-root to scan (default: docs).", + ) + parser.add_argument( + "--exclude", + type=str, + action="append", + default=None, + help=( + "Directory (relative to scope-dir) to skip. Repeatable. " + "Default exclude list strips ``analysis/`` (gitignored research " + "tree referencing local-only paths) and ``code_reviews/`` " + "(internal review docs). Pass empty list to scan everything." + ), + ) + parser.add_argument( + "--strict", + action="store_true", + help=( + "Strict mode: exit 1 on any broken link. Default is " + "advisory: report broken links to stdout but exit 0 so the " + "tool can land before the docs tree is clean. CI gate " + "wire-up uses --strict once Faz 30 broken-link cleanup is " + "complete." + ), + ) + parser.add_argument( + "--quiet", + action="store_true", + help="Suppress the OK summary on success.", + ) + args = parser.parse_args(argv) + + repo_root = args.repo_root.resolve() + scope_dir = (repo_root / args.scope).resolve() + if not scope_dir.is_dir(): + print(f"error: scope directory not found: {scope_dir}", file=sys.stderr) + return 1 + + # Default exclude list — `analysis/` is gitignored research with + # local-only path references; `code_reviews/` would be too were + # it not partially re-allowed in `.gitignore`, but its closure-plan + # docs cite real files at line-anchored locations that drift. + if args.exclude is None: + exclude_specs = ("analysis",) + else: + exclude_specs = tuple(args.exclude) + excluded_dirs = tuple((scope_dir / spec).resolve() for spec in exclude_specs) + + broken: list[BrokenLink] = [] + md_files = list(_walk_markdown_files(scope_dir, excluded_dirs)) + for md in md_files: + for link in _extract_links(md): + failure = _resolve_link(link, repo_root) + if failure is not None: + broken.append(failure) + + if broken: + verdict = "FAIL" if args.strict else "WARN" + print(f"{verdict}: broken anchor / relative-link references:") + for entry in broken: + print(f" {_format_broken(entry)}") + print(f"\n{len(broken)} broken link(s) across {len(md_files)} markdown file(s) under {args.scope}/.") + return 1 if args.strict else 0 + + if not args.quiet: + print(f"OK: {len(md_files)} markdown file(s) under {args.scope}/ have all anchors + relative links resolved.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 8f69cf1a5b4c0c0ca4d8635d6a0f7aed7f2b8c83 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 00:35:16 +0300 Subject: [PATCH 05/12] feat(wave-4-faz-30): partial final-documentation pass + closure-plan sync MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Faz 30 partial implementation: Tier 1 ghost-feature drift fixes + stat-block updates + closure-plan post-merge sync. Full Faz 30 sweep deferred to a follow-up commit (the closure plan's Faz 30 table A is 17 user-facing-feature rows; this commit closes the high-impact subset and documents the deferred work.) Tier 1 ghost-feature drift fixes (closure-plan task O subset) ============================================================= * GH-008 `verify-log` → `verify-audit` rename in `usermanuals/{en,tr}/compliance/audit-log.md` (Phase 6 shipped as `verify-audit`; the doc still cited the pre-rename name). * GH-021 `forgelm chat` slash commands aligned with parser (`forgelm chat --help` shows only `/reset`, `/save`, `/temperature`, `/system`, `/exit`): - removed `/load`, `/top_p`, `/max_tokens`, `/safety on|off` - `/quit` → `/exit` - "Saving and replaying" section reframed: `/save` is real; replay tooling is roadmapped for v0.6.0+ Pro CLI. EN+TR mirror. * GH-022 `q6_k` quant level removed from `usermanuals/{en,tr}/deployment/gguf-export.md`. The parser at `forgelm/cli/_parser.py:75-76` only supports `q2_k|q3_k_m|q4_k_m|q5_k_m|q8_0|f16`; `q6_k` was a fictional intermediate. Added the `f16` (lossless) row that was missing. * GH-024 `FORGELM_RESUME_TOKEN` env var removed from `usermanuals/{en,tr}/compliance/human-oversight.md`. The "API self-service automation" block citing this env var was pure fiction (no resume endpoint, no JWT-signed approval webhook callback). Replaced with the canonical CLI subcommand flow (`forgelm approve` / `reject` + `forgelm approvals --pending`) that actually exists, plus a v0.6.0+ Pro CLI roadmap callout for self-service automation. * GH-025 `FORGELM_CACHE_DIR` env var removed from `usermanuals/{en,tr}/getting-started/project-layout.md`. ForgeLM does not introduce its own cache-dir override; the canonical pattern is `HF_HOME` (matches Phase 35 `cache-models` + `cache-tasks` subcommand expectations). Stat-block updates (closure-plan task E) ======================================== * `docs/standards/testing.md` — "Current structure (48 test modules)" → "Current structure (67 test modules, one per feature area; ~1410 tests)". Reflects the post-Wave-3 + Wave-4 test- suite size. Closure-plan post-merge sync ============================ * `docs/analysis/code_reviews/closure-plan-202604300906.md`: - Faz 22, 23, 26, 30 status changed from `⏳ PENDING` to `✅ Wave 4 (PR pending)`. - Status legend gained a Wave 4 entry. - Added Wave 4 kapanış özeti naming the deliverables: 1 design doc (~865 lines), 4 new QMS docs, 6 existing-QMS TR mirrors, 4 new-QMS TR mirrors, 1 new guide (deployer cookbook EN+TR), 3 new reference docs (iso27001 / soc2 / supply_chain_security EN+TR), 3 new tools (check_pip_audit, check_bandit, check_anchor_resolution), `[security]` optional extra, nightly supply-chain-security job. - "Kalan iş" reframed: only Faz 33 release remains (post-wave), plus optional Faz 30 follow-up for the deferred items. Deferred to Faz 30 follow-up ============================ The closure plan's Faz 30 task A is a 17-row feature × doc-triplet table. Many rows are already covered from Wave 1-3; the rest is a focused doc-completion sweep that benefits from being its own commit. Specifically deferred: - GH-016 `--export-bundle` → `--compliance-export` rename in QMS SOPs (CLI flag is correct in code; only doc references drift). - GH-018 `kserve` / `triton` deploy targets removal from reference/cli.md. - GH-019 chat reference table cleanup in reference/cli.md. - GH-020 ingest reference table cleanup in reference/cli.md. - Sample-audit user-facing doc triplet completion (closure-plan Faz 30 task A's 17-row table — verify each feature's EN+TR guide + reference + usermanual triplet). - `tools/check_cli_help_consistency.py` addition + CI wire-up (closure-plan Faz 30 task J). - Full-suite `tools/check_anchor_resolution.py --strict` CI wire-up (currently advisory mode; 36 broken links in tree pre-Faz 26 — sweep + wire-up together). Validation ========== - ruff format + ruff check clean. - pytest 1397 passed, 14 skipped (no new tests; Faz 30 is doc-only). - python3 tools/check_bilingual_parity.py --strict → 13/13 OK. - python3 tools/check_anchor_resolution.py (advisory) → 36 broken-link warnings unchanged from pre-Faz-26 baseline; will shrink to 0 in Faz 30 follow-up. - forgelm --config config_template.yaml --dry-run → green. Closure-plan reference: Faz 30 task list at docs/analysis/code_reviews/closure-plan-202604300906.md:1203-1383. Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 51 +++++++++++++++++++ .../code_reviews/closure-plan-202604300906.md | 13 +++-- docs/standards/testing.md | 2 +- docs/usermanuals/en/compliance/audit-log.md | 4 +- .../en/compliance/human-oversight.md | 14 ++--- docs/usermanuals/en/deployment/chat.md | 12 ++--- docs/usermanuals/en/deployment/gguf-export.md | 4 +- .../en/getting-started/project-layout.md | 2 +- docs/usermanuals/tr/compliance/audit-log.md | 4 +- .../tr/compliance/human-oversight.md | 14 ++--- docs/usermanuals/tr/deployment/chat.md | 13 ++--- docs/usermanuals/tr/deployment/gguf-export.md | 4 +- .../tr/getting-started/project-layout.md | 2 +- 13 files changed, 93 insertions(+), 46 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4201da72..dce9c001 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -82,6 +82,57 @@ Cross-cutting: pairs (deployer guide + 3 reference docs); tally now 13 doc pairs (was 9). +**Faz 26 — QMS bilingual mirror + compliance_summary cleanup** + +- 6 existing QMS TR mirrors: `README-tr.md`, + `roles_responsibilities-tr.md`, `sop_model_training-tr.md`, + `sop_data_management-tr.md`, `sop_incident_response-tr.md`, + `sop_change_management-tr.md`. +- 4 new-Faz-23 QMS TR mirrors: `encryption_at_rest-tr.md`, + `access_control-tr.md`, `risk_treatment_plan-tr.md`, + `statement_of_applicability-tr.md`. +- `docs/qms/README.md` (+ `-tr.md`) — table extended with the 4 + Wave 4 QMS rows; "See also" section linking to compliance_summary + + ISO/SOC 2 deployer guide + audit event catalog. +- `docs/reference/compliance_summary.md` — full rewrite: H1 + scope + blockquote, 11 stale `forgelm/x.py#L33`-style line anchors + replaced with module-path + symbol-name references that survive + refactor, "50 prompts × 3 categories" → "140 × 6", linked from + QMS README. +- `tools/check_anchor_resolution.py` (new) — markdown link + resolution guard; advisory mode by default + `--strict` opt-in + for CI wire-up once tree is clean. 21 regression tests. + +**Faz 30 — Final documentation pass (partial)** + +Tier 1 ghost-feature drift fixes + stat-block updates: + +- GH-008 `verify-log` → `verify-audit` rename in + `usermanuals/{en,tr}/compliance/audit-log.md`. +- GH-021 `forgelm chat` slash commands aligned with parser: + removed `/load`, `/top_p`, `/max_tokens`, `/safety on|off`; + `/quit` → `/exit`. EN+TR mirror. +- GH-022 `q6_k` quant level removed from + `usermanuals/{en,tr}/deployment/gguf-export.md` (parser only + supports `q2_k|q3_k_m|q4_k_m|q5_k_m|q8_0|f16`); `f16` row + added. +- GH-024 `FORGELM_RESUME_TOKEN` env var removed from + `usermanuals/{en,tr}/compliance/human-oversight.md`; replaced + the API-automation block with the canonical CLI subcommand + flow (`forgelm approve` / `reject` + `forgelm approvals + --pending`); the v0.6.0+ Pro CLI roadmap callout flagged. +- GH-025 `FORGELM_CACHE_DIR` env var removed from + `usermanuals/{en,tr}/getting-started/project-layout.md`; + `HF_HOME` declared canonical. +- `docs/standards/testing.md` stat block: 48 → 67 test modules, + ~1410 tests. + +Deferred to Faz 30 follow-up (closure-plan task A + remaining O +items + tasks J + N): GH-016/018/019/020 reference/cli.md cleanup, +sample-audit user-facing doc triplet completion, +`tools/check_cli_help_consistency.py`, full-suite anchor-checker +`--strict` CI wire-up. + ### Wave 3 — Faz 24 + 28 + 38 (`closure/wave3-integration`) Single integration branch covering three closure-plan phases: diff --git a/docs/analysis/code_reviews/closure-plan-202604300906.md b/docs/analysis/code_reviews/closure-plan-202604300906.md index 9ac1635d..634a16bf 100644 --- a/docs/analysis/code_reviews/closure-plan-202604300906.md +++ b/docs/analysis/code_reviews/closure-plan-202604300906.md @@ -91,15 +91,15 @@ Her faz bir veya birden çok PR'la kapanır. Sıralama dependency-driven; parale | **19** | **Library API support — Implementation** | Tek PR | 1 | Faz 18 | ✅ Wave 2b (PR #30) | | **20** | **GDPR right-to-erasure — Analysis & design** | Tek PR | 1 | Faz 6 + 9 (audit + staging) | ✅ Wave 2a (PR #28) | | **21** | **GDPR right-to-erasure — Implementation** | Tek PR | 1 | Faz 20 | ✅ Wave 2b (PR #30) | -| **22** | **ISO 27001 / SOC 2 Type II — Analysis & gap assessment** | Tek PR | 1 | Faz 3, 6, 7, 8, 9, 21 (security + compliance baseline) | ⏳ PENDING | -| **23** | **ISO 27001 / SOC 2 Type II — Implementation** | Tek PR (büyük) | 1 | Faz 22 | ⏳ PENDING | +| **22** | **ISO 27001 / SOC 2 Type II — Analysis & gap assessment** | Tek PR | 1 | Faz 3, 6, 7, 8, 9, 21 (security + compliance baseline) | ✅ Wave 4 (PR pending) | +| **23** | **ISO 27001 / SOC 2 Type II — Implementation** | Tek PR (büyük) | 1 | Faz 22 | ✅ Wave 4 (PR pending) | | **24** | Bilingual TR mirror sweep (configuration-tr, distributed-tr, ingestion-tr, EN/TR user-manual H2 drift) | Tek PR | 1 | Faz 16 (description= migration ile config docs auto-gen) | ✅ Wave 3 (PR #31) | | **25** | Site picker honesty + site-as-tested-surface CI guard | Tek PR | 1 | Faz 1 (site honesty fixes önce) | ✅ DONE | -| **26** | QMS bilingual EN+TR mirror + compliance docs cleanup | Tek PR | 1 | Faz 23 (ISO/SOC 2 dokümanları QMS ile lockstep) | ⏳ PENDING | +| **26** | QMS bilingual EN+TR mirror + compliance docs cleanup | Tek PR | 1 | Faz 23 (ISO/SOC 2 dokümanları QMS ile lockstep) | ✅ Wave 4 (PR pending) | | **27** | Silent `except Exception:` sweep (Theme D residual) | Tek PR | 1 | — | ✅ DONE | | **28** | Remaining Major + Minor cleanup | Tek PR | 1 | Hepsi öncesindekilere bağlı | ✅ Wave 3 (PR #31; tier 1+2 absorbed) | | **29** | Nit sweep (40 madde mass-edit) | Tek PR | 1 | — | ✅ DONE | -| **30** | **Final documentation + site finalization (EN+TR only)** — tüm `docs/` + `docs/usermanuals/{en,tr}/` + `site/*` v0.5.5 nihai durumuna senkron | Tek PR | 1 | **Faz 1-29 + 34-38** (tüm fonksiyonel iş bitmiş olmalı) | ⏳ PENDING | +| **30** | **Final documentation + site finalization (EN+TR only)** — tüm `docs/` + `docs/usermanuals/{en,tr}/` + `site/*` v0.5.5 nihai durumuna senkron | Tek PR | 1 | **Faz 1-29 + 34-38** (tüm fonksiyonel iş bitmiş olmalı) | ✅ Wave 4 (PR pending; partial — Tier 1 ghost feature drift + stat blocks; full sweep deferred to Faz 30 follow-up) | | **31** | **Cross-OS release-tag matrix workflow** | Tek PR | 1 | — (release öncesi her şeyden) | ✅ DONE | | **32** | `.pre-commit-config.yaml` (opsiyonel — kullanıcı tercihi) | Tek PR | 1 | — | ✅ DONE | | **34** | **`forgelm doctor` env-check subcommand** (GH-001 — onboarding bloker) | Tek PR | 1 | Faz 15 (cli/ split) | ✅ Wave 2a (PR #28) | @@ -116,6 +116,7 @@ Her faz bir veya birden çok PR'la kapanır. Sıralama dependency-driven; parale - ✅ Wave 2a (PR #28) — `closure/wave2a-integration` → development, 2026-05-04; faz tamamlandı. - ✅ Wave 2b (PR #30) — `closure/wave2b-integration` → development, 2026-05-05 (merge commit `b05edb5`); faz tamamlandı. - ✅ Wave 3 (PR #31) — `closure/wave3-integration` → development, 2026-05-05 (merge commit `b87c872`); faz tamamlandı. +- ✅ Wave 4 (PR pending) — `closure/wave4-integration` → development, 2026-05-06; ISO 27001 / SOC 2 alignment dalı + QMS bilingual mirror sweep + Faz 30 partial (Tier 1 ghost-feature drift + stat blocks). - ⏳ PENDING — henüz başlanmadı; bağımlılıkları sağlanan ilk batch sıradaki Wave 4+'da açılır. - ⏳ POST-WAVE — release fazı (33), tüm fonksiyonel iş bittikten sonra son. @@ -127,7 +128,9 @@ Her faz bir veya birden çok PR'la kapanır. Sıralama dependency-driven; parale **Wave 3 kapanış özeti (post-merge, 2026-05-05):** 3 faz tamamlandı (Faz 24 bilingual TR mirror sweep + `tools/check_bilingual_parity.py` CI guard, Faz 28 curated tier 1+2 cleanup, Faz 38 `forgelm reverse-pii` GDPR Article 15 subcommand). PR #31 → development (merge commit `b87c872`). Yeni: 1 CLI subcommand (`reverse-pii`), 1 audit event (`data.access_request_query`), 1 yeni CI guard tool (`tools/check_bilingual_parity.py`). Behaviour change: high-risk + unacceptable + safety-disabled artık `ConfigError` raise eder (F-compliance-110, OR-across-fields strict gate). Webhook timeout default 5s → 10s (F-compliance-106). 12 commit (1 ana + 11 absorption rounds: 4-agent original review + 5 SonarCloud findings + 14 inline review comments across CodeRabbit/Gemini/inline + 5 follow-up multi-agent review absorptions). Suite 1298 → 1374 tests (+76 net). -**Kalan iş (Wave 4+):** 4 faz + 1 release. Pending: Faz 22 (ISO 27001/SOC 2 design), Faz 23 (ISO 27001/SOC 2 impl), Faz 26 (QMS bilingual mirror), Faz 30 (final docs + site finalization). Release: Faz 33 (v0.5.5 PyPI publish). Bağımlılık zinciri: 22 → 23 → 26 (QMS lockstep ISO/SOC 2 docs ile), Faz 30 tüm fonksiyonel iş bittikten sonra (yani 22+23+26 kapanmadan açılmaz), Faz 33 son. +**Wave 4 kapanış özeti (2026-05-06, PR pending):** 4 faz tamamlandı (Faz 22 ISO 27001 / SOC 2 alignment design, Faz 23 implementation, Faz 26 QMS bilingual mirror + compliance_summary cleanup + anchor checker, Faz 30 partial Tier 1 ghost-feature drift + stat blocks). Branch: `closure/wave4-integration`. Yeni: 1 design dosyası (~865 satır), 4 yeni QMS doküman (encryption_at_rest, access_control, risk_treatment_plan, statement_of_applicability), 6 yeni QMS TR mirror, 1 yeni guide (deployer cookbook EN+TR), 3 yeni reference (iso27001_control_mapping EN+TR, soc2_trust_criteria_mapping EN+TR, supply_chain_security EN+TR), 3 yeni tool (`tools/check_pip_audit.py`, `tools/check_bandit.py`, `tools/check_anchor_resolution.py`), 1 yeni `[security]` optional extra (pip-audit + bandit), 1 yeni nightly job (supply-chain-security). Behaviour change: yok. Suite 1374 → 1397 tests (+23 net: 16 supply-chain + 21 anchor checker − 14 displaced). compliance_summary.md tam rewrite (line-anchored references symbol-references'a geçti). Bilingual parity 9/9 → 13/13. + +**Kalan iş (Wave 4+):** 1 release. Pending: Faz 33 (v0.5.5 PyPI publish, post-wave). Optional follow-up before release: Faz 30 full sweep (kalan Tier 1 ghost-feature drift items GH-016/018/019/020, sample-audit user-facing doc triplet completion, tools/check_cli_help_consistency.py addition, full-suite anchor-checker --strict CI wire-up). --- diff --git a/docs/standards/testing.md b/docs/standards/testing.md index d66a0cf6..66a867c5 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -5,7 +5,7 @@ ## Layout -Current structure (48 test modules, one per feature area): +Current structure (67 test modules, one per feature area; ~1410 tests): ``` tests/ diff --git a/docs/usermanuals/en/compliance/audit-log.md b/docs/usermanuals/en/compliance/audit-log.md index 461b18a9..b4284fb8 100644 --- a/docs/usermanuals/en/compliance/audit-log.md +++ b/docs/usermanuals/en/compliance/audit-log.md @@ -56,13 +56,13 @@ ForgeLM never rewrites prior log entries. New events go at the end. The chained ## Verifying integrity ```shell -$ forgelm verify-log checkpoints/run/artifacts/audit_log.jsonl +$ forgelm verify-audit checkpoints/run/artifacts/audit_log.jsonl ✓ 87 entries, all timestamps monotonic ✓ all prev_hash chains valid ✓ no gaps in seq numbers ``` -If `verify-log` reports a chain break, the log was modified after generation. Investigate before treating it as evidence. +If `verify-audit` reports a chain break, the log was modified after generation. Investigate before treating it as evidence. ## Per-run vs per-project diff --git a/docs/usermanuals/en/compliance/human-oversight.md b/docs/usermanuals/en/compliance/human-oversight.md index 2d56ee11..e7e6c82a 100644 --- a/docs/usermanuals/en/compliance/human-oversight.md +++ b/docs/usermanuals/en/compliance/human-oversight.md @@ -70,17 +70,17 @@ approval: The trainer halts and posts the artifact bundle to your webhook. Your system handles the human review and POSTs back to ForgeLM's resume endpoint with a signed JWT. -### API +### CLI subcommand (canonical) -For self-service automation (e.g. a "promote this run" button in your dashboard): +The supported approval mechanism in v0.5.5 is the CLI subcommand pair `forgelm approve` / `forgelm reject`: -```yaml -approval: - signature_method: "api" - resume_token: "${FORGELM_RESUME_TOKEN}" +```bash +forgelm approvals --pending # list runs awaiting approval +forgelm approve --run-id # promote staging → final_model +forgelm reject --run-id --reason "..." # discard the staged model ``` -Your dashboard calls ForgeLM's resume endpoint directly with the run ID and reviewer identity. Signatures are recorded in the audit log. +Each invocation requires `FORGELM_OPERATOR` (the approver's identity) and writes a `human_approval.granted` / `human_approval.rejected` event to the chain. Self-service "promote this run" automation is roadmapped for v0.6.0+ Pro CLI (Phase 13 in the public roadmap); until then the CLI gate is the audit-grade interface. ## What's in an approval signature diff --git a/docs/usermanuals/en/deployment/chat.md b/docs/usermanuals/en/deployment/chat.md index 5bdaca75..8be93621 100644 --- a/docs/usermanuals/en/deployment/chat.md +++ b/docs/usermanuals/en/deployment/chat.md @@ -30,14 +30,10 @@ yolunu izleyebilirsiniz... |---|---| | `/reset` | Clear conversation history. | | `/save ` | Save the conversation to JSONL. | -| `/load ` | Load a previous conversation. | | `/system ` | Set or update the system prompt. | | `/temperature ` | Set sampling temperature (0.0 to 2.0). | -| `/top_p ` | Set nucleus sampling parameter. | -| `/max_tokens ` | Cap response length. | -| `/safety on|off` | Toggle Llama Guard pre/post screening. | | `/help` | Show this list. | -| `/quit` or `Ctrl+D` | Exit. | +| `/exit` or `Ctrl+D` | Exit. | ## Configuration @@ -97,11 +93,11 @@ To start fresh: `/reset`. ```shell forgelm> /save sessions/qa-1.jsonl [saved 6 turns to sessions/qa-1.jsonl] - -forgelm> /load sessions/qa-1.jsonl -[loaded 6 turns; ready to continue] ``` +The saved JSONL is replayable by feeding it back through a fresh +session's prompt history (replay tooling is roadmapped for v0.6.0+ Pro CLI). + Sessions are useful for: - Reproducing a bug you found during testing. - Building a benchmark prompt set from interactive exploration. diff --git a/docs/usermanuals/en/deployment/gguf-export.md b/docs/usermanuals/en/deployment/gguf-export.md index 736cb7d9..60371652 100644 --- a/docs/usermanuals/en/deployment/gguf-export.md +++ b/docs/usermanuals/en/deployment/gguf-export.md @@ -25,8 +25,8 @@ The output is a single `.gguf` file plus a `.sha256` manifest. | Level | Size (7B base) | Quality | Use case | |---|---|---|---| -| `q8_0` | 7.2 GB | Highest | Quality benchmark; production where memory is plentiful. | -| `q6_k` | 5.5 GB | Very high | Default for production. | +| `f16` | 13 GB | Lossless | Quality benchmark; full-precision archive. | +| `q8_0` | 7.2 GB | Highest | Production where memory is plentiful. | | `q5_k_m` | 4.8 GB | High | Sensible balance. | | `q4_k_m` | 4.1 GB | Good | **Default for local inference.** | | `q3_k_m` | 3.3 GB | Acceptable | Tight memory; some quality loss. | diff --git a/docs/usermanuals/en/getting-started/project-layout.md b/docs/usermanuals/en/getting-started/project-layout.md index 56eee374..915e7f90 100644 --- a/docs/usermanuals/en/getting-started/project-layout.md +++ b/docs/usermanuals/en/getting-started/project-layout.md @@ -80,7 +80,7 @@ ForgeLM uses these path conventions — change them with command-line flags but | Config file | `configs/.yaml` | `--config PATH` | | Audit output | `./audit/` | `forgelm audit --output PATH` | | Training output | `./checkpoints//` | `output.dir:` in YAML | -| Cache directory | `~/.forgelm/cache/` | `FORGELM_CACHE_DIR` env var | +| HuggingFace cache | `~/.cache/huggingface/` | `HF_HOME` env var (canonical; ForgeLM does not introduce its own cache-dir override) | | HuggingFace token | env `HF_TOKEN` | `auth.hf_token:` in YAML | ## Multi-config workflows diff --git a/docs/usermanuals/tr/compliance/audit-log.md b/docs/usermanuals/tr/compliance/audit-log.md index d6ec29af..4f932c8b 100644 --- a/docs/usermanuals/tr/compliance/audit-log.md +++ b/docs/usermanuals/tr/compliance/audit-log.md @@ -56,13 +56,13 @@ ForgeLM önceki log kayıtlarını asla yeniden yazmaz. Yeni olaylar sona ekleni ## Bütünlük doğrulama ```shell -$ forgelm verify-log checkpoints/run/artifacts/audit_log.jsonl +$ forgelm verify-audit checkpoints/run/artifacts/audit_log.jsonl ✓ 87 kayıt, tüm zaman damgaları monotonik ✓ tüm prev_hash zincirleri geçerli ✓ seq numaralarında boşluk yok ``` -`verify-log` zincir kırığı raporlarsa, log üretimden sonra değiştirilmiş demektir. Kanıt olarak işlem görmeden önce araştırın. +`verify-audit` zincir kırığı raporlarsa, log üretimden sonra değiştirilmiş demektir. Kanıt olarak işlem görmeden önce araştırın. ## Koşu başı vs proje başı diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index a67b2ff4..3e68b266 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -70,17 +70,17 @@ approval: Trainer durur ve artifact paketini webhook'unuza POST'lar. Sisteminiz insan incelemesini halleder ve imzalı JWT ile ForgeLM'in resume endpoint'ine geri POST'lar. -### API +### CLI subcommand (canonical) -Self-servis otomasyon için (ör. dashboard'unuzda "bu koşuyu terfi ettir" düğmesi): +v0.5.5'te desteklenen onay mekanizması CLI subcommand çifti `forgelm approve` / `forgelm reject`: -```yaml -approval: - signature_method: "api" - resume_token: "${FORGELM_RESUME_TOKEN}" +```bash +forgelm approvals --pending # onay bekleyen koşumları listele +forgelm approve --run-id # staging → final_model'e promote +forgelm reject --run-id --reason "..." # staged modeli at ``` -Dashboard'unuz doğrudan ForgeLM'in resume endpoint'ini koşu ID'si ve reviewer kimliğiyle çağırır. İmzalar audit log'a kaydedilir. +Her çağrı `FORGELM_OPERATOR` (onaylayanın kimliği) gerektirir ve zincire `human_approval.granted` / `human_approval.rejected` olayı yazar. Self-servis "bu koşuyu terfi ettir" otomasyonu v0.6.0+ Pro CLI (public roadmap'te Phase 13) için planlanmıştır; o zamana kadar CLI gate audit-grade arayüzdür. ## Onay imzasında ne var diff --git a/docs/usermanuals/tr/deployment/chat.md b/docs/usermanuals/tr/deployment/chat.md index 7f9db76e..8f50722c 100644 --- a/docs/usermanuals/tr/deployment/chat.md +++ b/docs/usermanuals/tr/deployment/chat.md @@ -30,14 +30,10 @@ Tarifenizi Ayarlar → Plan → Yükselt menüsünden değiştirebilirsiniz... |---|---| | `/reset` | Konuşma geçmişini temizle. | | `/save ` | Konuşmayı JSONL'a kaydet. | -| `/load ` | Önceki konuşmayı yükle. | | `/system ` | System prompt'u ayarla veya güncelle. | | `/temperature ` | Sampling sıcaklığını ayarla (0.0 - 2.0). | -| `/top_p ` | Nucleus sampling parametresi. | -| `/max_tokens ` | Yanıt uzunluğunu sınırla. | -| `/safety on|off` | Llama Guard pre/post tarayıcısını aç/kapa. | | `/help` | Bu listeyi göster. | -| `/quit` veya `Ctrl+D` | Çık. | +| `/exit` veya `Ctrl+D` | Çık. | ## Konfigürasyon @@ -97,11 +93,12 @@ Sıfırdan başlamak için: `/reset`. ```shell forgelm> /save sessions/qa-1.jsonl [6 tur sessions/qa-1.jsonl'a kaydedildi] - -forgelm> /load sessions/qa-1.jsonl -[6 tur yüklendi; devam etmeye hazır] ``` +Kaydedilen JSONL, yeni bir oturumun prompt history'sine geri +besleyerek tekrar oynatılabilir (replay tooling v0.6.0+ Pro CLI için +roadmap'tedir). + Oturumlar şu durumlar için faydalı: - Test sırasında bulduğunuz bir bug'ı yeniden üretmek. - İnteraktif keşiften benchmark prompt seti kurmak. diff --git a/docs/usermanuals/tr/deployment/gguf-export.md b/docs/usermanuals/tr/deployment/gguf-export.md index 003f67a8..d5269b75 100644 --- a/docs/usermanuals/tr/deployment/gguf-export.md +++ b/docs/usermanuals/tr/deployment/gguf-export.md @@ -25,8 +25,8 @@ $ forgelm export ./checkpoints/customer-support \ | Seviye | Boyut (7B base) | Kalite | Kullanım | |---|---|---|---| -| `q8_0` | 7.2 GB | En yüksek | Kalite benchmark; bellek bol üretim. | -| `q6_k` | 5.5 GB | Çok yüksek | Üretim için varsayılan. | +| `f16` | 13 GB | Kayıpsız | Kalite benchmark; tam-precision arşiv. | +| `q8_0` | 7.2 GB | En yüksek | Bellek bol üretim. | | `q5_k_m` | 4.8 GB | Yüksek | Mantıklı denge. | | `q4_k_m` | 4.1 GB | İyi | **Yerel inference için varsayılan.** | | `q3_k_m` | 3.3 GB | Kabul edilebilir | Dar bellek; biraz kalite kaybı. | diff --git a/docs/usermanuals/tr/getting-started/project-layout.md b/docs/usermanuals/tr/getting-started/project-layout.md index 6630301c..746de8ed 100644 --- a/docs/usermanuals/tr/getting-started/project-layout.md +++ b/docs/usermanuals/tr/getting-started/project-layout.md @@ -78,7 +78,7 @@ ingested/ | Config dosyası | `configs/.yaml` | `--config PATH` | | Audit çıktısı | `./audit/` | `forgelm audit --output PATH` | | Eğitim çıktısı | `./checkpoints//` | YAML'da `output.dir:` | -| Cache dizini | `~/.forgelm/cache/` | `FORGELM_CACHE_DIR` env var | +| HuggingFace cache | `~/.cache/huggingface/` | `HF_HOME` env var (canonical; ForgeLM kendi cache-dir override'ını eklemiyor) | | HuggingFace token | `HF_TOKEN` env | YAML'da `auth.hf_token:` | ## Çoklu config iş akışı From 9a1738a318e06e353763ed8ffb5a165c5330ee8a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 00:39:28 +0300 Subject: [PATCH 06/12] docs(wave-4): multi-agent review prompt for closure/wave4-integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Frozen at 8f69cf1 / PR #33. Same 4-agent split as Wave 3 (code-correctness / privacy-security / standards-compliance / test-rigour) per the established review cadence. Critical framing rules baked into the prompt: - D-22-01 enforcement: any "compliant" / "certified" wording found in the diff is a CRITICAL finding (alignment-vs- certified is the single most-impactful regression a reviewer can catch on this PR). - D-22-02 enforcement: SBOM stays CycloneDX 1.5; any SPDX claim is a finding. - Salt + HMAC chain are tamper-evidence, not non-repudiation — reject PKI-flavoured claims. - Wave 4 explicitly defers some Faz 30 items (GH-016/018/019/020 cli.md cleanup, sample-audit user-facing doc triplet, new check_cli_help_consistency tool, full-suite anchor-strict CI wire-up); reviewers should NOT flag those as Wave 4 omissions. Severity bar matches Wave 3 calibration; deliverable structure matches Wave 3 (one report per agent, saved as wave4-.md under docs/analysis/code_reviews/). Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + .../code_reviews/wave4-review-prompt.md | 419 ++++++++++++++++++ 2 files changed, 420 insertions(+) create mode 100644 docs/analysis/code_reviews/wave4-review-prompt.md diff --git a/.gitignore b/.gitignore index 0711b9d8..cee4d854 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,7 @@ docs/analysis/code_reviews/* !docs/analysis/code_reviews/wave3-review-prompt.md !docs/analysis/code_reviews/wave3-followup-review-prompt.md !docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md +!docs/analysis/code_reviews/wave4-review-prompt.md # Generated by tools/build_usermanuals.py — produced from # docs/usermanuals//**/*.md during the CI deploy. Not committed; diff --git a/docs/analysis/code_reviews/wave4-review-prompt.md b/docs/analysis/code_reviews/wave4-review-prompt.md new file mode 100644 index 00000000..c50899bb --- /dev/null +++ b/docs/analysis/code_reviews/wave4-review-prompt.md @@ -0,0 +1,419 @@ +# Wave 4 — Multi-Agent Code Review Prompt (PR #33) + +> **Use this prompt verbatim** to launch the multi-agent review on +> the consolidated Wave 4 PR. The prompt is self-contained: each +> reviewing agent can pick it up cold without context from prior +> conversations. +> +> Wave 4 consolidates four closure-plan phases (Faz 22 + 23 + 26 + +> Faz 30 partial). No prior absorption rounds yet — this is the +> FIRST review pass. Treat the bar accordingly: catch the issues +> now, before any operator-visible regressions land. + +--- + +## Repo & branch under review + +- **Repo:** `cemililik/ForgeLM` +- **PR:** [#33](https://github.com/cemililik/ForgeLM/pull/33) + (`closure/wave4-integration` → `development`) +- **Frozen HEAD:** `8f69cf1` +- **Diff vs `development`:** ~30 files, ~3500 insertions, ~50 + deletions. Five new code/tool files + (`tools/check_pip_audit.py`, `tools/check_bandit.py`, + `tools/check_anchor_resolution.py`, + `tests/test_supply_chain_security.py`, + `tests/test_check_anchor_resolution.py`), one new design doc + (~865 lines), four new QMS docs (EN), ten new QMS TR mirrors, + one new guide pair (EN+TR), three new reference pairs (EN+TR), + five modified usermanual EN+TR pairs (ghost-feature drift). +- **What it consolidates:** Faz 22 (ISO 27001 / SOC 2 alignment + design), Faz 23 (implementation: pip-audit + bandit CI, 4 new + QMS docs, deployer guide, 3 reference tables), Faz 26 (QMS + bilingual mirror sweep + `compliance_summary.md` cleanup + + `tools/check_anchor_resolution.py`), Faz 30 partial (Tier 1 + ghost-feature drift + stat blocks). + +## What ForgeLM is (60 seconds) + +A **config-driven, enterprise-grade LLM fine-tuning toolkit** — +YAML in, fine-tuned model + compliance artifacts out. Built for +CI/CD pipelines, not notebooks. Six alignment paradigms +(SFT/DPO/SimPO/KTO/ORPO/GRPO), integrated safety evaluation +(Llama Guard), EU AI Act compliance artefacts (Articles 9–17 + +Annex IV), append-only audit log, opt-in human approval gate, +auto-revert on quality regression, GDPR Article 15 + 17 subject +rights tooling. Wave 4 adds the deployer-side ISO 27001 / SOC 2 +Type II audit-evidence layer. Read [docs/product_strategy.md](../../product_strategy.md) +for the fuller background. + +Project rulebook lives under [docs/standards/](../../standards/) — +read [coding.md](../../standards/coding.md), +[error-handling.md](../../standards/error-handling.md), +[testing.md](../../standards/testing.md), +[logging-observability.md](../../standards/logging-observability.md), +[regex.md](../../standards/regex.md), +[localization.md](../../standards/localization.md), +[documentation.md](../../standards/documentation.md), and +[code-review.md](../../standards/code-review.md) before +commenting. Project-wide guidance for AI agents is in the root +`CLAUDE.md`. + +## Critical framing for this review (don't skip) + +**Wave 4's headline claim is ALIGNMENT, not CERTIFICATION.** +Software (a Python library) cannot be ISO 27001 certified — only +**organisations** can. Wave 4's design doc Decision D-22-01 makes +this explicit. If you find any text in this PR that says +"compliant" or "certified" instead of "aligned" / "supports the +deployer's certification" — flag it CRITICAL. This is the +single highest-impact regression a reviewer can catch. + +Additionally: + +- **Don't gate the auditor's deployer-side controls.** ForgeLM + contributes evidence; the deployer enforces. Reject any PR text + suggesting ForgeLM "enforces" things it merely records. +- **Salt + HMAC chain are tamper-evidence, not non-repudiation.** + Reject any text saying ForgeLM provides cryptographic + non-repudiation in a PKI sense. +- **No SPDX claim.** The SBOM emitter is CycloneDX 1.5 (Wave 2 + era). Decision D-22-02 commits to that. Reject any text + saying ForgeLM ships SPDX SBOMs. + +## What we want from this round + +**Goal:** decide whether `closure/wave4-integration` at `8f69cf1` +is mergeable to `development`. Block on real defects; do not gate +on style nits or hypothetical futures. + +Be concrete. Every finding must include: + +1. **Severity tag** — `CRITICAL` / `HIGH` / `MEDIUM` / `LOW` / `NIT`. +2. **Finding ID** — `F-W4-NN` (e.g. `F-W4-01`, `F-W4-12`). +3. **File:line citation** — `tools/check_pip_audit.py:48` or a + short range `:42-55`. +4. **One-paragraph reasoning** — what is wrong, what is the + user-visible consequence, why it qualifies for that severity. +5. **Suggested fix** — a code or text snippet, not a vague + direction. +6. **Test that would have caught it** — name the test file + a + stub of the assertion. For doc-only findings, name the + anchor checker rule that should have caught it. + +Severity bar: + +- `CRITICAL` — would corrupt data, silently lose audit events, + leak PII / secrets, break the public exit-code contract + (`0/1/2/3/4`), break the `--offline` / air-gap promise, + invalidate the audit-log hash chain, OR mislead the deployer + into believing they are ISO/SOC 2 *certified* (vs. *aligned*). + Blocks merge. +- `HIGH` — runtime crash on a documented happy path, missing test + for a documented contract, schema mismatch with `pyproject.toml` + / `config.py`, documentation claim with no code backing, or a + behaviour change that breaks 0.5.x users without a clear + migration path. Blocks merge. +- `MEDIUM` — logic bug in an off-the-happy-path branch, + observability gap, or design-doc statement that contradicts + shipped code. Should be fixed before merge but the call is + the maintainer's. +- `LOW` — defensive-coding improvement, minor inconsistency, + missing edge-case test. +- `NIT` — naming, line wrap, comment phrasing. Note in passing + only; do not stack these. + +## Areas to scrutinise + +These are the surfaces most likely to harbour real issues. Hit +them specifically. + +### Faz 22 design doc — `docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md` + +- **Citation accuracy.** Every cited symbol (`forgelm.compliance.AuditLogger`, + `_purge._resolve_salt`, `_reverse_pii._scan_file_with_alarm`, + etc.) must exist in the codebase at the cited file. Grep each + one. The doc claims to verify against HEAD — was it really? +- **Coverage tally honesty.** §3 claims "FL 11 / FL-helps 50 / OOS + 32 = 93". Walk the table and recount. An off-by-one is a + HIGH finding (deployer auditors trust the count). +- **Decision Log internal consistency.** D-22-01 (alignment vs + certified) must be reflected in every customer-facing artefact + in the diff. D-22-02 (CycloneDX not SPDX) must be reflected in + the supply_chain_security reference doc + README + CHANGELOG. + Cross-check. +- **FAQ accuracy.** The 10 FAQ Q/A pairs must match the cited + ForgeLM behaviour. E.g. Q5 (GDPR Article 15 response) must use + the actual `forgelm reverse-pii` flag set the parser exposes. +- **Length claim.** Closure plan acceptance bar was ≥800 lines; + the doc commits at 865. Confirm `wc -l` matches. + +### Faz 23 supply-chain tooling + +- **`tools/check_pip_audit.py`.** Walk the severity tier mapping + (`HIGH` / `CRITICAL` / `MEDIUM` / `MODERATE` / `LOW` / + `UNKNOWN`). Are there real-world pip-audit JSON shapes the + parser misses? Walk the GHSA imports that put severity in + `aliases[].severity[].type` (the 2.6.x shape) — is the fall- + through correct? +- **`tools/check_bandit.py`.** Same — walk the bandit JSON + shape. HIGH / MEDIUM / LOW + UNDEFINED tiering. Does the + parser handle bandit's confidence + severity orthogonally + correctly? +- **`tests/test_supply_chain_security.py::TestGenerateSbomDeterministic`.** + The test strips `serialNumber` + `metadata.timestamp` before + comparing. Are those the ONLY non-deterministic fields in + CycloneDX 1.5? Is the strip approach robust to future emitter + changes? +- **`pyproject.toml [project.optional-dependencies] security`.** + Pinning ranges (`pip-audit>=2.7.0,<3.0.0`, + `bandit[toml]>=1.7.0,<2.0.0`) — are these the right + upper-bounds? Will pip-audit 3.x break the JSON shape the + helper parses? +- **`.github/workflows/ci.yml` bandit step.** The step pipes + `|| true` to suppress bandit's exit code, then runs + `tools/check_bandit.py` for the actual gate. Confirm the + failure path: if the JSON file is malformed, does the helper + still exit 1 (preventing a green CI on a real bandit crash)? +- **`.github/workflows/nightly.yml supply-chain-security` job.** + The job uploads `/tmp/pip-audit.json` + `/tmp/bandit.json` as + artefacts. Confirm the path is correct on macOS / Windows + runners (the job is ubuntu-latest, so this is fine, but flag + if any cross-OS port is intended later). + +### Faz 23 QMS docs (EN-only, TR mirrors land in Faz 26) + +- **`docs/qms/encryption_at_rest.md`.** §3 threat model claims + "kripto-algoritma break out of scope". Is that a defensible + framing for an ISO 27001 doc? ISO A.8.24 explicitly addresses + algorithm agility — should we flag a residual risk? +- **`docs/qms/access_control.md`.** §3.4 claims `FORGELM_AUDIT_SECRET` + rotation should NOT happen mid-output-dir. Walk this against + the actual `_resolve_salt` code path: does mid-output-dir env + rotation actually break verify-audit, or does the salt-source + recording (Wave 3 followup F-W3-PS-07) make this safer than the + doc suggests? +- **`docs/qms/risk_treatment_plan.md`.** 12 risks documented. + R-05 (memorisation residual) is HIGH→MED. Is that an honest + reduction given the only "treatment" is operator-side + notification? Should it be HIGH→HIGH with explicit risk + acceptance? +- **`docs/qms/statement_of_applicability.md`.** 93-control SoA. + Walk against the §3 design-doc table — are the same controls + classified the same way? + +### Faz 23 deployer guide + reference docs + +- **`docs/guides/iso_soc2_deployer_guide.md` (+ TR).** 8 audit- + floor questions answered. Walk each: is the `jq` / `gh` + command syntactically correct? Will it work on macOS jq + + Linux jq? +- **`docs/reference/iso27001_control_mapping.md` (+ TR).** Is + every cell filled? Spot-check 5 random rows — does the + ForgeLM-evidence claim hold? +- **`docs/reference/soc2_trust_criteria_mapping.md` (+ TR).** + Same. +- **`docs/reference/supply_chain_security.md` (+ TR).** The + Dependency-Track ingestion command — verified against current + Dependency-Track API? + +### Faz 26 — QMS bilingual mirrors + +- **Structural parity.** 10 QMS docs now have TR mirrors but + QMS pairs are NOT in `tools/check_bilingual_parity.py::_PAIRS` + yet. Should they be? If yes, file as a finding for Faz 30 + follow-up to register them. +- **Translation faithfulness.** Spot-check 3 random TR mirrors + against the EN. Are technical terms (`audit chain`, + `genesis manifest`, `salted hash`) translated consistently? +- **Code-block transcription.** The shell snippets in the + audit-log + GDPR sections must match between EN + TR. Verify + the 5+ shell snippets in `iso_soc2_deployer_guide.md` / + `-tr.md` are byte-identical. + +### Faz 26 — `compliance_summary.md` rewrite + +- **Anchor-resolution claim.** The new doc claims + "module-path + symbol-name references that survive refactors" + — are the new references ACTUALLY symbol references, or did + some line-anchor sneak through? +- **Stat update.** "140 prompts × 6 categories" — verify + against `configs/safety_prompts/*.jsonl` count + the + in-`forgelm/safety.py` category list. + +### Faz 26 — `tools/check_anchor_resolution.py` + +- **Slugify accuracy.** GFM slug rules differ slightly from + GitLab. Does the helper produce the same slug GitHub renders + for these edge cases? + - "Section 1.2: Why?" + - "What's New" + - "Türkçe başlık" (Unicode) + - "Heading with `code`" +- **Skip patterns.** The helper skips `https://`, `mailto:`, + `tel:`, `javascript:`, `#/`. Is the SPA hash-router skip + comprehensive? E.g. does `#/data/ingestion?param=foo` get + skipped (it should — query strings on SPA paths)? +- **Advisory-mode default.** The closure plan called for a CI + gate. The maintainer's commit message defends advisory-mode + default with "36 broken links pre-Faz-26 baseline". Is that + acceptable? Or should the tool ship with `--strict` enabled + + the pre-existing breaks fixed in this commit? + +### Faz 30 partial — ghost-feature drift fixes + +- **GH-008 verify-log → verify-audit.** Are there OTHER docs + citing `verify-log`? Grep the tree. +- **GH-021 chat slash commands.** The fix removes + `/load`, `/top_p`, `/max_tokens`, `/safety on|off`. Confirm + these don't exist by running `forgelm chat --help` against + the parser. +- **GH-022 q6_k removal.** Confirm parser only supports + `q2_k|q3_k_m|q4_k_m|q5_k_m|q8_0|f16` at + `forgelm/cli/_parser.py:75-76`. +- **GH-024 FORGELM_RESUME_TOKEN.** Confirm there's NO real + resume endpoint or env-var-driven approval API in the code. + Grep `forgelm/`. +- **GH-025 FORGELM_CACHE_DIR.** Same — confirm no env var of + this name is consumed. + +### Cross-cutting (XPR) + +- **F-XPR-01 — alignment-vs-certified wording sweep.** `grep + -rni 'iso 27001 compliant\|iso 27001 certified\|soc 2 compliant\|soc 2 certified' + docs/ README.md CHANGELOG.md` should return zero (or only + "alignment with" / "deployer's certification" hits). +- **F-XPR-02 — bilingual parity strict.** `python3 tools/check_bilingual_parity.py + --strict` should exit 0 with 13 pairs. +- **F-XPR-03 — pytest count.** `pytest --collect-only -q` should + show ≥1397 tests. +- **F-XPR-04 — `forgelm --dry-run` green** on `config_template.yaml`. +- **F-XPR-05 — anchor-checker count.** Run + `tools/check_anchor_resolution.py` and confirm the broken-link + count matches the maintainer's claim ("36 baseline"). An + increase in broken links from this PR is a `MEDIUM` finding. +- **F-XPR-06 — closure-plan honesty.** `docs/analysis/code_reviews/closure-plan-202604300906.md` + Wave 4 kapanış özeti row enumerates the deliverables — every + named file should exist in the diff. +- **F-XPR-07 — `[security]` extra resolves.** `pip install -e + .[security]` (or equivalent dry-resolve) should resolve to + pip-audit + bandit pinned ranges without conflict. + +## Ground rules + +- **Read the relevant standard before commenting.** The project + [docs/standards/](../../standards/) directory is the rulebook. +- **Cite line numbers.** If you cannot point at a line, the + finding is not actionable. +- **Distinguish design-doc claim from runtime contract.** A + Decision Log entry that the code doesn't honour is a + CRITICAL — the maintainer commits to a specific design and + the code must follow. But "the design says X and the code + does X but X could be Y" is at most a MEDIUM design-rationale + question. +- **No phantom severity inflation.** A doc typo is `NIT`. A + user-visible CLI bug on the happy path is `HIGH` minimum. An + auditor-misleading wording (alignment vs certified) is + `CRITICAL`. +- **No re-litigating closed scope.** "Should ForgeLM also do X" + is not a finding. Stay in the diff. +- **No suggestions to add a Web UI / GUI / inference engine / + pretraining pipeline.** See + [docs/marketing/strategy/05-yapmayacaklarimiz.md] — these are + out of scope by design (read root `CLAUDE.md` "What ForgeLM is + not"). +- **Wave 4 explicitly defers some closure-plan Faz 30 items** + (GH-016/018/019/020 cli.md cleanup, sample-audit user-facing + doc completion, `tools/check_cli_help_consistency.py`, + full-suite anchor-strict CI wire-up). Don't flag those as + Wave 4 omissions — they're explicit Faz 30 follow-up scope. +- **No ISO/SOC 2 audit-domain claims you can't back from the + standards text.** This is a compliance-sensitive surface; if + you flag a control mapping as wrong, cite the ISO 27001:2022 + Annex A control description or the AICPA TSC clause that + contradicts ForgeLM's claim. + +## Required deliverable structure + +Each reviewing agent returns a single Markdown report with this +skeleton, saved as +`docs/analysis/code_reviews/wave4-.md`: + +```markdown +# Wave 4 — Review of `closure/wave4-integration` + +> Branch: `closure/wave4-integration` · HEAD `8f69cf1` · +> Reviewer focus: . Reviewing PR #33. + +## Summary +- Verdict: [Block / Conditional / Approve] +- CRITICAL: N · HIGH: N · MEDIUM: N · LOW: N · NIT: N +- One-sentence headline of the highest-severity finding. + +## Findings + +### F-W4-NN · · +- **File:** path:line[-line] +- **What's wrong:** ... +- **User-visible consequence:** ... +- **Suggested fix:** ```code``` +- **Regression test:** ```python ...``` + +(repeat per finding) + +## Verified absorptions + +For each closure-plan acceptance criterion (Faz 22 §13, Faz 23 §13, +Faz 26 task list, Faz 30 task O subset), state: + +- ** [verified]** — confirmed at ; + walked: . +- ** [partial]** — addressed at but + misses ; new finding F-W4-NN tracks it. +- ** [missing]** — not in the diff. Maintainer + documented as deferred to Faz 30 follow-up. + +## What this report deliberately did not cover + +(scope notes, deferred items, out-of-PR areas) + +## Merge recommendation + +(Block / Conditional / Approve, with the headline reasons) +``` + +## How to launch + +Spawn 4 parallel sub-agents (any harness with parallel workers; the +prior waves used Claude `Agent` calls in a single message). Pass +each agent the SAME prompt with one extra line at the top stating +its agent name + focus: + +1. **Code-correctness agent.** Focus areas: §A (Faz 22 design + citation accuracy), §B (Faz 23 supply-chain tooling logic), §G + (anchor-resolution checker correctness), §F (cross-cutting XPR- + 01/02/03/04/05/07). +2. **Privacy / security agent.** Focus areas: §C (Faz 23 QMS docs: + encryption / access / RTP / SoA), §B again from a security + posture lens (severity tiers correct? CVE policy reasonable?), + §F-XPR-01 (alignment-vs-certified wording — privacy bar). +3. **Standards / compliance agent.** Focus areas: §A (Faz 22 + coverage tally + decision-log internal consistency), §D + (deployer guide + reference docs accuracy), §C (RTP / SoA / + incident-response / change-mgmt SOP expansions). Cite + docs/standards/* extensively. +4. **Test-rigour agent.** Focus areas: §B + (`tests/test_supply_chain_security.py`), §G + (`tests/test_check_anchor_resolution.py`), §H ghost-feature + drift verification (do the doc-only fixes match the actual + parser output?). Vacuous-test bar reigns. + +Agents run independently. Maintainer absorbs in a single round +(per the established Wave 2-3 cadence: report → multi-round +absorption → followup review prompt). + +--- + +*Frozen at `8f69cf1` on 2026-05-06.* *If any reviewer finds the +prompt itself ambiguous, surface it as a `F-W4-PROMPT-NN` finding +before merging — the prompt itself is part of the audit trail.* From ffceaf786c59dca956996c5831fe9cf1f17e17bd Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 01:19:49 +0300 Subject: [PATCH 07/12] fix(wave-4-round1): absorb 4-agent + PR #33 + Sonar review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Convergent absorption of code-correctness, privacy-security, standards, and test-rigor reviews of `closure/wave4-integration` (HEAD 9a1738a), plus PR #33 inline comments (Gemini) and Sonar findings on the same PR. Wave 3 cadence: real bugs are fixed in the same round even when out of strict scope. HIGH (5) - Design doc §3 coverage tally was off-by-2 (claimed FL 11 / FL-helps 50 / OOS 32; row-by-row recount is 11 / 48 / 34). Corrected in design doc, FAQ Q1 ("~61" → 59 = 11 + 48), SoA EN+TR per-theme breakdown, ISO 27001 reference EN+TR header tally, and CHANGELOG. - Design doc cited non-existent `compute_artefact_sha256`; the real symbol is `generate_model_integrity` (delegates to `_hash_file` in `forgelm/compliance.py:466-499`). - Deployer guide cookbook (EN+TR) + access_control §6 (EN+TR) cited `forgelm verify-audit --output-dir ./outputs --json` — neither flag exists on the parser. Replaced with the shipping `forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac` form + a `--slurpfile`-based jq pipeline that actually parses. - access_control §3.4 (EN+TR) + RTP R-04 (EN+TR) misdescribed the HMAC chain key as the env secret XOR'd with `_resolve_salt`'s per-output-dir salt. The real derivation is `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` per `forgelm/compliance.py:104-114`; the per-output-dir salt is a distinct concern that salts purge / reverse-pii identifier hashes and does NOT participate in chain-key derivation. Mirrored in the ISO A.8.24 row of the design doc + reference (EN+TR). - access_control §3.4 (EN+TR) + deployer guide (EN+TR) advertised quarterly `FORGELM_AUDIT_SECRET` rotation. Rotation must occur *between* output-dir lifecycles — every entry's HMAC is bound to the secret live at emit time and `forgelm verify-audit --require-hmac` cannot verify a chain mixing secrets. Reframed to per-output-dir-lifecycle cadence with the procedure spelled out. MEDIUM (8) - `tools/check_pip_audit.py::_vuln_severity` was treating the OSV scoring-system label (`CVSS_V3`) as if it were a tier name, and missed the 2.6.x `aliases[].severity[]` shape its docstring promised. Added explicit score-type label set, CVSS-score → tier derivation per FIRST.org cut-points, 2.6.x aliases fallback, and a single-line summary annotation for UNKNOWN findings (was silent). - `tools/check_bandit.py` was masking malformed `"results": null` reports via `or []`. Now distinguishes missing-key / null-value / list shape explicitly + emits a single summary annotation when bandit's documented UNDEFINED tier appears. - `incident_response.md` lead-in (EN+TR) said "§6 covers" the security-incident playbook; the playbook is in §4. Fixed. - `incident_response.md` (EN+TR) §4.1 step 3 + §4.4 step 2 cited non-existent `--until-line N` / `--replay-since ` flags. Rewritten to use shipping primitives (positional log_path, manual `head -n N` bisect, jq filter-and-correlate). - `risk_treatment_plan.md` (EN+TR) R-05 residual rating was Med×Med presented as a passive reduction. ForgeLM contributes a detection signal only; deployer accepts the High×Med residual via the §5 acceptance log. Reframed honestly. - `encryption_at_rest.md` (EN+TR) §3 had algorithm-break wholly out-of-scope. Promoted to "Long-horizon (>5 years) — partial coverage" per ISO A.8.24 algorithm-agility expectation; cites NIST SP 800-131A. - `supply_chain_security.md` (EN+TR) cited ISO A.8.30 (outsourced development) as a control mapping for transitive PyPI deps. A.8.30 addresses outsourced *system development*, not consuming open-source packages; A.5.21 + A.8.8 already cover the supply-chain story honestly. A.8.30 row dropped. - `soc2_trust_criteria_mapping.md` (EN+TR) + design doc CC2.3 row conflated artefact production with communication channel. CC2.3 evidence is the deployer's regulator-contact process; ForgeLM produces the Annex IV bundle + `compliance_report.json` artefacts. Tooling + tests - `tools/check_anchor_resolution.py`: extracted helpers `_normalise_heading_body`, `_resolve_pure_anchor`, `_locate_target`, `_resolve_anchor_against_target`, `_build_argparser`, `_resolve_excludes`, `_collect_broken`, `_report_broken` to land `_resolve_link` and `main` below Sonar's cognitive-complexity 15 ceiling (was 17 each). Replaced the `(.+?)\s*#*\s*$` ATX-heading regex with `(.+)` + procedural trailing-whitespace + closing-hash trim to clear the python:S5852 ReDoS hotspot at line 57. - `tools/check_bandit.py::main`: extracted `_load_report`, `_extract_results`, `_classify_issues` to land below the cognitive-complexity ceiling (was 17). - `tools/check_bilingual_parity.py::_PAIRS`: registered the 10 QMS pairs (4 Faz-23 docs + 6 existing QMS files) so EN/TR drift is now CI-gated. Strict mode reports 23/23 OK. - `docs/standards/localization.md` `docs/qms/*.md` row flipped from "No" to "Yes" to reflect the bilingual sweep. - `tests/test_supply_chain_security.py`: added bandit UNDEFINED + null-results + missing-key tests, pip-audit UNKNOWN summary + CVSS-score derivation + 2.6.x aliases nested + list-form tier tests, SBOM serialNumber uniqueness test. - `tests/test_check_anchor_resolution.py`: added apostrophe + backtick slug edge tests, ATX closing-hash normalisation test, tel: + javascript: skip-pattern fixtures, image-link parity tests, strict mode broken-anchor exit-1 test. Other - `forgelm/cli/subcommands/_doctor.py`: `FORGELM_RESUME_TOKEN` inline comment reworded from "Phase 13 future" to "reserved name — masked defensively even if unused today". - `ghost-features-analysis-20260502.md` line 604: claim that `grep -rn "FORGELM_RESUME_TOKEN" forgelm/` returns no results corrected — there is one defensive secret-mask entry. - `closure-plan-202604300906.md` Wave 4 summary: 6 → 10 yeni QMS TR mirror (4 Faz-23 pair + 6 mevcut QMS dosyası); test count 1397 → 1411. Validation - `ruff format .` / `ruff check .` — clean. - `python3 tools/check_bilingual_parity.py --strict` — 23/23 OK. - `python3 -m pytest --no-cov -q` — 1414 passed, 14 skipped. - `forgelm --config config_template.yaml --dry-run` — green. - `python3 tools/check_anchor_resolution.py` (advisory) — 36 pre-existing broken (Faz 30 follow-up scope per closure-plan §10 acceptance criteria). Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 12 +- .../code_reviews/closure-plan-202604300906.md | 2 +- .../ghost-features-analysis-20260502.md | 2 +- .../iso27001-soc2-alignment-202605052315.md | 29 +-- docs/guides/iso_soc2_deployer_guide-tr.md | 28 ++- docs/guides/iso_soc2_deployer_guide.md | 27 ++- docs/qms/access_control-tr.md | 72 ++++--- docs/qms/access_control.md | 71 +++++-- docs/qms/encryption_at_rest-tr.md | 13 +- docs/qms/encryption_at_rest.md | 12 +- docs/qms/risk_treatment_plan-tr.md | 8 +- docs/qms/risk_treatment_plan.md | 8 +- docs/qms/sop_incident_response-tr.md | 26 ++- docs/qms/sop_incident_response.md | 26 ++- docs/qms/statement_of_applicability-tr.md | 14 +- docs/qms/statement_of_applicability.md | 14 +- docs/reference/iso27001_control_mapping-tr.md | 8 +- docs/reference/iso27001_control_mapping.md | 9 +- .../soc2_trust_criteria_mapping-tr.md | 2 +- docs/reference/soc2_trust_criteria_mapping.md | 2 +- docs/reference/supply_chain_security-tr.md | 3 +- docs/reference/supply_chain_security.md | 3 +- docs/standards/localization.md | 2 +- forgelm/cli/subcommands/_doctor.py | 2 +- tests/test_check_anchor_resolution.py | 54 ++++++ tests/test_supply_chain_security.py | 180 ++++++++++++++++++ tools/check_anchor_resolution.py | 161 ++++++++++------ tools/check_bandit.py | 65 +++++-- tools/check_bilingual_parity.py | 11 ++ tools/check_pip_audit.py | 99 +++++++++- 30 files changed, 757 insertions(+), 208 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dce9c001..cf792f14 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,9 @@ documentation pass. controls across A.5 + A.6 + A.7 + A.8) and every SOC 2 Trust Services Criterion (Common Criteria CC1.x–CC9.x mandatory + 4 optional categories) to the ForgeLM feature that produces audit - evidence. Coverage tally: FL (full) 11 / FL-helps 50 / OOS 32. + evidence. Coverage tally: FL (full) 11 / FL-helps 48 / OOS 34 + (row-by-row recount of §3.1–§3.4; per-theme split A.5: 3 / 24 / + 10, A.6: 0 / 5 / 3, A.7: 0 / 0 / 14, A.8: 8 / 19 / 7). 10-row Decision Log + 10-question deployer FAQ. **Faz 23 — ISO 27001 / SOC 2 alignment implementation** @@ -78,9 +80,11 @@ Cross-cutting: - `README.md` — new "ISO 27001 / SOC 2 Type II Alignment" bullet under Enterprise & MLOps with explicit "alignment, not certified" framing (closes the design doc's D-22-01 wording decision). -- `tools/check_bilingual_parity.py` `_PAIRS` registry — added 4 new - pairs (deployer guide + 3 reference docs); tally now 13 doc pairs - (was 9). +- `tools/check_bilingual_parity.py` `_PAIRS` registry — added 14 new + pairs total (4 Faz-23 pairs: deployer guide + 3 reference docs; + 10 Faz-26 QMS pairs); registry tally now 23 (was 9). +- `docs/standards/localization.md` — `docs/qms/*.md` row flipped + from "No (yet)" to "Yes" to reflect the bilingual sweep. **Faz 26 — QMS bilingual mirror + compliance_summary cleanup** diff --git a/docs/analysis/code_reviews/closure-plan-202604300906.md b/docs/analysis/code_reviews/closure-plan-202604300906.md index 634a16bf..1712a579 100644 --- a/docs/analysis/code_reviews/closure-plan-202604300906.md +++ b/docs/analysis/code_reviews/closure-plan-202604300906.md @@ -128,7 +128,7 @@ Her faz bir veya birden çok PR'la kapanır. Sıralama dependency-driven; parale **Wave 3 kapanış özeti (post-merge, 2026-05-05):** 3 faz tamamlandı (Faz 24 bilingual TR mirror sweep + `tools/check_bilingual_parity.py` CI guard, Faz 28 curated tier 1+2 cleanup, Faz 38 `forgelm reverse-pii` GDPR Article 15 subcommand). PR #31 → development (merge commit `b87c872`). Yeni: 1 CLI subcommand (`reverse-pii`), 1 audit event (`data.access_request_query`), 1 yeni CI guard tool (`tools/check_bilingual_parity.py`). Behaviour change: high-risk + unacceptable + safety-disabled artık `ConfigError` raise eder (F-compliance-110, OR-across-fields strict gate). Webhook timeout default 5s → 10s (F-compliance-106). 12 commit (1 ana + 11 absorption rounds: 4-agent original review + 5 SonarCloud findings + 14 inline review comments across CodeRabbit/Gemini/inline + 5 follow-up multi-agent review absorptions). Suite 1298 → 1374 tests (+76 net). -**Wave 4 kapanış özeti (2026-05-06, PR pending):** 4 faz tamamlandı (Faz 22 ISO 27001 / SOC 2 alignment design, Faz 23 implementation, Faz 26 QMS bilingual mirror + compliance_summary cleanup + anchor checker, Faz 30 partial Tier 1 ghost-feature drift + stat blocks). Branch: `closure/wave4-integration`. Yeni: 1 design dosyası (~865 satır), 4 yeni QMS doküman (encryption_at_rest, access_control, risk_treatment_plan, statement_of_applicability), 6 yeni QMS TR mirror, 1 yeni guide (deployer cookbook EN+TR), 3 yeni reference (iso27001_control_mapping EN+TR, soc2_trust_criteria_mapping EN+TR, supply_chain_security EN+TR), 3 yeni tool (`tools/check_pip_audit.py`, `tools/check_bandit.py`, `tools/check_anchor_resolution.py`), 1 yeni `[security]` optional extra (pip-audit + bandit), 1 yeni nightly job (supply-chain-security). Behaviour change: yok. Suite 1374 → 1397 tests (+23 net: 16 supply-chain + 21 anchor checker − 14 displaced). compliance_summary.md tam rewrite (line-anchored references symbol-references'a geçti). Bilingual parity 9/9 → 13/13. +**Wave 4 kapanış özeti (2026-05-06, PR #33 pending):** 4 faz tamamlandı (Faz 22 ISO 27001 / SOC 2 alignment design, Faz 23 implementation, Faz 26 QMS bilingual mirror + compliance_summary cleanup + anchor checker, Faz 30 partial Tier 1 ghost-feature drift + stat blocks). Branch: `closure/wave4-integration`. Yeni: 1 design dosyası (~865 satır), 4 yeni QMS doküman (encryption_at_rest, access_control, risk_treatment_plan, statement_of_applicability) + bunların TR mirror'ları + 6 mevcut QMS dosyası için yeni TR mirror (toplam 10 yeni QMS TR mirror), 1 yeni guide (deployer cookbook EN+TR), 3 yeni reference (iso27001_control_mapping EN+TR, soc2_trust_criteria_mapping EN+TR, supply_chain_security EN+TR), 3 yeni tool (`tools/check_pip_audit.py`, `tools/check_bandit.py`, `tools/check_anchor_resolution.py`), 1 yeni `[security]` optional extra (pip-audit + bandit), 1 yeni nightly job (supply-chain-security). Behaviour change: yok. Suite 1374 → 1411 tests (+37 net: 16 supply-chain + 21 anchor checker yeni dosyalardan). compliance_summary.md tam rewrite (line-anchored references symbol-references'a geçti). Bilingual parity 9/9 → 23/23 (4 Faz-23 + 10 Faz-26 QMS pair'i eklendi). **Kalan iş (Wave 4+):** 1 release. Pending: Faz 33 (v0.5.5 PyPI publish, post-wave). Optional follow-up before release: Faz 30 full sweep (kalan Tier 1 ghost-feature drift items GH-016/018/019/020, sample-audit user-facing doc triplet completion, tools/check_cli_help_consistency.py addition, full-suite anchor-checker --strict CI wire-up). diff --git a/docs/analysis/code_reviews/ghost-features-analysis-20260502.md b/docs/analysis/code_reviews/ghost-features-analysis-20260502.md index c6c9aa09..2c5bcabb 100644 --- a/docs/analysis/code_reviews/ghost-features-analysis-20260502.md +++ b/docs/analysis/code_reviews/ghost-features-analysis-20260502.md @@ -601,7 +601,7 @@ ingestion: **Documented:** "Token for the API-based human approval flow" -**Code reality:** `grep -rn "FORGELM_RESUME_TOKEN" forgelm/` → no results. Not used anywhere in the codebase. +**Code reality:** `grep -rn "FORGELM_RESUME_TOKEN" forgelm/` → one defensive reference at `forgelm/cli/subcommands/_doctor.py` (the var name appears in the secret-mask pre-list so a future probe never accidentally surfaces its value); the doctor probe never reads the env var. Not consumed anywhere in the codebase. The user-facing reference doc citation is the ghost; the defensive mask-list entry is intentional and stays. --- diff --git a/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md b/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md index 704e208d..f50c1913 100644 --- a/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md +++ b/docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md @@ -120,7 +120,7 @@ auditable evidence. Cited symbols are real (verified against | HTTP discipline | `safe_post` (Phase 7) | SSRF guard, TLS-only, no-redirect, header masking | | Data audit pipeline | `forgelm audit` (Phase 11) | `data_audit_report.json` (PII / secrets / dedup) | | Annex IV §8 governance | `_data_governance_block` (compliance.py) | `data_governance_report.json` Article 10 | -| Model integrity | `compute_artefact_sha256` | `model_integrity.json` | +| Model integrity | `generate_model_integrity` (delegates to `_hash_file`) | `model_integrity.json` | | SBOM | `tools/generate_sbom.py` (Wave 2 era) | CycloneDX 1.5 JSON per (OS, py-version), uploaded as release artefact | | Bilingual EN+TR docs | `tools/check_bilingual_parity.py` (Phase 24) | strict-mode diff exit code | @@ -249,7 +249,7 @@ ForgeLM contributes nothing here. | A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement; cert rotation | | A.8.22 Segregation of networks | OOS | — | VPC / subnet design | | A.8.23 Web filtering | OOS | — | Egress proxy | -| A.8.24 Use of cryptography | FL | SHA-256 + HMAC-SHA-256 (audit chain); salted SHA-256 identifier hashing; per-output-dir salt + env XOR derivation | KMS for `FORGELM_AUDIT_SECRET` | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC-SHA-256 (audit chain key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, see `forgelm/compliance.py:104-114`); salted SHA-256 identifier hashing for purge / reverse-pii (`_purge._resolve_salt`, distinct concern) | KMS for `FORGELM_AUDIT_SECRET` | | A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md` + `docs/standards/release.md` + CI gates (ruff, pytest, dry-run, parity, SBOM) | SDLC framework | | A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; Pydantic config validation; ReDoS guard in `_reverse_pii` | App-level threat modelling | | A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture (`flock`+`fsync` per line); audit chain HMAC; lazy import discipline; `safe_post` SSRF guard | Defence-in-depth design | @@ -261,10 +261,12 @@ ForgeLM contributes nothing here. | A.8.33 Test information | FL-helps | `forgelm audit` flags PII / secrets in test sets too | Test-data-handling policy | | A.8.34 Protection of information systems during audit testing | OOS | — | Read-only audit access | -**Coverage tally for §3:** 93 controls scored. **`FL` (full)**: 11 · -**`FL-helps`**: 50 · **`OOS`**: 32. The OOS bucket is dominated by -A.7 physical (14) + A.5 organisational governance (~10) + A.8 cloud / -network (~8) — all expected for a deployer-side toolkit boundary. +**Coverage tally for §3:** 93 controls scored, recounted row-by-row +across §3.1–§3.4. **`FL` (full)**: 11 · **`FL-helps`**: 48 · +**`OOS`**: 34. Per-theme split — A.5: 3 / 24 / 10; A.6: 0 / 5 / 3; +A.7: 0 / 0 / 14; A.8: 8 / 19 / 7. The OOS bucket is dominated by +A.7 physical (14) + A.5 organisational governance (10) + A.8 cloud / +network (7) — all expected for a deployer-side toolkit boundary. ## 4. SOC 2 Trust Services Criteria mapping @@ -291,7 +293,7 @@ categories are scoped per-engagement. | CC1.5 | Enforces accountability | `FORGELM_OPERATOR` attribution + audit chain | IdP integration | | CC2.1 | Communicates internally about controls | `audit_event_catalog.md` + `deployer_instructions.md` | Internal comms plan | | CC2.2 | Communicates externally | Article 13 `deployer_instructions.md`; Annex IV public summary | Customer-facing comms | -| CC2.3 | Communicates with regulators | Annex IV bundle ready to ship | Regulator contact list | +| CC2.3 | Communicates with regulators | Annex IV bundle + `compliance_report.json` are the artefacts the deployer ships to the regulator on request; ForgeLM does NOT operate the communication channel itself | Regulator contact list + ship process | | CC3.1 | Specifies suitable objectives | `compliance.intended_purpose`; risk classification | Adopt | | CC3.2 | Identifies and analyses risks | `risk_assessment` Pydantic block; safety eval | Risk register | | CC3.3 | Considers fraud risks | Audit log tamper-evidence; HMAC chain; manifest sidecar | Fraud taxonomy | @@ -788,9 +790,10 @@ Anticipated questions from the deployer's compliance team: **Q1.** *Is ForgeLM ISO 27001 certified?* **A.** No. ISO 27001 certifies organisations, not software libraries. ForgeLM is **aligned** with ISO 27001:2022 Annex A in that it produces -auditable evidence for ~61 of the 93 controls. The remaining 32 -controls (mainly A.7 physical, A.5 organisational governance, A.8 -network) are deployer-side. See §3 for the full mapping. +auditable evidence for 59 of the 93 controls (11 `FL` + 48 +`FL-helps`). The remaining 34 controls (mainly A.7 physical, +A.5 organisational governance, A.8 network / cloud) are +deployer-side. See §3 for the full mapping. **Q2.** *Is ForgeLM SOC 2 Type II compliant?* **A.** Same answer as Q1. ForgeLM is **aligned** with the SOC 2 @@ -802,7 +805,11 @@ approval-gate events, etc.) when assessing the deployer's controls. **A.** In your KMS / secrets vault (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Inject via env var at CI runner start. See `docs/qms/access_control.md` (Faz 23) for the rotation cadence -recommendation (quarterly). +recommendation. Rotation must occur **between output-dir lifecycles** +(after archiving the current `audit_log.jsonl` + `.manifest.json` +pair), not mid-output-dir — every entry's HMAC is bound to the secret +live at emit time, and `forgelm verify-audit --require-hmac` cannot +verify a chain that mixes secrets. **Q4.** *What encryption does ForgeLM apply at rest?* **A.** None — ForgeLM is encryption-substrate-agnostic. The deployer diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index 1cf58ac0..37c7486d 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -49,8 +49,13 @@ Denetçi sormadan **önce** indirmeniz gereken on iki madde. form için `docs/qms/access_control.md` §3'e bakın. - [ ] **`FORGELM_AUDIT_SECRET` üretin** KMS / Vault'unuzda (32+ rastgele bayt, AES-256-GCM-strength entropi). -- [ ] **Üç-aylık `FORGELM_AUDIT_SECRET` rotasyonunu zamanlayın** - KMS audit log'unuzda. +- [ ] **`FORGELM_AUDIT_SECRET` rotasyonunu output-dir lifecycle'lar + arasında planlayın** — her girişin HMAC'ı emit anındaki secret'a + bağlı olduğundan, rotasyon mevcut `audit_log.jsonl` + + `.manifest.json` çiftini arşivledikten *sonra* yapılmalıdır, + asla output-dir ortasında değil. (`forgelm verify-audit + --require-hmac` secret karışmış zinciri tasarım gereği + doğrulayamaz.) - [ ] **Approver identity ≠ trainer identity'yi yapılandırın** — Madde 14 staging gate sizin in-pipeline Change Advisory Board'unuzdur. @@ -90,8 +95,9 @@ ForgeLM artefaktı + grep / komut ile. ### S1: "Son 90 günde her model promotion için audit trail göster" ```bash -# Önce zincir bütünlüğünü doğrula. -forgelm verify-audit --output-dir ./outputs --json +# Önce zincir bütünlüğünü doğrula (positional log_path; bu +# subcommand'ta tasarım gereği --json / --output-dir flag'i yoktur). +forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac # Sonra promotion olaylarını çıkar. jq 'select(.event == "human_approval.granted")' ./outputs/audit_log.jsonl @@ -133,16 +139,18 @@ cat ./outputs/data_provenance.json # { # "dataset_id": "Acme/customer-support-v3", # "hf_revision": "9c7c8f3...", -# "fingerprint_sha256": "ab12...", -# "size_bytes": 14_982_011, -# "mtime": "2026-05-01T08:14:33Z", +# "sha256": "ab12...", +# "size_bytes": 14982011, +# "modified": "2026-05-01T08:14:33Z", # ... # } ``` -`fingerprint_sha256` + `hf_revision` birlikte korpusu deterministik -olarak pin'ler. Aynı girdi üzerinde `forgelm audit data/*.jsonl` -çalıştıran bir denetçi aynı fingerprint'i görmek zorundadır. +`sha256` + `hf_revision` birlikte korpusu deterministik olarak +pin'ler (yerel dosyalar ek olarak `size_bytes` + `modified` alanları +taşır; alan adları `forgelm.compliance._fingerprint_local_file`'dan +gelir). Aynı girdi üzerinde `forgelm audit data/*.jsonl` çalıştıran +bir denetçi aynı fingerprint'i görmek zorundadır. ### S4: "Supply chain göster" diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md index 37d26317..c1df4278 100644 --- a/docs/guides/iso_soc2_deployer_guide.md +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -48,8 +48,12 @@ Twelve items to land **before** the auditor asks. `docs/qms/access_control.md` §3 for the recommended form. - [ ] **Generate `FORGELM_AUDIT_SECRET`** in your KMS / Vault (32+ random bytes, AES-256-GCM-strength entropy). -- [ ] **Schedule quarterly `FORGELM_AUDIT_SECRET` rotation** in - your KMS audit log. +- [ ] **Plan `FORGELM_AUDIT_SECRET` rotation between output-dir + lifecycles** — every entry's HMAC is bound to the secret live + at emit time, so rotation must happen *after* archiving the + current `audit_log.jsonl` + `.manifest.json` pair, never + mid-output-dir. (`forgelm verify-audit --require-hmac` cannot + verify a chain that mixes secrets — by design.) - [ ] **Configure approver identity ≠ trainer identity** — the Article 14 staging gate is your in-pipeline Change Advisory Board. @@ -90,8 +94,9 @@ artefact + grep / command that produces the evidence. ### Q1: "Show me the audit trail for every model promotion in the past 90 days" ```bash -# Verify the chain integrity first. -forgelm verify-audit --output-dir ./outputs --json +# Verify the chain integrity first (positional log_path; no --json / +# --output-dir flags exist on this subcommand by design). +forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac # Then extract the promotion events. jq 'select(.event == "human_approval.granted")' ./outputs/audit_log.jsonl @@ -134,16 +139,18 @@ cat ./outputs/data_provenance.json # { # "dataset_id": "Acme/customer-support-v3", # "hf_revision": "9c7c8f3...", -# "fingerprint_sha256": "ab12...", -# "size_bytes": 14_982_011, -# "mtime": "2026-05-01T08:14:33Z", +# "sha256": "ab12...", +# "size_bytes": 14982011, +# "modified": "2026-05-01T08:14:33Z", # ... # } ``` -The `fingerprint_sha256` + `hf_revision` together pin the corpus -deterministically. An auditor running `forgelm audit data/*.jsonl` -on the same input must see the same fingerprint. +The `sha256` + `hf_revision` together pin the corpus deterministically +(local files also carry `size_bytes` + `modified`; the field shape +comes from `forgelm.compliance._fingerprint_local_file`). An auditor +running `forgelm audit data/*.jsonl` on the same input must see the +same fingerprint. ### Q4: "Show me the supply chain" diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md index 22a57924..664b8155 100644 --- a/docs/qms/access_control-tr.md +++ b/docs/qms/access_control-tr.md @@ -78,19 +78,32 @@ ID'leri sonsuza kadar chain'de kalır. Bir operatör ayrıldığında: ### 3.4 `FORGELM_AUDIT_SECRET` rotasyonu -HMAC-chain anahtarı (audit koşumu başına; gerçek imzalama anahtarını -türetmek için per-output-dir salt ile XOR'lanır — bkz. -`forgelm/cli/subcommands/_purge.py:_resolve_salt`). Tier-1 secret -olarak ele al: +HMAC-chain imzalama anahtarı **audit koşumu başına** +`SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` olarak türetilir +(birleştirme, bkz. `forgelm/compliance.py:104-114`). Not: +`/.forgelm_audit_salt`'ta yazılı per-output-dir salt +**ayrı bir konu**'dur — `forgelm purge` / `forgelm reverse-pii` +event'lerindeki tanımlayıcı hash'lemesini salt'lar +(`_purge._resolve_salt` / `_purge._hash_target_id`) ve chain-key +türetimine KATILMAZ. Secret'ı Tier-1 credential olarak ele al: - **Uzunluk:** 32+ rastgele bayt (256 bit entropi). - **Substrate:** KMS / Vault / eşdeğer. VCS'e checked-in `.env` asla. -- **Rotasyon cadence'i:** quarterly önerilen; şüpheli compromise'da - hemen. -- **Rotasyon prosedürü:** yeni değer sonraki pipeline koşumundan - ÖNCE env'e iner. Eski koşumların chain'leri eski anahtarla - doğrulanabilir kalır (audit log girişleri emit anındaki anahtarla - bound; ForgeLM chain üzerinde anahtar migration'u DESTEKLEMEZ — +- **Rotasyon cadence'i:** **output-dir lifecycle'lar arasında** — + her girişin HMAC'ı emit anındaki secret'a bağlı olduğundan, + rotasyon mevcut `audit_log.jsonl` + `.manifest.json` çiftini + arşivledikten SONRA yapılmalıdır. Output-dir ortasında rotasyon + karışık-secret aralığı için `forgelm verify-audit --require-hmac`'i + kırar (tasarım gereği — verifier'ın contract'ı "her girişin HMAC'ı + aynı secret'ı anahtarlar"). Cadence'ı taze `` + kestiğiniz sıklığa eşle (sürüm başına / çeyrek başına / proje + başına) ve şüpheli compromise'da hemen rotate et + aynı anda yeni + bir output-dir başlat. +- **Rotasyon prosedürü:** önceki ``'i write-once + storage'a arşivle, KMS'de taze bir secret üret, sonraki pipeline + koşumunu YENİ bir ``'e yönlendir. Önceki chain önceki + secret ile doğrulanabilir kalır (audit-log integrity per-output-dir; + ForgeLM chain üzerinde anahtar migration'u DESTEKLEMEZ — tasarımdan). ## 4. CI runner identity binding @@ -123,8 +136,13 @@ run sayfasına korelate edebilir. ```yaml train: + variables: + FORGELM_OPERATOR: "gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" + # FORGELM_AUDIT_SECRET'ı projenin "Settings → CI/CD → Variables" + # panelinden, operatörün secret manager'ından (HashiCorp Vault, AWS + # Secrets Manager vb.) beslenen *masked + protected* bir variable + # olarak inject et; literal secret'ı asla .gitlab-ci.yml'a yapıştırma. script: - - export FORGELM_OPERATOR="gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" - forgelm --config config.yaml ``` @@ -177,15 +195,24 @@ IdP kontrolüdür. Audit chain her ikisini kaydeder, böylece bir denetçi ihlalleri tespit edebilir: ```bash -# Trainer == approver olduğu approval'ları bul (görev ayrılığı ihlali) -forgelm verify-audit --output-dir ./outputs --json | jq ' - .events[] | - select(.event == "human_approval.granted") | - .matching_training_event = ( - [.|.run_id == .. .run_id and .event == "training.started"] | first - ) | - select(.operator == .matching_training_event.operator) | - .' +# 1. Önce zincir bütünlüğünü doğrula (positional log_path; bu +# subcommand'ta tasarım gereği --output-dir / --json flag'i yoktur). +forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac + +# 2. Koşum başına trainer. +jq -r 'select(.event == "training.started") | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl > /tmp/trainers.tsv + +# 3. Aynı koşum için trainer listesinde yer alan onaylayan-id'ye sahip +# approval'lar (görev ayrılığı ihlali). +jq -r --slurpfile t /tmp/trainers.tsv \ + 'select(.event == "human_approval.granted") | + . as $a | + $t[] | split("\t") as $row | + select($row[0] == $a.run_id and $row[1] == $a.operator) | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl ``` ## 7. Webhook secret ayrılığı @@ -215,8 +242,9 @@ Erişim-kontrolü kanıtını yürüyen operatör denetçi için: - [ ] İki aktif pipeline aynı `FORGELM_OPERATOR` değerini paylaşmıyor. - [ ] `FORGELM_AUDIT_SECRET` KMS / Vault substrate'inde yaşıyor, VCS'de veya düz `.env` dosyasında değil. -- [ ] Quarterly KMS audit log'u `FORGELM_AUDIT_SECRET` rotasyonu - gösteriyor. +- [ ] KMS audit log'u `FORGELM_AUDIT_SECRET` rotasyonunu output-dir + lifecycle sınırlarına hizalı gösteriyor (asla output-dir + ortasında değil). - [ ] Son 90 gündeki her `human_approval.granted` event'i için approval-gate kimliği training kimliğinden farklı (sample-audit, exhaustive değil). diff --git a/docs/qms/access_control.md b/docs/qms/access_control.md index 073aebb2..0c38f0e5 100644 --- a/docs/qms/access_control.md +++ b/docs/qms/access_control.md @@ -77,20 +77,34 @@ remain in the chain forever. When an operator leaves: ### 3.4 `FORGELM_AUDIT_SECRET` rotation -The HMAC-chain key (per audit run; XOR'd with the per-output-dir -salt to derive the actual signing key — see -`forgelm/cli/subcommands/_purge.py:_resolve_salt`). Treat as a Tier-1 -secret: +The HMAC-chain signing key is derived **per audit run** as +`SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (concatenation, see +`forgelm/compliance.py:104-114`). Note: the per-output-dir salt +written to `/.forgelm_audit_salt` is a **distinct +concern** — it salts identifier hashing inside `forgelm purge` / +`forgelm reverse-pii` events (`_purge._resolve_salt` / +`_purge._hash_target_id`) and does NOT participate in chain-key +derivation. Treat the secret as a Tier-1 credential: - **Length:** 32+ random bytes (256 bits of entropy). - **Substrate:** KMS / Vault / equivalent. Never in `.env` checked into VCS. -- **Rotation cadence:** quarterly recommended; immediately on - suspected compromise. -- **Rotation procedure:** new value lands in env BEFORE the next - pipeline run. Old runs' chains remain verifiable with the old - key (audit log entries are bound to the key live at emit time; - ForgeLM does NOT support key migration on the chain — by design). +- **Rotation cadence:** **between output-dir lifecycles** — every + entry's HMAC is bound to the secret live at emit time, so rotation + must occur AFTER archiving the current `audit_log.jsonl` + + `.manifest.json` pair. Mid-output-dir rotation breaks + `forgelm verify-audit --require-hmac` for the mixed-secret span + (by design — the verifier's contract is "every entry's HMAC keys + the same secret"). Set the cadence to match how often you cut a + fresh `` (per release / per quarter / per project), + and rotate immediately on suspected compromise + roll a new + output-dir at the same time. +- **Rotation procedure:** archive the prior `` to + write-once storage, generate a fresh secret in KMS, point the + next pipeline run at a NEW ``. The prior chain remains + verifiable with the prior secret (audit-log integrity is + per-output-dir; ForgeLM does NOT support key migration on the + chain — by design). ## 4. CI runner identity binding @@ -122,8 +136,13 @@ run page in seconds. ```yaml train: + variables: + FORGELM_OPERATOR: "gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" + # Inject FORGELM_AUDIT_SECRET from the project's "Settings → CI/CD + # → Variables" panel as a *masked + protected* variable sourced + # from the deployer's secret manager (HashiCorp Vault, AWS Secrets + # Manager, etc.); never paste the literal secret into .gitlab-ci.yml. script: - - export FORGELM_OPERATOR="gitlab:${CI_PROJECT_PATH}:${CI_PIPELINE_ID}:job-${CI_JOB_ID}" - forgelm --config config.yaml ``` @@ -176,15 +195,24 @@ deployer-side IdP control. The audit chain records both, so an auditor can detect violations: ```bash -# Find approvals where trainer == approver (segregation violation) -forgelm verify-audit --output-dir ./outputs --json | jq ' - .events[] | - select(.event == "human_approval.granted") | - .matching_training_event = ( - [.|.run_id == .. .run_id and .event == "training.started"] | first - ) | - select(.operator == .matching_training_event.operator) | - .' +# 1. Verify the chain integrity first (positional log_path; no +# --output-dir / --json flags exist on this subcommand by design). +forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac + +# 2. Trainer per run. +jq -r 'select(.event == "training.started") | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl > /tmp/trainers.tsv + +# 3. Approvals whose approver-id appears in the trainer list for the +# same run (segregation violation). +jq -r --slurpfile t /tmp/trainers.tsv \ + 'select(.event == "human_approval.granted") | + . as $a | + $t[] | split("\t") as $row | + select($row[0] == $a.run_id and $row[1] == $a.operator) | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl ``` ## 7. Webhook secret separation @@ -213,7 +241,8 @@ For a deployer auditor walking access-control evidence: - [ ] No two active pipelines share an `FORGELM_OPERATOR` value. - [ ] `FORGELM_AUDIT_SECRET` lives in a KMS / Vault substrate, never in VCS or a plain `.env` file. -- [ ] Quarterly KMS audit log shows `FORGELM_AUDIT_SECRET` rotation. +- [ ] KMS audit log shows `FORGELM_AUDIT_SECRET` rotation aligned to + output-dir lifecycle boundaries (never mid-output-dir). - [ ] Approval-gate identity differs from training identity for every `human_approval.granted` event in the past 90 days (sample-audit, not exhaustive). diff --git a/docs/qms/encryption_at_rest-tr.md b/docs/qms/encryption_at_rest-tr.md index 966cf981..145f2199 100644 --- a/docs/qms/encryption_at_rest-tr.md +++ b/docs/qms/encryption_at_rest-tr.md @@ -58,8 +58,17 @@ Açıkça kapsam DIŞI tehditler: endpoint detection / sandboxing bunu yönetir. - **Meşru decryption credential'lı insider** — şifreleme yetkili erişime karşı koruyamaz; cevap erişim kontrolü + audit log'dur. -- **Kripto-algoritma kırılması** — SHA-256 / AES-256-GCM'in QMS - doküman ömrü boyunca sağlam kalacağı varsayılır. + +Uzun-ufuk (>5 yıl) — kısmi kapsam: + +- **Kripto-algoritma kırılması (algoritma agility — ISO A.8.24).** + ForgeLM, mevcut QMS doküman ömrü boyunca SHA-256 / AES-256-GCM'in + sağlam kalacağını varsayar. Operatörün IT politikası NIST SP + 800-131A revizyonlarına atıf yapmalı ve 5-yıllık cadence'da + yeniden değerlendirmelidir; ForgeLM bir algoritma geçişi + yayınladığında (ör. SHA-3 audit chain) bu QMS dokümanı + güncellenir. O zamana kadar algoritma-kırılma exposure'u + residual'dir ve kabul edilmiştir. ## 4. Asset sınıfı başına önerilen kontroller diff --git a/docs/qms/encryption_at_rest.md b/docs/qms/encryption_at_rest.md index f550a298..4803ee87 100644 --- a/docs/qms/encryption_at_rest.md +++ b/docs/qms/encryption_at_rest.md @@ -57,8 +57,16 @@ Threats explicitly OUT of scope: - **Insider with legitimate decryption credentials** — encryption cannot defend against authorised access; access control + audit log is the answer. -- **Crypto-algorithm break** — assume SHA-256 / AES-256-GCM remain - intact for the QMS document's lifetime. + +Long-horizon (>5 years) — partial coverage: + +- **Crypto-algorithm break (algorithm agility — ISO A.8.24).** + ForgeLM assumes SHA-256 / AES-256-GCM remain intact for the + current QMS document lifetime. The deployer's IT policy should + reference NIST SP 800-131A revisions and reassess on a 5-year + cadence; this QMS document is updated when ForgeLM ships an + algorithm transition (e.g. SHA-3 audit chain). Until then, + algorithm-break exposure is residual and accepted. ## 4. Recommended controls per asset class diff --git a/docs/qms/risk_treatment_plan-tr.md b/docs/qms/risk_treatment_plan-tr.md index 79a23a8e..0a91ce8f 100644 --- a/docs/qms/risk_treatment_plan-tr.md +++ b/docs/qms/risk_treatment_plan-tr.md @@ -79,7 +79,7 @@ ekler. |---|---| | Açıklama | `audit_log.jsonl`'a write erişimi olan adversary girişleri yeniden yazıp bir dağıtımı gizler / onaylanmamış bir modeli onaylar | | L × I (inherent) | Low × High = MED | -| Treatment | Append-only `O_APPEND` + `flock` + per-line `fsync`; HMAC chain (per-line `_hmac` field per-output-dir salt + env secret ile XOR); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` truncate-and-resume reddeder); `forgelm verify-audit` uçtan uca doğrular | +| Treatment | Append-only `O_APPEND` + `flock` + per-line `fsync`; HMAC chain (per-line `_hmac` alanı, per-run imzalama anahtarı `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` olarak türetilir — `forgelm/compliance.py:104-114` — per-output-dir `.forgelm_audit_salt` ayrı bir konudur, purge / reverse-pii tanımlayıcı hash'lerini salt'lar ve chain-key türetimine KATILMAZ); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` truncate-and-resume reddeder); `forgelm verify-audit` uçtan uca doğrular | | Residual L × I | Low × Low = LOW | | Sahip | AI Officer | | Review cadence | Continuous (`forgelm verify-audit` cron) | @@ -90,8 +90,8 @@ ekler. |---|---| | Açıklama | Operatör `forgelm purge --row-id alice@example.com` corpusa karşı çalıştırır, ama haftalar önce satırı eğiten model öğrenilen ağırlıklardan onu hala üretebilir | | L × I (inherent) | High × Med = HIGH | -| Treatment | Silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` audit event raise edilir; operatör residual riski özneye bildirir; yüksek-stakes dağıtımlar için sıfırdan yeniden eğit | -| Residual L × I | Med × Med = MED | +| Treatment | ForgeLM, silinen satırı tüketen koşum bir `final_model/` artefaktına sahipse `data.erasure_warning_memorisation` event'ı yayar — bu bir **detection** sinyalidir, mitigation değildir. Operatör residual riski kabul eder ve organizasyonel kontrolleri uygular (özne bildirimi, yüksek-stakes dağıtımlar için sıfırdan yeniden eğit, DPIA güncellemesi). | +| Residual L × I | High × Med = HIGH (risk **kabul edildi**, bkz. §5) | | Sahip | Data Protection Officer (DPO) | | Review cadence | Madde 17 talebi başına | @@ -194,7 +194,7 @@ imzalamalıdır: | Risk ID | Inherent | Residual | Kabul eden | Tarih | Gerekçe | |---|---|---|---|---|---| -| R-05 | HIGH | MED | [AI Officer] | [TARİH] | Memorisation residual riski kabul edildi; operatör politikası: özneyi bilgilendir + yüksek-stakes dağıtımlar için sıfırdan yeniden eğit | +| R-05 | HIGH | HIGH (kabul edildi) | [AI Officer] | [TARİH] | ForgeLM yalnız `data.erasure_warning_memorisation` detection sinyalini katar; operatör residual memorisation riskini kabul eder ve özne bildirimi + yüksek-stakes dağıtımlar için sıfırdan yeniden eğit + DPIA güncellemesi yoluyla deşarj eder. | | ... | ... | ... | ... | ... | ... | ## 6. İnceleme diff --git a/docs/qms/risk_treatment_plan.md b/docs/qms/risk_treatment_plan.md index 5a85dd00..a92b8407 100644 --- a/docs/qms/risk_treatment_plan.md +++ b/docs/qms/risk_treatment_plan.md @@ -78,7 +78,7 @@ section §4. |---|---| | Description | Adversary with write access to `audit_log.jsonl` rewrites entries to hide a deployment / approve a model that wasn't approved | | L × I (inherent) | Low × High = MED | -| Treatment | Append-only `O_APPEND` + `flock` + `fsync` per line; HMAC chain (per-line `_hmac` field XOR'd with per-output-dir salt + env secret); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` refuses truncate-and-resume); `forgelm verify-audit` validates end-to-end | +| Treatment | Append-only `O_APPEND` + `flock` + `fsync` per line; HMAC chain (per-line `_hmac` field, per-run signing key derived as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` per `forgelm/compliance.py:104-114` — the per-output-dir `.forgelm_audit_salt` is a distinct concern that salts purge / reverse-pii identifier hashes and does NOT participate in chain-key derivation); SHA-256 prev-hash chain; genesis manifest sidecar (`_check_genesis_manifest` refuses truncate-and-resume); `forgelm verify-audit` validates end-to-end | | Residual L × I | Low × Low = LOW | | Owner | AI Officer | | Review cadence | Continuous (`forgelm verify-audit` cron) | @@ -89,8 +89,8 @@ section §4. |---|---| | Description | Operator runs `forgelm purge --row-id alice@example.com` against the corpus, but the model trained on the row weeks earlier may still reproduce it from learned weights | | L × I (inherent) | High × Med = HIGH | -| Treatment | `data.erasure_warning_memorisation` audit event raised when the run that consumed the deleted row has a `final_model/` artefact; deployer notifies subject of the residual risk; for high-stakes deployments retrain from scratch | -| Residual L × I | Med × Med = MED | +| Treatment | ForgeLM emits `data.erasure_warning_memorisation` when the run that consumed the deleted row has a `final_model/` artefact — this is a **detection** signal, not a mitigation. Deployer accepts the residual risk and applies organisational controls (subject notification, retrain from scratch for high-stakes deployments, DPIA update). | +| Residual L × I | High × Med = HIGH (risk **accepted**, see §5) | | Owner | Data Protection Officer (DPO) | | Review cadence | Per Article 17 request | @@ -193,7 +193,7 @@ acceptance: | Risk ID | Inherent | Residual | Accepted by | Date | Justification | |---|---|---|---|---|---| -| R-05 | HIGH | MED | [AI Officer] | [DATE] | Memorisation residual risk acknowledged; deployer policy: notify subject + retrain from scratch for high-stakes deployments | +| R-05 | HIGH | HIGH (accepted) | [AI Officer] | [DATE] | ForgeLM contributes the `data.erasure_warning_memorisation` detection signal only; the deployer accepts the residual memorisation risk and discharges it through subject notification + retrain-from-scratch for high-stakes deployments + DPIA update. | | ... | ... | ... | ... | ... | ... | ## 6. Review diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md index b017fad3..4653f0cf 100644 --- a/docs/qms/sop_incident_response-tr.md +++ b/docs/qms/sop_incident_response-tr.md @@ -9,7 +9,7 @@ Dağıtılan fine-tuned modeller için güvenlik olaylarını, model arızalarını, **güvenlik olaylarını** ve düzeltici eylemleri -yönetme prosedürünü tanımla. Wave 4 / Faz 23 genişletmesi: §6 +yönetme prosedürünü tanımla. Wave 4 / Faz 23 genişletmesi: §4 güvenlik-olay playbook'unu (audit-chain integrity, credential leak, supply-chain CVE, webhook target compromise, GDPR DSAR'lar) mevcut AI-safety olay akışı yanında kapsar. @@ -87,10 +87,14 @@ mismatch, manifest sidecar truncation, HMAC signature mismatch). `audit_log.manifest.json`, `.sha256` sidecar'ı (varsa) ve `/.forgelm_audit_salt`'ı write-once forensic substrate'e (S3 Object Lock, Azure Immutable Blob) kopyala. -3. [ ] **Son güvenilen girişi tanımla** — `forgelm verify-audit - --until-line N` ile bisect ederek ilk kötü satırı bul; o - satırdan önce her şey forensic olarak güvenilirdir, sonra - her şey tainted sayılmalıdır. +3. [ ] **Son güvenilen girişi tanımla** — + `forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac 2>&1 | tee verify.log` + çalıştır; verifier ilk başarısızlıkta exit 1 yapar ve hatalı + satır numarası stderr'e iner. Tam sınır gerekiyorsa, + `head -n N audit_log.jsonl > tmp.jsonl` ile manuel bisect + yap ve exit 0 veren son N bulunana kadar `verify-audit`'ı + `tmp.jsonl`'a karşı yeniden çalıştır. O satıra kadar her şey + forensic olarak güvenilirdir; sonrası tainted sayılmalıdır. 4. [ ] **Bildir** AI Officer + Güvenlik ekibi + DPO (kötü satırdan sonra herhangi bir PII-bearing event varsa). 5. [ ] **Karar ver** tainted-tail girişlerini kanıt olarak (önerilen) @@ -155,11 +159,13 @@ olabilir. **Runbook:** 1. [ ] **`webhook.secret_env`'i hemen rotate et**. -2. [ ] **Lifecycle olaylarını yeniden emit et** audit chain'den - saldırganın olayları recipient'a splice etmediğini teyit - etmek için. (`forgelm verify-audit --replay-since - ` v0.6.0+ aracıdır; v0.5.5 için chain'i manuel - yürü.) +2. [ ] **Audit chain'i yürü** saldırganın olayları recipient'a splice + etmediğini teyit için: `audit_log.jsonl`'ı event sınıfına göre + filtrele (`jq 'select(.event | startswith("notify_"))'`) ve + emit edilen her lifecycle girişinin aynı `run_id` içindeki + gerçek bir `pipeline.*` event'ına karşılık geldiğini doğrula. + Eşleşmeyen timestamp'lar veya yetim `notify_*` satırları + splice sinyalidir. 3. [ ] **`safe_post` error log'larını kontrol et** rotation sonrası redact'lı Authorization header'lar için — saldırganın artık geçerli token tutmadığını teyit et. diff --git a/docs/qms/sop_incident_response.md b/docs/qms/sop_incident_response.md index 330eff1e..d00de374 100644 --- a/docs/qms/sop_incident_response.md +++ b/docs/qms/sop_incident_response.md @@ -9,7 +9,7 @@ Define the procedure for handling safety incidents, model failures, **security incidents**, and corrective actions for deployed -fine-tuned models. Wave 4 / Faz 23 expansion: §6 covers the security- +fine-tuned models. Wave 4 / Faz 23 expansion: §4 covers the security- incident playbook (audit-chain integrity, credential leak, supply- chain CVE, webhook target compromise, GDPR DSARs) alongside the existing AI-safety incident flow. @@ -89,10 +89,15 @@ mismatch, manifest sidecar truncation, HMAC signature mismatch). present), and `/.forgelm_audit_salt` to a write-once forensic substrate (S3 Object Lock, Azure Immutable Blob). -3. [ ] **Identify the last trusted entry** — bisect with - `forgelm verify-audit --until-line N` to find the first bad - line; everything before that line is forensically trusted, - everything after must be considered tainted. +3. [ ] **Identify the last trusted entry** — run + `forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac 2>&1 | tee verify.log`; + the verifier exits 1 on first failure and the offending line + number lands in stderr. If you need the precise boundary, + bisect manually with `head -n N audit_log.jsonl > tmp.jsonl` + and re-run `verify-audit` against `tmp.jsonl` until the last + N that exits 0 is found. Everything up to that line is + forensically trusted; everything after must be considered + tainted. 4. [ ] **Notify** the AI Officer + Security team + DPO (if any PII-bearing event was after the bad line). 5. [ ] **Decide** whether to retain the tainted-tail entries as @@ -156,10 +161,13 @@ that an attacker may have observed. **Runbook:** 1. [ ] **Rotate `webhook.secret_env`** immediately. -2. [ ] **Re-emit lifecycle events** from the audit chain to confirm - the attacker did not splice events into the recipient. - (`forgelm verify-audit --replay-since ` is the - v0.6.0+ tool for this; for v0.5.5, manually walk the chain.) +2. [ ] **Walk the audit chain** to confirm the attacker did not splice + events into the recipient: filter + `audit_log.jsonl` by event class + (`jq 'select(.event | startswith("notify_"))'`) and confirm + every emitted lifecycle entry corresponds to a real + `pipeline.*` event in the same `run_id`. Mismatched timestamps + or orphan `notify_*` rows are the splice signal. 3. [ ] **Check `safe_post` error logs** for masked Authorization headers post-rotation — confirm the attacker no longer holds a valid token. diff --git a/docs/qms/statement_of_applicability-tr.md b/docs/qms/statement_of_applicability-tr.md index 09f6e746..37991723 100644 --- a/docs/qms/statement_of_applicability-tr.md +++ b/docs/qms/statement_of_applicability-tr.md @@ -146,11 +146,17 @@ ForgeLM-spesifik kontrol envanteriyle ilgilidir. | Tema | Toplam | Applicable | Excluded (ForgeLM scope) | FL-supported | FL-helps | |---|---|---|---|---|---| -| A.5 Organisational | 37 | 37 | 0 | 4 | 24 | -| A.6 People | 8 | 8 | 0 | 0 | 4 | +| A.5 Organisational | 37 | 37 | 0 | 3 | 24 | +| A.6 People | 8 | 8 | 0 | 0 | 5 | | A.7 Physical | 14 | 14 (operatör ISMS) | 14 (ForgeLM-spesifik) | 0 | 0 | -| A.8 Technological | 34 | 34 | 0 | 7 | 22 | -| **Toplam** | **93** | **93 (operatör ISMS)** | **14 (ForgeLM-spesifik)** | **11** | **50** | +| A.8 Technological | 34 | 34 | 0 | 8 | 19 | +| **Toplam** | **93** | **93 (operatör ISMS)** | **14 (ForgeLM-spesifik)** | **11** | **48** | + +§3.1–§3.4 (bu SoA'nın özetlediği design-doc tablosu) row-by-row +yeniden sayım. Tema başına tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 +/ 3 OOS; A.7: 0 / 0 / 14 OOS; A.8: 8 / 19 / 7 OOS — toplam 11 `FL` ++ 48 `FL-helps` + 34 OOS = 93. Design doc'un `§3` "Coverage tally" +paragrafıyla çapraz kontrol; ikisi eşleşmek zorunda. ## 4. İnceleme diff --git a/docs/qms/statement_of_applicability.md b/docs/qms/statement_of_applicability.md index c26ca8f1..38728025 100644 --- a/docs/qms/statement_of_applicability.md +++ b/docs/qms/statement_of_applicability.md @@ -145,11 +145,17 @@ ForgeLM-specific control inventory. | Theme | Total | Applicable | Excluded (ForgeLM scope) | FL-supported | FL-helps | |---|---|---|---|---|---| -| A.5 Organisational | 37 | 37 | 0 | 4 | 24 | -| A.6 People | 8 | 8 | 0 | 0 | 4 | +| A.5 Organisational | 37 | 37 | 0 | 3 | 24 | +| A.6 People | 8 | 8 | 0 | 0 | 5 | | A.7 Physical | 14 | 14 (deployer ISMS) | 14 (ForgeLM-specific) | 0 | 0 | -| A.8 Technological | 34 | 34 | 0 | 7 | 22 | -| **Total** | **93** | **93 (deployer ISMS)** | **14 (ForgeLM-specific)** | **11** | **50** | +| A.8 Technological | 34 | 34 | 0 | 8 | 19 | +| **Total** | **93** | **93 (deployer ISMS)** | **14 (ForgeLM-specific)** | **11** | **48** | + +Row-by-row recount of §3.1–§3.4 (the design-doc table this SoA +summarises). Per-theme tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 / 3 +OOS; A.7: 0 / 0 / 14 OOS; A.8: 8 / 19 / 7 OOS — sums to 11 `FL` + +48 `FL-helps` + 34 OOS = 93. Cross-check the design doc's `§3` +"Coverage tally" paragraph; the two must match. ## 4. Review diff --git a/docs/reference/iso27001_control_mapping-tr.md b/docs/reference/iso27001_control_mapping-tr.md index d8960a96..9cff9984 100644 --- a/docs/reference/iso27001_control_mapping-tr.md +++ b/docs/reference/iso27001_control_mapping-tr.md @@ -13,7 +13,9 @@ > operatörün diğer kaynaklarla birleştirdiği kısmi kanıt sağlar. > - **`OOS`** — *Out of scope*: yalnız operatör; ForgeLM hiçbir katkı yapmaz. > -> **Bu sürüm için kapsama özeti:** FL 11 / FL-helps 50 / OOS 32. +> **Bu sürüm için kapsama özeti:** FL 11 / FL-helps 48 / OOS 34 +> (design doc §3.1–§3.4 row-by-row yeniden sayım; tema başına +> A.5: 3 / 24 / 10, A.6: 0 / 5 / 3, A.7: 0 / 0 / 14, A.8: 8 / 19 / 7). ## A.5 Organisational controls (37) @@ -52,7 +54,7 @@ | A.5.31 Yasal, kanuni, düzenleyici ve sözleşmesel gerekliliklerin tanımlanması | FL-helps | EU AI Act + GDPR mappings; Annex IV bundle | | A.5.32 Fikri mülkiyet hakları | FL-helps | SBOM'da lisans çıkarımı; HF model-card metadata | | A.5.33 Kayıtların korunması | FL | Append-only + HMAC + manifest sidecar | -| A.5.34 Gizlilik ve PII korunması | FL | `forgelm reverse-pii` Madde 15; `forgelm purge` Madde 17 | +| A.5.34 Gizlilik ve PII korunması | FL | `forgelm reverse-pii` Madde 15; `forgelm purge` Madde 17 (ayrıca bkz. A.8.3) | | A.5.35 Bilgi güvenliğinin bağımsız incelenmesi | OOS | — | | A.5.36 Bilgi güvenliği için politikalar, kurallar ve standartlara uyum | FL-helps | Pydantic config validation; `forgelm doctor`; CI gates | | A.5.37 Dokümante edilmiş işletim prosedürleri | FL-helps | `docs/qms/` SOPs | @@ -119,7 +121,7 @@ SoA'sının uçtan uca auditable olması için tamlık adına listelenmiştir. | A.8.21 Ağ hizmetlerinin güvenliği | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | | A.8.22 Ağ ayrımı | OOS | — | | A.8.23 Web filtreleme | OOS | — | -| A.8.24 Kriptografi kullanımı | FL | SHA-256 + HMAC chain; salted SHA-256 identifier hashing | +| A.8.24 Kriptografi kullanımı | FL | SHA-256 + HMAC chain (per-run imzalama anahtarı = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`); ayrıca purge / reverse-pii için salted SHA-256 identifier hashing | | A.8.25 Güvenli geliştirme yaşam döngüsü | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | | A.8.26 Uygulama güvenliği gereksinimleri | FL-helps | F-compliance-110 strict gate; ReDoS guard | | A.8.27 Güvenli sistem mimarisi ve mühendislik prensipleri | FL-helps | Append-only audit log mimarisi | diff --git a/docs/reference/iso27001_control_mapping.md b/docs/reference/iso27001_control_mapping.md index 05fa4af5..0c2646e3 100644 --- a/docs/reference/iso27001_control_mapping.md +++ b/docs/reference/iso27001_control_mapping.md @@ -13,7 +13,10 @@ > provides partial evidence the deployer combines with other sources. > - **`OOS`** — *Out of scope*: deployer-only, ForgeLM contributes nothing. > -> **Coverage tally for this version:** FL 11 / FL-helps 50 / OOS 32. +> **Coverage tally for this version:** FL 11 / FL-helps 48 / OOS 34 +> (row-by-row recount of §3.1–§3.4 in the design doc; +> per-theme split A.5: 3 / 24 / 10, A.6: 0 / 5 / 3, A.7: 0 / 0 / 14, +> A.8: 8 / 19 / 7). ## A.5 Organisational controls (37) @@ -52,7 +55,7 @@ | A.5.31 Identification of legal, statutory, regulatory and contractual requirements | FL-helps | EU AI Act + GDPR mappings; Annex IV bundle | | A.5.32 Intellectual property rights | FL-helps | License extraction in SBOM; HF model-card metadata | | A.5.33 Protection of records | FL | Append-only + HMAC + manifest sidecar | -| A.5.34 Privacy and protection of PII | FL | `forgelm reverse-pii` Article 15; `forgelm purge` Article 17 | +| A.5.34 Privacy and protection of PII | FL | `forgelm reverse-pii` Article 15; `forgelm purge` Article 17 (see also A.8.3) | | A.5.35 Independent review of information security | OOS | — | | A.5.36 Compliance with policies, rules and standards for information security | FL-helps | Pydantic config validation; `forgelm doctor`; CI gates | | A.5.37 Documented operating procedures | FL-helps | `docs/qms/` SOPs | @@ -119,7 +122,7 @@ completeness so the deployer's SoA is auditable end-to-end. | A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | | A.8.22 Segregation of networks | OOS | — | | A.8.23 Web filtering | OOS | — | -| A.8.24 Use of cryptography | FL | SHA-256 + HMAC chain; salted SHA-256 identifier hashing | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC chain (per-run signing key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`); separately, salted SHA-256 identifier hashing for purge / reverse-pii | | A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | | A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; ReDoS guard | | A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture | diff --git a/docs/reference/soc2_trust_criteria_mapping-tr.md b/docs/reference/soc2_trust_criteria_mapping-tr.md index 6bef1e86..426156e9 100644 --- a/docs/reference/soc2_trust_criteria_mapping-tr.md +++ b/docs/reference/soc2_trust_criteria_mapping-tr.md @@ -32,7 +32,7 @@ kategoriler engagement-bazında scoplanır. | CC1.5 | Hesap verebilirliği uygular | `FORGELM_OPERATOR` attribution + audit chain | | CC2.1 | İç kontrolleri içe iletişim | `audit_event_catalog.md` + `deployer_instructions.md` | | CC2.2 | Dışa iletişim | Madde 13 `deployer_instructions.md`; Annex IV public summary | -| CC2.3 | Düzenleyicilerle iletişim | Annex IV bundle gönderime hazır | +| CC2.3 | Düzenleyicilerle iletişim | Annex IV bundle + `compliance_report.json` operatörün talep üzerine düzenleyiciye gönderdiği artefaktlardır; ForgeLM iletişim kanalını işletmez (operatör-tarafı kontrol) | | CC3.1 | Uygun hedefler belirler | `compliance.intended_purpose`; risk classification | | CC3.2 | Riskleri tanımlar ve analiz eder | `risk_assessment` Pydantic block; safety eval; `risk_treatment_plan.md` | | CC3.3 | Sahtekarlık risklerini değerlendirir | Audit log tamper-evidence; HMAC chain; manifest sidecar | diff --git a/docs/reference/soc2_trust_criteria_mapping.md b/docs/reference/soc2_trust_criteria_mapping.md index f0158cb1..d2c0d627 100644 --- a/docs/reference/soc2_trust_criteria_mapping.md +++ b/docs/reference/soc2_trust_criteria_mapping.md @@ -32,7 +32,7 @@ categories are scoped per-engagement. | CC1.5 | Enforces accountability | `FORGELM_OPERATOR` attribution + audit chain | | CC2.1 | Communicates internally about controls | `audit_event_catalog.md` + `deployer_instructions.md` | | CC2.2 | Communicates externally | Article 13 `deployer_instructions.md`; Annex IV public summary | -| CC2.3 | Communicates with regulators | Annex IV bundle ready to ship | +| CC2.3 | Communicates with regulators | Annex IV bundle + `compliance_report.json` are the artefacts the deployer ships to the regulator on request; ForgeLM does NOT operate the communication channel itself (deployer-side control) | | CC3.1 | Specifies suitable objectives | `compliance.intended_purpose`; risk classification | | CC3.2 | Identifies and analyses risks | `risk_assessment` Pydantic block; safety eval; `risk_treatment_plan.md` | | CC3.3 | Considers fraud risks | Audit log tamper-evidence; HMAC chain; manifest sidecar | diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md index 4b3098f0..ba4b379a 100644 --- a/docs/reference/supply_chain_security-tr.md +++ b/docs/reference/supply_chain_security-tr.md @@ -76,7 +76,7 @@ diff <(jq -S '.components | sort_by(.purl)' sbom-prev.json) \ # Dependency-Track'e ingest. curl -X POST -H "X-Api-Key: $DT_KEY" -H "Content-Type: application/octet-stream" \ --data-binary @sbom-ubuntu-latest-py3.11.json \ - https://depTrack.example.com/api/v1/bom + https://deptrack.example.com/api/v1/bom ``` ## pip-audit (transitive CVE taraması) @@ -177,7 +177,6 @@ getirmek için `pip-audit` ve `bandit`'i çeker. |---|---|---| | ISO 27001:2022 | A.5.21 | ICT tedarik zincirinde bilgi güvenliği yönetimi | | ISO 27001:2022 | A.8.8 | Teknik zafiyetlerin yönetimi | -| ISO 27001:2022 | A.8.30 | Dış kaynaklı geliştirme (transitive dep'ler outsourced kod olarak) | | SOC 2 | CC7.1 | Zafiyetleri tespit eder | | SOC 2 | CC8.1 | Değişimleri yetkilendirir (son release ile SBOM diff) | | SOC 2 | CC9.2 | Tedarikçi + iş ortağı riskini yönetir | diff --git a/docs/reference/supply_chain_security.md b/docs/reference/supply_chain_security.md index b71b165a..a4a93c19 100644 --- a/docs/reference/supply_chain_security.md +++ b/docs/reference/supply_chain_security.md @@ -76,7 +76,7 @@ diff <(jq -S '.components | sort_by(.purl)' sbom-prev.json) \ # Ingest into Dependency-Track. curl -X POST -H "X-Api-Key: $DT_KEY" -H "Content-Type: application/octet-stream" \ --data-binary @sbom-ubuntu-latest-py3.11.json \ - https://depTrack.example.com/api/v1/bom + https://deptrack.example.com/api/v1/bom ``` ## pip-audit (transitive CVE scan) @@ -178,7 +178,6 @@ PR. |---|---|---| | ISO 27001:2022 | A.5.21 | Managing information security in the ICT supply chain | | ISO 27001:2022 | A.8.8 | Management of technical vulnerabilities | -| ISO 27001:2022 | A.8.30 | Outsourced development (transitive deps as outsourced code) | | SOC 2 | CC7.1 | Detects vulnerabilities | | SOC 2 | CC8.1 | Authorises changes (SBOM diff against last release) | | SOC 2 | CC9.2 | Manages vendor + business-partner risk | diff --git a/docs/standards/localization.md b/docs/standards/localization.md index d8a8b950..ae99cd70 100644 --- a/docs/standards/localization.md +++ b/docs/standards/localization.md @@ -36,7 +36,7 @@ Rationale: | `docs/guides/*.md` | No (yet) | Turkish guides are an open task | | `docs/design/*.md` | No | Internal design history | | `docs/standards/*.md` | No | Contributor-facing | -| `docs/qms/*.md` | No | QMS templates in English (regulated orgs translate internally) | +| `docs/qms/*.md` | Yes | QMS templates ship bilingual EN+TR — regulated orgs in Turkey adopt the TR mirror as the canonical operating template | | `docs/marketing/**` | Mixed (strategy/ is TR, operational is EN) | Local-only; author's choice | | `docs/analysis/**` | Mixed | Analysis documents in whichever language suits the source | | `docs/roadmap/**` | No (yet) | English only; TR mirror summary at `docs/roadmap-tr.md` | diff --git a/forgelm/cli/subcommands/_doctor.py b/forgelm/cli/subcommands/_doctor.py index 7aebdab0..c81fe3e5 100644 --- a/forgelm/cli/subcommands/_doctor.py +++ b/forgelm/cli/subcommands/_doctor.py @@ -659,7 +659,7 @@ def _check_disk_space(path: str = ".") -> _CheckResult: "HF_TOKEN", # HuggingFace Hub auth token "HUGGING_FACE_HUB_TOKEN", # legacy alias of HF_TOKEN "HUGGINGFACE_TOKEN", # alternative spelling read by forgelm/utils.py - "FORGELM_RESUME_TOKEN", # API resume token (Phase 13 future) + "FORGELM_RESUME_TOKEN", # reserved name — masked defensively even if unused today # Defence-in-depth: third-party API keys ForgeLM accepts via YAML # interpolation (auth.openai_api_key etc.). No probe surfaces them # today, but pre-listing them means a future probe that adds env diff --git a/tests/test_check_anchor_resolution.py b/tests/test_check_anchor_resolution.py index d7cb316a..fb6c7102 100644 --- a/tests/test_check_anchor_resolution.py +++ b/tests/test_check_anchor_resolution.py @@ -72,6 +72,22 @@ def test_trims_leading_trailing_dashes(self): assert tool._slugify_heading(" a ") == "a" assert tool._slugify_heading("--a--") == "a" + def test_apostrophe_stripped(self): + # F-W4-TR-07 absorption: GFM strips apostrophes in "What's New". + tool = _load_tool() + assert tool._slugify_heading("What's New") == "whats-new" + + def test_backtick_stripped(self): + # F-W4-TR-07 absorption: GFM strips backticks in code-quoted headings. + tool = _load_tool() + assert tool._slugify_heading("Heading with `code`") == "heading-with-code" + + def test_atx_closing_hashes_stripped(self): + # ATX trailing-hash decoration must be stripped before slugify. + tool = _load_tool() + assert tool._normalise_heading_body("My Title #") == "My Title" + assert tool._normalise_heading_body("My Title ### ") == "My Title" + # --------------------------------------------------------------------------- # §2 — Resolution: relative paths + anchors @@ -111,6 +127,33 @@ def test_spa_hash_router_skipped(self, tmp_path: Path): source = _write(tmp_path / "src.md", "[ref](#/reference/usage)\n") assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + def test_tel_skipped(self, tmp_path: Path): + # F-W4-TR-04 absorption: tel: scheme must be in skip list. + tool = _load_tool() + source = _write(tmp_path / "src.md", "[call](tel:+15551234567)\n") + assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + + def test_javascript_skipped(self, tmp_path: Path): + # F-W4-TR-04 absorption: javascript: scheme must be in skip list. + tool = _load_tool() + source = _write(tmp_path / "src.md", "[js](javascript:void(0))\n") + assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + + def test_image_link_missing_target_flagged(self, tmp_path: Path): + # F-W4-TR-06 absorption: image-link regex parity with link form. + tool = _load_tool() + source = _write(tmp_path / "src.md", "![alt](missing.png)\n") + results = list(tool._extract_links(source)) + assert len(results) == 1, "regex must match the [alt](src) inside ![alt](src)" + broken = tool._resolve_link(results[0], tmp_path) + assert broken is not None and "target file not found" in broken.reason + + def test_image_link_present_target_resolves(self, tmp_path: Path): + tool = _load_tool() + _write(tmp_path / "ok.png", "") + source = _write(tmp_path / "src.md", "![ok](ok.png)\n") + assert tool._resolve_link(list(tool._extract_links(source))[0], tmp_path) is None + class TestResolveAnchor: def test_self_anchor_resolves(self, tmp_path: Path): @@ -195,6 +238,17 @@ def test_strict_mode_exits_one_on_broken(self, tmp_path: Path, capsys): captured = capsys.readouterr().out assert "FAIL" in captured + def test_strict_mode_exits_one_on_broken_anchor(self, tmp_path: Path, capsys): + # F-W4-TR-03 absorption: anchor-not-found and file-not-found must + # be symmetrically gated under --strict. + tool = _load_tool() + scope = tmp_path / "docs" + _write(scope / "target.md", "# Target\n\n## Real Section\n") + _write(scope / "src.md", "[link](target.md#nonexistent)\n") + rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs", "--strict"]) + assert rc == 1 + assert "FAIL" in capsys.readouterr().out + def test_quiet_suppresses_ok_summary(self, tmp_path: Path, capsys): tool = _load_tool() scope = tmp_path / "docs" diff --git a/tests/test_supply_chain_security.py b/tests/test_supply_chain_security.py index e21d827b..c5401e4e 100644 --- a/tests/test_supply_chain_security.py +++ b/tests/test_supply_chain_security.py @@ -338,3 +338,183 @@ def test_results_not_a_list_fails(self, tmp_path: Path, tool, capsys) -> None: rc = tool.main(["check_bandit", str(bad)]) assert rc == 1 assert "results" in capsys.readouterr().err + + def test_results_null_fails(self, tmp_path: Path, tool, capsys) -> None: + # F-W4-06 absorption: a report with ``"results": null`` is malformed + # and must NOT silently pass via ``or []``. + bad = self._write_report(tmp_path, {"results": None}) + rc = tool.main(["check_bandit", str(bad)]) + assert rc == 1 + assert "null" in capsys.readouterr().err + + def test_results_missing_key_fails(self, tmp_path: Path, tool, capsys) -> None: + # Missing ``results`` key — distinct from ``null`` value. + bad = self._write_report(tmp_path, {"errors": [], "metrics": {}}) + rc = tool.main(["check_bandit", str(bad)]) + assert rc == 1 + assert "missing 'results'" in capsys.readouterr().err + + def test_undefined_severity_summary_warning(self, tmp_path: Path, tool, capsys) -> None: + # F-W4-TR-01 absorption: bandit's documented UNDEFINED tier + # surfaces a single summary warning, not silence. + report = { + "results": [ + { + "test_id": "B999", + "test_name": "custom_rule_no_severity", + # No issue_severity field → bandit emits UNDEFINED. + "issue_confidence": "LOW", + "filename": "forgelm/x.py", + "line_number": 1, + "issue_text": "novel finding", + } + ] + } + rc = tool.main(["check_bandit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::bandit" in captured + assert "UNDEFINED" in captured + + +class TestCheckPipAuditExtraShapes: + """Extra pip-audit fixtures that pin documented contracts the + primary tier-suite leaves uncovered (F-W4-04 / F-W4-PS-07 / + F-W4-TR-02 / F-W4-TR-05 absorptions).""" + + @pytest.fixture + def tool(self): + return _load_tool_module(_PIP_AUDIT_TOOL, "check_pip_audit_extra") + + def _write_report(self, tmp_path: Path, payload: dict) -> Path: + path = tmp_path / "pip-audit.json" + path.write_text(json.dumps(payload), encoding="utf-8") + return path + + def test_missing_severity_field_summary_warning(self, tmp_path: Path, tool, capsys) -> None: + report = { + "dependencies": [ + { + "name": "u", + "version": "1.0.0", + "vulns": [{"id": "GHSA-no-sev"}], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::pip-audit" in captured + assert "without parseable severity" in captured + + def test_severity_list_with_explicit_tier(self, tmp_path: Path, tool) -> None: + # 2.7.x list-form carrying an explicit ``severity`` field. + report = { + "dependencies": [ + { + "name": "listpkg", + "version": "1.0.0", + "vulns": [ + { + "id": "GHSA-list", + "severity": [{"type": "CVSS_V3", "severity": "HIGH"}], + } + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + assert rc == 1, "list form with explicit HIGH tier must fail" + + def test_severity_list_with_cvss_score(self, tmp_path: Path, tool) -> None: + # CVSS_V3 type label + 9.5 score must derive CRITICAL. + report = { + "dependencies": [ + { + "name": "scorepkg", + "version": "1.0.0", + "vulns": [ + { + "id": "GHSA-score", + "severity": [{"type": "CVSS_V3", "score": "9.5"}], + } + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + assert rc == 1, "CVSS 9.5 must derive CRITICAL and fail the gate" + + def test_severity_list_cvss_score_medium(self, tmp_path: Path, tool, capsys) -> None: + report = { + "dependencies": [ + { + "name": "midscore", + "version": "1.0.0", + "vulns": [ + { + "id": "GHSA-mid-score", + "severity": [{"type": "CVSS_V3", "score": 5.5}], + } + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + captured = capsys.readouterr().out + assert rc == 0 + assert "::warning::pip-audit" in captured + + def test_aliases_nested_severity_2_6_shape(self, tmp_path: Path, tool) -> None: + # 2.6.x: severity buried under aliases[].severity[]. + report = { + "dependencies": [ + { + "name": "legacy", + "version": "1.0.0", + "vulns": [ + { + "id": "GHSA-old-shape", + "aliases": [ + { + "id": "CVE-2026-OLD", + "severity": [{"type": "CVSS_V3", "severity": "HIGH"}], + } + ], + } + ], + } + ] + } + rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + assert rc == 1, "2.6.x nested aliases shape must classify as HIGH" + + +class TestSbomSerialNumberUniqueness: + """F-W4-TR-08 absorption: the determinism contract strips + serialNumber before comparing; this test pins the *uniqueness* + side of CycloneDX 1.5 — two runs MUST emit different + ``serialNumber`` values so Dependency-Track ingest does not see + duplicate uploads.""" + + def test_serial_number_changes_between_runs(self) -> None: + first = subprocess.run( + [sys.executable, str(_SBOM_TOOL)], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + second = subprocess.run( + [sys.executable, str(_SBOM_TOOL)], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + sn_a = json.loads(first.stdout).get("serialNumber") + sn_b = json.loads(second.stdout).get("serialNumber") + assert sn_a and sn_b and sn_a != sn_b, ( + "serialNumber must be a fresh UUID per run per CycloneDX 1.5; " + "Dependency-Track rejects duplicate-serial uploads" + ) diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py index 2a3f23b8..074b30cf 100644 --- a/tools/check_anchor_resolution.py +++ b/tools/check_anchor_resolution.py @@ -54,7 +54,16 @@ # Match an ATX heading: 1-6 leading hashes + space + body. Setext # headings are out of scope for the anchor target check (the project # uses ATX exclusively per docs/standards/documentation.md). -_HEADING_RE = re.compile(r"^(#{1,6}) +(.+?)\s*#*\s*$") +# +# Backtracking analysis (Sonar python:S5852, ReDoS hotspot): +# the body is captured greedily and trailing whitespace + closing +# hashes are stripped procedurally in ``_normalise_heading_body`` +# below. This avoids the prior ``(.+?)\s*#*\s*$`` form whose three +# adjacent variable-length matchers are flagged as a polynomial +# backtracking risk on adversarial input. ATX headings do not +# legitimately exceed a few hundred characters so the worst-case +# linear scan is bounded by line length. +_HEADING_RE = re.compile(r"^(#{1,6}) +(.+)$") # Markdown image syntax (``![alt](src)``) reuses the link form but is # checked the same way — every ``src`` must resolve. @@ -83,6 +92,20 @@ class BrokenLink: reason: str +def _normalise_heading_body(body: str) -> str: + """Strip trailing whitespace + GFM closing-hash run from a heading body. + + ATX headings allow an optional sequence of trailing ``#`` characters + (e.g. ``# title #``); GFM treats the closing run as decoration and + the slug is derived from the body alone. Done procedurally to + avoid the multi-``\\s*`` regex pattern that triggers ReDoS warnings. + """ + trimmed = body.rstrip() + while trimmed.endswith("#"): + trimmed = trimmed[:-1].rstrip() + return trimmed + + def _slugify_heading(text: str) -> str: """GitHub-flavoured Markdown heading-anchor algorithm. @@ -115,7 +138,8 @@ def _extract_anchors(target: Path) -> set[str]: match = _HEADING_RE.match(line) if match is None: continue - anchors.add(_slugify_heading(match.group(2))) + body = _normalise_heading_body(match.group(2)) + anchors.add(_slugify_heading(body)) return anchors @@ -171,47 +195,61 @@ def _split_path_anchor(href: str) -> tuple[str, str]: return path, anchor +def _resolve_pure_anchor(link: Link, anchor_part: str) -> BrokenLink | None: + """Pure ``#section`` link — target is the same file.""" + if not anchor_part: + return BrokenLink(link, "empty href") + anchors = _extract_anchors(link.source) + if anchor_part not in anchors: + return BrokenLink(link, f"anchor #{anchor_part!r} not found in {link.source.name}") + return None + + +def _locate_target(link: Link, path_part: str, repo_root: Path) -> Path | BrokenLink: + """Resolve a repo-relative path against the source's parent or repo root.""" + target = (link.source.parent / path_part).resolve() + if target.exists(): + return target + # Fall back: maybe the path is repo-root-relative (legacy refs). + alt = (repo_root / path_part).resolve() + if alt.exists(): + return alt + return BrokenLink(link, f"target file not found: {path_part!r}") + + +def _resolve_anchor_against_target(link: Link, target: Path, path_part: str, anchor_part: str) -> BrokenLink | None: + """Validate ``anchor_part`` against the resolved target file.""" + if target.suffix.lower() != ".md": + # Code-file anchors (e.g. forgelm/x.py#L42) are stale-line + # references that drift on refactor. Fail-loud for the + # ``#L`` form, ignore other code-anchor forms (Sphinx + # ``?`` queries, language-specific fragment schemes). + if re.fullmatch(r"L\d+(?:-L\d+)?", anchor_part): + return BrokenLink( + link, + f"line-number anchor #{anchor_part!r} on code file is brittle (use symbol references)", + ) + return None + anchors = _extract_anchors(target) + if anchor_part not in anchors: + return BrokenLink(link, f"anchor #{anchor_part!r} not found in {path_part!r}") + return None + + def _resolve_link(link: Link, repo_root: Path) -> BrokenLink | None: """Return ``None`` if the link resolves; otherwise a ``BrokenLink``.""" href = link.href if _is_skipped(href): return None path_part, anchor_part = _split_path_anchor(href) - # Pure anchor (``#section``) — target is the same file. if not path_part: - if not anchor_part: - return BrokenLink(link, "empty href") - anchors = _extract_anchors(link.source) - if anchor_part not in anchors: - return BrokenLink(link, f"anchor #{anchor_part!r} not found in {link.source.name}") + return _resolve_pure_anchor(link, anchor_part) + target_or_broken = _locate_target(link, path_part, repo_root) + if isinstance(target_or_broken, BrokenLink): + return target_or_broken + if not anchor_part: return None - - # Repo-relative path. Resolve against the source file's parent. - target = (link.source.parent / path_part).resolve() - if not target.exists(): - # Fall back: maybe the path is repo-root-relative (legacy - # references). Try that before declaring broken. - alt = (repo_root / path_part).resolve() - if not alt.exists(): - return BrokenLink(link, f"target file not found: {path_part!r}") - target = alt - - if anchor_part: - # Anchors only meaningful for Markdown targets. - if target.suffix.lower() != ".md": - # Code-file anchors (e.g. forgelm/x.py#L42) are stale-line - # references — flag them since they drift on refactor. - # Fail-loud for ``#L`` form, ignore other code-anchor - # forms (e.g. Sphinx `?` queries). - if re.fullmatch(r"L\d+(?:-L\d+)?", anchor_part): - return BrokenLink( - link, f"line-number anchor #{anchor_part!r} on code file is brittle (use symbol references)" - ) - return None - anchors = _extract_anchors(target) - if anchor_part not in anchors: - return BrokenLink(link, f"anchor #{anchor_part!r} not found in {path_part!r}") - return None + return _resolve_anchor_against_target(link, target_or_broken, path_part, anchor_part) def _format_broken(broken: BrokenLink) -> str: @@ -219,7 +257,7 @@ def _format_broken(broken: BrokenLink) -> str: return f"{rel}:{broken.link.line} [{broken.link.text}]({broken.link.href}) → {broken.reason}" -def main(argv: list[str] | None = None) -> int: +def _build_argparser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description="Validate markdown anchor + relative-path links under docs/.", ) @@ -263,39 +301,56 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="Suppress the OK summary on success.", ) - args = parser.parse_args(argv) + return parser - repo_root = args.repo_root.resolve() - scope_dir = (repo_root / args.scope).resolve() - if not scope_dir.is_dir(): - print(f"error: scope directory not found: {scope_dir}", file=sys.stderr) - return 1 +def _resolve_excludes(scope_dir: Path, exclude_arg: list[str] | None) -> tuple[Path, ...]: # Default exclude list — `analysis/` is gitignored research with # local-only path references; `code_reviews/` would be too were # it not partially re-allowed in `.gitignore`, but its closure-plan # docs cite real files at line-anchored locations that drift. - if args.exclude is None: - exclude_specs = ("analysis",) + specs: tuple[str, ...] + if exclude_arg is None: + specs = ("analysis",) else: - exclude_specs = tuple(args.exclude) - excluded_dirs = tuple((scope_dir / spec).resolve() for spec in exclude_specs) + specs = tuple(exclude_arg) + return tuple((scope_dir / spec).resolve() for spec in specs) + +def _collect_broken(md_files: list[Path], repo_root: Path) -> list[BrokenLink]: broken: list[BrokenLink] = [] - md_files = list(_walk_markdown_files(scope_dir, excluded_dirs)) for md in md_files: for link in _extract_links(md): failure = _resolve_link(link, repo_root) if failure is not None: broken.append(failure) + return broken + + +def _report_broken(broken: list[BrokenLink], md_count: int, scope: str, strict: bool) -> int: + verdict = "FAIL" if strict else "WARN" + print(f"{verdict}: broken anchor / relative-link references:") + for entry in broken: + print(f" {_format_broken(entry)}") + print(f"\n{len(broken)} broken link(s) across {md_count} markdown file(s) under {scope}/.") + return 1 if strict else 0 + + +def main(argv: list[str] | None = None) -> int: + args = _build_argparser().parse_args(argv) + + repo_root = args.repo_root.resolve() + scope_dir = (repo_root / args.scope).resolve() + if not scope_dir.is_dir(): + print(f"error: scope directory not found: {scope_dir}", file=sys.stderr) + return 1 + + excluded_dirs = _resolve_excludes(scope_dir, args.exclude) + md_files = list(_walk_markdown_files(scope_dir, excluded_dirs)) + broken = _collect_broken(md_files, repo_root) if broken: - verdict = "FAIL" if args.strict else "WARN" - print(f"{verdict}: broken anchor / relative-link references:") - for entry in broken: - print(f" {_format_broken(entry)}") - print(f"\n{len(broken)} broken link(s) across {len(md_files)} markdown file(s) under {args.scope}/.") - return 1 if args.strict else 0 + return _report_broken(broken, len(md_files), args.scope, args.strict) if not args.quiet: print(f"OK: {len(md_files)} markdown file(s) under {args.scope}/ have all anchors + relative links resolved.") diff --git a/tools/check_bandit.py b/tools/check_bandit.py index 6a884ecc..1f41eb56 100644 --- a/tools/check_bandit.py +++ b/tools/check_bandit.py @@ -38,6 +38,7 @@ # can read it from the artefact. _HIGH = "HIGH" _MED = "MEDIUM" +_UNDEFINED = "UNDEFINED" def _format_issue(issue: dict[str, Any]) -> str: @@ -45,51 +46,89 @@ def _format_issue(issue: dict[str, Any]) -> str: test_name = issue.get("test_name") or "" filename = issue.get("filename") or "" line = issue.get("line_number") or "?" - severity = (issue.get("issue_severity") or "UNDEFINED").upper() - confidence = (issue.get("issue_confidence") or "UNDEFINED").upper() + severity = (issue.get("issue_severity") or _UNDEFINED).upper() + confidence = (issue.get("issue_confidence") or _UNDEFINED).upper() text = (issue.get("issue_text") or "").splitlines()[0] if issue.get("issue_text") else "" return f"[{severity}/{confidence}] {filename}:{line} {test_id} {test_name} — {text}" -def main(argv: list[str]) -> int: - if len(argv) != 2: - print(f"usage: {argv[0]} ", file=sys.stderr) - return 1 - - report_path = Path(argv[1]) +def _load_report(report_path: Path) -> dict[str, Any] | int: + """Read + parse the bandit JSON report; return the dict or an exit code.""" try: raw = report_path.read_text(encoding="utf-8") except OSError as exc: print(f"::error::bandit report not readable at {report_path}: {exc}", file=sys.stderr) return 1 - try: - report = json.loads(raw) + return json.loads(raw) except json.JSONDecodeError as exc: print(f"::error::bandit report at {report_path} is not valid JSON: {exc}", file=sys.stderr) return 1 - results = report.get("results") or [] + +def _extract_results(report: dict[str, Any]) -> list[Any] | int: + """Validate the ``results`` field shape; return list or an exit code. + + ``null``-valued ``results`` is rejected explicitly (a missing-key + report is not the same as an empty-results report — ``or []`` + would conflate them and let a malformed bandit run pass silently). + """ + if "results" not in report: + print("::error::bandit report missing 'results' field", file=sys.stderr) + return 1 + results = report["results"] + if results is None: + print("::error::bandit report 'results' is null (malformed)", file=sys.stderr) + return 1 if not isinstance(results, list): print("::error::bandit report 'results' field is not a list", file=sys.stderr) return 1 + return results + +def _classify_issues(results: list[Any]) -> tuple[list[str], list[str], int]: + """Bucket issues into (high-lines, medium-lines, undefined-count).""" high: list[str] = [] medium: list[str] = [] + undefined_count = 0 for issue in results: if not isinstance(issue, dict): continue - severity = (issue.get("issue_severity") or "UNDEFINED").upper() + severity = (issue.get("issue_severity") or _UNDEFINED).upper() line = _format_issue(issue) if severity == _HIGH: high.append(line) elif severity == _MED: medium.append(line) - # LOW / UNDEFINED are silent; the raw JSON remains in artefacts. + elif severity == _UNDEFINED: + undefined_count += 1 + # LOW is silent; the raw JSON remains in artefacts. + return high, medium, undefined_count + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print(f"usage: {argv[0]} ", file=sys.stderr) + return 1 + + report_or_code = _load_report(Path(argv[1])) + if isinstance(report_or_code, int): + return report_or_code + results_or_code = _extract_results(report_or_code) + if isinstance(results_or_code, int): + return results_or_code + + high, medium, undefined_count = _classify_issues(results_or_code) for line in medium: print(f"::warning::bandit {line}") + if undefined_count: + # One summary annotation rather than per-finding spam; UNDEFINED + # rules are bandit's "rule lacks severity metadata" fall-through + # and merit operator review without burying real signal. + print(f"::warning::bandit {undefined_count} issue(s) with UNDEFINED severity; review the raw report manually.") + if high: for line in high: print(f"::error::bandit {line}") diff --git a/tools/check_bilingual_parity.py b/tools/check_bilingual_parity.py index 4aed75f2..71199083 100644 --- a/tools/check_bilingual_parity.py +++ b/tools/check_bilingual_parity.py @@ -67,6 +67,17 @@ ("docs/guides/gdpr_erasure.md", "docs/guides/gdpr_erasure-tr.md"), ("docs/guides/ingestion.md", "docs/guides/ingestion-tr.md"), ("docs/guides/iso_soc2_deployer_guide.md", "docs/guides/iso_soc2_deployer_guide-tr.md"), + # docs/qms/ + ("docs/qms/README.md", "docs/qms/README-tr.md"), + ("docs/qms/access_control.md", "docs/qms/access_control-tr.md"), + ("docs/qms/encryption_at_rest.md", "docs/qms/encryption_at_rest-tr.md"), + ("docs/qms/risk_treatment_plan.md", "docs/qms/risk_treatment_plan-tr.md"), + ("docs/qms/roles_responsibilities.md", "docs/qms/roles_responsibilities-tr.md"), + ("docs/qms/sop_change_management.md", "docs/qms/sop_change_management-tr.md"), + ("docs/qms/sop_data_management.md", "docs/qms/sop_data_management-tr.md"), + ("docs/qms/sop_incident_response.md", "docs/qms/sop_incident_response-tr.md"), + ("docs/qms/sop_model_training.md", "docs/qms/sop_model_training-tr.md"), + ("docs/qms/statement_of_applicability.md", "docs/qms/statement_of_applicability-tr.md"), # docs/reference/ ("docs/reference/architecture.md", "docs/reference/architecture-tr.md"), ("docs/reference/audit_event_catalog.md", "docs/reference/audit_event_catalog-tr.md"), diff --git a/tools/check_pip_audit.py b/tools/check_pip_audit.py index f0539442..495dd797 100644 --- a/tools/check_pip_audit.py +++ b/tools/check_pip_audit.py @@ -39,6 +39,12 @@ _HIGH_TIERS: frozenset[str] = frozenset({"HIGH", "CRITICAL"}) _MED_TIERS: frozenset[str] = frozenset({"MEDIUM", "MODERATE"}) +# OSV severity-list ``type`` values are scoring-system labels, NOT tier +# names — recognising them lets us avoid mistaking the label for a +# severity (otherwise the original CVSS_V3 string would resolve to +# UNKNOWN by chance, but a renamed system would silently mis-tier). +_SCORE_TYPE_LABELS: frozenset[str] = frozenset({"CVSS", "CVSS_V2", "CVSS_V3", "CVSS_V4", "CVSS_V31", "CVSS_V40"}) + def _normalise_severity(raw: Optional[str]) -> str: """Upper-case + collapse synonyms; unknown/missing → ``UNKNOWN``.""" @@ -47,30 +53,93 @@ def _normalise_severity(raw: Optional[str]) -> str: upper = raw.upper().strip() if upper in {"MODERATE"}: return "MEDIUM" + if upper in _SCORE_TYPE_LABELS: + # CVSS_V3 etc. is a scoring-system label, not a tier name. + return "UNKNOWN" return upper +def _tier_from_cvss_score(score: Any) -> str: + """Map a CVSS base score to a severity tier per FIRST.org guidance. + + CVSS v3 / v4 cut-points: 0.0 NONE · 0.1–3.9 LOW · 4.0–6.9 MEDIUM · + 7.0–8.9 HIGH · 9.0–10.0 CRITICAL. Falls back to ``UNKNOWN`` on + any parse failure rather than guessing. + """ + if isinstance(score, (int, float)): + value = float(score) + elif isinstance(score, str): + try: + value = float(score.strip()) + except ValueError: + return "UNKNOWN" + else: + return "UNKNOWN" + if value >= 9.0: + return "CRITICAL" + if value >= 7.0: + return "HIGH" + if value >= 4.0: + return "MEDIUM" + if value > 0.0: + return "LOW" + return "UNKNOWN" + + +def _severity_from_entry(entry: Any) -> str: + """Extract a severity tier from a single ``severity[]`` element.""" + if isinstance(entry, dict): + # Only the ``severity`` field carries a tier label; ``type`` is + # the scoring-system identifier (CVSS_V3, ...). Try the tier + # first, then derive from the CVSS score when only the type + + # score pair is available. + explicit = _normalise_severity(entry.get("severity")) + if explicit != "UNKNOWN": + return explicit + return _tier_from_cvss_score(entry.get("score")) + if isinstance(entry, str): + return _normalise_severity(entry) + return "UNKNOWN" + + def _vuln_severity(vuln: dict[str, Any]) -> str: """Extract a single severity tier from a pip-audit vuln entry. Falls back through several fields because pip-audit's JSON shape has shifted across point releases: - 2.7.x: top-level ``severity`` string + - 2.7.x list form: top-level ``severity[]`` of ``{type, score}`` / + ``{severity}`` / plain string entries - 2.6.x and earlier: nested under ``aliases[].severity[]`` - - GHSA imports often carry only ``severity_score`` (CVSS) When no field is parseable we return ``"UNKNOWN"`` and let the - operator review the raw report manually. + caller surface a single summary annotation so the operator can + review the raw report. """ direct = vuln.get("severity") if isinstance(direct, str): return _normalise_severity(direct) - if isinstance(direct, list) and direct: - first = direct[0] - if isinstance(first, dict): - return _normalise_severity(first.get("type") or first.get("severity")) - if isinstance(first, str): - return _normalise_severity(first) + if isinstance(direct, list): + for entry in direct: + tier = _severity_from_entry(entry) + if tier != "UNKNOWN": + return tier + # 2.6.x fallback — severity buried inside aliases[].severity[]. + aliases = vuln.get("aliases") + if isinstance(aliases, list): + for alias in aliases: + if not isinstance(alias, dict): + continue + nested = alias.get("severity") + if isinstance(nested, list): + for entry in nested: + tier = _severity_from_entry(entry) + if tier != "UNKNOWN": + return tier + elif isinstance(nested, str): + tier = _normalise_severity(nested) + if tier != "UNKNOWN": + return tier return "UNKNOWN" @@ -120,6 +189,7 @@ def main(argv: list[str]) -> int: high: list[str] = [] medium: list[str] = [] + unknown: list[str] = [] for name, vuln in _iter_findings(report): severity = _vuln_severity(vuln) line = _format_finding(name, vuln, severity) @@ -127,13 +197,24 @@ def main(argv: list[str]) -> int: high.append(line) elif severity in _MED_TIERS: medium.append(line) - # LOW + UNKNOWN are silent; the raw JSON remains in artefacts. + elif severity == "UNKNOWN": + unknown.append(line) + # LOW is silent; the raw JSON remains in artefacts. for line in medium: # GitHub Actions annotation; surfaces in the run summary without # failing the build. print(f"::warning::pip-audit {line}") + if unknown: + # One summary annotation rather than per-finding spam — UNKNOWN + # findings need operator review, not noise that buries real + # signal (F-W4-PS-07 / F-W4-TR-05 absorption). + print( + f"::warning::pip-audit {len(unknown)} finding(s) without parseable " + "severity; review the raw report manually." + ) + if high: for line in high: print(f"::error::pip-audit {line}") From 105285ffb599983c0fc4ba6172ebd302cadfe13d Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 01:22:44 +0300 Subject: [PATCH 08/12] fix(wave-4-round1-followup): drop _HEADING_RE to clear Sonar python:S5852 hotspot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first absorption round (ffceaf7) replaced ``(.+?)\s*#*\s*$`` with ``^(#{1,6}) +(.+)$`` to remove three adjacent variable-length matchers, but Sonar's python:S5852 heuristic still flagged the new pattern: two sequential ``+`` quantifiers (`` +`` then ``(.+)``) trip the rule even though the leading ``{1,6}`` bound makes catastrophic backtracking impossible. Cleanest fix: drop the regex entirely. ATX heading parsing is now a single linear-pass procedural function (``_parse_atx_heading``) — count leading hashes (≤6), require a space, take the rest as body, hand off to the existing ``_normalise_heading_body`` for closing-hash trim. Worst-case runtime is linear in line length and there is no regex surface left for the heuristic to misclassify. Behaviour-equivalent: the 29 anchor-checker tests pass unchanged and the advisory baseline stays at 36 broken links. Co-Authored-By: Claude Opus 4.7 (1M context) --- tools/check_anchor_resolution.py | 55 ++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 17 deletions(-) diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py index 074b30cf..7d1910cc 100644 --- a/tools/check_anchor_resolution.py +++ b/tools/check_anchor_resolution.py @@ -51,19 +51,13 @@ # character classes only — no nested quantifiers. _LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]*)\)") -# Match an ATX heading: 1-6 leading hashes + space + body. Setext -# headings are out of scope for the anchor target check (the project -# uses ATX exclusively per docs/standards/documentation.md). -# -# Backtracking analysis (Sonar python:S5852, ReDoS hotspot): -# the body is captured greedily and trailing whitespace + closing -# hashes are stripped procedurally in ``_normalise_heading_body`` -# below. This avoids the prior ``(.+?)\s*#*\s*$`` form whose three -# adjacent variable-length matchers are flagged as a polynomial -# backtracking risk on adversarial input. ATX headings do not -# legitimately exceed a few hundred characters so the worst-case -# linear scan is bounded by line length. -_HEADING_RE = re.compile(r"^(#{1,6}) +(.+)$") +# ATX heading parsing is fully procedural — see ``_parse_atx_heading`` +# below. We do NOT use a regex here because Sonar's python:S5852 +# heuristic flags any pattern with two adjacent variable-length +# quantifiers (e.g. ``^(#{1,6}) +(.+)$``) as a polynomial-runtime +# ReDoS risk, even when the leading quantifier is bounded. The +# procedural form has worst-case linear runtime in line length and +# leaves no regex surface for the heuristic to misclassify. # Markdown image syntax (``![alt](src)``) reuses the link form but is # checked the same way — every ``src`` must resolve. @@ -92,13 +86,41 @@ class BrokenLink: reason: str +def _parse_atx_heading(line: str) -> str | None: + """Return the heading body for an ATX heading line, or None. + + An ATX heading is 1-6 leading ``#`` characters followed by at + least one whitespace character and then the body. GFM also + permits an optional trailing run of ``#`` decorators which we + strip after the leading-hashes scan. + + Implemented as a single linear pass to avoid the ReDoS risk + heuristic that flags adjacent variable-length quantifiers in + regex form (Sonar python:S5852). + """ + if not line or line[0] != "#": + return None + hash_count = 0 + while hash_count < len(line) and line[hash_count] == "#": + hash_count += 1 + if hash_count == 0 or hash_count > 6: + return None + if hash_count >= len(line) or line[hash_count] != " ": + return None + body = line[hash_count + 1 :].strip() + if not body: + return None + return _normalise_heading_body(body) + + def _normalise_heading_body(body: str) -> str: """Strip trailing whitespace + GFM closing-hash run from a heading body. ATX headings allow an optional sequence of trailing ``#`` characters (e.g. ``# title #``); GFM treats the closing run as decoration and the slug is derived from the body alone. Done procedurally to - avoid the multi-``\\s*`` regex pattern that triggers ReDoS warnings. + avoid the multi-``\\s*`` regex pattern that would trigger Sonar's + python:S5852 ReDoS heuristic. """ trimmed = body.rstrip() while trimmed.endswith("#"): @@ -135,10 +157,9 @@ def _extract_anchors(target: Path) -> set[str]: except OSError: return set() for line in text.splitlines(): - match = _HEADING_RE.match(line) - if match is None: + body = _parse_atx_heading(line) + if body is None: continue - body = _normalise_heading_body(match.group(2)) anchors.add(_slugify_heading(body)) return anchors From 27f4fe742536b7d357257ffab918ceee8ef146fe Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 01:36:55 +0300 Subject: [PATCH 09/12] docs(wave-4-followup): multi-agent review prompt for absorption commits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to wave4-review-prompt.md. Targets the two absorption commits ffceaf7 + 105285f that landed on PR #33 after the original 4-agent review completed. The prompt asks two complementary questions: 1. Did the absorptions correctly fix the original findings (every CRITICAL / HIGH from the 4 prior reports + the 2 inline PR comments + 4 SonarCloud items)? 2. Did the absorptions themselves introduce defects? 8 scrutiny surfaces, ~50 enumerated F-W4FU checks: - §A: §3 tally arithmetic (per-theme recount, FAQ Q1). - §B: HMAC chain key reframing — `SHA-256(secret ‖ run_id)` vs the pre-absorption XOR-with-salt claim, propagated to access_control / RTP / ISO A.8.24 / design doc. - §C: CLI cookbook fixes (verify-audit positional log_path; jq --slurpfile pipeline correctness end-to-end). - §D: check_pip_audit CVSS_V3 type label + score-to-tier derivation (CVSSv2 cut-point caveat) + 2.6.x aliases nested fallback + UNKNOWN summary annotation. - §E: check_bandit null-results / missing-key disambiguation + UNDEFINED summary annotation. - §F: check_anchor_resolution _HEADING_RE elimination — procedural ATX parser corner cases (CommonMark §4.2 conformance, indented headings, closing-hash decoration) + helper-extraction behaviour invariance + Sonar S5852 hotspot resolution verification. - §G: vacuous-test scan of the 17 new tests. - §H: cross-cutting (doc drift, end-to-end verify-audit smoke, closure-plan honesty, ruff-format equivalence, backwards compatibility, gemini-code-assist inline reply audit). .gitignore re-allows the new file alongside the prior planning artefacts. Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + .../wave4-followup-review-prompt.md | 831 ++++++++++++++++++ 2 files changed, 832 insertions(+) create mode 100644 docs/analysis/code_reviews/wave4-followup-review-prompt.md diff --git a/.gitignore b/.gitignore index cee4d854..391381cc 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,7 @@ docs/analysis/code_reviews/* !docs/analysis/code_reviews/wave3-followup-review-prompt.md !docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md !docs/analysis/code_reviews/wave4-review-prompt.md +!docs/analysis/code_reviews/wave4-followup-review-prompt.md # Generated by tools/build_usermanuals.py — produced from # docs/usermanuals//**/*.md during the CI deploy. Not committed; diff --git a/docs/analysis/code_reviews/wave4-followup-review-prompt.md b/docs/analysis/code_reviews/wave4-followup-review-prompt.md new file mode 100644 index 00000000..30ad868b --- /dev/null +++ b/docs/analysis/code_reviews/wave4-followup-review-prompt.md @@ -0,0 +1,831 @@ +# Wave 4 Follow-up — Multi-Agent Code Review Prompt (post-absorption) + +> **Use this prompt verbatim** to launch the multi-agent review on the +> 2 absorption commits that landed on PR #33 after the original +> 4-agent review completed. The prompt is self-contained: each +> reviewing agent can pick it up cold without context from the prior +> round. +> +> The original Wave 4 review (per +> [`wave4-review-prompt.md`](wave4-review-prompt.md), HEAD `9a1738a`) +> produced 4 reports under `docs/analysis/code_reviews/wave4-*.md`. +> The maintainer absorbed those reports plus inline PR-comment +> findings (gemini-code-assist) plus SonarCloud quality gate failures +> across **2 commits**. This second-pass review asks: *did the +> absorption correctly address the original findings, and did the +> absorption itself introduce any new defects?* + +--- + +## Repo & branch under review + +- **Repo:** `cemililik/ForgeLM` +- **PR:** [#33](https://github.com/cemililik/ForgeLM/pull/33) + (`closure/wave4-integration` → `development`) +- **HEAD SHA at prompt freeze:** `105285f` +- **Pre-absorption SHA (the snapshot the original 4-agent review + read):** `9a1738a` (working tree was at `8f69cf1` for §B/§G focus + agents; review prompt itself was `9a1738a`) +- **Absorption-only diff range:** `git diff 9a1738a..105285f` + — 30 files changed, ~783 insertions, ~213 deletions. +- **2 absorption commits in scope** (oldest → newest): + + | SHA | Title | Findings absorbed | + |-----------|--------------------------------------------------------------------------------|-------------------| + | `ffceaf7` | absorb 4-agent + PR #33 + Sonar review findings | F-W4-01..13, F-W4-PS-01..08, F-W4-NIT-01..02, F-W4-TR-01..10 + 2 inline (gemini-code-assist) + 3 Sonar code-smells (python:S3776 ×3) + 1 Sonar hotspot (python:S5852, partial) | + | `105285f` | drop `_HEADING_RE` to clear Sonar python:S5852 hotspot | python:S5852 (final — regex eliminated entirely) | + +## What ForgeLM is (60 seconds) + +A **config-driven, enterprise-grade LLM fine-tuning toolkit** — YAML +in, fine-tuned model + compliance artifacts out. Built for CI/CD +pipelines, not notebooks. Six alignment paradigms +(SFT/DPO/SimPO/KTO/ORPO/GRPO), integrated safety evaluation +(Llama Guard), EU AI Act compliance artefacts (Articles 9–17 + Annex +IV), append-only audit log, opt-in human approval gate, auto-revert +on quality regression, GDPR Article 15 + 17 subject rights tooling, +ISO 27001 / SOC 2 alignment evidence (Wave 4 / Faz 22 + 23). +Read [docs/product_strategy.md](../../product_strategy.md) for the +fuller background. + +Project rulebook lives under [docs/standards/](../../standards/) — read +[coding.md](../../standards/coding.md), +[error-handling.md](../../standards/error-handling.md), +[testing.md](../../standards/testing.md), +[logging-observability.md](../../standards/logging-observability.md), +[regex.md](../../standards/regex.md), +[localization.md](../../standards/localization.md), +[documentation.md](../../standards/documentation.md), and +[code-review.md](../../standards/code-review.md) before commenting. +Project-wide guidance for AI agents is in the root `CLAUDE.md`. + +The original 4 review reports under +[`docs/analysis/code_reviews/`](.) document the prior bar: + +- [`wave4-code-correctness.md`](wave4-code-correctness.md) — F-W4-01..F-W4-13. +- [`wave4-privacy-security.md`](wave4-privacy-security.md) — F-W4-PS-01..F-W4-PS-08. +- [`wave4-standards.md`](wave4-standards.md) — F-W4-01..F-W4-13 (re-numbered locally). +- [`wave4-test-rigor.md`](wave4-test-rigor.md) — F-W4-TR-01..F-W4-TR-10. + +The maintainer's absorption summary is at +[PR #33 the absorption commit message of `ffceaf7`](https://github.com/cemililik/ForgeLM/commit/ffceaf7). + +## What this round is for + +**Goal:** decide whether `closure/wave4-integration` at `105285f` is +mergeable to `development`, given the absorptions claim to have +addressed the original 4-agent review. + +**Question shape:** the previous review framed defects in code shipped +*before* absorption. This review asks two complementary questions: + +1. **Did the absorptions correctly fix the original findings?** + Walk every CRITICAL / HIGH from the 4 prior reports. For each, + verify the claimed fix lands at the cited file:line and actually + addresses the original concern (not "a fix that compiles" but + "the contract the original finding asked about now holds"). +2. **Did the absorptions themselves introduce defects?** The diff + touches 30 files; the largest absorption (`ffceaf7`) reframed + substantial portions of `docs/qms/` (HMAC misdescription, rotation + cadence) + rewrote the pip-audit / bandit / anchor-resolution + tooling logic + grew `tests/test_supply_chain_security.py` and + `tests/test_check_anchor_resolution.py`. The followup commit + (`105285f`) eliminated the `_HEADING_RE` regex entirely in favour + of a procedural ATX-heading parser. Treat both as new code — + same severity bar as the original review. + +Be concrete. Every finding must include: + +1. **Severity tag** — `CRITICAL` / `HIGH` / `MEDIUM` / `LOW` / `NIT`. +2. **Finding ID** — `F-W4FU-NN` (e.g. `F-W4FU-01`, `F-W4FU-12`). +3. **File:line citation** — `tools/check_pip_audit.py:62` + or a short range `:62-86`. +4. **One-paragraph reasoning** — what is wrong, what is the user-visible + consequence, and why it qualifies for that severity tier. +5. **Suggested fix** — a code or text snippet, not a vague direction. +6. **Test that would have caught it** — name the test file + a stub + of the assertion. +7. **Trace-back to original finding (if any)** — when re-evaluating + a claimed absorption, cite the original finding ID + (e.g. F-W4-01, F-W4-PS-01, F-W4-TR-01) so the maintainer + can compare against the pre-absorption code. + +Severity bar (unchanged from the original prompt — same calibration): + +- `CRITICAL` — would corrupt data, silently lose audit events, leak + PII / secrets, break the public exit-code contract (`0/1/2/3/4`), + break the `--offline` / air-gap promise, or invalidate the + audit-log hash chain. Blocks merge. +- `HIGH` — runtime crash on a documented happy path, missing test + for a documented contract, schema mismatch with `pyproject.toml` + / `config.py`, documentation claim with no code backing, or a + behaviour change that breaks 0.5.x users without a clear migration + path. Blocks merge. +- `MEDIUM` — logic bug in an off-the-happy-path branch, observability + gap, or design-doc statement that contradicts shipped code. + Should be fixed before merge but the call is the maintainer's. +- `LOW` — defensive-coding improvement, minor inconsistency, missing + edge-case test that would not have changed PR direction. +- `NIT` — naming, line wrap, comment phrasing. Note in passing + only; do not stack these. + +## Areas to scrutinise + +These are the surfaces most likely to harbour real issues *post-absorption*. + +### A. Coverage tally arithmetic (claims to absorb F-W4-01 / F-W4-02) + +The single most-cited claim across the absorption is the recount +`FL 11 / FL-helps 48 / OOS 34` (replacing the original `11 / 50 / +32`). Per-theme split: A.5: 3 / 24 / 10 · A.6: 0 / 5 / 3 · +A.7: 0 / 0 / 14 · A.8: 8 / 19 / 7. The recount is propagated to: + +- `iso27001-soc2-alignment-202605052315.md:264-269` (design doc §3 + closing paragraph). +- `iso27001-soc2-alignment-202605052315.md:790-795` (FAQ Q1, "59 + of 93"). +- `statement_of_applicability.md:148-152` + `-tr.md` (per-theme + table + total row). +- `iso27001_control_mapping.md:16` + `-tr.md` (header tally). +- `CHANGELOG.md:20-22` (release-notes line). + +#### A.1 — Per-theme arithmetic verifiability + +- **F-W4FU-XX-A1** — Walk §3.1 (A.5 — 37 rows), §3.2 (A.6 — 8 rows), + §3.3 (A.7 — 14 rows, prose only), §3.4 (A.8 — 34 rows) of the + design doc. For each theme, count `FL` / `FL-helps` / `OOS` + occurrences per row. The numbers must match the published + per-theme split exactly. If any theme is off by even 1, the + closing paragraph and SoA per-theme breakdown are both wrong + again. +- A.5: spot the `FL` rows (must be A.5.28, A.5.33, A.5.34). Confirm + 3 not 4. +- A.6: spot the `FL-helps` rows (must be A.6.3, A.6.4, A.6.5, + A.6.7, A.6.8). Confirm 5 not 4. `OOS` rows must be A.6.1, + A.6.2, A.6.6 (3 not 4). +- A.8: spot the `FL` rows (must be A.8.3, A.8.9, A.8.10, A.8.11, + A.8.12, A.8.15, A.8.24, A.8.32). Confirm 8 not 7. +- A.8.30 was OOS in the design doc and remains OOS — but the + supply_chain_security.md "Related controls" mapping previously + cited it; absorption dropped it. Confirm the design-doc OOS + count still accounts for A.8.30. +- A.8.34: design doc + ISO mapping list it; SoA table is the + contract auditor signs. Cross-check the SoA "Excluded" column + shows A.8.34 as included (the table only excludes A.7 14 rows + per the totals). + +#### A.2 — FAQ Q1 arithmetic + +- **F-W4FU-XX-A2** — `iso27001-soc2-alignment-202605052315.md:790-795` + now says "59 of 93". The arithmetic is `11 + 48 = 59`. Confirm + the surrounding prose ("The remaining 34 controls (mainly A.7 + physical, A.5 organisational governance, A.8 network / cloud)") + reconciles with the per-theme OOS breakdown (A.5 = 10, A.6 = 3, + A.7 = 14, A.8 = 7 → "mainly A.7 physical (14)" is honest; + the qualifier "A.5 governance (10) + A.8 network / cloud (7)" + must add to 17). Total OOS = 14 + 10 + 3 + 7 = 34. ✓ + +### B. HMAC chain key reframing (claims to absorb F-W4-PS-01 / F-W4-08) + +The most consequential narrative change. Before absorption, four +docs claimed the chain key was the env secret XOR'd with the +per-output-dir salt (`_resolve_salt`). The real derivation is +`SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` per +[`forgelm/compliance.py:104-114`](../../../forgelm/compliance.py#L104-L114), +and the per-output-dir salt is purge / reverse-pii's identifier-hash +salt, a distinct concern. The reframing is propagated to: + +- `access_control.md:78-100` + `-tr.md` (§3.4 expanded). +- `risk_treatment_plan.md:75-84` + `-tr.md` (R-04 row). +- `iso27001-soc2-alignment-202605052315.md:252` (A.8.24 row). +- `iso27001_control_mapping.md:122` + `-tr.md` (A.8.24 row). + +#### B.1 — Code citation accuracy + +- **F-W4FU-XX-B1** — Open `forgelm/compliance.py:104-114` and confirm: + - Line 112: `raw_secret = os.getenv("FORGELM_AUDIT_SECRET", "")`. + - Line 114: `self._hmac_key: Optional[bytes] = hashlib.sha256(raw_secret.encode() + self.run_id.encode()).digest()`. + - There is no XOR with `_resolve_salt`'s output anywhere in the + chain-key path. (`_resolve_salt` lives in + `forgelm/cli/subcommands/_purge.py:155-183` and is invoked only + by `forgelm purge` / `forgelm reverse-pii`, not the + `AuditLogger`.) +- The verifier path at `forgelm/compliance.py:1252` uses the same + derivation: `key = hashlib.sha256(hmac_secret.encode() + run_id.encode()).digest()`. + Confirm the absorbed prose's claim of `forgelm/compliance.py:104-114` + cite range covers the writer's derivation; cross-check the + verifier cite at `:1252` is implicit. + +#### B.2 — Per-output-dir-salt distinction + +- **F-W4FU-XX-B2** — The reframed prose calls the per-output-dir + salt "a distinct concern" that "salts purge / reverse-pii + identifier hashes and does NOT participate in chain-key + derivation." Walk: + - Is that *strictly* true at every code path? Search the codebase + for any usage of `_resolve_salt` outside `_purge.py` / + `_reverse_pii.py`. + - The audit event `salt_source` field (Wave 3 follow-up F-W3-PS-07) + — is its meaning unchanged? It records `per_dir` vs `env_var` + for the IDENTIFIER hash, not the chain key. Confirm the + audit-event catalog reflects this. + +#### B.3 — Rotation cadence reframing (claims to absorb F-W4-PS-02) + +- **F-W4FU-XX-B3** — Before absorption, the docs said "quarterly". + After: "between output-dir lifecycles". Walk: + - `access_control.md:78-100` (EN) and `-tr.md` — does the new + procedure correctly explain that rotating mid-output-dir would + break `forgelm verify-audit --require-hmac`? Test the claim by + writing a chain with secret A, rotating to secret B, appending + one entry, and running `verify-audit --require-hmac`: does it + actually fail? (If the verifier only checks line N's HMAC against + the secret it was emitted with, mixed-secret chains might still + verify.) + - `iso_soc2_deployer_guide.md:51-57` checklist (EN+TR) — same + question. + - The §8 audit checklist's bullet "KMS audit log shows + `FORGELM_AUDIT_SECRET` rotation aligned to output-dir lifecycle + boundaries" — is "lifecycle boundary" defined operationally? + A deployer auditor needs to be able to identify a boundary in + their KMS log; what's the signal? +- The CHANGELOG was NOT updated to call out this reframing as a + change in deployer guidance vs the prior `quarterly` advice. + Should it be? + +### C. CLI cookbook fixes (claims to absorb F-W4-01 / F-W4-02 / F-W4-PS-06) + +Four files cited `forgelm verify-audit --output-dir ./outputs --json` +— neither flag exists. The absorption replaced the invocation with +`forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac` +across: + +- `iso_soc2_deployer_guide.md:92-98` + `-tr.md`. +- `access_control.md:178-198` + `-tr.md` (§6 jq pipeline rewritten + with `--slurpfile`). + +#### C.1 — Parser surface confirmation + +- **F-W4FU-XX-C1** — Open `forgelm/cli/_parser.py:484-521` (the + `verify-audit` subparser definition) and confirm: + - Positional `log_path` exists. + - `--require-hmac` is a flag (no value). + - `--hmac-secret-env` exists with a default. + - `--output-dir` is NOT defined. + - `--json` is NOT defined. + Run `python3 -m forgelm.cli verify-audit --help` and confirm the + printed help matches the absorbed prose. + +#### C.2 — jq pipeline correctness (claims to absorb gemini-code-assist +medium) + +- **F-W4FU-XX-C2** — The new jq pipeline: + + ```bash + jq -r 'select(.event == "training.started") | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl > /tmp/trainers.tsv + + jq -r --slurpfile t /tmp/trainers.tsv \ + 'select(.event == "human_approval.granted") | + . as $a | + $t[] | split("\t") as $row | + select($row[0] == $a.run_id and $row[1] == $a.operator) | + [.run_id, .operator] | @tsv' \ + ./outputs/audit_log.jsonl + ``` + + Walk: + - The first pass writes TSV to `/tmp/trainers.tsv`. Tab-separated + `[.run_id, .operator]` — does jq's `@tsv` correctly emit the + raw fields? If `.operator` contains a tab character (rare but + possible for OIDC `oidc:gha:repo:...` strings), the second + pass's `split("\t")` would break. + - The second pass uses `--slurpfile t /tmp/trainers.tsv` — but + `--slurpfile` reads JSON, not TSV. This should fail loudly. + Test the pipeline against a synthetic audit log with one + `training.started` + one matching `human_approval.granted` and + confirm it actually finds the segregation violation. If the + pipeline doesn't run, this is HIGH (ships broken cookbook + again). + - Mirror the test on the TR file. + +#### C.3 — Reference doc consistency (claims to absorb F-W4-PS-06) + +- **F-W4FU-XX-C3** — `access_control.md:122-130` (EN+TR) GitLab CI + block — the absorption swapped from `script: - export + FORGELM_OPERATOR=...` to `variables: FORGELM_OPERATOR: "..."` + + inline comment about the secret manager. Walk: + - Is the GitLab CI `variables:` form actually the right shape? + GitLab supports both `variables:` block and `script: export` + pattern; the new form is cleaner but does not show the + `FORGELM_AUDIT_SECRET` injection. The comment says "inject + from Settings → CI/CD → Variables" — is that operator-actionable + without the YAML cue? + - Compare to the GitHub Actions block (`access_control.md:101-117`) + + Jenkins block (`:131-145`): both inject `FORGELM_AUDIT_SECRET` + in the YAML / Groovy. Symmetric across the three CI platforms? + +### D. `tools/check_pip_audit.py` — severity logic rewrite +(claims to absorb F-W4-04 / F-W4-PS-07 / F-W4-TR-02 / F-W4-TR-05) + +The pre-absorption helper had three documented contracts but only +honoured one. Rewrite added: CVSS-score-to-tier derivation per +FIRST.org cut-points, 2.6.x `aliases[].severity[]` fallback, and +a single-line UNKNOWN summary annotation. + +#### D.1 — Score-type label correctness + +- **F-W4FU-XX-D1** — + [`tools/check_pip_audit.py:46`](../../../tools/check_pip_audit.py#L46) + defines `_SCORE_TYPE_LABELS` containing + `{"CVSS", "CVSS_V2", "CVSS_V3", "CVSS_V4", "CVSS_V31", "CVSS_V40"}`. + Walk: + - Cross-reference against the OSV schema + (https://ossf.github.io/osv-schema/#severity-field) and the + GHSA documentation. Are these the canonical type labels? + Missing any (e.g. `CVSS_V21` for CVSSv2.1)? + - The `CVSS` standalone label (without version suffix) — does + OSV ever emit it? If not, dead defensive code; if yes, what + score format does it carry? + +#### D.2 — CVSS score → tier mapping + +- **F-W4FU-XX-D2** — `_tier_from_cvss_score` at `:62-86` uses + FIRST.org cut-points: 9.0+ CRITICAL, 7.0–8.9 HIGH, 4.0–6.9 + MEDIUM, 0.1–3.9 LOW. Walk: + - The cut-points apply to CVSS v3 / v4 base scores. CVSS v2 + has different cut-points (7.0–10.0 HIGH, 4.0–6.9 MEDIUM, + 0.0–3.9 LOW). The function does not distinguish. Should it? + A CVSSv2 base score 7.0 would be classed HIGH today (correct + by the FIRST.org table for v2 too — same cut-point), but + CVSSv2 has no CRITICAL tier at all (anything ≥7.0 is HIGH). + The current code maps CVSSv2 9.5 to CRITICAL — wrong per + CVSSv2. + - The score input format — `_tier_from_cvss_score` accepts + `int / float / str`. OSV's severity entries typically carry + `score` as either a CVSS vector string (e.g. + `"CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H"`) OR a + plain numeric base score. The string fallback `float(score.strip())` + only handles the latter. Verify: do any pip-audit reports + emit the vector string in `score`? If yes, the function + silently returns UNKNOWN for those. + +#### D.3 — 2.6.x aliases nested fallback + +- **F-W4FU-XX-D3** — `_vuln_severity` at `:127-143` falls through + to `aliases[].severity[]` if the top-level `severity` field is + empty. Walk: + - Is the 2.6.x JSON shape actually + `aliases: [{id: "...", severity: [{...}]}]`? Some OSV exports + nest under `aliases: [{...}]` with severity as a sibling of + each alias dict; others put severity at the vuln level. + Confirm against pip-audit 2.6.x source. + - The nested loop returns the FIRST non-UNKNOWN tier across all + aliases. If two aliases carry conflicting tiers (one HIGH, + one MEDIUM), the result depends on iteration order. Is that + deterministic? + - Could this fallback misfire on a vuln that has both a top-level + `severity` (string) and `aliases[].severity[]` (list)? The + top-level `isinstance(direct, str)` branch returns first, so + the alias path is unreachable. But what if `direct` is an + empty list `[]`? Walk: `isinstance(direct, list)` matches + empty list, the `for` loop yields nothing, and we fall + through to aliases. ✓ + +#### D.4 — UNKNOWN summary annotation + +- **F-W4FU-XX-D4** — The new behaviour emits a single + `::warning::pip-audit N finding(s) without parseable severity` + line. Walk: + - Is "without parseable severity" operator-meaningful? An auditor + walking the nightly run will need to grep the raw JSON + artefact to triage these. Does the artefact path appear in + the warning message? (No — by current code.) Does that + matter? + - Severity policy implication: pre-absorption, UNKNOWN was silent + AND counted toward LOW. New behaviour: UNKNOWN warns AND is + excluded from MEDIUM totals. Walk + `tests/test_supply_chain_security.py::TestCheckPipAuditExtraShapes::test_missing_severity_field_summary_warning`: + does it assert exit 0? (Yes, lines 178-179 of the test.) Is + that the right call given `pip-audit` operators expect + UNKNOWN to remain non-blocking? + +### E. `tools/check_bandit.py` — null-results + UNDEFINED handling +(claims to absorb F-W4-06 / F-W4-TR-01) + +Refactored into 3 helper functions + thin orchestrator. The +`results = report.get("results") or []` bug-line was replaced with +explicit "missing key" / "null value" / "non-list" handling. + +#### E.1 — Null-results disambiguation + +- **F-W4FU-XX-E1** — `_extract_results` at `:39-56`: + - Missing key → `::error::bandit report missing 'results' field` + + exit 1. + - `null` value → `::error::bandit report 'results' is null + (malformed)` + exit 1. + - Non-list → `::error::bandit report 'results' field is not a + list` + exit 1. + - Empty list → returns `[]`, no error. + Walk: + - Does pre-absorption test + `test_results_not_a_list_fails` still pass? (It asserts the + "not a list" string in stderr.) Confirm. + - The new tests `test_results_null_fails` + + `test_results_missing_key_fails` — are the assertion strings + distinct enough to catch a regression that collapsed the + three branches back to one? E.g. if a refactor merged "null" + + "missing" into one error message, the missing-key test would + still pass on the null fixture. + +#### E.2 — UNDEFINED summary annotation + +- **F-W4FU-XX-E2** — `_classify_issues` at `:59-77` separately + counts UNDEFINED severity (rather than letting it fall into + LOW silent). Output: `::warning::bandit N issue(s) with + UNDEFINED severity`. Walk: + - Same questions as D.4 — operator-meaningful? Should the + summary include the test_id list of UNDEFINED issues? The + current implementation drops the per-finding context. + - The test `test_undefined_severity_summary_warning` asserts + `"UNDEFINED" in captured` — is the assertion strong enough + to pin the summary phrasing? A regression that printed the + UNDEFINED finding line directly (with `[UNDEFINED/LOW]` + prefix) would also pass this assertion. + +### F. `tools/check_anchor_resolution.py` — regex-free ATX parser +(claims to absorb python:S5852 hotspot — both rounds) + +The pre-absorption regex `^(.+?)\s*#*\s*$` was replaced in `ffceaf7` +with `^(#{1,6}) +(.+)$` (still flagged by Sonar). The followup +commit `105285f` eliminated the regex entirely in favour of a +procedural `_parse_atx_heading`. Cognitive complexity in +`_resolve_link` + `main` was also reduced via helper extraction. + +#### F.1 — Procedural parser correctness + +- **F-W4FU-XX-F1** — `_parse_atx_heading` at `:95-119` walks each + candidate line: count leading hashes (must be 1-6), require a + space, take rest as body, hand off to `_normalise_heading_body`. + Walk corner cases against CommonMark §4.2: + - `# ` (hash + space + nothing) — body is empty; the helper's + `if not body: return None` rejects. ✓ + - `# heading` — count 1, body "heading". ✓ + - `####### too many hashes` — count 7 (loop continues past 6 to + count all consecutive hashes); helper rejects. ✓ Wait: + line 110 checks `hash_count > 6`, but line 108 increments + while `line[hash_count] == "#"` — for `#######`, `hash_count` + becomes 7. Then `7 > 6` is True, returns None. ✓ + - `#heading` (no space after hash) — `line[hash_count] != " "` + triggers the early return at `:113`. ✓ + - `#\theading` (tab after hash, not space) — early return. ✓ + But CommonMark §4.2 says "an opening sequence of 1–6 unescaped + `#` characters and a space or end-of-line"; tab is NOT + permitted, so reject is correct. Confirm. + - ` ## indented heading` (up to 3 leading spaces allowed by + CommonMark §4.2) — first char is " ", not "#", so + `_parse_atx_heading` returns None. CommonMark would parse this + AS a heading; ForgeLM's parser does not. Is that a real + concern for any project doc? Run a grep to confirm no shipped + doc starts a heading line with leading whitespace. + - `## heading ###` (closing-hash decoration) — body is + `"heading ###"`, then `_normalise_heading_body` strips + trailing `#`s and the surrounding whitespace. Confirm + `_normalise_heading_body("heading ###") == "heading"`. + - `## heading ###extra` (closing hashes followed by more text) — + `_normalise_heading_body` strips `#` greedily then `rstrip` + again. Walk: trimmed starts as `"heading ###extra"`, + `endswith("#")` is False (ends in `"a"`), so the while loop + never enters, and we return `"heading ###extra"`. ✓ The slug + will then strip the punctuation. Is that the GFM-correct + rendering? + +#### F.2 — Helper-extraction behaviour invariance + +- **F-W4FU-XX-F2** — `_resolve_link` was split into + `_resolve_pure_anchor`, `_locate_target`, + `_resolve_anchor_against_target`, plus the orchestrator. Walk: + - `_locate_target` returns `Path | BrokenLink`. The orchestrator + uses `isinstance(target_or_broken, BrokenLink)` to dispatch. + Is the type union safe across all call sites? (If `Path` + were ever subclassed, would `isinstance(... BrokenLink)` + misfire? `BrokenLink` is `@dataclass(frozen=True)`; it's not + a `Path`. ✓) + - `_resolve_anchor_against_target`'s `re.fullmatch(r"L\d+(?:-L\d+)?", anchor_part)` + is the only remaining regex in the resolver. Sonar's S5852 + heuristic doesn't flag it because the pattern is + backtracking-safe. Confirm by reading. + - `main` was split into `_build_argparser`, `_resolve_excludes`, + `_collect_broken`, `_report_broken`. Walk: the four helpers + + the four-line `main` should produce IDENTICAL stdout for + every input the pre-absorption main accepted. Run the existing + test suite — does it cover the strict + advisory + quiet + + exclude flag-combination matrix? + +#### F.3 — Sonar hotspot resolution + +- **F-W4FU-XX-F3** — The followup commit message claims: + "Sonar's next analysis on this PR should drop the hotspot to + zero." Verify: + - Run `curl -s "https://sonarcloud.io/api/hotspots/search?projectKey=cemililik_ForgeLM&pullRequest=33&ps=100"` + after the next CI run. Hotspot count must be 0. + - The remaining `_LINK_RE` (`\[([^\]]*)\]\(([^)]*)\)`) and + `_SKIP_HREF_PATTERNS` are still regex-based; confirm none + of them get newly flagged by S5852. + +### G. New test cases — vacuous-test scan (claims to absorb +F-W4-TR-01..F-W4-TR-08) + +17 new tests were added across `test_supply_chain_security.py` +(8 new) and `test_check_anchor_resolution.py` (9 new). The +test-rigour bar from the original prompt remains: a test that +only asserts schema shape (not behaviour contract) is vacuous. + +#### G.1 — Bandit UNDEFINED test (claims to absorb F-W4-TR-01) + +- **F-W4FU-XX-G1** — + `test_undefined_severity_summary_warning` at + `test_supply_chain_security.py`. Walk: + - The fixture omits `issue_severity`. The assertion is + `"::warning::bandit" in captured` and `"UNDEFINED" in + captured`. Is the assertion strong enough to pin both: + (a) the issue is NOT classified as HIGH/MEDIUM, AND + (b) the summary annotation explicitly mentions UNDEFINED? + - Could a regression that printed the per-finding line + `[UNDEFINED/LOW] forgelm/x.py:1 B999 ...` AS A WARNING (rather + than the summary) pass the test? Yes — both substrings would + appear. Tighten the assertion? + +#### G.2 — Pip-audit UNKNOWN summary test (claims to absorb F-W4-TR-05) + +- **F-W4FU-XX-G2** — + `test_missing_severity_field_summary_warning`. Walk: + - Fixture: vuln with no `severity` field. Assertion: `"::warning::pip-audit"` + + `"without parseable severity"`. The substring + `"without parseable severity"` is unique to the new code path + — any regression that reverted to silent UNKNOWN would fail. + ✓ + +#### G.3 — Pip-audit aliases nested test (claims to absorb F-W4-TR-02) + +- **F-W4FU-XX-G3** — `test_aliases_nested_severity_2_6_shape`. + Fixture: vuln with `aliases: [{id, severity: [{type: "CVSS_V3", + severity: "HIGH"}]}]`. Asserts exit 1. Walk: + - Does the fixture actually exercise the 2.6.x fallback (vs the + top-level `severity` field)? The vuln has no top-level + `severity` key, so `direct = vuln.get("severity")` is None, + `isinstance(None, str)` is False, `isinstance(None, list)` + is False → falls through to aliases. ✓ + - The nested `severity` list carries `severity: "HIGH"` (a + string field, not the list). The new code's + `_severity_from_entry` checks `entry.get("severity")` first, + which returns "HIGH", normalises, returns HIGH. ✓ + - But: does pip-audit 2.6.x actually emit `severity: "HIGH"` + inside the alias dict, or `severity: [{type, score}]`? + Re-confirm against pip-audit 2.6.x JSON output captured + from a real run. If the fixture shape is fictional, + the test pins a contract that doesn't match reality. + +#### G.4 — SBOM serialNumber uniqueness test (claims to absorb F-W4-TR-08) + +- **F-W4FU-XX-G4** — + `TestSbomSerialNumberUniqueness::test_serial_number_changes_between_runs`. + Walk: + - Spawns two `tools/generate_sbom.py` subprocesses and asserts + `sn_a != sn_b`. The SBOM emitter at + `tools/generate_sbom.py:106` uses `f"urn:uuid:{uuid.uuid4()}"`. + Confirm `uuid.uuid4()` IS the source of randomness (not a + seeded UUID5 derived from content); the test would silently + pass if the emitter swapped to UUID5 of a stable content + hash AND the content differed by a millisecond timestamp. + - The 60s subprocess timeout — adequate for a cold-import + + full SBOM generation? Run locally and confirm. + +#### G.5 — Anchor checker tests (claims to absorb F-W4-TR-03/04/06/07) + +- **F-W4FU-XX-G5** — The 6 new anchor tests cover apostrophe + + backtick + ATX-close + tel: + javascript: + image-link skip + patterns + strict-mode broken-anchor. Walk: + - `test_atx_closing_hashes_stripped`: asserts + `_normalise_heading_body("My Title #") == "My Title"` and + `_normalise_heading_body("My Title ### ") == "My Title"`. + Does the CommonMark §4.2 spec say the closing run must be + preceded by whitespace? If yes, the helper's behaviour for + `"foo###"` (no space before closing run) — current code + strips them anyway. Off-spec but defensible if no project + doc has unspaced closing runs. + - `test_strict_mode_exits_one_on_broken_anchor`: planted + `target.md#nonexistent` should fail strict. Confirm the test + fails when run against a regression that special-cased + anchor-not-found differently from file-not-found. + - `test_image_link_missing_target_flagged` + `_present_target_resolves`: + both rely on the regex `_LINK_RE.findall("![alt](image.png)")` + matching the embedded `[alt](image.png)`. Confirm by hand. + +### H. Cross-cutting (XPR) checks + +#### H.1 — Documentation drift (post-absorption) + +- **F-W4FU-XPR-01** — Did every absorption that touched code also + update the matching doc? + - `_DOCTOR_SECRET_ENV_NAMES` comment reword for `FORGELM_RESUME_TOKEN` + → ghost-features-analysis line 604 updated? Yes (per absorption). + Cross-check the comment text in `_doctor.py:662` against the + ghost-features description — must agree. + - Bilingual parity registry growth (4 → 14 new pairs, total + 23) → CHANGELOG line 84 reflects the new tally? (Updated.) + `localization.md:39` row "Yes" matches reality? Run the + parity strict mode and confirm 23/23. + - The closure-plan summary at `:131` says "1374 → 1411 tests + (+37 net)". After this absorption, the actual count grew to + 1414 (17 new tests minus 14 displaced = 3 additional from + the absorption, on top of the 1411 frozen at 9a1738a). Does + the closure plan need a third update, or is the 1411 baseline + still honest at the wave-4-merge boundary? + +#### H.2 — `forgelm verify-audit` cookbook end-to-end test + +- **F-W4FU-XPR-02** — The absorption replaced the broken + `--output-dir / --json` invocation with the shipping syntax. + Run end-to-end: + ```bash + cd /tmp && rm -rf wave4fu-test + mkdir wave4fu-test && cd wave4fu-test + FORGELM_AUDIT_SECRET=test FORGELM_OPERATOR=tester \ + python3 -m forgelm.cli --config /path/to/config_template.yaml \ + --dry-run + # Confirm audit_log.jsonl + audit_log.manifest.json exist. + python3 -m forgelm.cli verify-audit ./outputs/audit_log.jsonl --require-hmac + # Must exit 0. + ``` + Walk: does the deployer-guide cookbook actually produce a + verifiable chain from `--dry-run`? The dry-run skips training but + does emit the run-start audit entry; that single-entry chain + must verify. + +#### H.3 — Closure-plan honesty + +- **F-W4FU-XPR-03** — `closure-plan-202604300906.md:131` (the + Wave 4 closure summary) was updated from "6 yeni QMS TR mirror" + to the more honest "10 yeni QMS TR mirror (4 Faz-23 pair'leri + + 6 mevcut QMS dosyaları için)". Walk: + - Does the count `4 + 6 = 10` match the actual diff? List the + 10 TR mirrors against `git diff --name-only 9a1738a..HEAD -- + docs/qms/*-tr.md`. + - Test count `1374 → 1411 (+37 net)` — does the math work? + The frozen-tree count was 1411 per F-XPR-03 of the original + test-rigour report. (1374 + 37 = 1411 ✓.) But the post-absorption + count is 1414 — does the closure plan need updating again, + or is the absorption's test additions a separate accounting? + +#### H.4 — Lint / format equivalence + +- **F-W4FU-XPR-04** — The absorptions ran `ruff format` between + commits. Spot-check whether any auto-format change altered + *behaviour* vs purely cosmetic. E.g. changes to docstring + indentation, line splitting that affected an f-string's + embedded expression. (`tools/check_pip_audit.py` had its + `_SCORE_TYPE_LABELS` set definition reformatted from 6-line to + 1-line by ruff — confirm that's purely cosmetic.) + +#### H.5 — Backwards compatibility (post-absorption) + +- **F-W4FU-XPR-05** — Walk every public-API surface touched in + the diff: + - `forgelm.cli` — no facade re-exports added or removed by this + absorption. ✓ + - `tools/check_anchor_resolution.py` — `_HEADING_RE` removed, + `_parse_atx_heading` added. The `__all__` (none defined) is + unchanged. Are any third-party callers importing + `_HEADING_RE` from this module? (Internal `tools/` script; + not part of the `forgelm` package public surface. Unlikely.) + - `tools/check_pip_audit.py` — `_normalise_severity` semantics + changed (CVSS_V3 string now returns UNKNOWN, not "CVSS_V3"). + Any external import? (No `__all__`; private helper.) + - `tools/check_bandit.py` — `main` signature unchanged but + behaviour for `{"results": null}` reports flipped from + silent-pass to fail-loudly. Is that a *user-visible* behaviour + change that an existing CI consumer would notice? (Yes — + previously a malformed bandit run passed silently; now it + fails. The change is correct but worth a CHANGELOG line.) + +#### H.6 — gemini-code-assist inline reply audit + +- **F-W4FU-XPR-06** — The maintainer replied to two + gemini-code-assist comments via `gh api .../replies`. Walk: + - PR #33 thread on `tools/check_bilingual_parity.py:79` — reply + cites `ffceaf7` and "23/23 OK". Confirm the link resolves + and the SHA matches the absorption commit. + - PR #33 thread on `docs/qms/access_control.md:189` — reply + cites `ffceaf7` and the rewritten jq pipeline. Confirm. + +## Ground rules + +- **Read the relevant standard before commenting.** The project + [docs/standards/](../../standards/) directory is the rulebook. +- **Re-evaluate, don't just trust.** An "absorbed" finding in + the maintainer's commit message is a *claim*. Verify the + fix against the cited file:line. If the fix is correct, say + so explicitly in your "Verified absorptions" section. If the + fix is incomplete or wrong, that's a `F-W4FU-NN` finding. +- **Cite line numbers.** If you cannot point at a line, the + finding is not actionable. +- **Distinguish original vs newly introduced.** Tag every + finding with whether it traces back to an original + (`F-W4-NN` / `F-W4-PS-NN` / `F-W4-TR-NN`) or is *new* + (introduced by the absorption itself). Use the + `Trace-back to original finding` field in the deliverable. +- **No phantom severity inflation.** A doc typo is `NIT`. A + user-visible CLI bug on the happy path is `HIGH` minimum. +- **No re-litigating closed scope.** "Should ForgeLM also do X" + is not a finding. Stay in the diff. +- **Compliance / privacy claims demand code citations.** Any + prose change about the HMAC chain key, salt derivation, + rotation cadence, or audit fail-closed semantics MUST cite + the actual `forgelm/compliance.py` line where the contract + lives. If a doc change cannot point at the code that backs + it, that's a `HIGH` finding. + +## Required deliverable structure + +Each reviewing agent returns a single Markdown report with this +skeleton, saved as +`docs/analysis/code_reviews/wave4fu-.md`: + +```markdown +# Wave 4 Follow-up — Review of `closure/wave4-integration` + +> Branch: `closure/wave4-integration` · HEAD `105285f` · Reviewer focus: +> . Re-evaluating absorptions since `9a1738a`. + +## Summary +- Verdict: [Block / Conditional / Approve] +- CRITICAL: N · HIGH: N · MEDIUM: N · LOW: N · NIT: N +- One-sentence headline of the highest-severity finding. + +## Findings + +### F-W4FU-NN · · +- **File:** path:line[-line] +- **Trace-back:** F-W4-NN / F-W4-PS-NN / F-W4-TR-NN / NEW +- **What's wrong:** ... +- **User-visible consequence:** ... +- **Suggested fix:** ```code``` +- **Regression test:** ```python ...``` + +(repeat per finding) + +## Verified absorptions + +For each CRITICAL / HIGH from the four original reports + the +2 inline PR comments + 4 SonarCloud items (3 code-smells + +1 hotspot), state: + +- **F- [verified]** — fix lands at ; the + contract now holds. Walked: . +- **F- [partial]** — fix addresses but misses + ; new finding F-W4FU-NN tracks it. +- **F- [regressed]** — fix introduces ; new + finding F-W4FU-NN tracks it. + +## What this report deliberately did not cover + +(scope notes, deferred items, out-of-PR areas) + +## Merge recommendation + +(Block / Conditional / Approve, with the headline reasons) +``` + +## How to launch + +Spawn 4 parallel sub-agents (you may use any harness that supports +parallel workers; the original review used Claude `Agent` calls +in a single message). Pass each agent the SAME prompt, with one +extra line at the top stating its agent name + focus: + +1. **Code-correctness agent.** Focus areas: §C.1, C.2, D (all), + E (all), F (all), H.2, H.4, H.5. Walk the pip-audit / bandit / + anchor-checker rewrites for behaviour invariance. +2. **Privacy / security agent.** Focus areas: §B (all), C (all), + D.4, E.2, H.5. Privacy bar reigns — any prose change about + HMAC key derivation, salt, rotation cadence, or fail-closed + audit semantics must cite the actual code. +3. **Standards / compliance agent.** Focus areas: §A (all), B (all), + H.1, H.3, H.6. Cite [docs/standards/](../../standards/) + extensively. The §3 tally arithmetic is the headline check. +4. **Test-rigour agent.** Focus areas: §D (all), E (all), F (all), + G (all), H.4. Vacuous-test bar reigns — every new test must + pin a behaviour contract, not a schema shape. + +Agents run independently. Maintainer absorbs in a single round +(per the established Wave 3 / Wave 4 cadence: even out-of-scope +real bugs are fixed in the same round, not deferred). + +--- + +*Frozen at `105285f` on 2026-05-06.* *If any reviewer finds the +prompt itself ambiguous, surface it as a `F-W4FU-PROMPT-NN` +finding before merging — the prompt itself is part of the audit +trail.* From 44dbb2171ce8c7a7cb94cf4cba5fb5e0e8be230a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 08:33:12 +0300 Subject: [PATCH 10/12] fix(wave-4-round2): absorb 4-agent followup review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Convergent absorption of the 4 wave4-followup review reports (code-correctness, privacy-security, standards, test-rigor) on absorption commits ffceaf7 + 105285f. Wave 3 cadence: real bugs fixed in the same round, even out-of-strict-scope. HIGH (5) - F-W4FU-PS-01 / ST-01 (1/2) — `compliance_summary.md:90-92` still said "FORGELM_AUDIT_SECRET XOR'd against the per-output-dir salt". The Wave 4 round-1 absorption swept 4 of 6 HMAC-misdescription sites; reference summary (the customer-facing Article-12 contract surface) was missed. Reframed to cite `forgelm/compliance.py:104-114` with the per-output-dir-salt-is-distinct-primitive callout. - F-W4FU-PS-03 / ST-01 (2/2) — `statement_of_applicability.md:132` + `-tr.md:133` A.8.24 row still said "per-output-dir salt + env XOR" without disambiguating chain key from identifier hash. SoA is the auditor-signed document; rewritten to mirror the corrected iso27001_control_mapping cell. - F-W4FU-01 / PS-02 — `access_control.md` §6 segregation-of-duties jq cookbook (EN+TR) was non-runnable (`jq --slurpfile` rejects TSV; replaced flag-mismatch with format-mismatch) AND wrote operator identifiers to a world-readable `/tmp/trainers.tsv` (privacy leak on multi-tenant hosts, since §3.2 invites email identifiers). Replaced with a single-pass `jq -rs` form that joins trainers + approvals in-memory. End-to-end verified against a synthetic 4-entry chain: emits `r1\talice` (the segregation violation) at exit 0. - F-W4FU-TR-01 — `test_aliases_nested_severity_2_6_shape` pinned a fictional pip-audit JSON shape. Test-rigor agent's wheel inspection of pip-audit 2.6.0–2.9.0 confirmed `aliases` is `set[str]` of identifier strings; no version emits a nested `severity` field. Dropped the entire CVSS-from-score derivation + aliases nested fallback + 4 fictional fixtures. `_vuln_severity` now returns UNKNOWN for every real pip-audit vuln; the UNKNOWN summary annotation is the operator-triage path. Kept a `test_hand_crafted_top_level_severity_string_honoured` to pin the documented escape hatch for non-pip-audit scanners that emit a top-level `severity: "HIGH"`. - F-W4FU-TR-02 — `test_undefined_severity_summary_warning` would pass under the regression scenario the original review explicitly called out (per-finding `[UNDEFINED/LOW]` warning instead of summary). Tightened: pin the summary phrasing exactly + assert artefact path appears + negative assertion that `[UNDEFINED/LOW]` per-finding format MUST NOT appear. MEDIUM (4) - F-W4FU-PS-04 — CHANGELOG silent on rotation guidance reframing. Added Wave 4 "Migration notes" subsection covering quarterly → output-dir-lifecycle reframing with the failure-mode explanation. - F-W4FU-PS-05 (+ cc-04) — pip-audit/bandit UNKNOWN+UNDEFINED summary annotations now include the artefact path so an SRE on the GitHub Actions run summary can grep without walking the workflow YAML. - F-W4FU-PS-06 — §8 "lifecycle boundary" verification checklist bullet was a term of art; reframed to give the auditor a concrete signal (KMS rotation event must pair with a fresh `audit_log.manifest.json` genesis pin in the same timestamp window). Mirror in TR. - F-W4FU-TR-05 (+ cc-07) — `_parse_atx_heading` registered empty slug for all-hash headings (`## ##` → "" body after closing-hash strip). Added post-`_normalise_heading_body` empty guard + pinned in `test_atx_all_hashes_rejected`. LOW / NIT (4) - F-W4FU-cc-05 — CHANGELOG missing bandit `null`-results behaviour change line. Added under Migration notes subsection. - F-W4FU-PS-07 — `gdpr_erasure.md` (EN+TR) line 41 mentions identifier-salt XOR; added bridging sentence pointing readers to `access_control.md` §3.4 + `forgelm/compliance.py:104-114` so the XOR-vs-not-XOR asymmetry across docs is explained. - F-W4FU-TR-06 — `test_strict_mode_exits_one_on_broken_anchor` did not pin the broken-link reason. Tightened to assert `"anchor"` substring so a regression that printed the wrong reason class would fail. - F-W4FU-TR-09 — Pinned `_normalise_heading_body("foo###") == "foo"` to lock current off-spec-but-defensible behaviour against silent drift. - F-W4FU-ST-NIT-01 — `iso27001_control_mapping.md:125` (+ TR) A.8.24 row now cites `forgelm/compliance.py:104-114` for symmetry with the other 3 reframed locations. - F-W4FU-TR-08 — `_parse_atx_heading` docstring now explicitly notes the indented-heading CommonMark §4.2 divergence (intentional; pre-rewrite regex was column-anchored too). Validation - `ruff format .` / `ruff check .` — clean. - `python3 tools/check_bilingual_parity.py --strict` — 23/23 OK. - `python3 -m pytest --no-cov -q` — 1413 passed, 14 skipped. (Net -1 from round 1: dropped 4 fictional pip-audit tests, added 3 — 1 hand-crafted-severity, 1 ATX-empty-rejected, 1 ATX-no-space closing-hash, plus the UNDEFINED tightening doesn't add a test.) - Cookbook end-to-end: `jq -rs` snippet from `access_control.md` §6 emits `r1\talice` against a 4-entry synthetic chain with one segregation violation; exit 0. Fix verified live. - `forgelm verify-audit --help` confirms the cookbook syntax: positional `log_path` + `--require-hmac` + `--hmac-secret-env`; no `--output-dir`, no `--json`. - `python3 tools/check_anchor_resolution.py` (advisory) — 36 pre-existing baseline (Faz 30 follow-up scope). Co-Authored-By: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 19 +++ docs/guides/gdpr_erasure-tr.md | 2 +- docs/guides/gdpr_erasure.md | 2 +- docs/qms/access_control-tr.md | 39 +++--- docs/qms/access_control.md | 39 +++--- docs/qms/statement_of_applicability-tr.md | 2 +- docs/qms/statement_of_applicability.md | 2 +- docs/reference/compliance_summary.md | 11 +- docs/reference/iso27001_control_mapping-tr.md | 2 +- docs/reference/iso27001_control_mapping.md | 2 +- tests/test_check_anchor_resolution.py | 29 +++- tests/test_supply_chain_security.py | 126 +++++++----------- tools/check_anchor_resolution.py | 14 +- tools/check_bandit.py | 13 +- tools/check_pip_audit.py | 115 ++++------------ 15 files changed, 198 insertions(+), 219 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf792f14..e00d076c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -86,6 +86,25 @@ Cross-cutting: - `docs/standards/localization.md` — `docs/qms/*.md` row flipped from "No (yet)" to "Yes" to reflect the bilingual sweep. +**Migration notes (Wave 4)** + +- **Rotation guidance reframed (`FORGELM_AUDIT_SECRET`).** Prior + 0.5.x guidance recommended quarterly rotation. Wave 4 reframes + this to **between output-dir lifecycles** — every audit entry's + HMAC keys to the secret live at emit time, so rotating mid + output-dir breaks `forgelm verify-audit --require-hmac` for the + cross-secret span. Deployers running quarterly KMS-rotation + cron jobs today must align rotation to fresh `` + provisioning (rotate the secret + cut a new output-dir together). + See `docs/qms/access_control.md` §3.4 for the corrected procedure + and §8 verification checklist for the auditor-facing signal. +- **`tools/check_bandit.py` malformed-report behaviour.** A bandit + JSON report carrying `{"results": null}` (malformed; rare) now + fails the gate (exit 1) where the prior `or []` collapse silently + treated it as a clean run. Operators whose nightly runs previously + passed against a corrupt bandit invocation must now investigate + the upstream bandit failure instead of consuming the silent pass. + **Faz 26 — QMS bilingual mirror + compliance_summary cleanup** - 6 existing QMS TR mirrors: `README-tr.md`, diff --git a/docs/guides/gdpr_erasure-tr.md b/docs/guides/gdpr_erasure-tr.md index 56d5e9fa..089c6343 100644 --- a/docs/guides/gdpr_erasure-tr.md +++ b/docs/guides/gdpr_erasure-tr.md @@ -38,7 +38,7 @@ $ forgelm purge \ Tool şunları yapar: -1. Per-output-dir salt'ı `/.forgelm_audit_salt`'ta çözer (ilk kullanımda oluşturur; mode `0600`; `FORGELM_AUDIT_SECRET[:16]` set'liyse XOR'lar). +1. Per-output-dir salt'ı `/.forgelm_audit_salt`'ta çözer (ilk kullanımda oluşturur; mode `0600`; `FORGELM_AUDIT_SECRET[:16]` set'liyse XOR'lar). **Not:** bu XOR yalnızca **identifier hashing**'i besler — audit-chain HMAC anahtarı bağımsız olarak `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` şeklinde türetilir (bkz. [`docs/qms/access_control.md`](../qms/access_control.md) §3.4 / `forgelm/compliance.py:104-114`). İki primitif kasıtlı olarak ayrıdır. 2. Audit log'a `data.erasure_requested` yazar — **hash'lenmiş** target_id (`SHA-256(salt + value)`) ve operatör-sağlanan justification ile. 3. Eşleşen JSONL satırını `id` (veya `row_id`) field'ı ile bulur — re-order edilmiş bir dosyada sessiz yanlış-satır silmesini engellemek için satır-numarası fallback'i **reddedilir**. Id'siz corpus'lara sahip operatörler önce `forgelm audit --add-row-ids ` (Phase 28 follow-up) çalıştırmalı. 4. Corpus'u atomic olarak yeniden yazar (kardeş bir temp dosyaya yazar + `os.replace`); operatörler ya tam silme-öncesi dosyayı ya da tam silme-sonrası dosyayı görür, asla kısmi state'i değil. diff --git a/docs/guides/gdpr_erasure.md b/docs/guides/gdpr_erasure.md index 0c23aa37..4b496be7 100644 --- a/docs/guides/gdpr_erasure.md +++ b/docs/guides/gdpr_erasure.md @@ -38,7 +38,7 @@ $ forgelm purge \ The tool: -1. Resolves the per-output-dir salt at `/.forgelm_audit_salt` (creates it on first use; mode `0600`; XOR'd with `FORGELM_AUDIT_SECRET[:16]` if the env var is set). +1. Resolves the per-output-dir salt at `/.forgelm_audit_salt` (creates it on first use; mode `0600`; XOR'd with `FORGELM_AUDIT_SECRET[:16]` if the env var is set). **Note:** this XOR feeds **identifier hashing only** — the audit-chain HMAC key is derived independently as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (see [`docs/qms/access_control.md`](../qms/access_control.md) §3.4 / `forgelm/compliance.py:104-114`). The two primitives are intentionally separate. 2. Writes `data.erasure_requested` to the audit log with the **hashed** target_id (`SHA-256(salt + value)`) and the operator-supplied justification. 3. Locates the matching JSONL row by its `id` (or `row_id`) field — line-number fallback is **rejected** to defend against silent wrong-row deletion on a re-ordered file. Operators with id-less corpora must run `forgelm audit --add-row-ids ` (Phase 28 follow-up) first. 4. Atomically rewrites the corpus (writes a sibling temp file + `os.replace`); operators get either the full pre-erasure file or the full post-erasure file, never a partial state. diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md index 664b8155..ff1776ad 100644 --- a/docs/qms/access_control-tr.md +++ b/docs/qms/access_control-tr.md @@ -199,20 +199,21 @@ denetçi ihlalleri tespit edebilir: # subcommand'ta tasarım gereği --output-dir / --json flag'i yoktur). forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac -# 2. Koşum başına trainer. -jq -r 'select(.event == "training.started") | - [.run_id, .operator] | @tsv' \ - ./outputs/audit_log.jsonl > /tmp/trainers.tsv - -# 3. Aynı koşum için trainer listesinde yer alan onaylayan-id'ye sahip -# approval'lar (görev ayrılığı ihlali). -jq -r --slurpfile t /tmp/trainers.tsv \ - 'select(.event == "human_approval.granted") | - . as $a | - $t[] | split("\t") as $row | - select($row[0] == $a.run_id and $row[1] == $a.operator) | - [.run_id, .operator] | @tsv' \ - ./outputs/audit_log.jsonl +# 2. Single-pass jq: audit log'u slurp et, trainer'ları + approval'ları +# project et, in-memory join. Operatör tanımlayıcılarını paylaşılan +# /tmp/ dosyasından uzak tutar (multi-tenant host'larda 0644 / +# world-readable olur ve §3.2'ye göre operatörün +# FORGELM_OPERATOR'a koymuş olabileceği email'leri sızdırır). +# Çıktı: koşumun eğiteni ile aynı operatör tarafından onaylanmış +# her approval'ın TSV satırı (görev ayrılığı ihlali). +jq -rs ' + (map(select(.event == "training.started"))) as $trainers | + map(select(.event == "human_approval.granted"))[] | + . as $a | + $trainers[] | + select(.run_id == $a.run_id and .operator == $a.operator) | + [.run_id, .operator] | @tsv +' ./outputs/audit_log.jsonl ``` ## 7. Webhook secret ayrılığı @@ -242,9 +243,13 @@ Erişim-kontrolü kanıtını yürüyen operatör denetçi için: - [ ] İki aktif pipeline aynı `FORGELM_OPERATOR` değerini paylaşmıyor. - [ ] `FORGELM_AUDIT_SECRET` KMS / Vault substrate'inde yaşıyor, VCS'de veya düz `.env` dosyasında değil. -- [ ] KMS audit log'u `FORGELM_AUDIT_SECRET` rotasyonunu output-dir - lifecycle sınırlarına hizalı gösteriyor (asla output-dir - ortasında değil). +- [ ] KMS audit log'u `FORGELM_AUDIT_SECRET` rotasyonunu taze + `` provisioning event'ı ile eşli gösteriyor — her + KMS rotasyon event'ının aynı KMS-event zaman penceresi içinde + karşılık gelen yeni bir `/audit_log.manifest.json` + genesis pin'i olmak zorunda. Eşleşen genesis pin'i olmayan + rotasyonlar (yani output-dir-ortası rotasyonlar) cross-secret + aralık için `forgelm verify-audit --require-hmac`'i kırar. - [ ] Son 90 gündeki her `human_approval.granted` event'i için approval-gate kimliği training kimliğinden farklı (sample-audit, exhaustive değil). diff --git a/docs/qms/access_control.md b/docs/qms/access_control.md index 0c38f0e5..fa058d4d 100644 --- a/docs/qms/access_control.md +++ b/docs/qms/access_control.md @@ -199,20 +199,21 @@ auditor can detect violations: # --output-dir / --json flags exist on this subcommand by design). forgelm verify-audit ./outputs/audit_log.jsonl --require-hmac -# 2. Trainer per run. -jq -r 'select(.event == "training.started") | - [.run_id, .operator] | @tsv' \ - ./outputs/audit_log.jsonl > /tmp/trainers.tsv - -# 3. Approvals whose approver-id appears in the trainer list for the -# same run (segregation violation). -jq -r --slurpfile t /tmp/trainers.tsv \ - 'select(.event == "human_approval.granted") | - . as $a | - $t[] | split("\t") as $row | - select($row[0] == $a.run_id and $row[1] == $a.operator) | - [.run_id, .operator] | @tsv' \ - ./outputs/audit_log.jsonl +# 2. Single-pass jq: slurp the audit log, project trainers + approvals, +# join in-memory. Keeps operator identifiers out of any shared +# /tmp/ file (which would land 0644 / world-readable on multi-tenant +# hosts and leak emails the deployer might have set as +# FORGELM_OPERATOR per §3.2). Output: TSV rows of every approval +# where the approver matches the run's trainer (segregation +# violation). +jq -rs ' + (map(select(.event == "training.started"))) as $trainers | + map(select(.event == "human_approval.granted"))[] | + . as $a | + $trainers[] | + select(.run_id == $a.run_id and .operator == $a.operator) | + [.run_id, .operator] | @tsv +' ./outputs/audit_log.jsonl ``` ## 7. Webhook secret separation @@ -241,8 +242,14 @@ For a deployer auditor walking access-control evidence: - [ ] No two active pipelines share an `FORGELM_OPERATOR` value. - [ ] `FORGELM_AUDIT_SECRET` lives in a KMS / Vault substrate, never in VCS or a plain `.env` file. -- [ ] KMS audit log shows `FORGELM_AUDIT_SECRET` rotation aligned to - output-dir lifecycle boundaries (never mid-output-dir). +- [ ] KMS audit log shows `FORGELM_AUDIT_SECRET` rotation paired + with a fresh `` provisioning event — every KMS + rotation event must have a corresponding new + `/audit_log.manifest.json` genesis pin within the + same KMS-event timestamp window. Rotations with no matching + genesis pin (i.e. mid-output-dir rotations) break + `forgelm verify-audit --require-hmac` for the cross-secret + span. - [ ] Approval-gate identity differs from training identity for every `human_approval.granted` event in the past 90 days (sample-audit, not exhaustive). diff --git a/docs/qms/statement_of_applicability-tr.md b/docs/qms/statement_of_applicability-tr.md index 37991723..74b90930 100644 --- a/docs/qms/statement_of_applicability-tr.md +++ b/docs/qms/statement_of_applicability-tr.md @@ -130,7 +130,7 @@ ForgeLM-spesifik kontrol envanteriyle ilgilidir. | A.8.21 Ağ hizmetlerinin güvenliği | YES | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement | | A.8.22 Ağ ayrımı | YES | VPC / subnet design | Benimse | | A.8.23 Web filtreleme | YES | Egress proxy | Benimse | -| A.8.24 Kriptografi kullanımı | YES | SHA-256 + HMAC-SHA-256 chain; salted SHA-256 identifier hashing; per-output-dir salt + env XOR | `FORGELM_AUDIT_SECRET` için KMS | +| A.8.24 Kriptografi kullanımı | YES | SHA-256 + HMAC-SHA-256 (audit chain key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, bkz. `forgelm/compliance.py:104-114`); ayrıca `forgelm purge` / `forgelm reverse-pii` için salted SHA-256 identifier hashing (`_purge._resolve_salt`, ayrı bir konu — chain-key türetimine katılmaz) | `FORGELM_AUDIT_SECRET` için KMS | | A.8.25 Güvenli geliştirme yaşam döngüsü | YES | `docs/standards/code-review.md`, `release.md`, CI gate'leri | SDLC framework | | A.8.26 Uygulama güvenliği gereksinimleri | YES | F-compliance-110 strict gate; Pydantic validation; `_reverse_pii`'da ReDoS guard | App-seviyesi tehdit modelleme | | A.8.27 Güvenli sistem mimarisi ve mühendislik prensipleri | YES | Append-only audit log mimarisi; HMAC chain; lazy import; SSRF guard | Defence-in-depth | diff --git a/docs/qms/statement_of_applicability.md b/docs/qms/statement_of_applicability.md index 38728025..6fedb416 100644 --- a/docs/qms/statement_of_applicability.md +++ b/docs/qms/statement_of_applicability.md @@ -129,7 +129,7 @@ ForgeLM-specific control inventory. | A.8.21 Security of network services | YES | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | TLS 1.2+ enforcement | | A.8.22 Segregation of networks | YES | VPC / subnet design | Adopt | | A.8.23 Web filtering | YES | Egress proxy | Adopt | -| A.8.24 Use of cryptography | YES | SHA-256 + HMAC-SHA-256 chain; salted SHA-256 identifier hashing; per-output-dir salt + env XOR | KMS for `FORGELM_AUDIT_SECRET` | +| A.8.24 Use of cryptography | YES | SHA-256 + HMAC-SHA-256 (audit chain key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, see `forgelm/compliance.py:104-114`); separately, salted SHA-256 identifier hashing for `forgelm purge` / `forgelm reverse-pii` (`_purge._resolve_salt`, distinct concern — does not participate in chain-key derivation) | KMS for `FORGELM_AUDIT_SECRET` | | A.8.25 Secure development life cycle | YES | `docs/standards/code-review.md`, `release.md`, CI gates | SDLC framework | | A.8.26 Application security requirements | YES | F-compliance-110 strict gate; Pydantic validation; ReDoS guard in `_reverse_pii` | App-level threat modelling | | A.8.27 Secure system architecture and engineering principles | YES | Append-only audit log architecture; HMAC chain; lazy import; SSRF guard | Defence-in-depth | diff --git a/docs/reference/compliance_summary.md b/docs/reference/compliance_summary.md index 74d8480e..ddae7521 100644 --- a/docs/reference/compliance_summary.md +++ b/docs/reference/compliance_summary.md @@ -88,8 +88,13 @@ What the regulator asks vs. how ForgeLM answers: - Implementation: `forgelm.compliance.AuditLogger` — JSON Lines append-only log at `/audit_log.jsonl`, HMAC-chained - with `FORGELM_AUDIT_SECRET` XOR'd against the per-output-dir salt - in `/.forgelm_audit_salt`. Genesis manifest sidecar + with a per-run signing key derived as + `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (see + `forgelm/compliance.py:104-114`). The per-output-dir salt at + `/.forgelm_audit_salt` is a **distinct primitive** — + it salts identifier hashing in `forgelm purge` / + `forgelm reverse-pii` events (`_purge._resolve_salt`) and does + NOT participate in chain-key derivation. Genesis manifest sidecar (`audit_log.manifest.json`) refuses truncate-and-resume tampering. - Verification: `forgelm verify-audit [--require-hmac]` validates the chain end-to-end; exits 0/1/2/3. @@ -168,7 +173,7 @@ name; this survives refactors that the prior `#L33` form did not. `LlamaGuardClassifier`, `_evaluate_3_layer_gate`). - **Audit chain + HMAC + manifest**: `forgelm.compliance` (search for `AuditLogger`, `_check_genesis_manifest`, - `compute_artefact_sha256`). + `generate_model_integrity`). - **Salted identifier hashing**: `forgelm.cli.subcommands._purge` (search for `_resolve_salt`, `_read_persistent_salt`, `_hash_target_id`). diff --git a/docs/reference/iso27001_control_mapping-tr.md b/docs/reference/iso27001_control_mapping-tr.md index 9cff9984..50c89b82 100644 --- a/docs/reference/iso27001_control_mapping-tr.md +++ b/docs/reference/iso27001_control_mapping-tr.md @@ -121,7 +121,7 @@ SoA'sının uçtan uca auditable olması için tamlık adına listelenmiştir. | A.8.21 Ağ hizmetlerinin güvenliği | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | | A.8.22 Ağ ayrımı | OOS | — | | A.8.23 Web filtreleme | OOS | — | -| A.8.24 Kriptografi kullanımı | FL | SHA-256 + HMAC chain (per-run imzalama anahtarı = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`); ayrıca purge / reverse-pii için salted SHA-256 identifier hashing | +| A.8.24 Kriptografi kullanımı | FL | SHA-256 + HMAC chain (per-run imzalama anahtarı = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, bkz. `forgelm/compliance.py:104-114`); ayrıca purge / reverse-pii için salted SHA-256 identifier hashing | | A.8.25 Güvenli geliştirme yaşam döngüsü | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | | A.8.26 Uygulama güvenliği gereksinimleri | FL-helps | F-compliance-110 strict gate; ReDoS guard | | A.8.27 Güvenli sistem mimarisi ve mühendislik prensipleri | FL-helps | Append-only audit log mimarisi | diff --git a/docs/reference/iso27001_control_mapping.md b/docs/reference/iso27001_control_mapping.md index 0c2646e3..fce0b2e3 100644 --- a/docs/reference/iso27001_control_mapping.md +++ b/docs/reference/iso27001_control_mapping.md @@ -122,7 +122,7 @@ completeness so the deployer's SoA is auditable end-to-end. | A.8.21 Security of network services | FL-helps | TLS-only webhooks; `FORGELM_AUDIT_SECRET` HMAC | | A.8.22 Segregation of networks | OOS | — | | A.8.23 Web filtering | OOS | — | -| A.8.24 Use of cryptography | FL | SHA-256 + HMAC chain (per-run signing key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`); separately, salted SHA-256 identifier hashing for purge / reverse-pii | +| A.8.24 Use of cryptography | FL | SHA-256 + HMAC chain (per-run signing key = `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)`, see `forgelm/compliance.py:104-114`); separately, salted SHA-256 identifier hashing for purge / reverse-pii | | A.8.25 Secure development life cycle | FL-helps | `docs/standards/code-review.md`, `release.md`, CI gates | | A.8.26 Application security requirements | FL-helps | F-compliance-110 strict gate; ReDoS guard | | A.8.27 Secure system architecture and engineering principles | FL-helps | Append-only audit log architecture | diff --git a/tests/test_check_anchor_resolution.py b/tests/test_check_anchor_resolution.py index fb6c7102..60f17441 100644 --- a/tests/test_check_anchor_resolution.py +++ b/tests/test_check_anchor_resolution.py @@ -88,6 +88,25 @@ def test_atx_closing_hashes_stripped(self): assert tool._normalise_heading_body("My Title #") == "My Title" assert tool._normalise_heading_body("My Title ### ") == "My Title" + def test_atx_closing_hashes_without_preceding_space_still_stripped(self): + # F-W4FU-TR-09 absorption: the helper greedily strips the + # closing-hash run even without preceding whitespace. Off-spec + # per CommonMark §4.2 but defensible — pin current behaviour + # so a future tightening is a deliberate change, not silent + # drift. + tool = _load_tool() + assert tool._normalise_heading_body("foo###") == "foo" + + def test_atx_all_hashes_rejected(self): + # F-W4FU-TR-05 / F-W4FU-07 absorption: ``## ##`` (all-hash + # body) trims to "" after closing-hash strip; the parser must + # reject rather than register the empty slug as an anchor. + tool = _load_tool() + assert tool._parse_atx_heading("## ##") is None + assert tool._parse_atx_heading("### #") is None + # Symmetric happy path: real heading still parses. + assert tool._parse_atx_heading("## Real Heading") == "Real Heading" + # --------------------------------------------------------------------------- # §2 — Resolution: relative paths + anchors @@ -239,15 +258,19 @@ def test_strict_mode_exits_one_on_broken(self, tmp_path: Path, capsys): assert "FAIL" in captured def test_strict_mode_exits_one_on_broken_anchor(self, tmp_path: Path, capsys): - # F-W4-TR-03 absorption: anchor-not-found and file-not-found must - # be symmetrically gated under --strict. + # F-W4-TR-03 absorption (tightened in F-W4FU-TR-06): + # anchor-not-found and file-not-found must be symmetrically + # gated under --strict; the broken-link reason must + # disambiguate "anchor not found" from "target file not found". tool = _load_tool() scope = tmp_path / "docs" _write(scope / "target.md", "# Target\n\n## Real Section\n") _write(scope / "src.md", "[link](target.md#nonexistent)\n") rc = tool.main(["--repo-root", str(tmp_path), "--scope", "docs", "--strict"]) assert rc == 1 - assert "FAIL" in capsys.readouterr().out + out = capsys.readouterr().out + assert "FAIL" in out + assert "anchor" in out, f"reason should distinguish anchor-not-found from file-not-found; got: {out!r}" def test_quiet_suppresses_ok_summary(self, tmp_path: Path, capsys): tool = _load_tool() diff --git a/tests/test_supply_chain_security.py b/tests/test_supply_chain_security.py index c5401e4e..076118b9 100644 --- a/tests/test_supply_chain_security.py +++ b/tests/test_supply_chain_security.py @@ -355,8 +355,9 @@ def test_results_missing_key_fails(self, tmp_path: Path, tool, capsys) -> None: assert "missing 'results'" in capsys.readouterr().err def test_undefined_severity_summary_warning(self, tmp_path: Path, tool, capsys) -> None: - # F-W4-TR-01 absorption: bandit's documented UNDEFINED tier - # surfaces a single summary warning, not silence. + # F-W4-TR-01 absorption (tightened in F-W4FU-TR-02): bandit's + # documented UNDEFINED tier surfaces a single summary warning, + # not per-finding spam, not silence. report = { "results": [ { @@ -370,17 +371,33 @@ def test_undefined_severity_summary_warning(self, tmp_path: Path, tool, capsys) } ] } - rc = tool.main(["check_bandit", str(self._write_report(tmp_path, report))]) + report_path = self._write_report(tmp_path, report) + rc = tool.main(["check_bandit", str(report_path)]) captured = capsys.readouterr().out assert rc == 0 - assert "::warning::bandit" in captured - assert "UNDEFINED" in captured + # Pin the summary phrasing exactly — a regression that flipped + # UNDEFINED handling to per-finding ``[UNDEFINED/LOW]`` warnings + # would still satisfy a loose substring check. + assert "::warning::bandit 1 issue(s) with UNDEFINED severity" in captured, ( + f"expected single summary annotation; got: {captured!r}" + ) + assert str(report_path) in captured, "summary must include the artefact path for SRE triage" + # Negative: the per-finding format prefix MUST NOT appear. + assert "[UNDEFINED/LOW]" not in captured, ( + "UNDEFINED issues must surface only via the summary, not per-finding warnings" + ) class TestCheckPipAuditExtraShapes: - """Extra pip-audit fixtures that pin documented contracts the - primary tier-suite leaves uncovered (F-W4-04 / F-W4-PS-07 / - F-W4-TR-02 / F-W4-TR-05 absorptions).""" + """Extra pip-audit fixtures pinning the UNKNOWN summary contract. + + Wave 4 followup absorption (F-W4FU-TR-01): the speculative + CVSS-from-score derivation + aliases-nested fallback + 4 fictional + fixtures were dropped after the test-rigor agent's wheel inspection + confirmed pip-audit 2.6.0–2.9.0 never serialise severity into the + JSON output. ``_vuln_severity`` returns ``"UNKNOWN"`` for every + real pip-audit vuln; this class pins the operator-triage path. + """ @pytest.fixture def tool(self): @@ -392,102 +409,51 @@ def _write_report(self, tmp_path: Path, payload: dict) -> Path: return path def test_missing_severity_field_summary_warning(self, tmp_path: Path, tool, capsys) -> None: + # Real pip-audit JSON: vuln carries id + aliases (set of CVE/GHSA + # identifier strings) + fix_versions, NO severity field at any + # nesting level. Gate must surface UNKNOWN as a single summary + # annotation that includes the artefact path (F-W4FU-PS-05). report = { "dependencies": [ { "name": "u", "version": "1.0.0", - "vulns": [{"id": "GHSA-no-sev"}], - } - ] - } - rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) - captured = capsys.readouterr().out - assert rc == 0 - assert "::warning::pip-audit" in captured - assert "without parseable severity" in captured - - def test_severity_list_with_explicit_tier(self, tmp_path: Path, tool) -> None: - # 2.7.x list-form carrying an explicit ``severity`` field. - report = { - "dependencies": [ - { - "name": "listpkg", - "version": "1.0.0", "vulns": [ { - "id": "GHSA-list", - "severity": [{"type": "CVSS_V3", "severity": "HIGH"}], + "id": "GHSA-no-sev", + "fix_versions": ["1.0.1"], + "aliases": ["CVE-2026-NOSEV"], } ], } ] } - rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) - assert rc == 1, "list form with explicit HIGH tier must fail" - - def test_severity_list_with_cvss_score(self, tmp_path: Path, tool) -> None: - # CVSS_V3 type label + 9.5 score must derive CRITICAL. - report = { - "dependencies": [ - { - "name": "scorepkg", - "version": "1.0.0", - "vulns": [ - { - "id": "GHSA-score", - "severity": [{"type": "CVSS_V3", "score": "9.5"}], - } - ], - } - ] - } - rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) - assert rc == 1, "CVSS 9.5 must derive CRITICAL and fail the gate" - - def test_severity_list_cvss_score_medium(self, tmp_path: Path, tool, capsys) -> None: - report = { - "dependencies": [ - { - "name": "midscore", - "version": "1.0.0", - "vulns": [ - { - "id": "GHSA-mid-score", - "severity": [{"type": "CVSS_V3", "score": 5.5}], - } - ], - } - ] - } - rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) + report_path = self._write_report(tmp_path, report) + rc = tool.main(["check_pip_audit", str(report_path)]) captured = capsys.readouterr().out assert rc == 0 assert "::warning::pip-audit" in captured + assert "without parseable severity" in captured + assert str(report_path) in captured, ( + "UNKNOWN summary must include the artefact path so an " + "incident-triage SRE can grep without walking the workflow YAML" + ) - def test_aliases_nested_severity_2_6_shape(self, tmp_path: Path, tool) -> None: - # 2.6.x: severity buried under aliases[].severity[]. + def test_hand_crafted_top_level_severity_string_honoured(self, tmp_path: Path, tool) -> None: + # Hand-crafted JSON from non-pip-audit scanners that emit a + # top-level ``severity: "HIGH"`` must still fail the gate; this + # is the only severity path real ``_vuln_severity`` honours. report = { "dependencies": [ { - "name": "legacy", + "name": "external-scanner", "version": "1.0.0", - "vulns": [ - { - "id": "GHSA-old-shape", - "aliases": [ - { - "id": "CVE-2026-OLD", - "severity": [{"type": "CVSS_V3", "severity": "HIGH"}], - } - ], - } - ], + "vulns": [{"id": "EXT-2026-001", "severity": "HIGH", "fix_versions": []}], } ] } rc = tool.main(["check_pip_audit", str(self._write_report(tmp_path, report))]) - assert rc == 1, "2.6.x nested aliases shape must classify as HIGH" + assert rc == 1, "top-level severity string is the documented escape hatch" class TestSbomSerialNumberUniqueness: diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py index 7d1910cc..82ece048 100644 --- a/tools/check_anchor_resolution.py +++ b/tools/check_anchor_resolution.py @@ -97,6 +97,15 @@ def _parse_atx_heading(line: str) -> str | None: Implemented as a single linear pass to avoid the ReDoS risk heuristic that flags adjacent variable-length quantifiers in regex form (Sonar python:S5852). + + Known divergence from CommonMark §4.2 (intentional): indented + headings (1-3 leading spaces) are NOT recognised — project docs + must place ``#`` at column 0. The pre-rewrite regex form was + column-anchored too, so this preserves Wave 4 behaviour. + + Edge case (F-W4FU-TR-05): ``## ##`` (all-hash heading) trims to + an empty body after closing-hash strip; reject rather than + register the empty slug as an anchor. """ if not line or line[0] != "#": return None @@ -110,7 +119,10 @@ def _parse_atx_heading(line: str) -> str | None: body = line[hash_count + 1 :].strip() if not body: return None - return _normalise_heading_body(body) + body = _normalise_heading_body(body) + if not body: + return None + return body def _normalise_heading_body(body: str) -> str: diff --git a/tools/check_bandit.py b/tools/check_bandit.py index 1f41eb56..3f029469 100644 --- a/tools/check_bandit.py +++ b/tools/check_bandit.py @@ -111,7 +111,8 @@ def main(argv: list[str]) -> int: print(f"usage: {argv[0]} ", file=sys.stderr) return 1 - report_or_code = _load_report(Path(argv[1])) + report_path = Path(argv[1]) + report_or_code = _load_report(report_path) if isinstance(report_or_code, int): return report_or_code results_or_code = _extract_results(report_or_code) @@ -126,8 +127,14 @@ def main(argv: list[str]) -> int: if undefined_count: # One summary annotation rather than per-finding spam; UNDEFINED # rules are bandit's "rule lacks severity metadata" fall-through - # and merit operator review without burying real signal. - print(f"::warning::bandit {undefined_count} issue(s) with UNDEFINED severity; review the raw report manually.") + # and merit operator review without burying real signal. Includes + # the artefact path so an SRE on the GitHub Actions run summary + # can grep without walking the workflow YAML (F-W4FU-PS-05 + # absorption). + print( + f"::warning::bandit {undefined_count} issue(s) with UNDEFINED " + f"severity in {report_path}; review the raw report manually." + ) if high: for line in high: diff --git a/tools/check_pip_audit.py b/tools/check_pip_audit.py index 495dd797..0951a0d7 100644 --- a/tools/check_pip_audit.py +++ b/tools/check_pip_audit.py @@ -31,115 +31,48 @@ from pathlib import Path from typing import Any, Iterable, Optional -# pip-audit's JSON shape (≥2.7) puts findings under ``dependencies[].vulns``, +# pip-audit's JSON shape puts findings under ``dependencies[].vulns``, # each vuln carrying ``id``, ``aliases``, ``description``, ``fix_versions``. -# Severity is sourced from the OSV / GHSA aliases via ``severity`` (a list -# of {type, score} pairs) when present; pip-audit normalises into -# ``severity`` strings on the top-level vuln dict from 2.7.x onwards. +# +# Empirical note (verified against pip-audit 2.6.0–2.9.0 wheel sources +# during Wave 4 absorption round 2): pip-audit's ``_format/json.py`` +# does NOT serialise OSV severity into the JSON output — ``aliases`` +# is a flat list of CVE/GHSA identifier strings (per +# ``pip_audit/_service/interface.py``: ``aliases: set[str]``), no +# nested ``severity`` field appears at any nesting level. This means +# `_vuln_severity` returns ``"UNKNOWN"`` for every vuln in a real +# pip-audit JSON report, and the UNKNOWN summary annotation handles +# the operator-triage path. We retain the top-level string-severity +# branch to honour the documented CLAUDE.md / pyproject.toml schema +# (operators feeding hand-crafted JSON for non-pip-audit scanners can +# emit a top-level ``severity: "HIGH"`` and have the gate honour it). _HIGH_TIERS: frozenset[str] = frozenset({"HIGH", "CRITICAL"}) _MED_TIERS: frozenset[str] = frozenset({"MEDIUM", "MODERATE"}) -# OSV severity-list ``type`` values are scoring-system labels, NOT tier -# names — recognising them lets us avoid mistaking the label for a -# severity (otherwise the original CVSS_V3 string would resolve to -# UNKNOWN by chance, but a renamed system would silently mis-tier). -_SCORE_TYPE_LABELS: frozenset[str] = frozenset({"CVSS", "CVSS_V2", "CVSS_V3", "CVSS_V4", "CVSS_V31", "CVSS_V40"}) - def _normalise_severity(raw: Optional[str]) -> str: """Upper-case + collapse synonyms; unknown/missing → ``UNKNOWN``.""" if not raw: return "UNKNOWN" upper = raw.upper().strip() - if upper in {"MODERATE"}: + if upper == "MODERATE": return "MEDIUM" - if upper in _SCORE_TYPE_LABELS: - # CVSS_V3 etc. is a scoring-system label, not a tier name. - return "UNKNOWN" return upper -def _tier_from_cvss_score(score: Any) -> str: - """Map a CVSS base score to a severity tier per FIRST.org guidance. - - CVSS v3 / v4 cut-points: 0.0 NONE · 0.1–3.9 LOW · 4.0–6.9 MEDIUM · - 7.0–8.9 HIGH · 9.0–10.0 CRITICAL. Falls back to ``UNKNOWN`` on - any parse failure rather than guessing. - """ - if isinstance(score, (int, float)): - value = float(score) - elif isinstance(score, str): - try: - value = float(score.strip()) - except ValueError: - return "UNKNOWN" - else: - return "UNKNOWN" - if value >= 9.0: - return "CRITICAL" - if value >= 7.0: - return "HIGH" - if value >= 4.0: - return "MEDIUM" - if value > 0.0: - return "LOW" - return "UNKNOWN" - - -def _severity_from_entry(entry: Any) -> str: - """Extract a severity tier from a single ``severity[]`` element.""" - if isinstance(entry, dict): - # Only the ``severity`` field carries a tier label; ``type`` is - # the scoring-system identifier (CVSS_V3, ...). Try the tier - # first, then derive from the CVSS score when only the type + - # score pair is available. - explicit = _normalise_severity(entry.get("severity")) - if explicit != "UNKNOWN": - return explicit - return _tier_from_cvss_score(entry.get("score")) - if isinstance(entry, str): - return _normalise_severity(entry) - return "UNKNOWN" - - def _vuln_severity(vuln: dict[str, Any]) -> str: """Extract a single severity tier from a pip-audit vuln entry. - Falls back through several fields because pip-audit's JSON shape - has shifted across point releases: - - 2.7.x: top-level ``severity`` string - - 2.7.x list form: top-level ``severity[]`` of ``{type, score}`` / - ``{severity}`` / plain string entries - - 2.6.x and earlier: nested under ``aliases[].severity[]`` - - When no field is parseable we return ``"UNKNOWN"`` and let the - caller surface a single summary annotation so the operator can - review the raw report. + Honours only the top-level ``severity`` string — pip-audit's JSON + output never carries severity (verified against 2.6.0–2.9.0 wheel + sources). Hand-crafted JSON from non-pip-audit scanners can set + a top-level ``severity: "HIGH"`` and have the gate honour it. + Anything else falls through to ``"UNKNOWN"`` and surfaces via the + UNKNOWN summary annotation in ``main()``. """ direct = vuln.get("severity") if isinstance(direct, str): return _normalise_severity(direct) - if isinstance(direct, list): - for entry in direct: - tier = _severity_from_entry(entry) - if tier != "UNKNOWN": - return tier - # 2.6.x fallback — severity buried inside aliases[].severity[]. - aliases = vuln.get("aliases") - if isinstance(aliases, list): - for alias in aliases: - if not isinstance(alias, dict): - continue - nested = alias.get("severity") - if isinstance(nested, list): - for entry in nested: - tier = _severity_from_entry(entry) - if tier != "UNKNOWN": - return tier - elif isinstance(nested, str): - tier = _normalise_severity(nested) - if tier != "UNKNOWN": - return tier return "UNKNOWN" @@ -209,10 +142,12 @@ def main(argv: list[str]) -> int: if unknown: # One summary annotation rather than per-finding spam — UNKNOWN # findings need operator review, not noise that buries real - # signal (F-W4-PS-07 / F-W4-TR-05 absorption). + # signal. Includes the artefact path so an SRE on the GitHub + # Actions run summary can grep without walking the workflow YAML + # (F-W4FU-PS-05 absorption). print( f"::warning::pip-audit {len(unknown)} finding(s) without parseable " - "severity; review the raw report manually." + f"severity in {report_path}; review the raw report manually." ) if high: From 20e4245d1841c569c396aa9d3ed1b45c5c3e85ee Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 09:24:39 +0300 Subject: [PATCH 11/12] fix(wave-4-round3): absorb 16 inline review comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wave 4 round-3 absorption from inline review comments on PR #33. Per "Verify each finding against current code" methodology: 4 findings skipped as INVALID (CLI commands claimed non-shipped that verifiably ship in v0.5.x — `forgelm doctor`/`purge`/`reverse-pii`/ `safety-eval`/`verify-gguf`/`approvals` all confirmed via `forgelm --help`). Skipped (with reasons noted in this commit message): - RTP / iso27001_control_mapping / soc2_trust_criteria_mapping EN+TR — reviewer claimed `forgelm doctor`/`purge`/`reverse-pii`/ etc. as non-shipped; verified shipped via Phase 21/34/35/36/37/38 + live `forgelm --help` invocations. - iso_soc2_deployer_guide.md L149-152 — reviewer wanted `_fingerprint_local_file` → `_hash_file`; current cite is correct (`_fingerprint_local_file` IS the local-file fingerprint helper at compliance.py:510, distinct from `_hash_file` at :466 which is for model-artefact hashing under `model_integrity.json`). - nightly.yml v5→v7 bump — no justification given; v5 is the current pinned version per repo history. - check_anchor_resolution code-fence false-positives — wide-scope change requiring fence-tracking + inline-code stripping; the 36-link advisory baseline is Faz 30 follow-up territory. - _load_tool caching nit — performance gain negligible (tests run in 0.5s) for added complexity. Absorbed (16 fixes): - README.md L57 — HMAC integrity qualifier: "active only when FORGELM_AUDIT_SECRET is set + KMS-managed + rotated between output-dir lifecycles" with cross-link to access_control §3.4. - supply_chain_security.md + -tr.md severity policy — UNKNOWN flipped from "silent" to "single summary `::warning::` with count + report path" matching the round-2 pip-audit/bandit behaviour change. - human-oversight.md (EN) — `forgelm approve --run-id ` → positional `forgelm approve --output-dir --comment "..."` (parser uses positional run_id; --run-id is not a flag). - human-oversight.md / -tr.md — canonicalised event names from mixed underscore/dot to dotted form (`human_approval.required`, `.granted`, `.timeout`) matching the audit catalog. - standards/testing.md — "67 test modules, ~1410 tests" → "~70 modules, ~1413 tests" + added the 4 newest test files (supply_chain_security, check_anchor_resolution, check_bilingual_parity, gdpr_erasure) + clarified the listing is a representative subset (`git ls-files tests/` for full inventory). - gguf-export.md / -tr.md — `--quant fp16` → `--quant f16` (matches parser choices q2_k/q3_k_m/q4_k_m/q5_k_m/q8_0/f16); size 14 GB → 13 GB (matches the quantisation-table f16 row). - iso_soc2_deployer_guide.md / -tr.md — softened "verify-audit requires manifest" to "manifest is the truncate-and-resume detection signal; verifier walks chain end-to-end without it"; recommends backing up both for full tamper-detection coverage. - qms/README-tr.md — 9 internal QMS links updated from English filenames to `-tr.md` mirrors (sop_model_training-tr.md, sop_data_management-tr.md, etc.). - sop_incident_response-tr.md — removed duplicated "güvenlik olaylarını" phrase from §1 lead-in. - statement_of_applicability.md / -tr.md — internal pointer "§3.1–§3.4" corrected to "§2.1–§2.4" (the SoA's own matrix structure); kept the cross-reference to the design doc's §3 Coverage tally in a separate sentence with the absolute path. - compliance_summary.md + access_control-tr.md — replaced brittle `forgelm/compliance.py:104-114` line-range cite with the stable symbol path `AuditLogger.__init__` + `verify_audit_log` (per the Faz 26 line-anchors-to-symbol-references policy applied during the compliance_summary rewrite). - supply_chain_security-tr.md — 3 "See also" links updated from English QMS files to `-tr.md` mirrors (risk_treatment_plan-tr.md, sop_change_management-tr.md, sop_incident_response-tr.md). - sop_change_management.md / -tr.md — staging path `staging_model.` → `final_model.staging.` (matches config.py:482); retention key `evaluation.approval_retention_days` → `retention.staging_ttl_days` (matches config.py:1023); approve invocation flag-form `--run-id ` → positional ` --output-dir ` (matches the shipped parser surface). - ghost-features-analysis-20260502.md L110 — added Status update paragraph reconciling the original "no doctor / no _doctor.py" claim (true at 2026-05-02) with the current state (GH-001 closed via Phase 34; _doctor.py ships and contains the FORGELM_RESUME_TOKEN defensive secret-mask entry referenced at line 604). - check_bandit.py — `_format_issue` micro-opt: only call inside HIGH/MEDIUM branches (LOW is silent + UNDEFINED surfaces as a count, so per-finding formatting on those paths is wasted). - check_anchor_resolution.py — `--exclude` help text now matches actual behaviour (default = `("analysis",)` only; `analysis/code_reviews/` is excluded transitively as a child; passing `--exclude` REPLACES rather than augments). Validation - ruff format / check — clean. - python3 tools/check_bilingual_parity.py --strict — 23/23 OK. - python3 -m pytest --no-cov -q — 1413 passed, 14 skipped. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 2 +- .../ghost-features-analysis-20260502.md | 4 +++- docs/guides/iso_soc2_deployer_guide-tr.md | 13 ++++++++---- docs/guides/iso_soc2_deployer_guide.md | 13 ++++++++---- docs/qms/README-tr.md | 18 ++++++++--------- docs/qms/access_control-tr.md | 6 ++++-- docs/qms/sop_change_management-tr.md | 16 ++++++++------- docs/qms/sop_change_management.md | 10 ++++++---- docs/qms/sop_incident_response-tr.md | 10 +++++----- docs/qms/statement_of_applicability-tr.md | 11 +++++----- docs/qms/statement_of_applicability.md | 11 +++++----- docs/reference/compliance_summary.md | 18 +++++++++-------- docs/reference/supply_chain_security-tr.md | 13 ++++++++---- docs/reference/supply_chain_security.md | 7 ++++++- docs/standards/testing.md | 12 +++++++++-- .../en/compliance/human-oversight.md | 20 ++++++++++++------- docs/usermanuals/en/deployment/gguf-export.md | 6 +++--- .../tr/compliance/human-oversight.md | 8 ++++---- docs/usermanuals/tr/deployment/gguf-export.md | 6 +++--- tools/check_anchor_resolution.py | 19 ++++++++++++------ tools/check_bandit.py | 12 +++++++---- 21 files changed, 146 insertions(+), 89 deletions(-) diff --git a/README.md b/README.md index 58911ac9..72efe421 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,7 @@ ### Enterprise & MLOps - **Config-Driven**: Declarative YAML — built for CI/CD pipelines, not notebooks - **EU AI Act Compliance**: Auto-generated audit trails, data provenance (SHA-256), training manifests -- **ISO 27001 / SOC 2 Type II Alignment**: Software cannot be ISO/SOC 2 *certified* — only organisations can — but ForgeLM produces the audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor explicitly asks for. CycloneDX 1.5 SBOM per release, `pip-audit` nightly, `bandit` CI, append-only HMAC-chained audit log, Article 14 staging gate, Article 15/17 GDPR tooling. See [`docs/guides/iso_soc2_deployer_guide.md`](docs/guides/iso_soc2_deployer_guide.md) for the deployer audit cookbook + 93-control coverage map. +- **ISO 27001 / SOC 2 Type II Alignment**: Software cannot be ISO/SOC 2 *certified* — only organisations can — but ForgeLM produces the audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor explicitly asks for. CycloneDX 1.5 SBOM per release, `pip-audit` nightly, `bandit` CI, append-only HMAC-chained audit log (HMAC integrity is active **only when** `FORGELM_AUDIT_SECRET` is set + KMS-managed + rotated between output-dir lifecycles — see [`docs/qms/access_control.md`](docs/qms/access_control.md) §3.4), Article 14 staging gate, Article 15/17 GDPR tooling. See [`docs/guides/iso_soc2_deployer_guide.md`](docs/guides/iso_soc2_deployer_guide.md) for the deployer audit cookbook + 93-control coverage map. - **Docker**: Official Dockerfile and docker-compose for portable deployment - **Offline / Air-Gapped**: Full operation without internet for regulated industries - **JSON Output**: Machine-readable results with `--output-format json` for pipeline integration diff --git a/docs/analysis/code_reviews/ghost-features-analysis-20260502.md b/docs/analysis/code_reviews/ghost-features-analysis-20260502.md index 2c5bcabb..5052b4b2 100644 --- a/docs/analysis/code_reviews/ghost-features-analysis-20260502.md +++ b/docs/analysis/code_reviews/ghost-features-analysis-20260502.md @@ -107,7 +107,9 @@ **Documented behavior:** Environment check — verifies Python version, PyTorch, CUDA availability, GPU detection, optional dep install status. `--offline` variant checks cached resources for air-gapped deployments. -**Code reality:** No `doctor` in `forgelm/cli/_parser.py`, no `_doctor.py` in `forgelm/cli/subcommands/`. Running `forgelm doctor` exits with "unrecognized arguments". +**Code reality (2026-05-02 — original analysis):** No `doctor` in `forgelm/cli/_parser.py`, no `_doctor.py` in `forgelm/cli/subcommands/`. Running `forgelm doctor` exits with "unrecognized arguments". + +**Status update (post Wave 2a):** GH-001 closed via Phase 34. `forgelm doctor` ships in v0.5.x with `_doctor.py` at `forgelm/cli/subcommands/_doctor.py`; `forgelm doctor --help` lists Python / CUDA / GPU / extras / HF auth / disk / `FORGELM_OPERATOR` checks. The line-604 mention of `_doctor.py` (the `FORGELM_RESUME_TOKEN` defensive secret-mask entry) refers to this same shipped file — both line 110 and line 604 describe the same artefact at different points in time. **Closure plan:** Not mentioned in any phase. **Roadmap:** Not mentioned in any file under `docs/roadmap/`. diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index 37c7486d..fafdf908 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -258,10 +258,15 @@ Operatörlerin ilk denetimlerinde yanlış yaptığı şeyler: güveniyoruz" savunulabilir bir pozisyon değildir. Haftalık cron + sıfır-olmayan exit'e alarm zamanlayın; alarm geçmişini denetçiye gösterin. -4. **Manifest sidecar'ı unutmak.** `audit_log.jsonl` tek başına - yetersizdir — `forgelm verify-audit` truncate-and-resume - tampering'ini tespit etmek için `audit_log.manifest.json` - sidecar'ı gerektirir. İKİSİNİ DE backup'layın. +4. **Manifest sidecar'ı unutmak.** `forgelm verify-audit` zinciri + sidecar olmadan da uçtan uca yürür (manifest temel chain check + için kesin gerekli değildir), ama **truncate-and-resume + tampering**'ini ortaya çıkaran şey sidecar'dır — mevcut olduğunda + verifier, manifest'in pin'lenmiş first-entry SHA-256 + run_id + alanlarını canlı log'un ilk satırıyla çapraz kontrol eder. + Manifest olmadan bu saldırı sınıfı sessizce iner. Tam + tamper-detection kapsaması için **ikisini de** aynı write-once + substrate'e backup'layın. 5. **Üretim eğitiminde `auto_revert` yok.** Her zaman yeşil eğitime bahis koyuyorsanız, regulator-reportable incident'a bir safety-classifier degradation'ı uzaktasınız. `auto_revert: diff --git a/docs/guides/iso_soc2_deployer_guide.md b/docs/guides/iso_soc2_deployer_guide.md index c1df4278..a1fdb80a 100644 --- a/docs/guides/iso_soc2_deployer_guide.md +++ b/docs/guides/iso_soc2_deployer_guide.md @@ -259,10 +259,15 @@ Things deployers get wrong on their first audit: chain implicitly" is not a defensible position. Schedule the weekly cron + alert on non-zero exit; have the alert history to show the auditor. -4. **Forgetting the manifest sidecar.** `audit_log.jsonl` alone is - insufficient — `forgelm verify-audit` requires the - `audit_log.manifest.json` sidecar to detect truncate-and-resume - tampering. Back BOTH up. +4. **Forgetting the manifest sidecar.** `forgelm verify-audit` + walks the chain end-to-end without the sidecar (the manifest is + not strictly required for the basic chain check), but the + sidecar is what surfaces **truncate-and-resume tampering** — + when present, the verifier cross-checks the manifest's pinned + first-entry SHA-256 + run_id against the live log's first line. + Without the manifest, that class of attack lands silently. + Back **both** files up to the same write-once substrate for + full tamper-detection coverage. 5. **No `auto_revert` on production training.** If you're betting on always-green training, you're a single safety-classifier degradation away from a regulator-reportable incident. Enable diff --git a/docs/qms/README-tr.md b/docs/qms/README-tr.md index 71e2ed0f..30b0f0ef 100644 --- a/docs/qms/README-tr.md +++ b/docs/qms/README-tr.md @@ -6,15 +6,15 @@ | Şablon | EU AI Act Referansı | Amaç | |----------|-------------------|---------| -| [Model Eğitimi SOP](sop_model_training.md) | Md. 17(1)(b)(c)(d) | Fine-tuning onay, yürütme ve doğrulama için standart prosedür | -| [Veri Yönetimi SOP](sop_data_management.md) | Md. 17(1)(f), Md. 10 | Veri toplama, annotation, kalite güvencesi ve yönetişim | -| [Olay Yanıtı SOP](sop_incident_response.md) | Md. 17(1)(h)(i) | Model arızalarını, güvenlik olaylarını ve düzeltici eylemleri yönetme | -| [Değişim Yönetimi SOP](sop_change_management.md) | Md. 17(1)(b)(c) | Versiyonlama, inceleme, onay ve geri alma prosedürleri | -| [Roller & Sorumluluklar](roles_responsibilities.md) | Md. 17(1)(m) | AI Officer, Data Steward, ML Engineer, Compliance Officer rolleri | -| [Erişim Kontrolü](access_control.md) | Md. 17(1)(c) + ISO A.5.15-A.8.5 | Operatör kimliği + secret rotasyonu (Wave 4 / Faz 23) | -| [Encryption at Rest](encryption_at_rest.md) | Md. 17(1)(c) + ISO A.5.33 + A.8.10 + A.8.24 | Substrate-side şifreleme rehberi (Wave 4 / Faz 23) | -| [Risk Treatment Plan](risk_treatment_plan.md) | Md. 17(1)(c) + ISO A.5.7-A.8.30 | ISO 27005 risk register şablonu (Wave 4 / Faz 23) | -| [Statement of Applicability](statement_of_applicability.md) | Md. 17(1)(c) + ISO 6.1.3 d) | 93-kontrol uygulanabilirlik matrisi (Wave 4 / Faz 23) | +| [Model Eğitimi SOP](sop_model_training-tr.md) | Md. 17(1)(b)(c)(d) | Fine-tuning onay, yürütme ve doğrulama için standart prosedür | +| [Veri Yönetimi SOP](sop_data_management-tr.md) | Md. 17(1)(f), Md. 10 | Veri toplama, annotation, kalite güvencesi ve yönetişim | +| [Olay Yanıtı SOP](sop_incident_response-tr.md) | Md. 17(1)(h)(i) | Model arızalarını, güvenlik olaylarını ve düzeltici eylemleri yönetme | +| [Değişim Yönetimi SOP](sop_change_management-tr.md) | Md. 17(1)(b)(c) | Versiyonlama, inceleme, onay ve geri alma prosedürleri | +| [Roller & Sorumluluklar](roles_responsibilities-tr.md) | Md. 17(1)(m) | AI Officer, Data Steward, ML Engineer, Compliance Officer rolleri | +| [Erişim Kontrolü](access_control-tr.md) | Md. 17(1)(c) + ISO A.5.15-A.8.5 | Operatör kimliği + secret rotasyonu (Wave 4 / Faz 23) | +| [Encryption at Rest](encryption_at_rest-tr.md) | Md. 17(1)(c) + ISO A.5.33 + A.8.10 + A.8.24 | Substrate-side şifreleme rehberi (Wave 4 / Faz 23) | +| [Risk Treatment Plan](risk_treatment_plan-tr.md) | Md. 17(1)(c) + ISO A.5.7-A.8.30 | ISO 27005 risk register şablonu (Wave 4 / Faz 23) | +| [Statement of Applicability](statement_of_applicability-tr.md) | Md. 17(1)(c) + ISO 6.1.3 d) | 93-kontrol uygulanabilirlik matrisi (Wave 4 / Faz 23) | ## Nasıl Kullanılır diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md index ff1776ad..17313f47 100644 --- a/docs/qms/access_control-tr.md +++ b/docs/qms/access_control-tr.md @@ -79,8 +79,10 @@ ID'leri sonsuza kadar chain'de kalır. Bir operatör ayrıldığında: ### 3.4 `FORGELM_AUDIT_SECRET` rotasyonu HMAC-chain imzalama anahtarı **audit koşumu başına** -`SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` olarak türetilir -(birleştirme, bkz. `forgelm/compliance.py:104-114`). Not: +`AuditLogger.__init__` içinde `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` +olarak türetilir (birleştirme; verifier +`forgelm.compliance.verify_audit_log` aynı türetimi byte-byte +yansıtır). Not: `/.forgelm_audit_salt`'ta yazılı per-output-dir salt **ayrı bir konu**'dur — `forgelm purge` / `forgelm reverse-pii` event'lerindeki tanımlayıcı hash'lemesini salt'lar diff --git a/docs/qms/sop_change_management-tr.md b/docs/qms/sop_change_management-tr.md index 16c1b237..be1f2b11 100644 --- a/docs/qms/sop_change_management-tr.md +++ b/docs/qms/sop_change_management-tr.md @@ -96,16 +96,18 @@ Yeni eğitim koşumu üreten değişimler için (yapılandırma-yalnız değişimler vs.), **Madde 14 staging gate** operatörün in-pipeline CAB'sıdır: -1. CI eğitim job'u modeli `/staging_model./`'e - yerleştirir. +1. CI eğitim job'u modeli + `/final_model.staging./`'e yerleştirir. 2. `human_approval.required` audit event yangın eder. -3. Bir reviewer (eğiten DEĞİL) `forgelm approve --run-id ` - çalıştırır. +3. Bir reviewer (eğiten DEĞİL) + `forgelm approve --output-dir ` çalıştırır + (not: `run_id` positional'dır — `--run-id` flag'i yoktur). 4. `human_approval.granted` audit event yangın eder; model - `/final_model/`'e promote olur. + atomik rename ile `/final_model/`'e promote olur. 5. Reddedilirse `human_approval.rejected` event yangın eder; model - retention süresi dolana kadar staging'de kalır - (`evaluation.approval_retention_days`). + forensic inceleme için `final_model.staging./`'de kalır, + retention süresi dolana kadar (`retention.staging_ttl_days` — + config'te). Bu, operatöre SOC 2 CC8.1 kanıtı verir: her model promotion attribute, çift kontrollü ve forensic olarak kayıtlıdır. diff --git a/docs/qms/sop_change_management.md b/docs/qms/sop_change_management.md index 42003c10..8162314b 100644 --- a/docs/qms/sop_change_management.md +++ b/docs/qms/sop_change_management.md @@ -94,13 +94,15 @@ For changes that produce a new training run (vs. configuration-only changes), the **Article 14 staging gate** is the deployer's in-pipeline CAB: -1. CI training job lands the model in `/staging_model./`. +1. CI training job lands the model in `/final_model.staging./`. 2. `human_approval.required` audit event fires. -3. A reviewer (NOT the trainer) runs `forgelm approve --run-id `. +3. A reviewer (NOT the trainer) runs `forgelm approve --output-dir ` + (note: `run_id` is positional — `--run-id` is not a flag). 4. `human_approval.granted` audit event fires; model promotes to - `/final_model/`. + `/final_model/` via atomic rename. 5. If rejected, `human_approval.rejected` event fires; model stays - in staging until retention expires (`evaluation.approval_retention_days`). + in `final_model.staging./` for forensic review until + retention expires (`retention.staging_ttl_days` in the config). This gives a deployer SOC 2 CC8.1 evidence: every model promotion is attributed, dual-controlled, and forensically recorded. diff --git a/docs/qms/sop_incident_response-tr.md b/docs/qms/sop_incident_response-tr.md index 4653f0cf..8358ad4a 100644 --- a/docs/qms/sop_incident_response-tr.md +++ b/docs/qms/sop_incident_response-tr.md @@ -8,11 +8,11 @@ ## 1. Amaç Dağıtılan fine-tuned modeller için güvenlik olaylarını, model -arızalarını, **güvenlik olaylarını** ve düzeltici eylemleri -yönetme prosedürünü tanımla. Wave 4 / Faz 23 genişletmesi: §4 -güvenlik-olay playbook'unu (audit-chain integrity, credential leak, -supply-chain CVE, webhook target compromise, GDPR DSAR'lar) mevcut -AI-safety olay akışı yanında kapsar. +arızalarını ve düzeltici eylemleri yönetme prosedürünü tanımla. +Wave 4 / Faz 23 genişletmesi: §4 güvenlik-olay playbook'unu +(audit-chain integrity, credential leak, supply-chain CVE, webhook +target compromise, GDPR DSAR'lar) mevcut AI-safety olay akışı +yanında kapsar. ## 2. Olay Sınıflandırması diff --git a/docs/qms/statement_of_applicability-tr.md b/docs/qms/statement_of_applicability-tr.md index 74b90930..5e7148bc 100644 --- a/docs/qms/statement_of_applicability-tr.md +++ b/docs/qms/statement_of_applicability-tr.md @@ -152,11 +152,12 @@ ForgeLM-spesifik kontrol envanteriyle ilgilidir. | A.8 Technological | 34 | 34 | 0 | 8 | 19 | | **Toplam** | **93** | **93 (operatör ISMS)** | **14 (ForgeLM-spesifik)** | **11** | **48** | -§3.1–§3.4 (bu SoA'nın özetlediği design-doc tablosu) row-by-row -yeniden sayım. Tema başına tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 -/ 3 OOS; A.7: 0 / 0 / 14 OOS; A.8: 8 / 19 / 7 OOS — toplam 11 `FL` -+ 48 `FL-helps` + 34 OOS = 93. Design doc'un `§3` "Coverage tally" -paragrafıyla çapraz kontrol; ikisi eşleşmek zorunda. +Yukarıdaki §2.1–§2.4 (SoA matrisi) row-by-row yeniden sayım. Tema +başına tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 / 3 OOS; A.7: 0 / +0 / 14 OOS; A.8: 8 / 19 / 7 OOS — toplam 11 `FL` + 48 `FL-helps` ++ 34 OOS = 93. Design doc'un §3 "Coverage tally" paragrafıyla +(`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`) +çapraz kontrol; ikisi eşleşmek zorunda. ## 4. İnceleme diff --git a/docs/qms/statement_of_applicability.md b/docs/qms/statement_of_applicability.md index 6fedb416..eef5fbfe 100644 --- a/docs/qms/statement_of_applicability.md +++ b/docs/qms/statement_of_applicability.md @@ -151,11 +151,12 @@ ForgeLM-specific control inventory. | A.8 Technological | 34 | 34 | 0 | 8 | 19 | | **Total** | **93** | **93 (deployer ISMS)** | **14 (ForgeLM-specific)** | **11** | **48** | -Row-by-row recount of §3.1–§3.4 (the design-doc table this SoA -summarises). Per-theme tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 / 3 -OOS; A.7: 0 / 0 / 14 OOS; A.8: 8 / 19 / 7 OOS — sums to 11 `FL` + -48 `FL-helps` + 34 OOS = 93. Cross-check the design doc's `§3` -"Coverage tally" paragraph; the two must match. +Row-by-row recount of §2.1–§2.4 above (the SoA matrix). Per-theme +tally — A.5: 3 / 24 / 10 OOS; A.6: 0 / 5 / 3 OOS; A.7: 0 / 0 / 14 +OOS; A.8: 8 / 19 / 7 OOS — sums to 11 `FL` + 48 `FL-helps` + 34 OOS += 93. Cross-check the design doc's §3 "Coverage tally" paragraph +(`docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`); +the two must match. ## 4. Review diff --git a/docs/reference/compliance_summary.md b/docs/reference/compliance_summary.md index ddae7521..4dc0e994 100644 --- a/docs/reference/compliance_summary.md +++ b/docs/reference/compliance_summary.md @@ -88,14 +88,16 @@ What the regulator asks vs. how ForgeLM answers: - Implementation: `forgelm.compliance.AuditLogger` — JSON Lines append-only log at `/audit_log.jsonl`, HMAC-chained - with a per-run signing key derived as - `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` (see - `forgelm/compliance.py:104-114`). The per-output-dir salt at - `/.forgelm_audit_salt` is a **distinct primitive** — - it salts identifier hashing in `forgelm purge` / - `forgelm reverse-pii` events (`_purge._resolve_salt`) and does - NOT participate in chain-key derivation. Genesis manifest sidecar - (`audit_log.manifest.json`) refuses truncate-and-resume tampering. + with a per-run signing key derived inside + `AuditLogger.__init__` as `SHA-256(FORGELM_AUDIT_SECRET ‖ run_id)` + (the writer at `AuditLogger.log_event` and the verifier at + `forgelm.compliance.verify_audit_log` mirror the same derivation). + The per-output-dir salt at `/.forgelm_audit_salt` + is a **distinct primitive** — it salts identifier hashing in + `forgelm purge` / `forgelm reverse-pii` events + (`_purge._resolve_salt`) and does NOT participate in chain-key + derivation. Genesis manifest sidecar (`audit_log.manifest.json`) + refuses truncate-and-resume tampering. - Verification: `forgelm verify-audit [--require-hmac]` validates the chain end-to-end; exits 0/1/2/3. diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md index ba4b379a..5c1d5dc2 100644 --- a/docs/reference/supply_chain_security-tr.md +++ b/docs/reference/supply_chain_security-tr.md @@ -90,7 +90,12 @@ Wave 4 / Faz 23 nightly workflow'a `pip-audit` ekler. Davranış: notify-failure job'u üzerinden GitHub issue açar). - **MEDIUM / MODERATE** → `::warning::` annotation; nightly yeşil kalır. - - **LOW / UNKNOWN** → sessiz. + - **LOW** → sessiz. + - **UNKNOWN** → adet + rapor path'ini listeleyen tek bir özet + `::warning::` annotation'ı; operator-triage SRE'leri workflow + YAML'ını dolaşmadan artefactı grep'leyebilsin diye. Nightly + yeşil kalır (pip-audit'in JSON'u severity taşımadığı için + gerçek raporlarda çoğu bulgu buraya düşer). - OSV / GHSA veritabanlarını kullanır (pip-audit varsayılanı). Operatörler aynı tooling'i lokalde kurar: @@ -187,9 +192,9 @@ getirmek için `pip-audit` ve `bandit`'i çeker. - [`../guides/iso_soc2_deployer_guide-tr.md`](../guides/iso_soc2_deployer_guide-tr.md) — operatör denetim cookbook'u. - [`iso27001_control_mapping-tr.md`](iso27001_control_mapping-tr.md) — ISO 27001 Annex A kontrolleri × ForgeLM kanıtı. - [`soc2_trust_criteria_mapping-tr.md`](soc2_trust_criteria_mapping-tr.md) — SOC 2 Trust Services Criteria × ForgeLM kanıtı. -- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. -- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook. -- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook. +- [`../qms/risk_treatment_plan-tr.md`](../qms/risk_treatment_plan-tr.md) — pre-populated risk register. +- [`../qms/sop_change_management-tr.md`](../qms/sop_change_management-tr.md) — change management runbook. +- [`../qms/sop_incident_response-tr.md`](../qms/sop_incident_response-tr.md) — incident response runbook. - `tools/generate_sbom.py` — CycloneDX 1.5 emitter. - `tools/check_pip_audit.py` — pip-audit severity gate. - `tools/check_bandit.py` — bandit severity gate. diff --git a/docs/reference/supply_chain_security.md b/docs/reference/supply_chain_security.md index a4a93c19..4879a7b2 100644 --- a/docs/reference/supply_chain_security.md +++ b/docs/reference/supply_chain_security.md @@ -90,7 +90,12 @@ Wave 4 / Faz 23 adds `pip-audit` to the nightly workflow. Behaviour: via the existing notify-failure job). - **MEDIUM / MODERATE** → `::warning::` annotation; nightly stays green. - - **LOW / UNKNOWN** → silent. + - **LOW** → silent. + - **UNKNOWN** → single summary `::warning::` annotation listing + the count + report path, so operator-triage SREs can grep the + artefact without walking the workflow YAML; nightly stays green + (pip-audit's JSON does not carry severity, so most findings land + here on real reports). - Uses the OSV / GHSA databases (pip-audit's default). Operators install the same tooling locally: diff --git a/docs/standards/testing.md b/docs/standards/testing.md index 66a867c5..426476f0 100644 --- a/docs/standards/testing.md +++ b/docs/standards/testing.md @@ -5,7 +5,10 @@ ## Layout -Current structure (67 test modules, one per feature area; ~1410 tests): +Current structure (post Wave 4 round-2 absorption: ~70 test modules, +one per feature area; ~1413 tests collected). The tree below is a +**representative subset** — see `git ls-files tests/` for the full +inventory: ``` tests/ @@ -33,7 +36,12 @@ tests/ ├── test_cost_estimation.py # GPU cost heuristics ├── test_webhook.py # Slack/Teams notifier ├── test_distributed.py # DeepSpeed / FSDP config -└── test_data_edge_cases.py # Malformed datasets, edge cases +├── test_data_edge_cases.py # Malformed datasets, edge cases +├── test_supply_chain_security.py # Wave 4 / Faz 23 — pip-audit + bandit + SBOM +├── test_check_anchor_resolution.py # Wave 4 / Faz 26 — markdown anchor resolver +├── test_check_bilingual_parity.py # Bilingual EN/TR mirror parity +├── test_gdpr_erasure.py # GDPR Article 17 (forgelm purge) +└── … ``` **Rules:** diff --git a/docs/usermanuals/en/compliance/human-oversight.md b/docs/usermanuals/en/compliance/human-oversight.md index e7e6c82a..444d4fc7 100644 --- a/docs/usermanuals/en/compliance/human-oversight.md +++ b/docs/usermanuals/en/compliance/human-oversight.md @@ -19,10 +19,10 @@ sequenceDiagram Train->>Eval: Training complete, run eval Eval->>Eval: Benchmarks + safety pass - Eval->>Audit: Append "human_approval_request" + Eval->>Audit: Append "human_approval.required" Eval->>Approver: Webhook + structured request Approver-->>Eval: Sign approval (CLI / webhook callback) - Eval->>Audit: Append "human_approval_granted" with signature + Eval->>Audit: Append "human_approval.granted" with signature Eval->>Output: Promote checkpoint ``` @@ -75,11 +75,17 @@ The trainer halts and posts the artifact bundle to your webhook. Your system han The supported approval mechanism in v0.5.5 is the CLI subcommand pair `forgelm approve` / `forgelm reject`: ```bash -forgelm approvals --pending # list runs awaiting approval -forgelm approve --run-id # promote staging → final_model -forgelm reject --run-id --reason "..." # discard the staged model +forgelm approvals --pending --output-dir # list runs awaiting approval +forgelm approve --output-dir --comment "..." # promote staging → final_model +forgelm reject --output-dir --comment "..." # discard the staged model ``` +**Note:** `approve` and `reject` take a positional `run_id` (not +`--run-id`); `--comment "..."` is the reviewer note that lands in +the `human_approval.granted` / `human_approval.rejected` event. +`--output-dir ` is required and points at the training output +directory containing `audit_log.jsonl` and `final_model.staging/`. + Each invocation requires `FORGELM_OPERATOR` (the approver's identity) and writes a `human_approval.granted` / `human_approval.rejected` event to the chain. Self-service "promote this run" automation is roadmapped for v0.6.0+ Pro CLI (Phase 13 in the public roadmap); until then the CLI gate is the audit-grade interface. ## What's in an approval signature @@ -90,7 +96,7 @@ Every approval (or rejection) appends to `audit_log.jsonl`: { "ts": "2026-04-29T15:18:42Z", "seq": 87, - "event": "human_approval_granted", + "event": "human_approval.granted", "run_id": "abc123", "reviewer": "Cemil Ilik ", "role": "ml-compliance-lead", @@ -120,7 +126,7 @@ Each approver runs the CLI command independently. Promotion happens after the qu After `timeout_hours`, an unsigned run auto-fails with exit code 4 + a structured event: ```json -{"event": "human_approval_timeout", "expired_at": "2026-04-30T14:33:10Z"} +{"event": "human_approval.timeout", "expired_at": "2026-04-30T14:33:10Z"} ``` Default is 48 hours. Set to 0 for "no timeout — wait forever" (not recommended in CI). diff --git a/docs/usermanuals/en/deployment/gguf-export.md b/docs/usermanuals/en/deployment/gguf-export.md index 60371652..9cabe158 100644 --- a/docs/usermanuals/en/deployment/gguf-export.md +++ b/docs/usermanuals/en/deployment/gguf-export.md @@ -112,13 +112,13 @@ $ ./main -m model.q4_k_m.gguf -p "Hello, how are you?" -n 256 ## Direct conversion (no quantisation) -For the rare case where you want fp16 GGUF (e.g. for a quantisation-sensitive inference engine): +For the rare case where you want full-precision GGUF (e.g. for a quantisation-sensitive inference engine): ```shell -$ forgelm export ./checkpoints/run --output model.gguf --quant fp16 +$ forgelm export ./checkpoints/run --output model.gguf --quant f16 ``` -The result is full-precision GGUF, ~14 GB for a 7B model. +The result is full-precision GGUF, ~13 GB for a 7B model. ## Common pitfalls diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index 3e68b266..9c68d5e7 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -19,10 +19,10 @@ sequenceDiagram Train->>Eval: Eğitim tamam, eval koştur Eval->>Eval: Benchmark + güvenlik geçti - Eval->>Audit: "human_approval_request" ekle + Eval->>Audit: "human_approval.required" ekle Eval->>Approver: Webhook + yapılandırılmış istek Approver-->>Eval: Onayı imzala (CLI / webhook callback) - Eval->>Audit: İmzalı "human_approval_granted" ekle + Eval->>Audit: İmzalı "human_approval.granted" ekle Eval->>Output: Checkpoint'i terfi ettir ``` @@ -90,7 +90,7 @@ Her onay (veya red) `audit_log.jsonl`'a eklenir: { "ts": "2026-04-29T15:18:42Z", "seq": 87, - "event": "human_approval_granted", + "event": "human_approval.granted", "run_id": "abc123", "reviewer": "Cemil Ilik ", "role": "ml-compliance-lead", @@ -120,7 +120,7 @@ Her onaylayıcı CLI komutunu bağımsız çalıştırır. Quorum imzaladıktan `timeout_hours` sonrası imzasız koşu yapılandırılmış olayla exit 4 + auto-fail: ```json -{"event": "human_approval_timeout", "expired_at": "2026-04-30T14:33:10Z"} +{"event": "human_approval.timeout", "expired_at": "2026-04-30T14:33:10Z"} ``` Varsayılan 48 saat. "Timeout yok — sonsuza kadar bekle" için 0 (CI'da önerilmez). diff --git a/docs/usermanuals/tr/deployment/gguf-export.md b/docs/usermanuals/tr/deployment/gguf-export.md index d5269b75..6d6a240a 100644 --- a/docs/usermanuals/tr/deployment/gguf-export.md +++ b/docs/usermanuals/tr/deployment/gguf-export.md @@ -112,13 +112,13 @@ $ ./main -m model.q4_k_m.gguf -p "Merhaba, nasılsın?" -n 256 ## Doğrudan dönüştürme (kuantizasyon yok) -Nadir olarak fp16 GGUF isterseniz (kuantizasyon-duyarlı bir inference engine için): +Nadir olarak full-precision GGUF isterseniz (kuantizasyon-duyarlı bir inference engine için): ```shell -$ forgelm export ./checkpoints/run --output model.gguf --quant fp16 +$ forgelm export ./checkpoints/run --output model.gguf --quant f16 ``` -Sonuç full-precision GGUF, 7B model için ~14 GB. +Sonuç full-precision GGUF, 7B model için ~13 GB. ## Sık hatalar diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py index 82ece048..8bf54ff2 100644 --- a/tools/check_anchor_resolution.py +++ b/tools/check_anchor_resolution.py @@ -313,9 +313,12 @@ def _build_argparser() -> argparse.ArgumentParser: default=None, help=( "Directory (relative to scope-dir) to skip. Repeatable. " - "Default exclude list strips ``analysis/`` (gitignored research " - "tree referencing local-only paths) and ``code_reviews/`` " - "(internal review docs). Pass empty list to scan everything." + "Default exclude list strips ``analysis/`` only — the " + "gitignored research tree references local-only paths and " + "is excluded by default; ``analysis/code_reviews/`` is a " + "subdirectory that gets caught transitively. Passing " + "``--exclude`` REPLACES the default (e.g. pass " + "``--exclude ''`` or omit the default to scan analysis/ too)." ), ) parser.add_argument( @@ -339,9 +342,13 @@ def _build_argparser() -> argparse.ArgumentParser: def _resolve_excludes(scope_dir: Path, exclude_arg: list[str] | None) -> tuple[Path, ...]: # Default exclude list — `analysis/` is gitignored research with - # local-only path references; `code_reviews/` would be too were - # it not partially re-allowed in `.gitignore`, but its closure-plan - # docs cite real files at line-anchored locations that drift. + # local-only path references; `code_reviews/` lives under it and + # is caught transitively (its closure-plan docs cite real files at + # line-anchored locations that drift). + # + # Passing --exclude REPLACES the default rather than augmenting it + # (call sites that need additional excludes pass the full list, + # including ``analysis``); this matches the help-text contract. specs: tuple[str, ...] if exclude_arg is None: specs = ("analysis",) diff --git a/tools/check_bandit.py b/tools/check_bandit.py index 3f029469..685d15ed 100644 --- a/tools/check_bandit.py +++ b/tools/check_bandit.py @@ -87,7 +87,12 @@ def _extract_results(report: dict[str, Any]) -> list[Any] | int: def _classify_issues(results: list[Any]) -> tuple[list[str], list[str], int]: - """Bucket issues into (high-lines, medium-lines, undefined-count).""" + """Bucket issues into (high-lines, medium-lines, undefined-count). + + ``_format_issue`` is only called for the HIGH / MEDIUM branches — + LOW is silent and UNDEFINED surfaces as a single summary count, so + formatting LOW / UNDEFINED issues per-finding would be wasted work. + """ high: list[str] = [] medium: list[str] = [] undefined_count = 0 @@ -95,11 +100,10 @@ def _classify_issues(results: list[Any]) -> tuple[list[str], list[str], int]: if not isinstance(issue, dict): continue severity = (issue.get("issue_severity") or _UNDEFINED).upper() - line = _format_issue(issue) if severity == _HIGH: - high.append(line) + high.append(_format_issue(issue)) elif severity == _MED: - medium.append(line) + medium.append(_format_issue(issue)) elif severity == _UNDEFINED: undefined_count += 1 # LOW is silent; the raw JSON remains in artefacts. From dcd37b5df3479639e88fcd51c06006164011bc55 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 10:20:38 +0300 Subject: [PATCH 12/12] fix(wave-4-round4): absorb 7 inline review findings (TR mirror drift + anchor checker hardening) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 7 findings verified valid against current code; round 3 caught the EN side of the approve/reject + staging-path drift but missed several TR mirror sites + the compliance_summary EN reference. Documentation (5) - supply_chain_security-tr.md L3 — "dokuman" → "doküman" (TR spelling). - usermanuals/tr/compliance/human-oversight.md L77-80 — round 3 fixed the EN CLI block but missed the TR mirror; now uses the shipped contract: forgelm approvals --pending --output-dir forgelm approve --output-dir --comment "..." forgelm reject --output-dir --comment "..." + explicit note about positional run_id. - qms/access_control-tr.md L190-193 + qms/access_control.md L190 — segregation-of-duties Job 2 example used `forgelm approve --run-id `; replaced with positional `forgelm approve --output-dir ` (parser contract). Both EN + TR. - reference/compliance_summary.md L106-109 — Article 14 staging description used the deprecated `staging_model./` path AND the broken `--run-id ` flag form. Now uses `final_model.staging./` (matches `forgelm/config.py:482`) and `forgelm approve --output-dir ` (positional). Mirror of round-2/3 fixes propagated to the compliance_summary surface. - guides/iso_soc2_deployer_guide-tr.md L282-291 — "Bkz." list linked to 9 EN files when TR mirrors exist. Updated all 9 to `-tr.md` variants (qms × 6, reference × 4 + audit_event_catalog). Left the analysis design doc link as EN (no TR mirror exists for research artefacts under `docs/analysis/`). Tooling (2) - tools/check_anchor_resolution.py::_locate_target — both candidate paths (`target` from source.parent + `alt` from repo_root) are now constrained to live under `repo_root.resolve()` via `Path.relative_to`. A `../../../etc/passwd`-style path traversal that resolves outside the repo tree is now treated as broken rather than silently followed. Defensive correctness fix; no shipped doc trips the new branch (anchor-checker baseline still reports the same 36 broken links). - tools/check_anchor_resolution.py::_resolve_excludes — empty / whitespace-only entries in `--exclude` are now filtered (an `--exclude ""` previously resolved to `scope_dir` itself and silently excluded the entire tree). When the cleaned list is empty (all entries filtered out), falls back to the default ("analysis",) rather than scanning everything. Validation - ruff format / check — clean. - python3 tools/check_bilingual_parity.py --strict — 23/23 OK. - python3 -m pytest tests/test_check_anchor_resolution.py — 31/31 pass. - python3 -m pytest --no-cov -q — 1413 passed, 14 skipped. - Anchor advisory baseline unchanged at 36 (Faz 30 territory). Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/guides/iso_soc2_deployer_guide-tr.md | 20 +++++------ docs/qms/access_control-tr.md | 4 +-- docs/qms/access_control.md | 2 +- docs/reference/compliance_summary.md | 8 +++-- docs/reference/supply_chain_security-tr.md | 2 +- .../tr/compliance/human-oversight.md | 12 +++++-- tools/check_anchor_resolution.py | 33 +++++++++++++++---- 7 files changed, 55 insertions(+), 26 deletions(-) diff --git a/docs/guides/iso_soc2_deployer_guide-tr.md b/docs/guides/iso_soc2_deployer_guide-tr.md index fafdf908..3f23a3b6 100644 --- a/docs/guides/iso_soc2_deployer_guide-tr.md +++ b/docs/guides/iso_soc2_deployer_guide-tr.md @@ -280,13 +280,13 @@ Operatörlerin ilk denetimlerinde yanlış yaptığı şeyler: ## Bkz. - [`../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md`](../analysis/code_reviews/iso27001-soc2-alignment-202605052315.md) — tam tasarım gerekçesi + 93 kontrol haritası. -- [`../qms/encryption_at_rest.md`](../qms/encryption_at_rest.md) — substrate-side şifreleme rehberi. -- [`../qms/access_control.md`](../qms/access_control.md) — operator identity + secrets management. -- [`../qms/risk_treatment_plan.md`](../qms/risk_treatment_plan.md) — pre-populated risk register. -- [`../qms/statement_of_applicability.md`](../qms/statement_of_applicability.md) — 93-kontrol SoA matrisi. -- [`../qms/sop_incident_response.md`](../qms/sop_incident_response.md) — incident response runbook (Wave 4 genişletme). -- [`../qms/sop_change_management.md`](../qms/sop_change_management.md) — change management runbook (Wave 4 genişletme). -- [`../reference/iso27001_control_mapping.md`](../reference/iso27001_control_mapping.md) — ISO 27001:2022 Annex A kontrolleri × ForgeLM kanıtı. -- [`../reference/soc2_trust_criteria_mapping.md`](../reference/soc2_trust_criteria_mapping.md) — SOC 2 Trust Services Criteria × ForgeLM kanıtı. -- [`../reference/supply_chain_security.md`](../reference/supply_chain_security.md) — SBOM + pip-audit + bandit overview. -- [`../reference/audit_event_catalog.md`](../reference/audit_event_catalog.md) — audit-event vocabulary. +- [`../qms/encryption_at_rest-tr.md`](../qms/encryption_at_rest-tr.md) — substrate-side şifreleme rehberi. +- [`../qms/access_control-tr.md`](../qms/access_control-tr.md) — operator identity + secrets management. +- [`../qms/risk_treatment_plan-tr.md`](../qms/risk_treatment_plan-tr.md) — pre-populated risk register. +- [`../qms/statement_of_applicability-tr.md`](../qms/statement_of_applicability-tr.md) — 93-kontrol SoA matrisi. +- [`../qms/sop_incident_response-tr.md`](../qms/sop_incident_response-tr.md) — incident response runbook (Wave 4 genişletme). +- [`../qms/sop_change_management-tr.md`](../qms/sop_change_management-tr.md) — change management runbook (Wave 4 genişletme). +- [`../reference/iso27001_control_mapping-tr.md`](../reference/iso27001_control_mapping-tr.md) — ISO 27001:2022 Annex A kontrolleri × ForgeLM kanıtı. +- [`../reference/soc2_trust_criteria_mapping-tr.md`](../reference/soc2_trust_criteria_mapping-tr.md) — SOC 2 Trust Services Criteria × ForgeLM kanıtı. +- [`../reference/supply_chain_security-tr.md`](../reference/supply_chain_security-tr.md) — SBOM + pip-audit + bandit overview. +- [`../reference/audit_event_catalog-tr.md`](../reference/audit_event_catalog-tr.md) — audit-event vocabulary. diff --git a/docs/qms/access_control-tr.md b/docs/qms/access_control-tr.md index 17313f47..0bd36de7 100644 --- a/docs/qms/access_control-tr.md +++ b/docs/qms/access_control-tr.md @@ -187,9 +187,9 @@ olmasını gerektirir (görev ayrılığı — ISO A.5.3, SOC 2 CC1.5): FORGELM_OPERATOR="gha:Acme/pipelines:training:run-42" \ forgelm --config config.yaml -# Job 2 — onay (insan reviewer identity) +# Job 2 — onay (insan reviewer identity; positional run_id) FORGELM_OPERATOR="alice@acme.example" \ - forgelm approve --run-id + forgelm approve --output-dir ``` ForgeLM iki kimliğin farklı olduğunu zorlamaz — bu operatör-tarafı diff --git a/docs/qms/access_control.md b/docs/qms/access_control.md index fa058d4d..a745e4d1 100644 --- a/docs/qms/access_control.md +++ b/docs/qms/access_control.md @@ -187,7 +187,7 @@ FORGELM_OPERATOR="gha:Acme/pipelines:training:run-42" \ # Job 2 — approval (human reviewer identity) FORGELM_OPERATOR="alice@acme.example" \ - forgelm approve --run-id + forgelm approve --output-dir ``` ForgeLM does not enforce that the two identities differ — that is a diff --git a/docs/reference/compliance_summary.md b/docs/reference/compliance_summary.md index 4dc0e994..b0c71ead 100644 --- a/docs/reference/compliance_summary.md +++ b/docs/reference/compliance_summary.md @@ -104,9 +104,11 @@ What the regulator asks vs. how ForgeLM answers: ### Article 14 staging gate - Implementation: when `evaluation.require_human_approval: true` - the trained model lands in `/staging_model./` - awaiting `forgelm approve --run-id ` from a non-trainer - operator. + the trained model lands in + `/final_model.staging./` awaiting + `forgelm approve --output-dir ` from a + non-trainer operator (positional `run_id`; `--run-id` is not a + flag). - Listing: `forgelm approvals --pending` (Phase 37). - Audit: `human_approval.required/granted/rejected` events. diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md index 5c1d5dc2..4017f28f 100644 --- a/docs/reference/supply_chain_security-tr.md +++ b/docs/reference/supply_chain_security-tr.md @@ -1,6 +1,6 @@ # Supply-chain güvenliği — SBOM + zafiyet taraması -> ForgeLM'in supply-chain güvenlik tooling'i için referans dokuman: +> ForgeLM'in supply-chain güvenlik tooling'i için referans doküman: > SBOM üretimi (Wave 2 dönemi), pip-audit nightly (Wave 4 / Faz 23), > bandit static analysis (Wave 4 / Faz 23). Hedef kitle: operatör > compliance ekibi + denetçi. diff --git a/docs/usermanuals/tr/compliance/human-oversight.md b/docs/usermanuals/tr/compliance/human-oversight.md index 9c68d5e7..2d0ed2b4 100644 --- a/docs/usermanuals/tr/compliance/human-oversight.md +++ b/docs/usermanuals/tr/compliance/human-oversight.md @@ -75,11 +75,17 @@ Trainer durur ve artifact paketini webhook'unuza POST'lar. Sisteminiz insan ince v0.5.5'te desteklenen onay mekanizması CLI subcommand çifti `forgelm approve` / `forgelm reject`: ```bash -forgelm approvals --pending # onay bekleyen koşumları listele -forgelm approve --run-id # staging → final_model'e promote -forgelm reject --run-id --reason "..." # staged modeli at +forgelm approvals --pending --output-dir # onay bekleyen koşumları listele +forgelm approve --output-dir --comment "..." # staging → final_model'e promote +forgelm reject --output-dir --comment "..." # staged modeli at ``` +**Not:** `approve` ve `reject` positional `run_id` alır (`--run-id` +flag'i yoktur); `--comment "..."` reviewer notunu +`human_approval.granted` / `human_approval.rejected` event'ına yazar. +`--output-dir ` zorunludur ve `audit_log.jsonl` + +`final_model.staging/` içeren training output dizinini gösterir. + Her çağrı `FORGELM_OPERATOR` (onaylayanın kimliği) gerektirir ve zincire `human_approval.granted` / `human_approval.rejected` olayı yazar. Self-servis "bu koşuyu terfi ettir" otomasyonu v0.6.0+ Pro CLI (public roadmap'te Phase 13) için planlanmıştır; o zamana kadar CLI gate audit-grade arayüzdür. ## Onay imzasında ne var diff --git a/tools/check_anchor_resolution.py b/tools/check_anchor_resolution.py index 8bf54ff2..1b359067 100644 --- a/tools/check_anchor_resolution.py +++ b/tools/check_anchor_resolution.py @@ -239,13 +239,27 @@ def _resolve_pure_anchor(link: Link, anchor_part: str) -> BrokenLink | None: def _locate_target(link: Link, path_part: str, repo_root: Path) -> Path | BrokenLink: - """Resolve a repo-relative path against the source's parent or repo root.""" + """Resolve a repo-relative path against the source's parent or repo root. + + Both candidate paths are constrained to live under ``repo_root`` — + a `../../../etc/passwd`-style traversal that resolves outside the + tree is treated as broken rather than silently followed. + """ + repo_root_resolved = repo_root.resolve() + + def _under_repo_root(candidate: Path) -> bool: + try: + candidate.relative_to(repo_root_resolved) + except ValueError: + return False + return True + target = (link.source.parent / path_part).resolve() - if target.exists(): + if target.exists() and _under_repo_root(target): return target # Fall back: maybe the path is repo-root-relative (legacy refs). alt = (repo_root / path_part).resolve() - if alt.exists(): + if alt.exists() and _under_repo_root(alt): return alt return BrokenLink(link, f"target file not found: {path_part!r}") @@ -349,11 +363,18 @@ def _resolve_excludes(scope_dir: Path, exclude_arg: list[str] | None) -> tuple[P # Passing --exclude REPLACES the default rather than augmenting it # (call sites that need additional excludes pass the full list, # including ``analysis``); this matches the help-text contract. - specs: tuple[str, ...] + # + # Empty / whitespace-only entries are filtered (an `--exclude ""` + # would otherwise resolve to ``scope_dir`` itself and silently + # exclude the entire tree). When all entries filter out, fall + # back to the default rather than scanning everything — the + # opt-out for "scan analysis/ too" is `--exclude doesnotexist` + # or any other path that won't match. if exclude_arg is None: - specs = ("analysis",) + specs: tuple[str, ...] = ("analysis",) else: - specs = tuple(exclude_arg) + cleaned = tuple(s for s in (entry.strip() for entry in exclude_arg) if s) + specs = cleaned if cleaned else ("analysis",) return tuple((scope_dir / spec).resolve() for spec in specs)