Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ jobs:
# the moment a dependency advisory drops, even before the team
# re-bumps versions:
#
# - pip-audit (transitive CVEs against the OSV / GHSA databases)
# - pip-audit (transitive CVEs against PyPI's vulnerability data, its default service)
# - bandit (static-analysis on forgelm/ production code only)
#
# Severity policy (per tools/check_pip_audit.py + tools/check_bandit.py):
Expand Down
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,49 @@ All notable changes to ForgeLM are documented here.

### Fixed

- Stop describing the human-approval gate as enforcing segregation of duties (English and Turkish): it records the
trainer and approver identities and does not compare them, and both are self-asserted. The ISO 27001 and SOC 2
mappings, the Statement of Applicability, the change-management SOP, the deployer guide and the user manual now
say what the gate records and that keeping the two people different is the deployer's control. The guide also no
longer says `require_human_approval` is enforced for strict risk tiers: it is not (#73, #99, #90).
- Rewrite the GDPR / KVKK user-manual page (English and Turkish) to what ForgeLM does. It claimed that "by the time
data lands in JSONL, identifiable subjects are removed", that "no specific person's data is recoverable from the model"
and that the audit report "confirms this", that the audit flags memorisation, that a retention setting "auto-deletes"
originals and "records the intended TTL in the audit log", and it showed an `ingestion: pii_mask:` block and an
`international_transfers.enabled` setting that do not exist. Masking now reads as reducing the eight detected identifier
formats (names, addresses and free text are not detected), the audit report as the plain JSON it is (no source
attribution, no hash manifest), and access and erasure requests as `forgelm reverse-pii` and `forgelm purge` (#101).
- Document the one case where a failed safety gate still exits 0 with `auto_revert: true`: an evaluation that produced no
verdict (`safety.evaluation_completed: false` — classifier not loaded, probes file missing or empty, or most verdicts
unreadable) keeps the model and the run exits 0, or 4 with the approval gate. The error-handling standard, the exit-codes and
JSON-output pages, the audit-event catalogue, the safety guide, the deployer guide and the risk-treatment plan said that exit
0 meant every gate passed, that a classifier load failure "halts" the pipeline and leaves no `final_model/`, and that
classification errors count as unsafe. The behaviour is unchanged; making it fail closed is a breaking exit-code change
left for a major version (#87).
- Say where the per-release SBOMs actually are and what they contain. The supply-chain reference, the ISO 27001 / SOC 2 deployer guide,
the change-management SOP and the supply-chain and deployer user-manual pages told an auditor to run `gh release download … --pattern 'sbom-*'`
and to "re-emit the SBOM from the git tag and diff it against the one attached to the release". Nothing attaches an SBOM to a
GitHub Release: `publish.yml` uploads the twelve files as workflow-run artifacts, which GitHub deletes after 90 days. The docs now give
the working commands (`gh run list --workflow publish.yml --branch v<X.Y.Z> --status success`, then `gh run download`), say that a
release SBOM is the CI runner's `pip list` (test tooling such as `pytest` included, no dependency graph, no licence data) and so is
not reproducible from the tag, and drop "license extraction in SBOM" from the ISO A.5.32, SOC 2 CC9.2 and statement-of-applicability
rows. The nightly `pip-audit` is described as querying PyPI's vulnerability data (its default service, not OSV), and the manual's
"exit 1 means HIGH/CRITICAL" now includes UNKNOWN-severity findings. English and Turkish. Attaching the SBOMs to the release and
generating them from a clean runtime-only environment are code changes and stay open on the issue (#114).
- Stop saying that ForgeLM's audit log is the record-keeping artefact "mandated by EU AI Act Article 12" and that the approval gate
"implements" Article 14. Article 12 asks the deployed high-risk system to log events automatically over its lifetime, and Article 14's
oversight (monitor, interpret, override, stop) happens while the system is in use. `audit_log.jsonl` is development-time evidence of the
training pipeline, and `evaluation.require_human_approval` is a pre-promotion sign-off that supports design-time oversight documentation.
The audit-event catalogue, the audit-log and human-oversight user-manual pages (English and Turkish) now say that (#75).
- Correct site copy that promised more than the code does, in all six languages (`site/js/translations.js`, the HTML fallbacks and a code
comment in `site/js/wizard.js`): a failing benchmark gate exits 3 only with `evaluation.auto_revert: true` (and a safety evaluation with
no verdict keeps the model and exits 0); the approval gate records the approver, it is not "dual-controlled" and does not implement
Article 14's in-use oversight; webhook alerts are Slack-format, Teams and Discord need a relay; PII masking covers common formats, not
names or free text, and `reverse-pii` does not "satisfy" an access request; the wizard's retention "enforce" option is reserved and
deletes nothing; GGUF export writes a SHA-256 sidecar (K-quants need `llama-quantize`) rather than a "manifest"; DeepSpeed preset names
need a source checkout. A new site test keeps those phrases from coming back. English and Turkish are the maintained tiers; the German,
French, Spanish and Chinese strings were written by the assistant, not a native reviewer, and need review (#86, #89, #92, #94, #98, #99,
#101, #110, #112). The `forgelm doctor` sample on the quickstart page now uses the real renderer's format (#534).
- Make `.github/dependabot.yml` parse. It set `versioning-strategy: widen` for `pip`, which that ecosystem does not accept (GitHub's
"Dependabot" check on any pull request touching the file said so), so Dependabot opened no pull requests at all. The setting is removed, the
header comment explains the restriction.
Expand Down
48 changes: 32 additions & 16 deletions docs/guides/iso_soc2_deployer_guide-tr.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,20 @@ ForgeLM kanıtına sahip:
2. **Change control** — Madde 14 staging gate (`forgelm approve` /
`reject`) + `human_approval.required/granted/rejected` audit
olayları + koşum başına damgalanan `config_hash` (manifest sidecar
alanı). Her model promotion çift kontrollü ve forensic olarak
attribute edilmiştir.
alanı). Onay
kapısı etkinleştirildiğinde her promotion, audit log'a kaydedilen onaylayıcı kimliğine atfedilir. ForgeLM
eğitici ve onaylayıcı kimliklerini kaydeder ama karşılaştırmaz: bunların farklı kişiler olmasını kimlik
sağlayıcınızda uygularsınız.
3. **Data lineage** — `data_provenance.json` (SHA-256 fingerprint +
size + mtime + `hf_revision_source` ile derecelendirilmiş dataset HF
Hub commit SHA'sı); `data_governance_report.json`
(collection_method, annotation_process, known_biases,
personal_data_included, dpia_completed).
4. **Supply chain** — yayın etiketi başına yayın matrisinin her (OS
× Python-version) hücresi için emit edilen CycloneDX 1.5 SBOM.
Wave 4 statik + dinamik güvenlik taraması için `pip-audit`
× Python-version) hücresi için emit edilen CycloneDX 1.5 SBOM;
90 gün boyunca workflow-run artifact olarak saklanır (GitHub
release'e eklenmez; CI runner'ının `pip list`'ini, test araçları
dahil, lisans verisi olmadan listeler). Wave 4 statik + dinamik güvenlik taraması için `pip-audit`
nightly + `bandit` CI ekler.

## Denetim gözlem dönemi öncesi setup checklist'i
Expand Down Expand Up @@ -65,7 +69,9 @@ Denetçi sormadan **önce** indirmeniz gereken on iki madde.

- [ ] **`evaluation.require_human_approval: true`** her
`risk_classification` ∈ `{high-risk, unacceptable}` koşumu
için. Unutursanız F-compliance-110 `ConfigError` raise eder.
için. ForgeLM bunu zorlamaz: F-compliance-110 yalnızca bu katmanlarda `evaluation.safety.enabled`
kapalıysa `ConfigError` raise eder; dolayısıyla onay kapısı olmayan, elle yazılmış bir strict-katman
YAML'ı kabul edilir. CLI sihirbazı (`forgelm --wizard`) bu katmanlarda kapıyı açar.
- [ ] **Webhook lifecycle yapılandır** (`url_env`, `secret_env`,
`notify_on_*`) böylece SIEM her durum geçişini içe alır.
- [ ] **`auto_revert: true` etkinleştir** üretim eğitimi için —
Expand Down Expand Up @@ -155,8 +161,9 @@ jq -r 'select(.run_id == "X" and (.event == "training.started" or .event == "hum
./outputs/audit_log.jsonl | sort -u
```

İki farklı operator ID görev ayrılığını kanıtlar (ISO A.5.3, SOC 2
CC1.5).
`training.started` üzerindeki operatörü `human_approval.granted` üzerindeki onaylayıcıyla karşılaştırın. İki kimlik de
kendi beyanıdır (`FORGELM_OPERATOR`, yoksa OS kullanıcısı); dolayısıyla görev ayrılığı (ISO A.5.3, SOC 2 CC1.5) ancak
kimlik sağlayıcınız bunları farklı kişiler yapıyorsa geçerlidir.

### S3: "Data lineage göster"

Expand Down Expand Up @@ -198,19 +205,25 @@ eğittiğinin kanıtı saymayın.
### S4: "Supply chain göster"

```bash
# SBOM artefaktını GitHub release sayfasından indir.
gh release download v0.5.5 --pattern 'sbom-*'

# Veya istek üzerine yeniden üret.
# Tag'in publish çalıştırmasından bir SBOM indir. Bunlar workflow-run
# artifact'larıdır, 90 günde silinir; GitHub release asset'i değildir.
gh run list --repo HodeTech/ForgeLM --workflow publish.yml --branch v0.11.0 --status success \
--json databaseId,createdAt
gh run download <databaseId> --repo HodeTech/ForgeLM --name sbom-ubuntu-latest-py3.11

# Veya kendi ortamınızı tanımlayın: modeli eğiten ya da sunan ortamda,
# bir ForgeLM checkout'undan çalıştırın.
python3 tools/generate_sbom.py > sbom.json

# Son yayınla diff al.
diff <(jq -S . sbom-prev.json) <(jq -S . sbom.json)
```

CycloneDX 1.5 JSON her transitive bağımlılığı purl
(`pkg:pypi/...`) ve sürümle listeler. Dependency-Track CVE
korelasyonu için bunu doğal olarak içe alır.
CycloneDX 1.5 JSON kurulu her paketi purl (`pkg:pypi/...`) ve
sürümle listeler; bağımlılık grafiği ve lisans verisi yoktur.
Release SBOM'ları CI runner'ını tanımlar, bu yüzden `pytest` gibi
test araçlarını da adlandırır. Dependency-Track dosyayı CVE
korelasyonu için doğal olarak içe alır.

### S5: "Erişim kontrolleri göster — yalnız yetkili reviewer'ların model onayladığını nasıl ispatlıyorsunuz?"

Expand Down Expand Up @@ -249,8 +262,11 @@ Referans:

- `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23
genişletme) güvenlik-incident playbook için.
- Zincirin kendisi: `audit.classifier_load_failed` event yangın +
`pipeline.failed` propagate + koşum `final_model/` dizini ÜRETMEZ.
- Zincirin kendisi: `audit.classifier_load_failed` event'i tetiklenir
ve `safety.evaluation_completed` `evaluation_completed: false` kaydeder.
Koşu **durdurulmaz**: model `final_model/` içinde kalır ve koşu 0 ile
çıkar (onay kapısıyla 4); bu yüzden exit koduna değil, bu event'lere ya da
JSON zarfının `safety` bloğuna göre alarm kurun.
- F-compliance-110 strict gate: `risk_classification ∈ {high-risk,
unacceptable}` AND `evaluation.safety.enabled = false` ise config
validator koşum başlamadan ÖNCE `ConfigError` raise eder. Denetçi
Expand Down
48 changes: 32 additions & 16 deletions docs/guides/iso_soc2_deployer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,20 @@ direct ForgeLM evidence:
`forgelm verify-audit` validates the chain end-to-end.
2. **Change control** — Article 14 staging gate (`forgelm approve` /
`reject`) + `human_approval.required/granted/rejected` audit
events + `config_hash` (per-run manifest sidecar field) stamped per run. Every
model promotion is dual-controlled and forensically attributed.
events + `config_hash` (per-run manifest sidecar field) stamped per run. When
the approval gate is enabled, every promotion is attributed to the approver identity recorded in the audit
log. ForgeLM records the trainer and approver identities but does not compare them: keeping them different
people is a control you enforce at your identity provider.
3. **Data lineage** — `data_provenance.json` (SHA-256 fingerprint +
size + mtime + the dataset's HF Hub commit SHA, graded by
`hf_revision_source`); `data_governance_report.json`
(collection_method, annotation_process, known_biases,
personal_data_included, dpia_completed).
4. **Supply chain** — CycloneDX 1.5 SBOM emitted per release tag for
every (OS × Python-version) cell of the publish matrix. Wave 4
every (OS × Python-version) cell of the publish matrix, kept as a
workflow-run artifact for 90 days (not attached to the GitHub
release; it lists the CI runner's `pip list`, test tooling
included, with no licence data). Wave 4
adds `pip-audit` nightly + `bandit` CI for static + dynamic
security scanning.

Expand Down Expand Up @@ -63,7 +68,9 @@ Twelve items to land **before** the auditor asks.

- [ ] **`evaluation.require_human_approval: true`** for every
`risk_classification` ∈ `{high-risk, unacceptable}` run.
F-compliance-110 raises `ConfigError` if you forget.
ForgeLM does not enforce this: F-compliance-110 only raises `ConfigError` when
`evaluation.safety.enabled` is off for these tiers, so a hand-written strict-tier YAML without the
approval gate is accepted. The CLI wizard (`forgelm --wizard`) switches the gate on for these tiers.
- [ ] **Configure webhook lifecycle** (`url_env`, `secret_env`,
`notify_on_*`) so SIEM ingests every state transition.
- [ ] **Enable `auto_revert: true`** for production training so
Expand Down Expand Up @@ -155,8 +162,9 @@ jq -r 'select(.run_id == "X" and (.event == "training.started" or .event == "hum
./outputs/audit_log.jsonl | sort -u
```

Two distinct operator IDs prove segregation of duties (ISO A.5.3,
SOC 2 CC1.5).
Compare the operator on `training.started` with the approver on `human_approval.granted`. Both identities are
self-asserted (`FORGELM_OPERATOR`, else the OS user), so segregation of duties (ISO A.5.3, SOC 2 CC1.5) holds only
if your identity provider makes them different people.

### Q3: "Show me the data lineage"

Expand Down Expand Up @@ -198,19 +206,25 @@ corpus trained the model.
### Q4: "Show me the supply chain"

```bash
# Download the SBOM artefact from the GitHub release page.
gh release download v0.5.5 --pattern 'sbom-*'

# Or regenerate on demand.
# Download one SBOM from the tag's publish run. They are workflow-run
# artifacts that expire after 90 days, not GitHub release assets.
gh run list --repo HodeTech/ForgeLM --workflow publish.yml --branch v0.11.0 --status success \
--json databaseId,createdAt
gh run download <databaseId> --repo HodeTech/ForgeLM --name sbom-ubuntu-latest-py3.11

# Or describe your own environment: run this from a ForgeLM checkout in the
# environment that trains or serves the model.
python3 tools/generate_sbom.py > sbom.json

# Diff against last release.
diff <(jq -S . sbom-prev.json) <(jq -S . sbom.json)
```

The CycloneDX 1.5 JSON lists every transitive dependency with its
purl (`pkg:pypi/...`) and version. Dependency-Track ingests this
natively for CVE correlation.
The CycloneDX 1.5 JSON lists every installed package with its purl
(`pkg:pypi/...`) and version; it has no dependency graph and no licence
data. The release SBOMs describe the CI runner, so they also name test
tooling such as `pytest`. Dependency-Track ingests the file natively for
CVE correlation.

### Q5: "Show me the access controls — how do you prove only authorised reviewers approve models?"

Expand Down Expand Up @@ -248,9 +262,11 @@ Reference:

- `docs/qms/sop_incident_response.md` §4 (Wave 4 / Faz 23 expansion)
for the security-incident playbook.
- The audit chain itself: `audit.classifier_load_failed` event fires
+ `pipeline.failed` propagates + the run does NOT produce a
`final_model/` directory.
- The audit chain itself: the `audit.classifier_load_failed` event fires
and `safety.evaluation_completed` records `evaluation_completed: false`.
The run is **not** halted: the model stays in `final_model/` and the
run exits 0 (4 with the approval gate), so alert on those events or on
the JSON envelope's `safety` block, not on the exit code.
- F-compliance-110 strict gate: if `risk_classification ∈
{high-risk, unacceptable}` AND `evaluation.safety.enabled = false`,
the config validator raises `ConfigError` BEFORE the run starts.
Expand Down
14 changes: 7 additions & 7 deletions docs/guides/safety_compliance-tr.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,7 @@ Kapı 2: Confidence-weighted → safety_score < min_safety_score?
Kapı 3: Şiddet → critical_count > 0? high_count > eşik?
```

Üç kapı bağımsızdır. Herhangi bir kapının başarısızlığı (etkinse)
auto-revert tetikler.
Üç kapı bağımsızdır. Bir kapının başarısızlığı (etkinse) auto-revert tetikler; güvenlik değerlendirmesi hiç karar üretemediyse bu geçerli değildir (bkz. [Fail-safe davranış](#fail-safe-davranış)).

### Zarar kategorileri (Llama Guard 3)

Expand Down Expand Up @@ -204,13 +203,12 @@ iyileştirip bozmadığını izlemesine yardım eder.

### Fail-safe davranış

- Sınıflandırma hataları **güvensiz** olarak değerlendirilir (fail-safe
prensibi)
- Sınıflandırıcının yanıtlayamadığı bir çift **skorlanmamış** sayılır, güvenli değil
- Düşük-güvenli sınıflandırmalar manuel inceleme için işaretlenir
(uyarı log'u)
- Güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise
model otomatik olarak silinir
- Hat entegrasyonu için `3` exit kodu döner
- Skorlanmış bir güvenlik değerlendirmesi başarısız olursa ve `auto_revert: true` ise
model otomatik olarak silinir ve hat entegrasyonu için `3` exit kodu döner
- **Değerlendirme karar üretemezse** (`safety.evaluation_completed: false` — sınıflandırıcı yüklenemedi, probe dosyası yok ya da boş, ya da çoğu karar okunamadı), model `auto_revert: true` olsa bile **korunur** ve koşu `0` ile çıkar (onay kapısıyla `4`); çünkü okunmamış bir karar zarar kanıtı değildir. Yalnızca exit koduna değil, JSON çıktısındaki `safety.evaluation_completed` ve `safety.passed`'e bakın

---

Expand Down Expand Up @@ -522,6 +520,8 @@ Herhangi bir adım başarısız olursa ve `auto_revert: true` ise:
Model silinir + Webhook bildirimi (başarısızlık) + Exit kodu 3
```

Karar üretemeyen bir güvenlik değerlendirmesi (`evaluation_completed: false`) istisnadır: model korunur ve koşu 0 ile çıkar.

### İnsan onay kapısı (Madde 14)

Tüm otomatik kontroller geçtikten sonra hattı duraklatmak için
Expand Down
Loading
Loading