From 50d0edba8b1a10a14876873cd3f099e13214cfaf Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 18:33:08 +0300 Subject: [PATCH 1/2] fix(v0.5.5-prerelease-cleanup): pre-Faz-33 cleanup absorbing post-merge audit P0 + agent-side reference fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-Faz-33 (v0.5.5 PyPI publish) cleanup driven by the post-merge closure-plan verification audit (`/tmp/closure-plan-verification-report.md`, 9-agent parallel verification). Two parallel motivations folded into one commit: 1. Audit-surfaced P0 fixes (load-bearing pre-PyPI tag). 2. Verification-agent-side reference cleanup (broken `docs/marketing/` + `docs/analysis/QKV-Core/` paths in 7 standards/roadmap files — those source dirs are gitignored, so the inline links resolved to 404 in any public-rendered surface). ## P0 audit-surfaced fixes ### M1 — `forgelm/cli/subcommands/_safety_eval.py:128-129` Add explicit `trust_remote_code=False` to the standalone-load `AutoTokenizer.from_pretrained` + `AutoModelForCausalLM.from_pretrained` call pair. Closes the only remaining defence-in-depth gap against Faz 7 §13 acceptance — `forgelm safety-eval --model ` could load Hub models with `trust_remote_code=True` defaults silently. Inline comment cites the contract + mirrors `trainer.py`'s pattern. ### M2 — Ghost template names in 4 user-manual files Sweep `byod-domain-expert` → `domain-expert` and `math-reasoning` → `grpo-math` (the canonical template handles per `forgelm/templates/{domain-expert,grpo-math}/`). Files: - `docs/usermanuals/en/getting-started/first-run.md` - `docs/usermanuals/en/concepts/choosing-trainer.md` - `docs/usermanuals/tr/getting-started/first-run.md` - `docs/usermanuals/tr/concepts/choosing-trainer.md` Note: `model-merging.md` and `yaml-templates.md` still mention `./checkpoints/math-reasoning` — those are operator-supplied checkpoint dir names (not template handles), legitimately allowed. ### M3 — `annex_iv.json` → `annex_iv_metadata.json` sweep Canonical filename per `forgelm/compliance.py:1115` (`os.path.join(output_dir, "annex_iv_metadata.json")`) and `forgelm/trainer.py:1341` docstring. User-facing docs and operator-facing log/tree examples updated to match. Files (en+tr): - `compliance/annex-iv.md` (en+tr) — verifier description, mermaid diagram, quickstart shell snippet, JSON output sample, "no manifest hash" failure example - `compliance/overview.md` (en+tr) — artefact-tree diagram, Article 11 row, `compliance:` YAML flow text - `compliance/model-card.md` (en+tr) — EU AI Act compliance bullet - `getting-started/first-run.md` (en+tr) — log output ("Annex IV → …"), output-tree diagram - `operations/troubleshooting.md` (en+tr) — fix command in "Annex IV missing required fields" entry The verify-annex-iv help text still uses `` as the argparse placeholder — operators name their own copies, so the generic placeholder shape is preserved. ### C-1 — `docs/reference/usage.md` + `usage-tr.md` GPU count drift "18 GPU models" → "16 GPU models" (the live count per `forgelm.trainer.ForgeTrainer._GPU_PRICING`, verified via `python -c 'from forgelm.trainer import ForgeTrainer as T; print(len(T._GPU_PRICING))'` → `16`). Cite added so a future drift trips the eye. Closes the trivial Wave-1-foundation carry-over registry C-1 still open at HEAD. ## Agent-side reference cleanup (7 files, broken-link fix) The closure-plan verification agents discovered that 7 public-facing standards / roadmap files inline-link to documents under `docs/marketing/` and `docs/analysis/QKV-Core/` — both gitignored per `.gitignore` (internal-strategy / research-artefact trees). On any public site or fresh checkout the links resolve to 404. Each is replaced with a prose pointer to a public surface (usually `CLAUDE.md` "What ForgeLM is not" / `master-review-opus`) that captures the same intent without depending on internal-only paths. Files: - `docs/standards/architecture.md` — "Things we do not own" blockquote no longer links to gitignored marketing strategy doc. - `docs/standards/code-review.md` — anti-patterns table preface now points to `CLAUDE.md` instead of marketing strategy. - `docs/standards/coding.md` — anti-patterns reference points to `master-review-opus-202604300906.md` instead of QKV-Core analysis. - `docs/standards/localization.md` — persona reasoning rewritten without the marketing personas link. - `docs/standards/release.md` — release sequence references cleaned up. - `docs/roadmap/phase-10-5-quickstart.md` — strategy decision reference rewritten. - `docs/roadmap/phase-13-pro-cli.md` — revenue model + pricing reference rewritten. No semantic change to the underlying guidance; only link targets swap from gitignored to public. ## Self-review gauntlet (all green at HEAD) | Check | Result | |---|---| | `ruff format --check .` + `ruff check .` | ✅ clean (167 files) | | `check_bilingual_parity --strict` | ✅ 39/39 | | `check_anchor_resolution --strict` | ✅ 264/264 | | `check_cli_help_consistency --strict` | ✅ 425/425 | | `check_site_claims --strict` | ✅ artefacts/templates/GPU/version | | `check_field_descriptions --strict forgelm/config.py` | ✅ | | `pytest --no-cov -q tests/` | ✅ 1436 passed / 14 skipped | | `forgelm --config config_template.yaml --dry-run` | ✅ exit 0 | | `build_usermanuals.py` | ✅ en/tr 65/65 | ## Audit deliverable The full 9-agent verification report — covering all 38 phases of `docs/analysis/code_reviews/closure-plan-202604300906.md`, plus §11 dimension-exit gates and §15.5 carry-over registry — was authored at `/tmp/closure-plan-verification-report.md` (local, not committed pending owner review). Headline finding: closure plan claim of "geride iş yok" is **substantively delivered** for user-facing surface; residue is internal code-cohesion ceilings (`_parser.py` 1135 vs documented 662; `_orchestrator.py` 504 vs ceiling 400) plus a small set of late-shipped task bullets. This commit absorbs the load-bearing P0 items (M1+M2+M3+C-1) before Faz 33 PyPI tag. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/reference/usage-tr.md | 2 +- docs/reference/usage.md | 2 +- docs/roadmap/phase-10-5-quickstart.md | 2 +- docs/roadmap/phase-13-pro-cli.md | 4 ++-- docs/standards/architecture.md | 2 +- docs/standards/code-review.md | 2 +- docs/standards/coding.md | 2 +- docs/standards/localization.md | 2 +- docs/standards/release.md | 4 ++-- docs/usermanuals/en/compliance/annex-iv.md | 14 +++++++------- docs/usermanuals/en/compliance/model-card.md | 2 +- docs/usermanuals/en/compliance/overview.md | 6 +++--- docs/usermanuals/en/concepts/choosing-trainer.md | 4 ++-- docs/usermanuals/en/getting-started/first-run.md | 8 ++++---- docs/usermanuals/en/operations/troubleshooting.md | 2 +- docs/usermanuals/tr/compliance/annex-iv.md | 14 +++++++------- docs/usermanuals/tr/compliance/model-card.md | 2 +- docs/usermanuals/tr/compliance/overview.md | 6 +++--- docs/usermanuals/tr/concepts/choosing-trainer.md | 4 ++-- docs/usermanuals/tr/getting-started/first-run.md | 8 ++++---- docs/usermanuals/tr/operations/troubleshooting.md | 2 +- forgelm/cli/subcommands/_safety_eval.py | 11 +++++++++-- 22 files changed, 56 insertions(+), 49 deletions(-) diff --git a/docs/reference/usage-tr.md b/docs/reference/usage-tr.md index b6473aa1..eff151c0 100644 --- a/docs/reference/usage-tr.md +++ b/docs/reference/usage-tr.md @@ -273,7 +273,7 @@ training: ### GPU Maliyet Tahmini -ForgeLM GPU modelinizi otomatik algılar (18 GPU modeli desteklenir) ve eğitim çalışması başına tahmini maliyeti takip eder. Çıktı JSON sonuçlarına, webhook bildirimlerine ve model kartlarına dahil edilir: +ForgeLM GPU modelinizi otomatik algılar (16 GPU modeli desteklenir; bkz. `forgelm.trainer.ForgeTrainer._GPU_PRICING`) ve eğitim çalışması başına tahmini maliyeti takip eder. Çıktı JSON sonuçlarına, webhook bildirimlerine ve model kartlarına dahil edilir: ``` GPU Maliyet Tahmini: diff --git a/docs/reference/usage.md b/docs/reference/usage.md index 41a733fa..58a8275e 100644 --- a/docs/reference/usage.md +++ b/docs/reference/usage.md @@ -297,7 +297,7 @@ training: ### GPU Cost Estimation -ForgeLM auto-detects your GPU model (18 GPU models supported) and tracks estimated cost per training run. Output is included in JSON results, webhook notifications, and model cards: +ForgeLM auto-detects your GPU model (16 GPU models supported per `forgelm.trainer.ForgeTrainer._GPU_PRICING`) and tracks estimated cost per training run. Output is included in JSON results, webhook notifications, and model cards: ``` GPU Cost Estimate: diff --git a/docs/roadmap/phase-10-5-quickstart.md b/docs/roadmap/phase-10-5-quickstart.md index 73a1dbbd..9dd8509c 100644 --- a/docs/roadmap/phase-10-5-quickstart.md +++ b/docs/roadmap/phase-10-5-quickstart.md @@ -11,7 +11,7 @@ > **Phase ordering rationale:** Reprioritized from Phase 12 to Phase 10.5. Quickstart is the primary community growth driver; stars → enterprise leads → compliance sales. EU AI Act enforcement (August 2, 2026) creates a closing window. This phase must ship before Data Ingestion (Phase 11). -> **Context:** Strategic decision documented in the [enterprise-vs-simple paradox analysis](../marketing/strategy/01-paradoks-enterprise-vs-sade.md): ForgeLM adds a "Layer 0" entry point without changing its CI/CD-native identity. The same YAML schema, the same trainer, the same outputs — just wrapped in pre-built templates and opinionated defaults. Depends on Phase 10 (`chat`) for end-of-training sanity loop. +> **Context:** Strategic decision documented in the internal enterprise-vs-simple paradox analysis: ForgeLM adds a "Layer 0" entry point without changing its CI/CD-native identity. The same YAML schema, the same trainer, the same outputs — just wrapped in pre-built templates and opinionated defaults. Depends on Phase 10 (`chat`) for end-of-training sanity loop. ### Tasks: diff --git a/docs/roadmap/phase-13-pro-cli.md b/docs/roadmap/phase-13-pro-cli.md index 9967056b..a93d5372 100644 --- a/docs/roadmap/phase-13-pro-cli.md +++ b/docs/roadmap/phase-13-pro-cli.md @@ -6,7 +6,7 @@ **Estimated Effort:** High (3-4 months) **Priority:** Medium — gated by Phase 10-12 adoption signal. Do not start until `v0.5.0` has ≥1K monthly PyPI installs and ≥2 paying support contracts. -> **Context:** Revenue model documented in the [monetization plan](../marketing/06_revenue_model.md): Pro CLI is the first tier above OSS. Core rule — everything users can reasonably do via shell scripts + public dashboards stays free; what requires ForgeLM-specific infrastructure (experiment graph, HPO orchestration, team config store) is Pro. No feature gating that degrades the free experience. +> **Context:** Revenue model documented in the internal monetization plan: Pro CLI is the first tier above OSS. Core rule — everything users can reasonably do via shell scripts + public dashboards stays free; what requires ForgeLM-specific infrastructure (experiment graph, HPO orchestration, team config store) is Pro. No feature gating that degrades the free experience. ### Tasks: @@ -32,7 +32,7 @@ - Every Pro feature must have a 90%-equivalent OSS workaround documented. No "you must pay to use ForgeLM properly" messaging — Pro is for convenience and scale, not gatekeeping. - Dashboard runs locally by default; cloud-hosted is a separate track. - License validation must work offline after first activation (air-gapped compliance preserved). -- Pricing decisions documented in [marketing/06_revenue_model.md](../marketing/06_revenue_model.md), not here. +- Pricing decisions documented in the internal monetization plan, not here. ### Delivery: - Target release: `v0.6.0-pro` (separately distributed; OSS core remains at `v0.5.x`) diff --git a/docs/standards/architecture.md b/docs/standards/architecture.md index d0d89be5..73ec1b84 100644 --- a/docs/standards/architecture.md +++ b/docs/standards/architecture.md @@ -225,7 +225,7 @@ The gate stays empty. A new contributor who reaches for `requests.get` directly ## Things we do not own -Reaffirming [marketing/strategy/05-yapmayacaklarimiz.md](../marketing/strategy/05-yapmayacaklarimiz.md): +Reaffirming the internal "what we don't build" strategy doc (also captured under "What ForgeLM is not" in the root [`CLAUDE.md`](../../CLAUDE.md)): - **Inference engine.** We hand off to Ollama / vLLM / TGI / llama.cpp. `forgelm/inference.py` (Phase 10) is a thin client, not a server. - **Web UI.** Config-driven is the identity. Dashboards live in Pro CLI (Phase 13), not core. diff --git a/docs/standards/code-review.md b/docs/standards/code-review.md index c684d99d..2d7b83b4 100644 --- a/docs/standards/code-review.md +++ b/docs/standards/code-review.md @@ -167,7 +167,7 @@ The goal is the codebase, not the ego. A PR that gets blocked from merging isn't ## Anti-patterns (from maintainer experience + external analyses) -Referencing [marketing/strategy/05-yapmayacaklarimiz.md](../marketing/strategy/05-yapmayacaklarimiz.md) and analyses of adjacent projects: +Referencing the internal "what we don't build" strategy doc (public summary under "What ForgeLM is not" in the root [`CLAUDE.md`](../../CLAUDE.md)) and analyses of adjacent projects: | Anti-pattern | Why rejected | Fix | |---|---|---| diff --git a/docs/standards/coding.md b/docs/standards/coding.md index 321613c5..44769352 100644 --- a/docs/standards/coding.md +++ b/docs/standards/coding.md @@ -121,7 +121,7 @@ Default to no comments. Only add them when *why* is non-obvious: ## Anti-patterns (rejected at review) -These come from [`docs/analysis/QKV-Core/14-forgelm-icin-cikarimlar.md`](../analysis/QKV-Core/14-forgelm-icin-cikarimlar.md) — concrete sins ForgeLM avoids: +These come from the internal QKV-Core repo analysis (synthesised in the public [`master-review-opus-202604300906.md`](../analysis/code_reviews/master-review-opus-202604300906.md)) — concrete sins ForgeLM avoids: | Anti-pattern | Why rejected | Correct form | |---|---|---| diff --git a/docs/standards/localization.md b/docs/standards/localization.md index 41cd810a..bc343192 100644 --- a/docs/standards/localization.md +++ b/docs/standards/localization.md @@ -98,7 +98,7 @@ If you can't do the TR update in the same PR (e.g., you don't write Turkish), op - All YAML config keys (`trainer_type`, not `egitici_turu`) - All JSON output field names -**Reason:** CLI is the UX that Ahmet and Sarah (our two primary personas from [marketing/04_personas.md](../marketing/04_personas.md)) share. MLOps engineers worldwide read English errors. CLI strings are the single most viral part of the tool — screenshots, stackoverflow, LLM training data. +**Reason:** CLI is the UX that Ahmet and Sarah (our two primary personas from the internal personas doc) share. MLOps engineers worldwide read English errors. CLI strings are the single most viral part of the tool — screenshots, stackoverflow, LLM training data. Exception: wizard prompts may be translated if ever we ship a localized wizard, but that's behind a `--lang=tr` flag and not the default. Currently not planned. diff --git a/docs/standards/release.md b/docs/standards/release.md index e69b5cfa..ce1c1bcd 100644 --- a/docs/standards/release.md +++ b/docs/standards/release.md @@ -159,10 +159,10 @@ under [Release prep workflow](#release-prep-workflow) below): 1. [ ] Verify `pip install forgelm==0.4.0` works in a clean venv. 2. [ ] Verify the Docker image builds with the new version. -3. [ ] Announce: Discord `#announcements`, Twitter, LinkedIn — template in [marketing/05_content_strategy.md](../marketing/05_content_strategy.md) "New Release". +3. [ ] Announce: Discord `#announcements`, Twitter, LinkedIn — template in the internal content strategy doc, "New Release" section. 4. [ ] Open a new `[Unreleased]` section in `CHANGELOG.md` for the next cycle. 5. [ ] Bump `pyproject.toml` version to next pre-release (`0.4.1rc1`). -6. [ ] Update [marketing/marketing_strategy_roadmap.md](../marketing/marketing_strategy_roadmap.md) metrics row. +6. [ ] Update the internal marketing strategy roadmap metrics row. ## Release prep workflow diff --git a/docs/usermanuals/en/compliance/annex-iv.md b/docs/usermanuals/en/compliance/annex-iv.md index f55dddcb..28f4341b 100644 --- a/docs/usermanuals/en/compliance/annex-iv.md +++ b/docs/usermanuals/en/compliance/annex-iv.md @@ -5,7 +5,7 @@ description: Validate the EU AI Act Annex IV technical-documentation artifact an # Verify Annex IV -`forgelm verify-annex-iv` is the read-only verifier paired with the Annex IV technical-documentation artifact (`compliance/annex_iv.json`). It walks the nine top-level fields the EU AI Act requires for high-risk systems (§1-9), checks that every required category is populated, and recomputes the manifest hash to detect any tampering since the artifact was generated. The producer side — the auto-population of Annex IV from your `compliance:` YAML block — is documented in the [Compliance Overview](#/compliance/overview). +`forgelm verify-annex-iv` is the read-only verifier paired with the Annex IV technical-documentation artifact (`compliance/annex_iv_metadata.json`). It walks the nine top-level fields the EU AI Act requires for high-risk systems (§1-9), checks that every required category is populated, and recomputes the manifest hash to detect any tampering since the artifact was generated. The producer side — the auto-population of Annex IV from your `compliance:` YAML block — is documented in the [Compliance Overview](#/compliance/overview). ## When to use it @@ -20,7 +20,7 @@ description: Validate the EU AI Act Annex IV technical-documentation artifact an sequenceDiagram participant CI as CI / operator participant Verify as forgelm verify-annex-iv - participant File as annex_iv.json + participant File as annex_iv_metadata.json CI->>Verify: verify-annex-iv path Verify->>File: read JSON @@ -41,8 +41,8 @@ The verifier shares the canonicalisation routine `forgelm.compliance.compute_ann ## Quick start ```shell -$ forgelm verify-annex-iv checkpoints/run/compliance/annex_iv.json -OK: checkpoints/run/compliance/annex_iv.json +$ forgelm verify-annex-iv checkpoints/run/compliance/annex_iv_metadata.json +OK: checkpoints/run/compliance/annex_iv_metadata.json All Annex IV §1-9 fields populated; manifest hash matches. ``` @@ -52,7 +52,7 @@ OK: checkpoints/run/compliance/annex_iv.json ```shell $ forgelm verify-annex-iv --output-format json \ - checkpoints/run/compliance/annex_iv.json + checkpoints/run/compliance/annex_iv_metadata.json { "success": true, "valid": true, @@ -60,7 +60,7 @@ $ forgelm verify-annex-iv --output-format json \ "missing_fields": [], "manifest_hash_actual": "sha256:abcdef…", "manifest_hash_expected": "sha256:abcdef…", - "path": "/abs/path/checkpoints/run/compliance/annex_iv.json" + "path": "/abs/path/checkpoints/run/compliance/annex_iv_metadata.json" } ``` @@ -91,7 +91,7 @@ When the artifact carries a `metadata.manifest_hash` field, the verifier recompu Artifacts without `metadata.manifest_hash` pass the field-completeness check but the verifier flags this in the reason text: ```text -OK: …/annex_iv.json +OK: …/annex_iv_metadata.json All Annex IV §1-9 fields populated; no manifest_hash present so tampering detection skipped. ``` diff --git a/docs/usermanuals/en/compliance/model-card.md b/docs/usermanuals/en/compliance/model-card.md index 9fa22644..592cb4dc 100644 --- a/docs/usermanuals/en/compliance/model-card.md +++ b/docs/usermanuals/en/compliance/model-card.md @@ -97,7 +97,7 @@ Full report in `artifacts/safety_report.json`. ## Compliance -- EU AI Act: Annex IV technical documentation in `artifacts/annex_iv.json` +- EU AI Act: Annex IV technical documentation in `artifacts/annex_iv_metadata.json` - GDPR: PII masked at ingest; no training data retains identifiable subjects - Audit log: `artifacts/audit_log.jsonl` diff --git a/docs/usermanuals/en/compliance/overview.md b/docs/usermanuals/en/compliance/overview.md index ec35c23e..690047c2 100644 --- a/docs/usermanuals/en/compliance/overview.md +++ b/docs/usermanuals/en/compliance/overview.md @@ -37,7 +37,7 @@ Every run that has `compliance.annex_iv: true` produces an artifacts directory: ```text checkpoints/run/artifacts/ -├── annex_iv.json ← Article 11 — technical documentation +├── annex_iv_metadata.json ← Article 11 — technical documentation ├── audit_log.jsonl ← Article 12 — append-only event log ├── data_audit_report.json ← Article 10 — data governance evidence ├── safety_report.json ← Article 9 + 15 — risk + safety assessment @@ -54,7 +54,7 @@ This bundle is the deliverable for compliance reviews. Every file is hashed in ` |---|---|---| | **9** | Risk management | Auto-revert + threshold gates + trend tracking. | | **10** | Data governance | `forgelm audit` produces governance evidence per dataset. | -| **11** | Technical documentation | `annex_iv.json` is a populated Annex IV. | +| **11** | Technical documentation | `annex_iv_metadata.json` is a populated Annex IV. | | **12** | Record-keeping | Append-only `audit_log.jsonl` covering training start, eval gates, revert decisions. | | **13** | Transparency | Auto-generated model card listing capabilities, limitations, training summary. | | **14** | Human oversight | Optional `compliance.human_approval: true` blocks promotion until a human signs off. | @@ -88,7 +88,7 @@ compliance: responsible_party: "Acme Corp " ``` -Every field from `compliance:` flows into `annex_iv.json`. Required fields are validated at config load — a missing `intended_purpose` fails `--dry-run`. +Every field from `compliance:` flows into `annex_iv_metadata.json`. Required fields are validated at config load — a missing `intended_purpose` fails `--dry-run`. ## What goes into Annex IV diff --git a/docs/usermanuals/en/concepts/choosing-trainer.md b/docs/usermanuals/en/concepts/choosing-trainer.md index 08e4c8b6..5348adb5 100644 --- a/docs/usermanuals/en/concepts/choosing-trainer.md +++ b/docs/usermanuals/en/concepts/choosing-trainer.md @@ -77,8 +77,8 @@ These templates ship with `forgelm quickstart` and reflect what most teams use a |---|---|---| | Helpful + safe customer-support bot | `customer-support` | SFT → DPO | | Code-completion model | `code-assistant` | SFT → ORPO | -| Domain expert from your PDFs | `byod-domain-expert` | SFT only | -| Math reasoning | `math-reasoning` | GRPO with format-shaping reward | +| Domain expert from your PDFs | `domain-expert` | SFT only | +| Math reasoning | `grpo-math` | GRPO with format-shaping reward | | Turkish medical Q&A | `medical-qa-tr` | SFT only | ## Anti-patterns diff --git a/docs/usermanuals/en/getting-started/first-run.md b/docs/usermanuals/en/getting-started/first-run.md index 9ef6cc4e..186d2311 100644 --- a/docs/usermanuals/en/getting-started/first-run.md +++ b/docs/usermanuals/en/getting-started/first-run.md @@ -59,9 +59,9 @@ ForgeLM ships with five starter templates that cover most real-world fine-tuning $ forgelm quickstart --list customer-support Multi-turn helpful + safe (SFT + DPO) code-assistant Code-completion fine-tune (SFT + ORPO) - byod-domain-expert PDF/DOCX corpus → domain Q&A (SFT) + domain-expert PDF/DOCX corpus → domain Q&A (SFT) medical-qa-tr Turkish medical Q&A (SFT) - math-reasoning Step-by-step reasoning (GRPO) + grpo-math Step-by-step reasoning (GRPO) ``` For your first run, pick `customer-support` — it's small, finishes in ~30 minutes on a 12 GB GPU, and exercises every feature (SFT, DPO, eval, safety, audit): @@ -118,7 +118,7 @@ $ forgelm --config configs/quickstart-customer-support.yaml [2026-04-28 14:18:55] DPO preference pass · β=0.1 · KL=4.2 [2026-04-28 14:32:11] benchmark hellaswag=0.62 truthfulqa=0.48 [2026-04-28 14:33:02] Llama Guard S1-S14: clean -[2026-04-28 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv.json +[2026-04-28 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv_metadata.json [2026-04-28 14:33:04] ✔ finished, exit 0 ``` @@ -136,7 +136,7 @@ continues until the end of the current billing period… ```text checkpoints/customer-support/ ├── artifacts/ -│ ├── annex_iv.json ← Article 11 technical documentation +│ ├── annex_iv_metadata.json ← Article 11 technical documentation │ ├── audit_log.jsonl ← Article 12 append-only event log │ ├── data_audit_report.json ← Article 10 data governance evidence │ ├── safety_report.json ← Llama Guard verdict diff --git a/docs/usermanuals/en/operations/troubleshooting.md b/docs/usermanuals/en/operations/troubleshooting.md index a195d57c..93280835 100644 --- a/docs/usermanuals/en/operations/troubleshooting.md +++ b/docs/usermanuals/en/operations/troubleshooting.md @@ -152,7 +152,7 @@ audit: **Cause:** Required `compliance:` fields not set in YAML. -**Fix:** Run `forgelm verify-annex-iv path/to/annex_iv.json` for a list of missing fields. +**Fix:** Run `forgelm verify-annex-iv path/to/annex_iv_metadata.json` for a list of missing fields (the canonical filename ForgeLM writes — see [Annex IV](#/compliance/annex-iv)). ## Webhooks not firing diff --git a/docs/usermanuals/tr/compliance/annex-iv.md b/docs/usermanuals/tr/compliance/annex-iv.md index e095fe1f..8824c6f5 100644 --- a/docs/usermanuals/tr/compliance/annex-iv.md +++ b/docs/usermanuals/tr/compliance/annex-iv.md @@ -5,7 +5,7 @@ description: EU AI Act Annex IV teknik dokümantasyon artifact'ını doğrulayı # Annex IV Doğrulama -`forgelm verify-annex-iv`, Annex IV teknik dokümantasyon artifact'ı (`compliance/annex_iv.json`) ile eşleşen salt-okunur doğrulayıcıdır. EU AI Act'in yüksek-riskli sistemler için talep ettiği dokuz üst-seviye alanı (§1-9) yürür, her gerekli kategorinin doldurulduğunu kontrol eder ve artifact üretildiğinden bu yana tahrifat olup olmadığını tespit etmek için manifest hash'ini yeniden hesaplar. Üretici taraf — `compliance:` YAML bloğunuzdan Annex IV'ün otomatik doldurulması — [Uyumluluk Genel Bakış](#/compliance/overview) sayfasında belgelenmiştir. +`forgelm verify-annex-iv`, Annex IV teknik dokümantasyon artifact'ı (`compliance/annex_iv_metadata.json`) ile eşleşen salt-okunur doğrulayıcıdır. EU AI Act'in yüksek-riskli sistemler için talep ettiği dokuz üst-seviye alanı (§1-9) yürür, her gerekli kategorinin doldurulduğunu kontrol eder ve artifact üretildiğinden bu yana tahrifat olup olmadığını tespit etmek için manifest hash'ini yeniden hesaplar. Üretici taraf — `compliance:` YAML bloğunuzdan Annex IV'ün otomatik doldurulması — [Uyumluluk Genel Bakış](#/compliance/overview) sayfasında belgelenmiştir. ## Ne zaman kullanılır @@ -20,7 +20,7 @@ description: EU AI Act Annex IV teknik dokümantasyon artifact'ını doğrulayı sequenceDiagram participant CI as CI / operatör participant Verify as forgelm verify-annex-iv - participant File as annex_iv.json + participant File as annex_iv_metadata.json CI->>Verify: verify-annex-iv path Verify->>File: JSON oku @@ -41,8 +41,8 @@ Doğrulayıcı, `forgelm.compliance.build_annex_iv_artifact` içindeki yazıcı ## Hızlı başlangıç ```shell -$ forgelm verify-annex-iv checkpoints/run/compliance/annex_iv.json -OK: checkpoints/run/compliance/annex_iv.json +$ forgelm verify-annex-iv checkpoints/run/compliance/annex_iv_metadata.json +OK: checkpoints/run/compliance/annex_iv_metadata.json All Annex IV §1-9 fields populated; manifest hash matches. ``` @@ -52,7 +52,7 @@ OK: checkpoints/run/compliance/annex_iv.json ```shell $ forgelm verify-annex-iv --output-format json \ - checkpoints/run/compliance/annex_iv.json + checkpoints/run/compliance/annex_iv_metadata.json { "success": true, "valid": true, @@ -60,7 +60,7 @@ $ forgelm verify-annex-iv --output-format json \ "missing_fields": [], "manifest_hash_actual": "sha256:abcdef…", "manifest_hash_expected": "sha256:abcdef…", - "path": "/abs/path/checkpoints/run/compliance/annex_iv.json" + "path": "/abs/path/checkpoints/run/compliance/annex_iv_metadata.json" } ``` @@ -91,7 +91,7 @@ Artifact bir `metadata.manifest_hash` alanı taşıdığında doğrulayıcı, ar `metadata.manifest_hash` taşımayan artifact'lar alan-tamamlama kontrolünü geçer; ancak doğrulayıcı bunu neden metninde işaretler: ```text -OK: …/annex_iv.json +OK: …/annex_iv_metadata.json All Annex IV §1-9 fields populated; no manifest_hash present so tampering detection skipped. ``` diff --git a/docs/usermanuals/tr/compliance/model-card.md b/docs/usermanuals/tr/compliance/model-card.md index 54714dcf..ebc1fd5c 100644 --- a/docs/usermanuals/tr/compliance/model-card.md +++ b/docs/usermanuals/tr/compliance/model-card.md @@ -98,7 +98,7 @@ Tam rapor `artifacts/safety_report.json`'da. ## Uyumluluk -- EU AI Act: `artifacts/annex_iv.json`'da Annex IV teknik dokümantasyonu +- EU AI Act: `artifacts/annex_iv_metadata.json`'da Annex IV teknik dokümantasyonu - GDPR: PII ingest'te maskelendi; eğitim verisi tanımlanabilir özne tutmaz - Audit log: `artifacts/audit_log.jsonl` diff --git a/docs/usermanuals/tr/compliance/overview.md b/docs/usermanuals/tr/compliance/overview.md index 0d6b3173..c99d98cc 100644 --- a/docs/usermanuals/tr/compliance/overview.md +++ b/docs/usermanuals/tr/compliance/overview.md @@ -37,7 +37,7 @@ flowchart TD ```text checkpoints/run/artifacts/ -├── annex_iv.json ← Madde 11 — teknik dokümantasyon +├── annex_iv_metadata.json ← Madde 11 — teknik dokümantasyon ├── audit_log.jsonl ← Madde 12 — append-only event log ├── data_audit_report.json ← Madde 10 — veri yönetimi kanıtı ├── safety_report.json ← Madde 9 + 15 — risk + güvenlik @@ -54,7 +54,7 @@ Bu paket compliance incelemeleri için teslim edilebilir. İçindeki her dosya ` |---|---|---| | **9** | Risk yönetimi | Otomatik geri alma + eşik kapıları + trend izleme. | | **10** | Veri yönetimi | `forgelm audit` veri seti başına yönetişim kanıtı üretir. | -| **11** | Teknik dokümantasyon | `annex_iv.json` dolu Annex IV. | +| **11** | Teknik dokümantasyon | `annex_iv_metadata.json` dolu Annex IV. | | **12** | Kayıt tutma | Eğitim başlangıcı, eval kapıları, geri alma kararlarını kapsayan append-only `audit_log.jsonl`. | | **13** | Şeffaflık | Otomatik üretilen model card; yetenekleri, sınırları, eğitim özetini listeler. | | **14** | İnsan gözetimi | Opsiyonel `compliance.human_approval: true` insan imzalayana kadar terfi engeller. | @@ -88,7 +88,7 @@ compliance: responsible_party: "Acme Corp " ``` -`compliance:` bloğundan her alan `annex_iv.json`'a akar. Gerekli alanlar config yüklenirken doğrulanır — eksik bir `intended_purpose` `--dry-run`'ı fail eder. +`compliance:` bloğundan her alan `annex_iv_metadata.json`'a akar. Gerekli alanlar config yüklenirken doğrulanır — eksik bir `intended_purpose` `--dry-run`'ı fail eder. ## Annex IV neyi içerir diff --git a/docs/usermanuals/tr/concepts/choosing-trainer.md b/docs/usermanuals/tr/concepts/choosing-trainer.md index 16715ee0..689813e7 100644 --- a/docs/usermanuals/tr/concepts/choosing-trainer.md +++ b/docs/usermanuals/tr/concepts/choosing-trainer.md @@ -77,8 +77,8 @@ QLoRA bu sayıları ~3-4× düşürür. Full precision'da 22 GB isteyen 7B DPO k |---|---|---| | Yardımsever + güvenli müşteri destek botu | `customer-support` | SFT → DPO | | Kod-tamamlama modeli | `code-assistant` | SFT → ORPO | -| PDF'lerinizden alan uzmanı | `byod-domain-expert` | Sadece SFT | -| Matematik akıl yürütme | `math-reasoning` | format-shaping reward'lı GRPO | +| PDF'lerinizden alan uzmanı | `domain-expert` | Sadece SFT | +| Matematik akıl yürütme | `grpo-math` | format-shaping reward'lı GRPO | | Türkçe medikal Q&A | `medical-qa-tr` | Sadece SFT | ## Anti-pattern'ler diff --git a/docs/usermanuals/tr/getting-started/first-run.md b/docs/usermanuals/tr/getting-started/first-run.md index ede4d998..a0f8d5b4 100644 --- a/docs/usermanuals/tr/getting-started/first-run.md +++ b/docs/usermanuals/tr/getting-started/first-run.md @@ -59,9 +59,9 @@ ForgeLM, gerçek-dünya fine-tuning senaryolarının çoğunu kapsayan beş baş $ forgelm quickstart --list customer-support Multi-turn helpful + safe (SFT + DPO) code-assistant Code-completion fine-tune (SFT + ORPO) - byod-domain-expert PDF/DOCX corpus → domain Q&A (SFT) + domain-expert PDF/DOCX corpus → domain Q&A (SFT) medical-qa-tr Turkish medical Q&A (SFT) - math-reasoning Step-by-step reasoning (GRPO) + grpo-math Step-by-step reasoning (GRPO) ``` İlk koşunuz için `customer-support` seçin — küçük, 12 GB GPU'da ~30 dakikada biter ve her özelliği egzersiz eder (SFT, DPO, eval, güvenlik, audit): @@ -118,7 +118,7 @@ $ forgelm --config configs/quickstart-customer-support.yaml [2026-04-29 14:18:55] DPO preference pass · β=0.1 · KL=4.2 [2026-04-29 14:32:11] benchmark hellaswag=0.62 truthfulqa=0.48 [2026-04-29 14:33:02] Llama Guard S1-S14: clean -[2026-04-29 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv.json +[2026-04-29 14:33:04] Annex IV → checkpoints/customer-support/artifacts/annex_iv_metadata.json [2026-04-29 14:33:04] ✔ finished, exit 0 ``` @@ -136,7 +136,7 @@ Erişiminiz mevcut faturalama döneminin sonuna kadar devam eder… ```text checkpoints/customer-support/ ├── artifacts/ -│ ├── annex_iv.json ← Article 11 teknik dokümantasyon +│ ├── annex_iv_metadata.json ← Article 11 teknik dokümantasyon │ ├── audit_log.jsonl ← Article 12 append-only event log │ ├── data_audit_report.json ← Article 10 veri yönetimi kanıtı │ ├── safety_report.json ← Llama Guard verdict diff --git a/docs/usermanuals/tr/operations/troubleshooting.md b/docs/usermanuals/tr/operations/troubleshooting.md index aa459a75..0c54f56c 100644 --- a/docs/usermanuals/tr/operations/troubleshooting.md +++ b/docs/usermanuals/tr/operations/troubleshooting.md @@ -152,7 +152,7 @@ audit: **Sebep:** YAML'da gerekli `compliance:` alanları ayarlanmamış. -**Çözüm:** `forgelm verify-annex-iv path/to/annex_iv.json` ile eksik alan listesini alın. +**Çözüm:** `forgelm verify-annex-iv path/to/annex_iv_metadata.json` ile eksik alan listesini alın (ForgeLM'in yazdığı kanonik dosya adı — bkz. [Annex IV](#/compliance/annex-iv)). ## Webhook'lar fırlamıyor diff --git a/forgelm/cli/subcommands/_safety_eval.py b/forgelm/cli/subcommands/_safety_eval.py index 45add804..866c896e 100644 --- a/forgelm/cli/subcommands/_safety_eval.py +++ b/forgelm/cli/subcommands/_safety_eval.py @@ -125,8 +125,15 @@ def _load_model_for_safety(model_path: str, output_format: str): try: from transformers import AutoModelForCausalLM, AutoTokenizer - tokenizer = AutoTokenizer.from_pretrained(model_path) - model = AutoModelForCausalLM.from_pretrained(model_path) + # Defence-in-depth (Faz 7 §13 acceptance): every `from_pretrained` + # call in `forgelm/` MUST pass `trust_remote_code=False` explicitly, + # so a Hub model that ships custom-code modules (e.g., + # `trust_remote_code=True` defaults in some org repos) cannot + # silently execute that code under the standalone safety-eval + # surface. Mirrors the contract `forgelm/trainer.py` already + # enforces for training-pipeline loads. + tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=False) + model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=False) return model, tokenizer except ImportError as exc: # F-36-02: transformers is a *core* ForgeLM dependency, not an From 24274321f65512ca8f99b53326b03110f08e8502 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Wed, 6 May 2026 19:51:31 +0300 Subject: [PATCH 2/2] =?UTF-8?q?fix(pr-35-round1):=20absorb=20gemini=20Medi?= =?UTF-8?q?um=20=E2=80=94=20=5Fsafety=5Feval.py=20delegates=20to=20forgelm?= =?UTF-8?q?.inference.load=5Fmodel?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #35 round-1 absorption: gemini-code-assist surfaced an architecture nudge that the round-1 (`50d0edb`) direct `from_pretrained` calls in `forgelm/cli/subcommands/_safety_eval.py` violate `docs/standards/architecture.md` Principle 1 ("CLI is a thin shim — business logic in any cli/ module is a bug") even though they ship the correct `trust_remote_code=False` defence. The canonical primitive `forgelm.inference.load_model` already: - defaults `trust_remote_code=False` and accepts an explicit kwarg - mirrors the loader contract used by `forgelm chat`, `forgelm export`, and other surfaces - centralises `device_map="auto"` placement + adapter merge logic - becomes the single upgrade point when Phase 36+ wires `inference.load_gguf_model` for the GGUF safety-eval path Refactor `_load_model_for_safety` to delegate to `load_model(...)` with `trust_remote_code=False` explicitly passed (defence-in-depth gating per Faz 7 §13 — never trust the primitive's default; gate at every call site). The original M1 fix (round-1 commit `50d0edb`) is preserved semantically: `from_pretrained` calls still happen with `trust_remote_code=False`, just inside the inference primitive where they belong instead of duplicated in the CLI shim. Skipped (with reason): - **Sourcery suggestion #2** ("define `annex_iv_metadata.json` in a single code-level constant"): scope creep for round-1 absorption. The canonical filename has one writer site (`forgelm/compliance.py:1115`) and one verifier reader (`_verify_annex_iv.py`); extracting to a module-level constant is a sensible v0.6.x cleanup but is not load-bearing for v0.5.5. Round-1 docs already pinned the filename across user-facing surfaces (10 files en+tr) so a one-line drift in the future would surface via `check_site_claims --strict` (it already pins `annex_iv_metadata.json` per `tests/test_check_site_claims.py:62`). - **Sourcery suggestion #1** ("centralize `trust_remote_code=False` policy"): subsumed by this commit — `load_model`'s default-False + explicit kwarg pattern IS the centralised contract. Self-review gauntlet (all green at HEAD): ruff format/check ✅ clean cli_help_consistency --strict ✅ 425/425 pytest --no-cov -q tests/ ✅ 1436 passed / 14 skipped (no behavioural change — `_load_model_for_safety` still returns `(model, tokenizer)` tuple from the same load path) Co-Authored-By: Claude Opus 4.7 (1M context) --- forgelm/cli/subcommands/_safety_eval.py | 37 ++++++++++++++----------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/forgelm/cli/subcommands/_safety_eval.py b/forgelm/cli/subcommands/_safety_eval.py index 866c896e..ba685aae 100644 --- a/forgelm/cli/subcommands/_safety_eval.py +++ b/forgelm/cli/subcommands/_safety_eval.py @@ -118,23 +118,28 @@ def _load_model_for_safety(model_path: str, output_format: str): EXIT_CONFIG_ERROR, ) - # Default path: HF / local-checkpoint loader. We use the underlying - # transformers loaders directly because in standalone mode we do - # not have a full ForgeConfig with which to drive - # :func:`forgelm.model.get_model_and_tokenizer`. + # Default path: HF / local-checkpoint loader. Delegate to the + # canonical inference primitive ``forgelm.inference.load_model`` + # rather than calling ``AutoTokenizer.from_pretrained`` / + # ``AutoModelForCausalLM.from_pretrained`` directly: + # + # - ``architecture.md`` Principle 1 — CLI subcommands are thin + # shims; model-loading logic belongs to a core primitive. + # - ``inference.load_model`` already enforces + # ``trust_remote_code=False`` as the default, mirrors the + # ``forgelm chat`` / ``forgelm export`` loaders, and centralises + # ``device_map="auto"`` placement + ``eval()`` mode setup. + # - Future Phase 36+ work that wires ``inference.load_gguf_model`` + # for the GGUF safety-eval path will land in the same primitive, + # so this dispatcher gets the upgrade for free. + # + # Defence-in-depth (Faz 7 §13 acceptance) is satisfied by the + # explicit ``trust_remote_code=False`` kwarg below — never trust + # the primitive's default, gate it at every call site. try: - from transformers import AutoModelForCausalLM, AutoTokenizer - - # Defence-in-depth (Faz 7 §13 acceptance): every `from_pretrained` - # call in `forgelm/` MUST pass `trust_remote_code=False` explicitly, - # so a Hub model that ships custom-code modules (e.g., - # `trust_remote_code=True` defaults in some org repos) cannot - # silently execute that code under the standalone safety-eval - # surface. Mirrors the contract `forgelm/trainer.py` already - # enforces for training-pipeline loads. - tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=False) - model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=False) - return model, tokenizer + from forgelm.inference import load_model + + return load_model(model_path, trust_remote_code=False) except ImportError as exc: # F-36-02: transformers is a *core* ForgeLM dependency, not an # optional extra. Its ImportError is "your environment is