diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 066ebf88..b692db67 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -205,6 +205,28 @@ jobs: # automatically the next run. run: python3 tools/check_bilingual_parity.py --strict + - name: Markdown anchor resolution check (strict) + # Wave 5 / Faz 30 Task N: every Markdown anchor + relative-link + # under docs/ must resolve. Catches renamed/removed targets, + # slug-case drift, and forward-refs to pages that never landed + # (the GH-014/017/019 bug class). Strict mode wired up after + # the Wave 5 cleanup zeroed the live drift count from 36 → 0; + # the tool itself remains usable in advisory mode (no flag) for + # local feedback during a docs PR. See tools/check_anchor_resolution.py + # for what's checked and the GFM slug approximation used. + run: python3 tools/check_anchor_resolution.py --strict + + - name: CLI / docs help-consistency drift check (strict) + # Wave 5 / Faz 30 Task J: catches docs that cite ghost flags + # or non-existent subcommands (the GH-008/011/016/018/020 + # bug class). Strict mode wired up after the Wave 5 cleanup + # commit zeroed the live drift count from 40 → 0 (matches the + # check_anchor_resolution.py precedent: gate lands advisory, + # baseline cleanup follows, later commit flips to --strict). + # The tool itself remains usable in advisory mode (no flag) + # for local feedback during a docs PR. + run: python3 tools/check_cli_help_consistency.py --strict + - name: License check run: | test -f LICENSE || (echo "LICENSE file missing" && exit 1) diff --git a/.gitignore b/.gitignore index 391381cc..5d837dd9 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,7 @@ docs/analysis/code_reviews/* !docs/analysis/code_reviews/iso27001-soc2-alignment-202605052315.md !docs/analysis/code_reviews/wave4-review-prompt.md !docs/analysis/code_reviews/wave4-followup-review-prompt.md +!docs/analysis/code_reviews/wave5-review-prompt.md # Generated by tools/build_usermanuals.py — produced from # docs/usermanuals//**/*.md during the CI deploy. Not committed; diff --git a/CHANGELOG.md b/CHANGELOG.md index e00d076c..ce732a7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,158 @@ All notable changes to ForgeLM are documented here. ## [Unreleased] +### Wave 5 — Faz 30 full sweep (`closure/wave5-integration`) + +Final pre-release documentation + tooling closure. Wave 4 partial +landed Faz 30 Tier 1 ghost-feature drift + stat blocks; Wave 5 +completes Faz 30 Tasks A through O end-to-end. + +**Faz 30 Task A — feature × {guide, reference, usermanual} EN+TR +triplets (50 new doc files):** + +- 11 new reference docs: `docs/reference/{verify_audit, + verify_annex_iv_subcommand, verify_gguf_subcommand, + purge_subcommand, reverse_pii_subcommand, approve_subcommand, + approvals_subcommand, doctor_subcommand, cache_subcommands, + safety_eval_subcommand, library_api_reference}.md` (+ TR + mirrors). Each carries the live parser surface (flags + exit + codes + audit events emitted), worked examples, and stable + symbol citations. +- 5 new guides: `docs/guides/{getting-started, air_gap_deployment, + human_approval_gate, library_api, performance}.md` (+ TR + mirrors). `getting-started.md` is the v0.5.5 onboarding canonical + (opens with `forgelm doctor`); `air_gap_deployment.md` is the + deep deployer cookbook for `cache-models` / `cache-tasks`; + `human_approval_gate.md` cross-links access_control.md §6's + `jq -rs` segregation-of-duties pipeline. +- 9 new usermanual pages (`docs/usermanuals/{en,tr}/...`): + `compliance/{verify-audit, annex-iv, gdpr-erasure, + human-approval-gate}`, `deployment/verify-gguf`, + `operations/{iso-soc2-deployer, supply-chain}`, + `reference/{library-api, performance}`. +- `docs/usermanuals/_meta.yaml` registers the 8 new pages with + full 6-language title sets (EN+TR mandatory; DE/FR/ES/ZH + fallback to EN per Task L policy). +- `tools/check_bilingual_parity.py::_PAIRS` grew from 23 to 39 + pairs (16 new pairs across guides + reference). + +**Faz 30 Task J — `tools/check_cli_help_consistency.py` (new +guard):** + +- 749 LOC + 15 pinned tests (`tests/test_check_cli_help_consistency.py`). +- Discovers the live parser surface by spawning + `forgelm --help` for every shipped subcommand; + walks docs / README for fenced bash/shell `forgelm` invocations; + reports three drift classes (subcommand not in parser; flag not + in parser for that subcommand; flag value not in parser's + `choices` list). Skip patterns honour forward-reference framing + (`planned`/`roadmap`/`future`/`not in v0.5.5`) and `Wrong:` / + `Don't:` / `Anti-pattern:` / `Legacy:` / `Yanlış:`-tagged code + blocks. +- `.github/workflows/ci.yml` runs the gate in `--strict` mode in + the validate job (advisory in commit `c7bedc9`, flipped to + strict in commit `fbb082d` after baseline cleanup). + +**Faz 30 Task N — `tools/check_anchor_resolution.py` strict CI +flip:** + +- Drift baseline cleanup: 36 → 0 broken anchor / relative-link + references. Fixes cluster in: `forgelm/data_audit.py` / + `forgelm/cli.py` references (Faz 14 / 15 package splits — paths + updated to `forgelm/data_audit/.py` / + `forgelm/cli/.py` per symbol); slug-case mismatches + (double-dash collapse: `#v050--document-ingestion-...` → + `#v050-document-ingestion-...`); Turkish-letter slug drift + (`#dil-secimi` → `#dil-seçimi` Unicode-preserving canonical); + documentation.md self-illustrative `[Other doc 1](other.md)` + examples rephrased as plain prose. +- `.github/workflows/ci.yml` runs the gate in `--strict` mode. + +**Faz 30 Task O — Tier 1 ghost-feature drift residuals:** + +- GH-011: `docs/usermanuals/{en,tr}/evaluation/benchmarks.md` + rewritten — `forgelm benchmark --model "..."` (subcommand form + that doesn't exist) → `forgelm --config + --benchmark-only ` (the shipped flag form). +- GH-016: `--export-bundle` → `--compliance-export` rename in + `docs/qms/sop_model_training.md`, + `docs/qms/roles_responsibilities.md`, + `docs/roadmap/completed-phases.md`. +- GH-018: `kserve` / `triton` rows + dedicated example sections + removed from `docs/usermanuals/{en,tr}/deployment/deploy-targets.md` + (parser only ships `{ollama,vllm,tgi,hf-endpoints}`); replaced + with hand-authored manifest pointer note. +- GH-020: ingest flag drift in + `docs/usermanuals/{en,tr}/data/{ingestion, pii-masking, + language-detection, synthetic-data}.md` — `--max-tokens` → + `--chunk-tokens`; removed non-existent `--language` / + `--include` / `--exclude` / `--format` / `--pii-locale` flags; + `--strategy` choices clarified to `{sliding,paragraph,markdown}` + (no `tokens` / `sentence`); `--pii-locale` example replaced + with the shipped `forgelm audit --pii-ml --pii-ml-language LANG` + invocation. + +**Faz 30 Tasks B + C + D + E + F + G:** + +- `docs/usermanuals/_meta.yaml` +8 page entries (Task B); JS + bundle rebuild via `tools/build_usermanuals.py` 56 → 64 pages + EN+TR (Task C; artefacts gitignored, deployed by CI). +- `site/{index,features,compliance,quickstart,privacy}.html` + brought to v0.5.5 final state with new Wave 2-5 feature blocks + (Library API, GDPR rights tooling, doctor, air-gap, ISO/SOC 2 + alignment, supply-chain security, etc.); `site/js/translations.js` + 6-language i18n updates (Task D). +- `README.md` + `CONTRIBUTING.md` + `CLAUDE.md` final pass: stat + block refreshed (~72 modules / 1428 tests); feature list + reflects Wave 2-5 deliverables; CI guard list updated for the + two new strict gates (Task E). +- `docs/roadmap.md` + `-tr.md`: Phase 12.6 closure cycle ✅ Done; + v0.5.5 next-up; `docs/roadmap/releases.md` v0.5.5 row added; + `docs/roadmap/risks-and-decisions.md` Wave 5 cycle decisions + appended; new `docs/roadmap/phase-12-6-closure-cycle.md` + summary (Task F). +- 10 standards files final pass: `architecture.md` reflects + package splits; `documentation.md` cites the new CI guards; + `localization.md` formalises EN+TR mandatory + DE/FR/ES/ZH + deferred; `release.md` documents v0.5.5 release sequence; + `testing.md` test count refreshed (Task G). + +**Faz 30 Tasks H + I + K + L + M:** all green / closed. + +- H (parity strict CI): 39/39 OK. +- I (site-claim CI strict): all green (5 templates, 16 GPU + profiles, version 0.5.5). +- K (config doc auto-gen): closed without building the + speculatively-named `tools/regenerate_config_doc.py`. The + description→doc-source-of-truth contract is gated by + `check_field_descriptions.py --strict` + + `check_bilingual_parity.py --strict` + + `check_cli_help_consistency.py --strict` together; + configuration.md + `-tr.md` are hand-maintained. +- L (locale policy): formalised in `docs/standards/localization.md` + via Task G. +- M (diagram / asset sweep): user-facing tree image-free; no + drift. + +**Migration notes (Wave 5):** + +- **`pyproject.toml` version `0.5.1rc1` → `0.5.5`.** Brought + forward from Faz 33 because `tools/check_site_claims.py + --strict` enforces site/code version match (site cites + `pip install forgelm==0.5.5`). The Faz 33 release commit + pushes the tag + PyPI publish; v0.5.5 will be installable from + PyPI within minutes of that release. +- **Two new strict CI gates land.** Operators running ForgeLM + CI workflows on top of the project's existing `ci.yml` gain + `tools/check_anchor_resolution.py --strict` (Wave 4 / Faz 26 + tool, advisory until this commit) and + `tools/check_cli_help_consistency.py --strict` (Wave 5 / Faz + 30 Task J new tool). Both fail-the-build on drift; baselines + are at zero before the strict flip lands. +- **8 new usermanual pages + 16 new bilingual doc pairs.** No + user-facing behaviour change; the navigation surface grows + but the underlying capabilities all shipped in Wave 1-4. + ### Wave 4 — Faz 22 + 23 + 26 + 30 (`closure/wave4-integration`) ISO 27001 / SOC 2 Type II alignment + QMS bilingual sweep + final diff --git a/CLAUDE.md b/CLAUDE.md index 80232345..122e1cd9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,26 +46,46 @@ Each skill's `SKILL.md` has the full checklist. Follow it; don't skip steps to s ``` ForgeLM/ -├── forgelm/ # Source code (17 single-file modules) -│ ├── cli.py # Entry point +├── forgelm/ # Source code: ~22 single-file modules + 2 sub-packages +│ ├── cli/ # CLI package (Phase 15 split): _parser, _dispatch, +│ │ # _exit_codes, subcommands/{ingest, audit, chat, +│ │ # export, deploy, quickstart, doctor, cache, +│ │ # purge, reverse_pii, approve, approvals, +│ │ # safety_eval, verify_audit, verify_annex_iv, +│ │ # verify_gguf, ...} +│ ├── data_audit/ # Audit package (Phase 14 split): _orchestrator, +│ │ # _aggregator, _streaming, _simhash, _minhash, +│ │ # _pii_regex, _pii_ml, _secrets, _quality, +│ │ # _croissant, _summary, _splits │ ├── config.py # Pydantic schemas (19 models) │ ├── trainer.py # TRL wrapper (SFT/DPO/SimPO/KTO/ORPO/GRPO) │ ├── model.py # HF + PEFT model loading │ ├── data.py # Dataset loading + format detection +│ ├── ingestion.py # Raw docs → SFT JSONL (`forgelm ingest`) │ ├── safety.py # Llama Guard + harm categories + auto-revert -│ ├── compliance.py # EU AI Act Articles 9-17 + Annex IV -│ ├── webhook.py # Slack/Teams notifications +│ ├── compliance.py # EU AI Act Articles 9-17 + Annex IV + GDPR purge / reverse-pii primitives +│ ├── webhook.py # Slack/Teams notifications (5-event vocabulary) │ ├── grpo_rewards.py # Built-in GRPO format/length shaping reward fallback -│ └── ... # benchmark, judge, merging, synthetic, wizard, ... -├── tests/ # pytest test modules +│ ├── _http.py # SSRF-guarded HTTP chokepoint (safe_post / safe_get) +│ ├── _version.py # `__version__` + `__api_version__` (decoupled) +│ └── ... # benchmark, judge, merging, synthetic, wizard, +│ # quickstart, model_card, fit_check, deploy, chat, +│ # export, inference, results, utils +├── tests/ # 68 test modules, ~1442 tests +├── tools/ # CI guards: check_anchor_resolution, +│ # check_bilingual_parity, check_cli_help_consistency, +│ # check_field_descriptions, check_pip_audit, +│ # check_bandit, check_site_claims, +│ # generate_sbom, build_usermanuals ├── docs/ │ ├── roadmap.md # Public roadmap (short index) │ ├── roadmap/ # Detailed phase files + archive │ ├── reference/ # User-facing API/config reference │ ├── guides/ # User-facing tutorials +│ ├── usermanuals/{en,tr}/ # 4-section user manual (training, eval, deploy, ref) │ ├── design/ # Design specs (internal) │ ├── standards/ # Engineering standards (this project's rulebook) -│ ├── qms/ # Quality management SOPs (EU AI Act Art. 17) +│ ├── qms/ # Quality management SOPs (EU AI Act Art. 17, EN+TR) │ ├── analysis/ # Research, code reviews, external repo analyses │ └── marketing/ # Local-only (gitignored): marketing + strategy ├── config_template.yaml # Canonical YAML example — CI dry-runs this @@ -127,10 +147,16 @@ Default workflow for a non-trivial change: ```bash ruff format . && ruff check . && pytest tests/ && \ - forgelm --config config_template.yaml --dry-run + forgelm --config config_template.yaml --dry-run && \ + python3 tools/check_bilingual_parity.py --strict && \ + python3 tools/check_anchor_resolution.py --strict && \ + python3 tools/check_cli_help_consistency.py --strict ``` - All four must pass. + All seven must pass. The first four are the historical gauntlet; the + three doc guards (Wave 3 / Wave 4 / Wave 5 additions) catch bilingual + structural drift, broken markdown anchors, and CLI ↔ docs help-text + drift before the PR opens. ## Etiquette when communicating with the user diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2d1e93ce..a12e5f5e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -38,19 +38,31 @@ Branch naming: `feat/`, `fix/`, `docs/`, `test/`, `chore/` + short description. ### 4. Make your changes -Edit the code, then verify: +Edit the code, then run the full validation gauntlet (every guard CI also +enforces — passing locally means CI will too): ```bash -# Run tests -pytest tests/ -q +# 1. Lint + format +ruff format . && ruff check . -# Run linter + format check -ruff check . && ruff format --check . +# 2. Test suite +pytest tests/ -q -# Quick smoke test +# 3. Config dry-run smoke test forgelm --config config_template.yaml --dry-run + +# 4. Doc-side guards (Wave 4 / Wave 5 additions) +python3 tools/check_bilingual_parity.py --strict +python3 tools/check_anchor_resolution.py --strict +python3 tools/check_cli_help_consistency.py --strict ``` +The four-tool sequence at the top of the gauntlet (`ruff` + `pytest` + +`--dry-run`) is the historical "self-review" command from +[`docs/standards/code-review.md`](docs/standards/code-review.md). The three +doc guards landed in Waves 3-5 and run on every PR via `.github/workflows/`; +running them locally before pushing avoids CI round-trips. + ### 5. Submit a PR Push your branch and open a Pull Request against `main`. @@ -59,10 +71,13 @@ Push your branch and open a Pull Request against `main`. ### Project Structure -ForgeLM is a single-package layout: ~26 single-file modules under `forgelm/`, -~47 test files under `tests/`, plus `configs/`, `docs/`, and `notebooks/`. For -the authoritative module-by-module map (purpose, public surface, dependency -arrows), see [`docs/reference/architecture.md`](docs/reference/architecture.md). +ForgeLM is a single-package layout: a mix of single-file modules and two +focused sub-packages (`forgelm/cli/` post-Phase-15 split and +`forgelm/data_audit/` post-Phase-14 split) under `forgelm/`, ~68 test files / +~1442 tests under `tests/`, plus `configs/`, `docs/`, `tools/` (CI guards), +and `notebooks/`. For the authoritative module-by-module map (purpose, public +surface, dependency arrows), see +[`docs/reference/architecture.md`](docs/reference/architecture.md). ### Running Tests diff --git a/README.md b/README.md index 72efe421..569b9ed7 100644 --- a/README.md +++ b/README.md @@ -55,13 +55,17 @@ - **Config-Driven**: Declarative YAML — built for CI/CD pipelines, not notebooks - **EU AI Act Compliance**: Auto-generated audit trails, data provenance (SHA-256), training manifests - **ISO 27001 / SOC 2 Type II Alignment**: Software cannot be ISO/SOC 2 *certified* — only organisations can — but ForgeLM produces the audit-trail, change-management, data-lineage, and supply-chain evidence the deployer's auditor explicitly asks for. CycloneDX 1.5 SBOM per release, `pip-audit` nightly, `bandit` CI, append-only HMAC-chained audit log (HMAC integrity is active **only when** `FORGELM_AUDIT_SECRET` is set + KMS-managed + rotated between output-dir lifecycles — see [`docs/qms/access_control.md`](docs/qms/access_control.md) §3.4), Article 14 staging gate, Article 15/17 GDPR tooling. See [`docs/guides/iso_soc2_deployer_guide.md`](docs/guides/iso_soc2_deployer_guide.md) for the deployer audit cookbook + 93-control coverage map. +- **Library API**: `from forgelm import ForgeTrainer, audit_dataset, verify_audit_log, verify_annex_iv_artifact, verify_gguf, mask_pii, mask_secrets, ...` — every CLI surface has a stable Python entry point under `forgelm.__all__`, version-pinned via `forgelm.__api_version__` (decoupled from `__version__`). Run `python -c "import forgelm; print(sorted(forgelm.__all__))"` for the full surface. +- **GDPR Tooling**: `forgelm purge --subject-id ` (Article 17 right-to-erasure: redacts data + adapters in place; the audit log itself is append-only — Article 17(3)(b) preservation — and the erasure is recorded by appending six compensating events (`data.erasure_requested`, `data.erasure_completed`, etc.) so the hash chain remains intact) and `forgelm reverse-pii --query ` (Article 15 right-of-access: locates PII matches across artefacts without re-loading raw data). +- **Operational Subcommands**: `forgelm doctor` (env / GPU / CUDA / extras pre-flight), `forgelm cache-models` + `forgelm cache-tasks` (air-gap pre-cache for HF models + lm-eval tasks), `forgelm safety-eval` (standalone Llama Guard run), `forgelm verify-audit` / `verify-annex-iv` / `verify-gguf` (compliance + artefact verification toolbelt), `forgelm approvals` (list pending Article 14 staging-gate runs). - **Docker**: Official Dockerfile and docker-compose for portable deployment - **Offline / Air-Gapped**: Full operation without internet for regulated industries -- **JSON Output**: Machine-readable results with `--output-format json` for pipeline integration -- **Webhook Notifications**: Slack/Teams alerts on training start, success, or failure +- **JSON Output**: Machine-readable results with `--output-format json` for pipeline integration; the per-subcommand schema is locked in [`docs/usermanuals/en/reference/json-output.md`](docs/usermanuals/en/reference/json-output.md). +- **Webhook Notifications**: Slack/Teams alerts on training start, success, failure, reverted, or awaiting_approval (5-event vocabulary, paired with audit events) - **W&B / MLflow / TensorBoard**: Flexible experiment tracking via `report_to` - **Model Card Generation**: Auto-generated HF-compatible model cards with metrics and benchmarks - **Model Merging**: TIES, DARE, SLERP, linear merge of multiple adapters via `--merge` +- **Supply-Chain Security**: CycloneDX 1.5 SBOM per release-tag matrix combo (12 SBOMs per tag), `pip-audit` + `bandit` nightly + on-tag, `gitleaks` pre-commit. See [`docs/reference/supply_chain_security.md`](docs/reference/supply_chain_security.md). --- @@ -233,10 +237,29 @@ forgelm --version # Show version ``` forgelm/ -├── cli.py # CLI entry point (train, dry-run, merge, benchmark, wizard, ingest, audit, ...) +├── cli/ # CLI package (split out of legacy single-file cli.py) +│ ├── __main__.py # `python -m forgelm.cli` entrypoint +│ ├── _parser.py # argparse wiring for the top-level + every subcommand +│ ├── _dispatch.py # subcommand router; one entry per *_subcommand* below +│ ├── _exit_codes.py# 0/1/2/3/4 — public CLI contract +│ └── subcommands/ # _audit, _ingest, _chat, _export, _deploy, _quickstart, +│ # _doctor, _cache, _purge, _reverse_pii, _approve, +│ # _approvals, _safety_eval, _verify_audit, +│ # _verify_annex_iv, _verify_gguf, ... +├── data_audit/ # Audit package (split out of legacy single-file data_audit.py) +│ ├── _orchestrator.py # `run_audit` entry point + parallel split walker +│ ├── _aggregator.py # per-split metric aggregator +│ ├── _streaming.py # streaming JSONL reader +│ ├── _simhash.py # 64-bit simhash + LSH banding (default dedup) +│ ├── _minhash.py # MinHash LSH dedup (`[ingestion-scale]` extra) +│ ├── _pii_regex.py # PII regex engine (Luhn, IBAN, TC Kimlik validators) +│ ├── _pii_ml.py # Presidio NER adapter (`[ingestion-pii-ml]` extra) +│ ├── _secrets.py # 9-family credential scan (always-on) +│ ├── _quality.py # Gopher / C4 / RefinedWeb-style heuristics +│ ├── _croissant.py # Croissant 1.0 dataset card emitter +│ └── _summary.py # `summarize_report` truncation policy ├── config.py # Pydantic config (19 models: training, evaluation, distributed, ...) ├── data.py # Dataset loading (SFT, DPO, KTO, GRPO formats + multi-dataset) -├── data_audit.py # Audit pipeline (length / language / dedup / leakage / PII / secrets / quality) — `forgelm audit` ├── ingestion.py # Raw docs → SFT JSONL (PDF/DOCX/EPUB/TXT/Markdown + chunking + masking) — `forgelm ingest` ├── model.py # Model loading (transformers, unsloth, MoE, PEFT) ├── trainer.py # Training orchestration (6 trainer types via TRL, GaLore, long-context) @@ -245,26 +268,33 @@ forgelm/ ├── export.py # GGUF export via llama-cpp-python ├── fit_check.py # Pre-flight VRAM estimator (FITS / TIGHT / OOM / UNKNOWN) ├── deploy.py # Deployment config generator (Ollama, vLLM, TGI, HF Endpoints) -├── results.py # TrainResult dataclass (AuditReport lives in data_audit.py, IngestionResult in ingestion.py) +├── results.py # TrainResult dataclass (AuditReport lives in data_audit/, IngestionResult in ingestion.py) ├── benchmark.py # lm-evaluation-harness integration ├── safety.py # Post-training safety evaluation (Llama Guard) ├── judge.py # LLM-as-Judge evaluation (API + local) -├── compliance.py # EU AI Act compliance export & data provenance +├── compliance.py # EU AI Act compliance export, audit log + HMAC chain, GDPR purge / reverse-pii primitives ├── model_card.py # Auto-generated HF model cards ├── merging.py # Model merging (TIES, DARE, SLERP, linear) ├── synthetic.py # Synthetic data generation (teacher→student distillation) ├── grpo_rewards.py # Built-in GRPO reward shapers (format / length fallbacks) ├── quickstart.py # `forgelm quickstart