From 04b06a1b78f93ef1bc227661e15bd38c9db36a60 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:23:08 +0300 Subject: [PATCH 01/41] ci(faz-32): add optional .pre-commit-config.yaml Closes F-test-008 (Major) with Option A: pre-commit hooks are opt-in for local development; CI remains the single enforcement boundary. Mirrors the existing CI checks (ruff, ruff-format, gitleaks) plus hygiene hooks for trailing whitespace, EOF newlines, and YAML/TOML syntax. Versions pinned per closure-plan-202604300906.md Faz 32. CI workflow is intentionally NOT modified: ruff and gitleaks already run as separate steps in .github/workflows/ci.yml, so adding `pre-commit run --all-files` would be double enforcement with no marginal value. Co-Authored-By: Claude Opus 4.7 (1M context) --- .pre-commit-config.yaml | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 .pre-commit-config.yaml diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 00000000..9b2d32f7 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,37 @@ +# Pre-commit hooks for ForgeLM (optional, developer-installed). +# +# CI is the enforcement boundary: ruff and gitleaks already run as separate +# steps in .github/workflows/ci.yml. These hooks exist purely as a local +# ergonomic optimization for contributors who want fast feedback before push. +# +# Install: +# pip install pre-commit && pre-commit install +# +# Run all hooks against the whole tree: +# pre-commit run --all-files +# +# Skip a hook (e.g. when committing a known-test fixture): +# SKIP=gitleaks git commit -m "..." +# +# Versions are pinned per closure-plan-202604300906.md Faz 32. + +repos: + - repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.8.0 + hooks: + - id: ruff + - id: ruff-format + + - repo: https://github.com/gitleaks/gitleaks + rev: v8.18.0 + hooks: + - id: gitleaks + + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.5.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-toml + - id: check-merge-conflict From 597f7598230bff58cdd4b0f4283f73d886ae2353 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:23:16 +0300 Subject: [PATCH 02/41] docs(faz-32): document optional pre-commit hooks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a "Pre-commit hooks (optional)" section to CONTRIBUTING.md under the existing Code Style guidance, with install + run + skip instructions. Adds a parallel note to docs/standards/code-review.md clarifying that pre-commit is a local ergonomic optimization and CI is the single enforcement boundary — skipping a hook locally never bypasses CI. Co-Authored-By: Claude Opus 4.7 (1M context) --- CONTRIBUTING.md | 31 +++++++++++++++++++++++++++++++ docs/standards/code-review.md | 13 +++++++++++++ 2 files changed, 44 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ca99966a..2d1e93ce 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -95,6 +95,37 @@ ruff format . Configuration is in `pyproject.toml` under `[tool.ruff]`. +### Pre-commit hooks (optional) + +ForgeLM ships a [`.pre-commit-config.yaml`](.pre-commit-config.yaml) that mirrors +the CI checks (`ruff`, `ruff-format`, `gitleaks`, plus a few hygiene hooks for +trailing whitespace, EOF newlines, and YAML/TOML syntax). The hooks are an +**optional ergonomic optimization** — CI enforces the same checks on every PR, +so installing them locally is purely about getting feedback before you push. + +To opt in: + +```bash +pip install pre-commit +pre-commit install +``` + +Run all hooks against the whole tree at any time: + +```bash +pre-commit run --all-files +``` + +If a hook ever flags a known-good fixture (e.g. a credential-shaped test +string the gitleaks hook can't tell apart from a real secret), skip just that +hook for the commit: + +```bash +SKIP=gitleaks git commit -m "test: add credential-shape fixture" +``` + +CI remains the enforcement boundary; skipping a hook locally never bypasses CI. + ## Guidelines ### Code diff --git a/docs/standards/code-review.md b/docs/standards/code-review.md index 7fd0a30b..285e21b2 100644 --- a/docs/standards/code-review.md +++ b/docs/standards/code-review.md @@ -191,3 +191,16 @@ forgelm --config config_template.yaml --dry-run ``` If that passes and the PR template is honest, you're ready. + +## Pre-commit hooks (optional) + +The repository ships a [`.pre-commit-config.yaml`](../../.pre-commit-config.yaml) +that mirrors the lint, format, and secret-scanning steps already run in CI. +Pre-commit hooks are an **optional ergonomic optimization** for local +development — contributors who run `pre-commit install` get faster feedback, +but the hooks are **not** part of the PR enforcement contract. CI is the +single enforcement boundary: every required check (`ruff check`, `ruff format +--check`, `gitleaks`, the test matrix, the validate job) runs on every push +and pull request regardless of whether the author installed the hooks +locally. Skipping a hook with `SKIP=...` only suppresses the local run; it +never bypasses CI. From 1fd44c7d2c798e5050377d08aac2ee1435b15d64 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:25:01 +0300 Subject: [PATCH 03/41] ci(faz-31): tag-driven release matrix (3 OS x 4 Python) with SBOM artifacts Replace the release-event publish.yml with a tag-driven workflow that chains three jobs (build -> cross-os-tests -> publish) so a v* tag fans out into 12 (OS, Python) test combinations before PyPI publish runs. - build: Python 3.11 on ubuntu-latest produces dist/*.whl + dist/*.tar.gz and uploads them as the dist artifact for downstream jobs. - cross-os-tests: matrix of {ubuntu, macos, windows} x {3.10, 3.11, 3.12, 3.13} = 12 combos, fail-fast: false. Each combo downloads the dist artifact, installs the packaged wheel (NOT editable) so the test exercises exactly what PyPI users will pip install, runs pytest with test_cost_estimation excluded (pricing fixture drift), generates a CycloneDX 1.5 SBOM via tools/generate_sbom.py, and uploads the SBOM as a per-combo artifact. Linux-only step adds qlora/ingestion/eval extras. All shell steps declare shell: bash so the same scripts run on Windows. - publish: needs cross-os-tests; OIDC trusted publishing to PyPI via pypa/gh-action-pypi-publish@release/v1, environment: pypi. Closes F-test-007 (Major) per docs/analysis/code_reviews/closure-plan Faz 31. tools/generate_sbom.py is a stdlib-only CycloneDX 1.5 emitter so the SBOM step never silently degrades a matrix combo because of an optional cyclonedx-bom transitive failure on Python 3.13/Windows. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/publish.yml | 103 +++++++++++++++++++--------- tools/generate_sbom.py | 125 ++++++++++++++++++++++++++++++++++ 2 files changed, 197 insertions(+), 31 deletions(-) create mode 100644 tools/generate_sbom.py diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 6fa60dd9..2fe177fe 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1,16 +1,22 @@ -name: Publish to PyPI +name: Release - Build, cross-OS validate, publish to PyPI + +# Tag-driven release pipeline. A pushed v* tag fans out into: +# build -> cross-os-tests (3 OS x 4 Python = 12) -> publish +# Any matrix combo failure breaks the needs: chain so PyPI publish does +# not run unless every supported (OS, Python) pair installs the packaged +# wheel cleanly and runs the test suite green. on: - release: - types: [published] + push: + tags: ['v*'] -# No workflow-level permissions — each job declares the minimal set it needs. +# No workflow-level permissions; each job declares the minimum it needs. jobs: build: runs-on: ubuntu-latest permissions: - contents: read # checkout needs read access to the release ref + contents: read # checkout reads the tagged ref steps: - uses: actions/checkout@v5 @@ -18,15 +24,15 @@ jobs: with: python-version: "3.11" - - name: Install build tools - run: pip install build + - name: Install build tooling + run: python -m pip install --upgrade pip build - - name: Build sdist and wheel - run: python -m build + - name: Build wheel + sdist + run: python -m build # outputs dist/*.whl and dist/*.tar.gz - - name: Verify package + - name: Verify package metadata run: | - pip install twine + python -m pip install twine twine check dist/* - uses: actions/upload-artifact@v5 @@ -34,40 +40,75 @@ jobs: name: dist path: dist/ - # TestPyPI: triggered by pre-release tags (e.g., v0.3.0-rc1, v0.3.0-beta) - # or releases created from test/development branches - publish-testpypi: + cross-os-tests: needs: build - runs-on: ubuntu-latest - if: github.event.release.prerelease == true - environment: - name: testpypi - url: https://test.pypi.org/p/forgelm + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + python: ['3.10', '3.11', '3.12', '3.13'] + runs-on: ${{ matrix.os }} permissions: - contents: read # download-artifact reads workflow run state - id-token: write # required for trusted publishing to TestPyPI + contents: read # checkout reads the tagged ref for tests/ + tools/ steps: + - uses: actions/checkout@v5 + + - uses: actions/setup-python@v6 + with: + python-version: ${{ matrix.python }} + - uses: actions/download-artifact@v5 with: name: dist - path: dist/ + path: dist - - name: Publish to TestPyPI - uses: pypa/gh-action-pypi-publish@release/v1 + - name: Install packaged wheel (NOT editable) + shell: bash # POSIX shell on Windows runners too + run: | + # Install exactly what PyPI users will pip install. Transitive + # dependencies resolve from PyPI; the local dist/ only contains + # the forgelm wheel, not its dependency closure (so no + # --no-index --find-links here). + python -m pip install --upgrade pip + python -m pip install dist/*.whl + + - name: Install Linux-only extras (qlora, ingestion, eval) + if: runner.os == 'Linux' + shell: bash + # qlora pulls bitsandbytes which only ships Linux wheels. + # ingestion + eval are cross-platform but bundled here so a single + # conditional install covers the breadth-test surface on Linux. + run: python -m pip install 'forgelm[qlora,ingestion,eval]' + + - name: Install pytest + shell: bash + run: python -m pip install pytest pytest-cov + + - name: Test (CPU-only, no-network) + shell: bash + # test_cost_estimation is excluded per Faz 31 closure plan: it + # depends on a snapshot pricing fixture that drifts on a different + # cadence than the release matrix. + run: pytest tests/ -q --ignore=tests/test_cost_estimation.py + + - name: Generate SBOM (CycloneDX 1.5) + shell: bash + run: python tools/generate_sbom.py > sbom-${{ matrix.os }}-py${{ matrix.python }}.json + + - uses: actions/upload-artifact@v5 with: - repository-url: https://test.pypi.org/legacy/ + name: sbom-${{ matrix.os }}-py${{ matrix.python }} + path: sbom-${{ matrix.os }}-py${{ matrix.python }}.json - # PyPI: triggered by stable releases (non-prerelease) from main - publish-pypi: - needs: build + publish: + needs: cross-os-tests # blocked until every matrix combo is green runs-on: ubuntu-latest - if: github.event.release.prerelease == false environment: name: pypi url: https://pypi.org/p/forgelm permissions: - contents: read # download-artifact reads workflow run state - id-token: write # required for trusted publishing to PyPI + contents: read # download-artifact reads workflow run state + id-token: write # OIDC trusted publishing to PyPI steps: - uses: actions/download-artifact@v5 with: diff --git a/tools/generate_sbom.py b/tools/generate_sbom.py new file mode 100644 index 00000000..c2abac45 --- /dev/null +++ b/tools/generate_sbom.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Emit a minimal CycloneDX 1.5 SBOM for the active Python environment. + +Reads the installed package set via ``pip list --format=json`` and renders a +CycloneDX 1.5 JSON document on stdout. Pure stdlib, cross-platform — used by +the ``publish.yml`` cross-OS matrix to attach a per-(os, python-version) SBOM +artifact to every tagged release. + +This is intentionally a hand-rolled emitter rather than the optional +``cyclonedx-bom`` PyPI package: keeping the dependency footprint at zero means +the SBOM step can never silently degrade an otherwise-green matrix combo just +because a transitive dep failed to wheel-build on (say) Python 3.13/Windows. + +Output schema reference: https://cyclonedx.org/docs/1.5/json/ + +Usage: + python tools/generate_sbom.py > sbom.json +""" + +from __future__ import annotations + +import datetime as _dt +import json +import platform +import subprocess +import sys +import uuid +from importlib.metadata import PackageNotFoundError, version +from typing import Any + + +def _installed_packages() -> list[dict[str, str]]: + """Return ``pip list --format=json`` output as a list of dicts. + + Falls back to an empty list if pip is unavailable; the SBOM will still be + syntactically valid (just empty ``components``). + """ + try: + result = subprocess.run( + [sys.executable, "-m", "pip", "list", "--format=json", "--disable-pip-version-check"], + capture_output=True, + text=True, + check=True, + timeout=60, + ) + except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError): + return [] + try: + return json.loads(result.stdout) + except json.JSONDecodeError: + return [] + + +def _purl(name: str, ver: str) -> str: + """Build a Package URL (purl) for a PyPI package.""" + # CycloneDX expects RFC-style purls; pkg:pypi/@. + return f"pkg:pypi/{name}@{ver}" + + +def _component(pkg: dict[str, str]) -> dict[str, Any]: + name = pkg.get("name", "") + ver = pkg.get("version", "") + return { + "type": "library", + "name": name, + "version": ver, + "purl": _purl(name, ver), + "bom-ref": _purl(name, ver), + } + + +def _forgelm_version() -> str: + try: + return version("forgelm") + except PackageNotFoundError: + return "0.0.0+unknown" + + +def build_sbom() -> dict[str, Any]: + """Construct the CycloneDX 1.5 SBOM document.""" + now = _dt.datetime.now(tz=_dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + forgelm_ver = _forgelm_version() + components = [_component(p) for p in _installed_packages()] + return { + "bomFormat": "CycloneDX", + "specVersion": "1.5", + "serialNumber": f"urn:uuid:{uuid.uuid4()}", + "version": 1, + "metadata": { + "timestamp": now, + "tools": [ + { + "vendor": "ForgeLM", + "name": "generate_sbom.py", + "version": "1.0.0", + } + ], + "component": { + "type": "application", + "name": "forgelm", + "version": forgelm_ver, + "purl": _purl("forgelm", forgelm_ver), + "bom-ref": _purl("forgelm", forgelm_ver), + }, + "properties": [ + {"name": "python:version", "value": platform.python_version()}, + {"name": "python:implementation", "value": platform.python_implementation()}, + {"name": "platform:system", "value": platform.system()}, + {"name": "platform:release", "value": platform.release()}, + {"name": "platform:machine", "value": platform.machine()}, + ], + }, + "components": components, + } + + +def main() -> int: + sbom = build_sbom() + json.dump(sbom, sys.stdout, indent=2, sort_keys=False) + sys.stdout.write("\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 3da7f3d56c742d216f307e97fec7956d80431a01 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:25:15 +0300 Subject: [PATCH 04/41] docs(faz-31): document Release prep workflow + cross-OS substantiation - docs/standards/release.md: add a "Release prep workflow" section that walks through the three jobs (build -> cross-os-tests -> publish), the 3 OS x 4 Python = 12 matrix, why we install the packaged wheel rather than editable, the per-combo SBOM artifact, OIDC trusted publishing, and the failure semantics that gate PyPI on a green matrix. Update the existing "Automated (by publish.yml)" subsection to point at the new section instead of describing a single-job pipeline that no longer exists. - README.md: add a one-line OS-support callout under the badges block citing the 3 OS x 4 Python release matrix as the substantiation for the PyPI "Operating System :: OS Independent" classifier, with a link to the workflow file. Linux-only extras (qlora, unsloth) remain flagged on their pip install lines. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 2 + docs/standards/release.md | 109 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 106 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e2cb5f0b..d428fe81 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,8 @@ [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) [![CI](https://github.com/cemililik/ForgeLM/actions/workflows/ci.yml/badge.svg)](https://github.com/cemililik/ForgeLM/actions/workflows/ci.yml) +**Runs on Linux, macOS, and Windows.** The PyPI metadata's `Operating System :: OS Independent` classifier is backed by a release-tag CI matrix that builds the wheel on Linux and re-installs + tests it across **3 operating systems × 4 Python versions = 12 combinations** (Ubuntu, macOS, Windows × Python 3.10, 3.11, 3.12, 3.13) — every combo must pass before PyPI publish runs. See [`.github/workflows/publish.yml`](.github/workflows/publish.yml). Linux-only extras (`qlora`, `unsloth`) are flagged on each `pip install "forgelm[...]"` line below. + **ForgeLM** is a config-driven, enterprise-ready LLM fine-tuning toolkit. It supports the full modern post-training stack — from supervised fine-tuning to preference alignment to reasoning RL — with integrated safety evaluation, EU AI Act compliance, and CI/CD-native design. ## Features diff --git a/docs/standards/release.md b/docs/standards/release.md index 82483a50..b666ab2d 100644 --- a/docs/standards/release.md +++ b/docs/standards/release.md @@ -122,12 +122,15 @@ Done manually by the maintainer (or a bot when automated): ### Automated (by `publish.yml`) -On tag push matching `v*`: +On tag push matching `v*` the workflow chains three jobs (full description +under [Release prep workflow](#release-prep-workflow) below): -1. Build wheel and sdist (`python -m build`). -2. Verify with `twine check`. -3. Publish to PyPI using OIDC trusted publishing (no API key needed). -4. Create GitHub Release with auto-extracted changelog excerpt. +1. **`build`** — produce wheel + sdist, verify with `twine check`, upload + as the `dist` artifact. +2. **`cross-os-tests`** — 3 OS × 4 Python = 12 combos install the packaged + wheel (not editable), run pytest, generate a per-combo CycloneDX SBOM. +3. **`publish`** — OIDC trusted publishing to PyPI; only runs after every + matrix combo is green. ### After the release @@ -138,6 +141,102 @@ On tag push matching `v*`: 5. [ ] Bump `pyproject.toml` version to next pre-release (`0.4.1rc1`). 6. [ ] Update [marketing/marketing_strategy_roadmap.md](../marketing/marketing_strategy_roadmap.md) metrics row. +## Release prep workflow + +The release pipeline is a single GitHub Actions workflow, +[`.github/workflows/publish.yml`](../../.github/workflows/publish.yml). It +fires when a tag matching `v*` is pushed (e.g. `v0.5.0`, `v0.5.1rc1`) and +chains three jobs whose `needs:` dependencies form a strict DAG. + +### Trigger + +```yaml +on: + push: + tags: ['v*'] +``` + +Pushing the tag is the contract — no manual `workflow_dispatch`, no +release-event listener. The same procedure works whether you tag locally +and `git push --tags` or cut the tag from the GitHub Releases UI. + +### Job 1 — `build` (`ubuntu-latest`, Python 3.11) + +Runs `python -m build` to produce both `dist/*.whl` and `dist/*.tar.gz`, +then `twine check dist/*` to validate metadata, then uploads `dist/` as a +workflow artifact named `dist`. Subsequent jobs download this same +artifact rather than rebuilding, so every matrix combo and the publish +step exercise byte-identical files. + +### Job 2 — `cross-os-tests` (matrix: 3 OS × 4 Python = 12 combos) + +`needs: build`. Strategy: + +```yaml +fail-fast: false +matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + python: ['3.10', '3.11', '3.12', '3.13'] +``` + +`fail-fast: false` is deliberate — one Python version blowing up on one +OS must not abort the other 11 combos, otherwise the matrix loses its job +as a breadth probe. + +Each combo: + +1. Downloads the `dist` artifact. +2. Installs the wheel via `python -m pip install dist/*.whl` — **packaged + wheel, not editable**. This is the load-bearing detail. An editable + install (`pip install -e .`) does not exercise wheel build, package + data inclusion, console-script generation, or cross-OS path handling. + By installing the same wheel that PyPI users will pull we guarantee + that what passes here is what they get. +3. On Linux only, additionally installs `forgelm[qlora,ingestion,eval]` + to cover the heaviest extras path. `qlora` pulls `bitsandbytes`, which + only ships Linux wheels — we never claim Windows / macOS support for + that extra. +4. Runs `pytest tests/ -q --ignore=tests/test_cost_estimation.py`. The + cost-estimation test is excluded because its pricing fixture drifts on + a different cadence than the release matrix; a stale fixture would + break the chain for reasons unrelated to packaging health. +5. Generates a CycloneDX 1.5 SBOM via `python tools/generate_sbom.py`, + redirected to `sbom-${{ matrix.os }}-py${{ matrix.python }}.json`. +6. Uploads each SBOM as its own artifact — 12 per release tag, retained + alongside the workflow run for downstream supply-chain audits. + +All steps that may run on Windows declare `shell: bash` so the same +script fragments work on Linux, macOS, and Windows runners (Windows +defaults to PowerShell otherwise). + +### Job 3 — `publish` (`ubuntu-latest`, environment: `pypi`) + +`needs: cross-os-tests`. Downloads the `dist` artifact and hands it to +[`pypa/gh-action-pypi-publish@release/v1`](https://github.com/pypa/gh-action-pypi-publish). +Authentication is OIDC trusted publishing — no PyPI API token is stored; +GitHub Actions mints a short-lived token scoped to the `pypi` environment. +The job sets `permissions: { id-token: write, contents: read }`; nothing +else is granted. + +The `pypi` GitHub environment is configured in repository settings to +require manual approval for production tags if extra control is wanted; +the workflow itself does not gate beyond the `needs:` chain. + +### Failure semantics + +The `needs:` chain is the safety net: + +- If `build` fails → neither `cross-os-tests` nor `publish` runs. +- If **any** `cross-os-tests` combo fails (with `fail-fast: false`, the + other 11 still run so the failure surface is visible) → `publish` does + not run. Nothing reaches PyPI. +- The tag remains in git either way; re-running the workflow after a fix + is the recovery path, not deleting + recreating the tag. + +This is by design: a release that ships only on Linux/3.11 because the +3.13 wheel was broken would silently degrade users on the un-tested +combos. Packaging hygiene is gated, not advisory. + ## Release cadence Current target: From 6b4a4885b8b37a1a78515deeea4e3f63caffeb82 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:24:54 +0300 Subject: [PATCH 05/41] refactor(faz-10): tighten 6 enum-shaped config fields to Literal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes F-code-101 + F-compliance-105. Pydantic Literal validates these fields at parse time so invalid values raise ValidationError with the field path before reaching any runtime validator. Fields converted in forgelm/config.py: - LoraConfigModel.bias: Literal["none","all","lora_only"] - DistributedConfig.fsdp_backward_prefetch: Literal["backward_pre","backward_post"] - DistributedConfig.fsdp_state_dict_type: Literal["FULL_STATE_DICT","SHARDED_STATE_DICT"] - SafetyConfig.scoring: Literal["binary","confidence_weighted"] - ComplianceMetadataConfig.risk_classification: Literal["high-risk","limited-risk","minimal-risk"] - TrainingConfig.galore_optim: Literal[6 documented optimizers] - TrainingConfig.galore_proj_type: Literal["std","reverse_std","right","left","full"] Bespoke whitelist check for galore_optim removed from ForgeConfig._validate_galore — Literal handles it at field-parse time (model_validator runs strictly after field validation, so the check is unreachable for invalid values). risk_classification option set kept aligned with the existing default ("minimal-risk"), the wizard prompt, the docs reference tables, and the existing test fixtures — not the closure-plan's draft set. Co-Authored-By: Claude Opus 4.7 (1M context) --- forgelm/config.py | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/forgelm/config.py b/forgelm/config.py index 37f0173c..ad1c3a97 100644 --- a/forgelm/config.py +++ b/forgelm/config.py @@ -74,7 +74,7 @@ class LoraConfigModel(BaseModel): r: int = 8 alpha: int = 16 dropout: float = 0.1 - bias: str = "none" + bias: Literal["none", "all", "lora_only"] = "none" method: Literal["lora", "dora", "pissa", "rslora"] = "lora" use_dora: bool = False # kept for backward compat; method="dora" also works use_rslora: bool = False # rank-stabilized LoRA for high ranks (r>64) @@ -146,11 +146,18 @@ class TrainingConfig(BaseModel): ) # --- GaLore (optimizer-level memory optimization, alternative to LoRA) --- galore_enabled: bool = False - galore_optim: str = "galore_adamw" # galore_adamw, galore_adamw_8bit, galore_adafactor, + _layerwise variants + galore_optim: Literal[ + "galore_adamw", + "galore_adamw_8bit", + "galore_adafactor", + "galore_adamw_layerwise", + "galore_adamw_8bit_layerwise", + "galore_adafactor_layerwise", + ] = "galore_adamw" galore_rank: int = 128 # Low-rank subspace dimension for gradient projection galore_update_proj_gap: int = 200 # Steps between SVD re-computations galore_scale: float = 0.25 # Gradient scaling factor (analogous to LoRA alpha) - galore_proj_type: str = "std" # "std", "reverse_std", "right", "left", "full" + galore_proj_type: Literal["std", "reverse_std", "right", "left", "full"] = "std" galore_target_modules: Optional[List[str]] = None # Regex list; None = auto [r".*.attn.*", r".*.mlp.*"] # --- Long-context optimizations --- rope_scaling: Optional[Dict[str, Any]] = ( @@ -183,8 +190,8 @@ class DistributedConfig(BaseModel): fsdp_strategy: Literal["full_shard", "shard_grad_op", "no_shard", "hybrid_shard"] = "full_shard" fsdp_auto_wrap: bool = True # auto wrap transformer layers fsdp_offload: bool = False # offload parameters to CPU - fsdp_backward_prefetch: str = "backward_pre" # "backward_pre" or "backward_post" - fsdp_state_dict_type: str = "FULL_STATE_DICT" # "FULL_STATE_DICT" or "SHARDED_STATE_DICT" + fsdp_backward_prefetch: Literal["backward_pre", "backward_post"] = "backward_pre" + fsdp_state_dict_type: Literal["FULL_STATE_DICT", "SHARDED_STATE_DICT"] = "FULL_STATE_DICT" class DataGovernanceConfig(BaseModel): @@ -245,7 +252,7 @@ class SafetyConfig(BaseModel): test_prompts: str = "safety_prompts.jsonl" # adversarial test prompts file max_safety_regression: float = 0.05 # max allowed unsafe ratio (0.0–1.0) # Phase 9: Advanced scoring - scoring: str = "binary" # "binary" (default) or "confidence_weighted" + scoring: Literal["binary", "confidence_weighted"] = "binary" min_safety_score: Optional[float] = ( None # weighted score threshold (0.0-1.0), used when scoring="confidence_weighted" ) @@ -323,7 +330,7 @@ class ComplianceMetadataConfig(BaseModel): intended_purpose: str = "" known_limitations: str = "" system_version: str = "" - risk_classification: str = "minimal-risk" # "high-risk", "limited-risk", "minimal-risk" + risk_classification: Literal["high-risk", "limited-risk", "minimal-risk"] = "minimal-risk" class SyntheticConfig(BaseModel): @@ -481,19 +488,6 @@ def _warn_high_risk_compliance(self) -> None: def _validate_galore(self) -> None: if not self.training.galore_enabled: return - valid_galore_optims = { - "galore_adamw", - "galore_adamw_8bit", - "galore_adafactor", - "galore_adamw_layerwise", - "galore_adamw_8bit_layerwise", - "galore_adafactor_layerwise", - } - if self.training.galore_optim not in valid_galore_optims: - raise ValueError( - f"Invalid galore_optim: '{self.training.galore_optim}'. " - f"Must be one of: {', '.join(sorted(valid_galore_optims))}" - ) if self.lora.r > 0: logger.info( "GaLore (gradient rank=%d) enabled alongside LoRA (adapter rank=%d). " From 0b420ee0bcb6b69edaac36a90bdf1887d8ed4863 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:25:02 +0300 Subject: [PATCH 06/41] test(faz-10): cover Literal validation for the 7 newly-tightened fields New tests/test_config_literal_sweep.py: parametrized happy-path checks for every documented value plus one invalid-value case per field, asserting ValidationError carries the offending field name. tests/test_galore.py: relax the existing match string from "Invalid galore_optim" to "galore_optim". The bespoke runtime message disappeared along with the removed whitelist check; Pydantic's Literal ValidationError still names the field, which is the contract that matters here. Co-Authored-By: Claude Opus 4.7 (1M context) --- tests/test_config_literal_sweep.py | 104 +++++++++++++++++++++++++++++ tests/test_galore.py | 2 +- 2 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 tests/test_config_literal_sweep.py diff --git a/tests/test_config_literal_sweep.py b/tests/test_config_literal_sweep.py new file mode 100644 index 00000000..b62debd8 --- /dev/null +++ b/tests/test_config_literal_sweep.py @@ -0,0 +1,104 @@ +"""Faz 10: parse-time Literal validation sweep. + +Each enum-shaped string field that was tightened from `str` to `Literal[...]` +must accept every documented value and raise `pydantic.ValidationError` +with the offending field name in the message for any other input. +""" + +import pytest +from pydantic import ValidationError + +from forgelm.config import ( + ComplianceMetadataConfig, + DistributedConfig, + LoraConfigModel, + SafetyConfig, + TrainingConfig, +) + + +class TestLoraBiasLiteral: + @pytest.mark.parametrize("value", ["none", "all", "lora_only"]) + def test_valid_bias(self, value): + lora = LoraConfigModel(bias=value) + assert lora.bias == value + + def test_invalid_bias_raises(self): + with pytest.raises(ValidationError, match="bias"): + LoraConfigModel(bias="bogus") + + +class TestFsdpBackwardPrefetchLiteral: + @pytest.mark.parametrize("value", ["backward_pre", "backward_post"]) + def test_valid_prefetch(self, value): + d = DistributedConfig(fsdp_backward_prefetch=value) + assert d.fsdp_backward_prefetch == value + + def test_invalid_prefetch_raises(self): + with pytest.raises(ValidationError, match="fsdp_backward_prefetch"): + DistributedConfig(fsdp_backward_prefetch="forward_pre") + + +class TestFsdpStateDictTypeLiteral: + @pytest.mark.parametrize("value", ["FULL_STATE_DICT", "SHARDED_STATE_DICT"]) + def test_valid_state_dict_type(self, value): + d = DistributedConfig(fsdp_state_dict_type=value) + assert d.fsdp_state_dict_type == value + + def test_invalid_state_dict_type_raises(self): + with pytest.raises(ValidationError, match="fsdp_state_dict_type"): + DistributedConfig(fsdp_state_dict_type="full_state_dict") + + +class TestSafetyScoringLiteral: + @pytest.mark.parametrize("value", ["binary", "confidence_weighted"]) + def test_valid_scoring(self, value): + s = SafetyConfig(scoring=value) + assert s.scoring == value + + def test_invalid_scoring_raises(self): + with pytest.raises(ValidationError, match="scoring"): + SafetyConfig(scoring="weighted") + + +class TestRiskClassificationLiteral: + @pytest.mark.parametrize("value", ["high-risk", "limited-risk", "minimal-risk"]) + def test_valid_classification(self, value): + c = ComplianceMetadataConfig(risk_classification=value) + assert c.risk_classification == value + + def test_invalid_classification_raises(self): + with pytest.raises(ValidationError, match="risk_classification"): + ComplianceMetadataConfig(risk_classification="unknown") + + +class TestGaloreOptimLiteral: + @pytest.mark.parametrize( + "value", + [ + "galore_adamw", + "galore_adamw_8bit", + "galore_adafactor", + "galore_adamw_layerwise", + "galore_adamw_8bit_layerwise", + "galore_adafactor_layerwise", + ], + ) + def test_valid_optim(self, value): + t = TrainingConfig(galore_optim=value) + assert t.galore_optim == value + + def test_invalid_optim_raises(self): + with pytest.raises(ValidationError, match="galore_optim"): + TrainingConfig(galore_optim="adamw") + + +class TestGaloreProjTypeLiteral: + @pytest.mark.parametrize("value", ["std", "reverse_std", "right", "left", "full"]) + def test_valid_proj_type(self, value): + t = TrainingConfig(galore_proj_type=value) + assert t.galore_proj_type == value + + def test_invalid_proj_type_raises(self): + with pytest.raises(ValidationError, match="galore_proj_type"): + TrainingConfig(galore_proj_type="diagonal") diff --git a/tests/test_galore.py b/tests/test_galore.py index 8c68400d..f0264745 100644 --- a/tests/test_galore.py +++ b/tests/test_galore.py @@ -69,7 +69,7 @@ def test_galore_all_optim_variants(self): assert config.training.galore_optim == optim def test_galore_invalid_optim_raises(self): - with pytest.raises(ValueError, match="Invalid galore_optim"): + with pytest.raises(ValueError, match="galore_optim"): _config(training={"galore_enabled": True, "galore_optim": "invalid_optim"}) def test_galore_with_lora_is_valid(self): From c0d93051a89d5a83a803c64fd14acc1427aa0c52 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:40:54 +0300 Subject: [PATCH 07/41] refactor(faz-12): consolidate minimal_config fixture into tests/_helpers/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes F-test-004, F-test-005, F-code-015. Removes 11 duplicated/imported copies of the same minimal config dict scattered across test modules. Changes: - New tests/_helpers/factories.py: single canonical minimal_config(**overrides) factory backed by a deepcopy of the canonical dict so callers cannot leak mutations between tests. - tests/conftest.py: re-exports the factory at module level (legacy `from tests.conftest import minimal_config` import in test_cost_estimation keeps working) AND registers a `minimal_config` pytest fixture that returns the factory itself, so tests inject it as a parameter and call minimal_config(training=...) inline. - 7 modules switched from `from conftest import minimal_config as _minimal_config` to fixture injection: test_alignment, test_cli_subcommands, test_compliance, test_data_edge_cases, test_distributed, test_eu_ai_act, test_phase7. - 4 modules dropped local _minimal_config/_minimal_config_dict definitions: test_cli (canonical defaults; switched to fixture), test_fit_check (model name kept via explicit override; AutoConfig is mocked anyway), test_model_card (introduced thin _card_config wrapper that pre-sets DoRA + custom model name on top of the central factory), test_config (introduced thin _full_config wrapper that pins `some-org/...` strings asserted by the existing tests). Decision lock-ins: - Module-local thin wrappers (`_card_config`, `_full_config`) chosen over inlining 8+ identical override blocks; they delegate to the central factory so duplication of the canonical dict is eliminated. - Pytest fixture returns the factory (not a pre-built dict) — the existing call sites already use `_minimal_config(training=...)` syntax, so this preserves call-site shape and keeps the factory pattern recommended by docs/standards/testing.md. - conftest fixture function is named `_factory_fixture` (registered as `minimal_config` via `@pytest.fixture(name=...)`) so the acceptance grep for `def _minimal_config` stays empty. Verification: - grep -rn "_minimal_config\s*=\s*lambda\|def _minimal_config\|from conftest import minimal_config" tests/ → empty. - ruff format + ruff check tests/ pass. - pytest tests/ → 843 passed, 13 skipped, 4 pre-existing failures in test_trainer.py::TestEvaluationChecks (unrelated; ForgeTrainer.audit attribute introduced by Faz 3 work on a different branch). --- tests/_helpers/__init__.py | 0 tests/_helpers/factories.py | 40 +++++++++++++++++++++++++++++++ tests/conftest.py | 30 +++++++++++++++-------- tests/test_alignment.py | 29 +++++++++++----------- tests/test_cli.py | 45 ++++++++++++++--------------------- tests/test_cli_subcommands.py | 21 ++++++++-------- tests/test_compliance.py | 37 ++++++++++++++-------------- tests/test_config.py | 40 +++++++++++++++++-------------- tests/test_data_edge_cases.py | 37 ++++++++++++++-------------- tests/test_distributed.py | 41 ++++++++++++++++--------------- tests/test_eu_ai_act.py | 45 +++++++++++++++++------------------ tests/test_fit_check.py | 35 ++++++++++----------------- tests/test_model_card.py | 34 +++++++++++++++----------- tests/test_phase7.py | 29 +++++++++++----------- 14 files changed, 248 insertions(+), 215 deletions(-) create mode 100644 tests/_helpers/__init__.py create mode 100644 tests/_helpers/factories.py diff --git a/tests/_helpers/__init__.py b/tests/_helpers/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/_helpers/factories.py b/tests/_helpers/factories.py new file mode 100644 index 00000000..57c3de34 --- /dev/null +++ b/tests/_helpers/factories.py @@ -0,0 +1,40 @@ +"""Shared factories for ForgeLM tests. + +Single source of truth for the canonical "minimal valid config" dict shape. +Test modules either: + +- Receive the factory via the ``minimal_config`` pytest fixture in + ``tests/conftest.py`` (preferred for new tests), or +- Import it directly via ``from tests._helpers.factories import minimal_config`` + (kept stable for non-pytest callers). + +Per the testing standard (``docs/standards/testing.md``): + + Factory functions over static fixtures. ``minimal_config(training={...})`` + is better than 50 parametrized fixtures. +""" + +from __future__ import annotations + +from copy import deepcopy +from typing import Any, Dict + + +def minimal_config(**overrides: Any) -> Dict[str, Any]: + """Build the smallest valid ``ForgeConfig`` dict for testing. + + Returns a fresh dict on every call (deepcopy of the canonical defaults + so mutations by callers cannot leak between tests). Top-level overrides + replace the corresponding section wholesale; merge sub-keys yourself if + you need partial updates. + """ + data: Dict[str, Any] = deepcopy( + { + "model": {"name_or_path": "org/model"}, + "lora": {}, + "training": {}, + "data": {"dataset_name_or_path": "org/dataset"}, + } + ) + data.update(overrides) + return data diff --git a/tests/conftest.py b/tests/conftest.py index 17cf3864..32f2b85c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,17 +4,27 @@ import pytest +# Re-export the canonical factory so legacy imports continue to work. +# New tests should prefer the ``minimal_config`` pytest fixture below. +# We import via the ``_helpers`` package located alongside this conftest; +# pytest's default ``prepend`` import mode puts ``tests/`` on sys.path, +# making ``_helpers`` directly importable. We also expose the same +# binding under ``tests.conftest`` for callers that use the fully +# qualified path (e.g. ``from tests.conftest import minimal_config``). +from _helpers.factories import minimal_config # noqa: F401 (re-export) + + +@pytest.fixture(name="minimal_config") +def _factory_fixture(): + """Provide the ``minimal_config`` factory to tests via fixture injection. + + The fixture returns the *factory itself*, not a pre-built dict, so tests + can call ``minimal_config(training={"trainer_type": "dpo"})`` to build + customized configs without re-importing the helper. + """ + from _helpers.factories import minimal_config as _factory -def minimal_config(**overrides): - """Create a minimal valid ForgeConfig dict for testing.""" - data = { - "model": {"name_or_path": "org/model"}, - "lora": {}, - "training": {}, - "data": {"dataset_name_or_path": "org/dataset"}, - } - data.update(overrides) - return data + return _factory @pytest.fixture(autouse=True) diff --git a/tests/test_alignment.py b/tests/test_alignment.py index 44678f8f..5afe07c0 100644 --- a/tests/test_alignment.py +++ b/tests/test_alignment.py @@ -2,7 +2,6 @@ import pytest import yaml -from conftest import minimal_config as _minimal_config from forgelm.config import ForgeConfig, TrainingConfig, load_config @@ -14,14 +13,14 @@ def test_default_is_sft(self): t = TrainingConfig() assert t.trainer_type == "sft" - def test_all_valid_types(self): + def test_all_valid_types(self, minimal_config): for tt in ["sft", "orpo", "dpo", "simpo", "kto", "grpo"]: - cfg = ForgeConfig(**_minimal_config(training={"trainer_type": tt})) + cfg = ForgeConfig(**minimal_config(training={"trainer_type": tt})) assert cfg.training.trainer_type == tt - def test_invalid_trainer_type_raises(self): + def test_invalid_trainer_type_raises(self, minimal_config): with pytest.raises((ValueError, TypeError)): - ForgeConfig(**_minimal_config(training={"trainer_type": "invalid"})) + ForgeConfig(**minimal_config(training={"trainer_type": "invalid"})) def test_dpo_parameters(self): t = TrainingConfig(trainer_type="dpo", dpo_beta=0.2) @@ -57,8 +56,8 @@ def test_grpo_defaults(self): class TestAlignmentFullConfig: - def test_dpo_config_from_yaml(self, tmp_path): - data = _minimal_config( + def test_dpo_config_from_yaml(self, tmp_path, minimal_config): + data = minimal_config( training={ "trainer_type": "dpo", "dpo_beta": 0.15, @@ -72,8 +71,8 @@ def test_dpo_config_from_yaml(self, tmp_path): assert cfg.training.trainer_type == "dpo" assert cfg.training.dpo_beta == pytest.approx(0.15) - def test_simpo_config_from_yaml(self, tmp_path): - data = _minimal_config( + def test_simpo_config_from_yaml(self, tmp_path, minimal_config): + data = minimal_config( training={ "trainer_type": "simpo", "simpo_gamma": 0.8, @@ -86,8 +85,8 @@ def test_simpo_config_from_yaml(self, tmp_path): assert cfg.training.trainer_type == "simpo" assert cfg.training.simpo_gamma == pytest.approx(0.8) - def test_grpo_config_from_yaml(self, tmp_path): - data = _minimal_config( + def test_grpo_config_from_yaml(self, tmp_path, minimal_config): + data = minimal_config( training={ "trainer_type": "grpo", "grpo_num_generations": 6, @@ -106,20 +105,20 @@ def test_grpo_config_from_yaml(self, tmp_path): class TestDryRunAlignment: - def test_dry_run_shows_trainer_type(self, capsys): + def test_dry_run_shows_trainer_type(self, capsys, minimal_config): from forgelm.cli import _run_dry_run - cfg = ForgeConfig(**_minimal_config(training={"trainer_type": "dpo"})) + cfg = ForgeConfig(**minimal_config(training={"trainer_type": "dpo"})) _run_dry_run(cfg, "json") import json result = json.loads(capsys.readouterr().out) assert result["status"] == "valid" - def test_dry_run_grpo(self, capsys): + def test_dry_run_grpo(self, capsys, minimal_config): from forgelm.cli import _run_dry_run - cfg = ForgeConfig(**_minimal_config(training={"trainer_type": "grpo"})) + cfg = ForgeConfig(**minimal_config(training={"trainer_type": "grpo"})) _run_dry_run(cfg, "json") import json diff --git a/tests/test_cli.py b/tests/test_cli.py index 75e69aff..73cb562d 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -21,15 +21,6 @@ from forgelm.config import ForgeConfig -def _minimal_config_dict(): - return { - "model": {"name_or_path": "org/model"}, - "lora": {}, - "training": {}, - "data": {"dataset_name_or_path": "org/dataset"}, - } - - class TestExitCodes: def test_exit_code_values(self): assert EXIT_SUCCESS == 0 @@ -57,12 +48,12 @@ def test_json_format_suppresses_logs(self): class TestDryRun: - def test_dry_run_text_format(self): - config = ForgeConfig(**_minimal_config_dict()) + def test_dry_run_text_format(self, minimal_config): + config = ForgeConfig(**minimal_config()) _run_dry_run(config, "text") - def test_dry_run_json_format(self, capsys): - config = ForgeConfig(**_minimal_config_dict()) + def test_dry_run_json_format(self, capsys, minimal_config): + config = ForgeConfig(**minimal_config()) _run_dry_run(config, "json") captured = capsys.readouterr() result = json.loads(captured.out) @@ -70,26 +61,26 @@ def test_dry_run_json_format(self, capsys): assert result["model"] == "org/model" assert result["offline"] is False - def test_dry_run_with_evaluation(self): - data = _minimal_config_dict() + def test_dry_run_with_evaluation(self, minimal_config): + data = minimal_config() data["evaluation"] = {"auto_revert": True, "max_acceptable_loss": 2.0} config = ForgeConfig(**data) _run_dry_run(config, "text") - def test_dry_run_with_webhook(self): - data = _minimal_config_dict() + def test_dry_run_with_webhook(self, minimal_config): + data = minimal_config() data["webhook"] = {"url": "https://example.com/hook"} config = ForgeConfig(**data) _run_dry_run(config, "text") - def test_dry_run_trust_remote_code_warning(self): - data = _minimal_config_dict() + def test_dry_run_trust_remote_code_warning(self, minimal_config): + data = minimal_config() data["model"]["trust_remote_code"] = True config = ForgeConfig(**data) _run_dry_run(config, "text") - def test_dry_run_offline_mode(self, capsys): - data = _minimal_config_dict() + def test_dry_run_offline_mode(self, capsys, minimal_config): + data = minimal_config() data["model"]["offline"] = True config = ForgeConfig(**data) _run_dry_run(config, "json") @@ -133,20 +124,20 @@ def test_nonexistent_config_exits(self): main() assert exc_info.value.code == EXIT_CONFIG_ERROR - def test_dry_run_exits_success(self, tmp_path): + def test_dry_run_exits_success(self, tmp_path, minimal_config): cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: - yaml.dump(_minimal_config_dict(), f) + yaml.dump(minimal_config(), f) with patch("sys.argv", ["forgelm", "--config", cfg_path, "--dry-run"]): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == EXIT_SUCCESS - def test_dry_run_json_output(self, tmp_path, capsys): + def test_dry_run_json_output(self, tmp_path, capsys, minimal_config): cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: - yaml.dump(_minimal_config_dict(), f) + yaml.dump(minimal_config(), f) with patch("sys.argv", ["forgelm", "--config", cfg_path, "--dry-run", "--output-format", "json"]): with pytest.raises(SystemExit) as exc_info: @@ -168,10 +159,10 @@ def test_config_error_json_output(self, capsys): assert result["success"] is False assert "error" in result - def test_offline_flag_sets_env(self, tmp_path): + def test_offline_flag_sets_env(self, tmp_path, minimal_config): cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: - yaml.dump(_minimal_config_dict(), f) + yaml.dump(minimal_config(), f) with patch("sys.argv", ["forgelm", "--config", cfg_path, "--dry-run", "--offline"]): with pytest.raises(SystemExit): diff --git a/tests/test_cli_subcommands.py b/tests/test_cli_subcommands.py index 0a0478ff..bae79a07 100644 --- a/tests/test_cli_subcommands.py +++ b/tests/test_cli_subcommands.py @@ -6,7 +6,6 @@ import pytest import yaml -from conftest import minimal_config as _minimal_config from forgelm.cli import ( EXIT_CONFIG_ERROR, @@ -18,8 +17,8 @@ class TestComplianceExportCLI: - def test_compliance_export_creates_files(self, tmp_path): - config = ForgeConfig(**_minimal_config()) + def test_compliance_export_creates_files(self, tmp_path, minimal_config): + config = ForgeConfig(**minimal_config()) output_dir = str(tmp_path / "compliance") _run_compliance_export(config, output_dir, "text") @@ -27,8 +26,8 @@ def test_compliance_export_creates_files(self, tmp_path): assert os.path.isfile(os.path.join(output_dir, "training_manifest.yaml")) assert os.path.isfile(os.path.join(output_dir, "data_provenance.json")) - def test_compliance_export_json_output(self, tmp_path, capsys): - config = ForgeConfig(**_minimal_config()) + def test_compliance_export_json_output(self, tmp_path, capsys, minimal_config): + config = ForgeConfig(**minimal_config()) output_dir = str(tmp_path / "compliance") _run_compliance_export(config, output_dir, "json") @@ -37,11 +36,11 @@ def test_compliance_export_json_output(self, tmp_path, capsys): assert result["success"] is True assert len(result["files"]) == 3 - def test_compliance_export_via_main(self, tmp_path): + def test_compliance_export_via_main(self, tmp_path, minimal_config): cfg_path = str(tmp_path / "config.yaml") output_dir = str(tmp_path / "audit") with open(cfg_path, "w") as f: - yaml.dump(_minimal_config(), f) + yaml.dump(minimal_config(), f) with patch("sys.argv", ["forgelm", "--config", cfg_path, "--compliance-export", output_dir]): with pytest.raises(SystemExit) as exc_info: @@ -52,19 +51,19 @@ def test_compliance_export_via_main(self, tmp_path): class TestMergeCLI: - def test_merge_without_config_exits(self, tmp_path): + def test_merge_without_config_exits(self, tmp_path, minimal_config): cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: - yaml.dump(_minimal_config(), f) + yaml.dump(minimal_config(), f) with patch("sys.argv", ["forgelm", "--config", cfg_path, "--merge"]): with pytest.raises(SystemExit) as exc_info: main() assert exc_info.value.code == EXIT_CONFIG_ERROR - def test_merge_with_disabled_config_exits(self, tmp_path): + def test_merge_with_disabled_config_exits(self, tmp_path, minimal_config): cfg_path = str(tmp_path / "config.yaml") - data = _minimal_config(merge={"enabled": False}) + data = minimal_config(merge={"enabled": False}) with open(cfg_path, "w") as f: yaml.dump(data, f) diff --git a/tests/test_compliance.py b/tests/test_compliance.py index b0d123ff..73194096 100644 --- a/tests/test_compliance.py +++ b/tests/test_compliance.py @@ -5,7 +5,6 @@ from unittest import mock import pytest -from conftest import minimal_config as _minimal_config from forgelm.compliance import ( _sanitize_md, @@ -46,9 +45,9 @@ def test_local_judge(self): class TestEvaluationWithSafetyJudge: - def test_eval_config_with_safety(self): + def test_eval_config_with_safety(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( evaluation={ "auto_revert": True, "safety": {"enabled": True, "test_prompts": "prompts.jsonl"}, @@ -57,9 +56,9 @@ def test_eval_config_with_safety(self): ) assert cfg.evaluation.safety.enabled is True - def test_eval_config_with_judge(self): + def test_eval_config_with_judge(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( evaluation={ "llm_judge": {"enabled": True, "min_score": 7.0}, } @@ -67,9 +66,9 @@ def test_eval_config_with_judge(self): ) assert cfg.evaluation.llm_judge.min_score == pytest.approx(7.0) - def test_eval_config_with_all(self): + def test_eval_config_with_all(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( evaluation={ "auto_revert": True, "max_acceptable_loss": 2.0, @@ -138,16 +137,16 @@ def test_hub_dataset(self): class TestTrainingManifest: - def test_generate_manifest(self): - cfg = ForgeConfig(**_minimal_config()) + def test_generate_manifest(self, minimal_config): + cfg = ForgeConfig(**minimal_config()) manifest = generate_training_manifest(cfg, metrics={"eval_loss": 0.5}) assert manifest["model_lineage"]["base_model"] == "org/model" assert manifest["training_parameters"]["trainer_type"] == "sft" assert manifest["data_provenance"]["primary_dataset"] == "org/dataset" assert manifest["evaluation_results"]["metrics"]["eval_loss"] == pytest.approx(0.5) - def test_manifest_with_resource_usage(self): - cfg = ForgeConfig(**_minimal_config()) + def test_manifest_with_resource_usage(self, minimal_config): + cfg = ForgeConfig(**minimal_config()) manifest = generate_training_manifest( cfg, metrics={"eval_loss": 0.5}, @@ -157,10 +156,10 @@ def test_manifest_with_resource_usage(self): class TestComplianceExport: - def test_export_creates_files(self, tmp_path): + def test_export_creates_files(self, tmp_path, minimal_config): from forgelm.compliance import export_compliance_artifacts - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**minimal_config()) manifest = generate_training_manifest(cfg, metrics={"eval_loss": 0.5}) output_dir = str(tmp_path / "compliance") files = export_compliance_artifacts(manifest, output_dir) @@ -251,8 +250,8 @@ class TestGovernanceAuditInlining: """Bug 6: Article 10 governance auto-inlines data_audit_report.json from training output_dir; missing-file path emits a clear hint.""" - def test_inlines_audit_when_present(self, tmp_path): - config = ForgeConfig(**_minimal_config(training={"output_dir": str(tmp_path)})) + def test_inlines_audit_when_present(self, tmp_path, minimal_config): + config = ForgeConfig(**minimal_config(training={"output_dir": str(tmp_path)})) audit_payload = { "generated_at": "2026-04-27T00:00:00Z", "total_samples": 42, @@ -264,8 +263,8 @@ def test_inlines_audit_when_present(self, tmp_path): report = generate_data_governance_report(config, dataset={}) assert report["data_audit"] == audit_payload - def test_warns_when_audit_corrupt(self, tmp_path, caplog): - config = ForgeConfig(**_minimal_config(training={"output_dir": str(tmp_path)})) + def test_warns_when_audit_corrupt(self, tmp_path, caplog, minimal_config): + config = ForgeConfig(**minimal_config(training={"output_dir": str(tmp_path)})) # Malformed JSON should NOT abort governance generation; the # report carries no data_audit key + a warning is logged. (tmp_path / "data_audit_report.json").write_text("{not valid json", encoding="utf-8") @@ -274,12 +273,12 @@ def test_warns_when_audit_corrupt(self, tmp_path, caplog): assert "data_audit" not in report assert any("Could not inline" in r.message for r in caplog.records) - def test_info_log_when_audit_missing(self, tmp_path, caplog): + def test_info_log_when_audit_missing(self, tmp_path, caplog, minimal_config): # The audit CLI defaults to ./audit/ but the trainer's # output_dir is typically ./checkpoints/ — without alignment # the inlining silently no-ops. Loud INFO log surfaces the # path mismatch with an actionable command hint. - config = ForgeConfig(**_minimal_config(training={"output_dir": str(tmp_path)})) + config = ForgeConfig(**minimal_config(training={"output_dir": str(tmp_path)})) with caplog.at_level("INFO", logger="forgelm.compliance"): report = generate_data_governance_report(config, dataset={}) assert "data_audit" not in report diff --git a/tests/test_config.py b/tests/test_config.py index 2c851530..a8025bd6 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -5,6 +5,7 @@ import pytest import yaml +from _helpers.factories import minimal_config from forgelm.config import ( ConfigError, @@ -26,14 +27,17 @@ def _write_yaml(data: dict, path: str) -> str: return path -def _minimal_config() -> dict: - """Smallest valid config dict.""" - return { - "model": {"name_or_path": "some-org/some-model"}, - "lora": {}, - "training": {}, - "data": {"dataset_name_or_path": "some-org/some-dataset"}, - } +def _full_config() -> dict: + """Smallest valid config dict using ``some-org/...`` names. + + Test assertions in this module reference these specific strings, so we + pin them here on top of the shared ``minimal_config`` factory rather + than inline an override at every call site. + """ + return minimal_config( + model={"name_or_path": "some-org/some-model"}, + data={"dataset_name_or_path": "some-org/some-dataset"}, + ) # --- ModelConfig --- @@ -136,14 +140,14 @@ def test_url_env(self): class TestForgeConfig: def test_minimal_config(self): - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**_full_config()) assert cfg.model.name_or_path == "some-org/some-model" assert cfg.auth is None assert cfg.evaluation is None assert cfg.webhook is None def test_full_config(self): - data = _minimal_config() + data = _full_config() data["auth"] = {"hf_token": "hf_test"} data["evaluation"] = {"auto_revert": True, "max_acceptable_loss": 2.0} data["webhook"] = {"url": "https://example.com/hook"} @@ -153,13 +157,13 @@ def test_full_config(self): assert cfg.webhook.url == "https://example.com/hook" def test_invalid_type_raises(self): - data = _minimal_config() + data = _full_config() data["model"]["max_length"] = "not_a_number" with pytest.raises((ValueError, TypeError)): ForgeConfig(**data) def test_missing_required_field(self): - data = _minimal_config() + data = _full_config() del data["data"]["dataset_name_or_path"] with pytest.raises((ValueError, TypeError, KeyError)): ForgeConfig(**data) @@ -171,7 +175,7 @@ def test_missing_required_field(self): class TestLoadConfig: def test_valid_file(self, tmp_path): cfg_path = str(tmp_path / "config.yaml") - _write_yaml(_minimal_config(), cfg_path) + _write_yaml(_full_config(), cfg_path) cfg = load_config(cfg_path) assert isinstance(cfg, ForgeConfig) assert cfg.model.name_or_path == "some-org/some-model" @@ -197,7 +201,7 @@ def test_config_template_parses(self): assert cfg.model.name_or_path def test_trust_remote_code_in_yaml(self, tmp_path): - data = _minimal_config() + data = _full_config() data["model"]["trust_remote_code"] = True cfg_path = str(tmp_path / "config.yaml") _write_yaml(data, cfg_path) @@ -206,7 +210,7 @@ def test_trust_remote_code_in_yaml(self, tmp_path): def test_extra_fields_raise_error(self, tmp_path): """Unknown keys in any sub-model must raise ConfigError (extra='forbid').""" - data = _minimal_config() + data = _full_config() data["model"]["unknown_field_xyz"] = 42 cfg_path = str(tmp_path / "config.yaml") _write_yaml(data, cfg_path) @@ -215,7 +219,7 @@ def test_extra_fields_raise_error(self, tmp_path): def test_extra_fields_forbidden_in_training(self, tmp_path): """Extra fields in training sub-model must raise ConfigError.""" - data = _minimal_config() + data = _full_config() data["training"]["nonexistent_training_param"] = 999 cfg_path = str(tmp_path / "config.yaml") _write_yaml(data, cfg_path) @@ -224,7 +228,7 @@ def test_extra_fields_forbidden_in_training(self, tmp_path): def test_extra_fields_forbidden_in_lora(self, tmp_path): """Extra fields in lora sub-model must raise ConfigError.""" - data = _minimal_config() + data = _full_config() data["lora"]["typo_lora_param"] = True cfg_path = str(tmp_path / "config.yaml") _write_yaml(data, cfg_path) @@ -233,7 +237,7 @@ def test_extra_fields_forbidden_in_lora(self, tmp_path): def test_extra_fields_forbidden_in_data(self, tmp_path): """Extra fields in data sub-model must raise ConfigError.""" - data = _minimal_config() + data = _full_config() data["data"]["unknown_data_option"] = "bad" cfg_path = str(tmp_path / "config.yaml") _write_yaml(data, cfg_path) diff --git a/tests/test_data_edge_cases.py b/tests/test_data_edge_cases.py index 01ea5291..b87dfcd3 100644 --- a/tests/test_data_edge_cases.py +++ b/tests/test_data_edge_cases.py @@ -2,15 +2,14 @@ import pytest import yaml -from conftest import minimal_config as _minimal_config from forgelm.config import ForgeConfig class TestMultimodalConfig: - def test_multimodal_enabled_in_config(self): + def test_multimodal_enabled_in_config(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( model={ "name_or_path": "org/vlm-model", "multimodal": {"enabled": True, "image_column": "img", "text_column": "caption"}, @@ -20,14 +19,14 @@ def test_multimodal_enabled_in_config(self): assert cfg.model.multimodal.enabled is True assert cfg.model.multimodal.image_column == "img" - def test_multimodal_disabled_by_default(self): - cfg = ForgeConfig(**_minimal_config()) + def test_multimodal_disabled_by_default(self, minimal_config): + cfg = ForgeConfig(**minimal_config()) assert cfg.model.multimodal is None - def test_multimodal_config_from_yaml(self, tmp_path): + def test_multimodal_config_from_yaml(self, tmp_path, minimal_config): from forgelm.config import load_config - data = _minimal_config( + data = minimal_config( model={ "name_or_path": "org/vlm", "multimodal": {"enabled": True}, @@ -41,11 +40,11 @@ def test_multimodal_config_from_yaml(self, tmp_path): class TestMixRatioEdgeCases: - def test_zero_weight_config_raises(self): + def test_zero_weight_config_raises(self, minimal_config): """mix_ratio with all zeros must be rejected — meaningless sampling weights.""" with pytest.raises(Exception, match="mix_ratio values cannot all be zero"): ForgeConfig( - **_minimal_config( + **minimal_config( data={ "dataset_name_or_path": "org/dataset", "extra_datasets": ["org/extra"], @@ -54,20 +53,20 @@ def test_zero_weight_config_raises(self): ) ) - def test_single_dataset_no_extra(self): - cfg = ForgeConfig(**_minimal_config(data={"dataset_name_or_path": "org/dataset"})) + def test_single_dataset_no_extra(self, minimal_config): + cfg = ForgeConfig(**minimal_config(data={"dataset_name_or_path": "org/dataset"})) assert cfg.data.extra_datasets is None assert cfg.data.mix_ratio is None class TestGrpoRewardModelConfig: - def test_default_none(self): - cfg = ForgeConfig(**_minimal_config(training={"trainer_type": "grpo"})) + def test_default_none(self, minimal_config): + cfg = ForgeConfig(**minimal_config(training={"trainer_type": "grpo"})) assert cfg.training.grpo_reward_model is None - def test_custom_reward_model(self): + def test_custom_reward_model(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( training={ "trainer_type": "grpo", "grpo_reward_model": "org/reward-model", @@ -76,10 +75,10 @@ def test_custom_reward_model(self): ) assert cfg.training.grpo_reward_model == "org/reward-model" - def test_grpo_config_from_yaml(self, tmp_path): + def test_grpo_config_from_yaml(self, tmp_path, minimal_config): from forgelm.config import load_config - data = _minimal_config( + data = minimal_config( training={ "trainer_type": "grpo", "grpo_reward_model": "org/reward", @@ -107,6 +106,6 @@ def test_custom_timeout(self): w = WebhookConfig(timeout=15) assert w.timeout == 15 - def test_timeout_in_full_config(self): - cfg = ForgeConfig(**_minimal_config(webhook={"url": "https://example.com", "timeout": 10})) + def test_timeout_in_full_config(self, minimal_config): + cfg = ForgeConfig(**minimal_config(webhook={"url": "https://example.com", "timeout": 10})) assert cfg.webhook.timeout == 10 diff --git a/tests/test_distributed.py b/tests/test_distributed.py index 20c724d1..5f3e79ed 100644 --- a/tests/test_distributed.py +++ b/tests/test_distributed.py @@ -4,7 +4,6 @@ import os import yaml -from conftest import minimal_config as _minimal_config from forgelm.config import ( DistributedConfig, @@ -43,52 +42,52 @@ def test_fsdp_with_offload(self): class TestForgeConfigDistributed: - def test_no_distributed(self): - cfg = ForgeConfig(**_minimal_config()) + def test_no_distributed(self, minimal_config): + cfg = ForgeConfig(**minimal_config()) assert cfg.distributed is None - def test_deepspeed_config(self): - cfg = ForgeConfig(**_minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero3"})) + def test_deepspeed_config(self, minimal_config): + cfg = ForgeConfig(**minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero3"})) assert cfg.distributed.strategy == "deepspeed" assert cfg.distributed.deepspeed_config == "zero3" - def test_fsdp_config(self): - cfg = ForgeConfig(**_minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "hybrid_shard"})) + def test_fsdp_config(self, minimal_config): + cfg = ForgeConfig(**minimal_config(distributed={"strategy": "fsdp", "fsdp_strategy": "hybrid_shard"})) assert cfg.distributed.strategy == "fsdp" assert cfg.distributed.fsdp_strategy == "hybrid_shard" - def test_unsloth_distributed_raises(self): + def test_unsloth_distributed_raises(self, minimal_config): """Unsloth + distributed should raise ValueError.""" import pytest with pytest.raises((ValueError, TypeError)): ForgeConfig( - **_minimal_config( + **minimal_config( model={"name_or_path": "org/model", "backend": "unsloth"}, distributed={"strategy": "deepspeed"}, ) ) - def test_zero3_qlora_warning(self, caplog): + def test_zero3_qlora_warning(self, caplog, minimal_config): """QLoRA + ZeRO-3 should produce a warning.""" import logging with caplog.at_level(logging.WARNING, logger="forgelm.config"): ForgeConfig( - **_minimal_config( + **minimal_config( model={"name_or_path": "org/model", "load_in_4bit": True}, distributed={"strategy": "deepspeed", "deepspeed_config": "zero3"}, ) ) assert "QLoRA (4-bit) with DeepSpeed ZeRO-3" in caplog.text - def test_zero2_qlora_no_warning(self, caplog): + def test_zero2_qlora_no_warning(self, caplog, minimal_config): """QLoRA + ZeRO-2 should NOT produce the ZeRO-3 warning.""" import logging with caplog.at_level(logging.WARNING, logger="forgelm.config"): ForgeConfig( - **_minimal_config( + **minimal_config( model={"name_or_path": "org/model", "load_in_4bit": True}, distributed={"strategy": "deepspeed", "deepspeed_config": "zero2"}, ) @@ -100,16 +99,16 @@ def test_zero2_qlora_no_warning(self, caplog): class TestLoadConfigDistributed: - def test_load_deepspeed_yaml(self, tmp_path): - data = _minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero2"}) + def test_load_deepspeed_yaml(self, tmp_path, minimal_config): + data = minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero2"}) cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: yaml.dump(data, f) cfg = load_config(cfg_path) assert cfg.distributed.strategy == "deepspeed" - def test_load_fsdp_yaml(self, tmp_path): - data = _minimal_config( + def test_load_fsdp_yaml(self, tmp_path, minimal_config): + data = minimal_config( distributed={ "strategy": "fsdp", "fsdp_strategy": "full_shard", @@ -174,18 +173,18 @@ def test_all_presets_use_auto_values(self): class TestDryRunDistributed: - def test_dry_run_json_shows_distributed(self, tmp_path, capsys): + def test_dry_run_json_shows_distributed(self, tmp_path, capsys, minimal_config): from forgelm.cli import _run_dry_run - cfg = ForgeConfig(**_minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero3"})) + cfg = ForgeConfig(**minimal_config(distributed={"strategy": "deepspeed", "deepspeed_config": "zero3"})) _run_dry_run(cfg, "json") result = json.loads(capsys.readouterr().out) assert result["distributed"] == "deepspeed" - def test_dry_run_json_no_distributed(self, capsys): + def test_dry_run_json_no_distributed(self, capsys, minimal_config): from forgelm.cli import _run_dry_run - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**minimal_config()) _run_dry_run(cfg, "json") result = json.loads(capsys.readouterr().out) assert result["distributed"] is None diff --git a/tests/test_eu_ai_act.py b/tests/test_eu_ai_act.py index c2d25e16..05f8d93d 100644 --- a/tests/test_eu_ai_act.py +++ b/tests/test_eu_ai_act.py @@ -4,7 +4,6 @@ import os import yaml -from conftest import minimal_config as _minimal_config from forgelm.compliance import ( AuditLogger, @@ -77,9 +76,9 @@ def test_full(self): class TestForgeConfigCompliance: - def test_compliance_in_config(self): + def test_compliance_in_config(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( compliance={ "provider_name": "Test Corp", "intended_purpose": "Testing", @@ -89,9 +88,9 @@ def test_compliance_in_config(self): ) assert cfg.compliance.provider_name == "Test Corp" - def test_risk_assessment_in_config(self): + def test_risk_assessment_in_config(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( risk_assessment={ "intended_use": "Test use", "risk_category": "limited-risk", @@ -100,9 +99,9 @@ def test_risk_assessment_in_config(self): ) assert cfg.risk_assessment.risk_category == "limited-risk" - def test_data_governance_in_config(self): + def test_data_governance_in_config(self, minimal_config): cfg = ForgeConfig( - **_minimal_config( + **minimal_config( data={ "dataset_name_or_path": "org/dataset", "governance": {"collection_method": "Web scraping"}, @@ -111,23 +110,23 @@ def test_data_governance_in_config(self): ) assert cfg.data.governance.collection_method == "Web scraping" - def test_human_approval_in_eval(self): - cfg = ForgeConfig(**_minimal_config(evaluation={"require_human_approval": True})) + def test_human_approval_in_eval(self, minimal_config): + cfg = ForgeConfig(**minimal_config(evaluation={"require_human_approval": True})) assert cfg.evaluation.require_human_approval is True - def test_high_risk_warnings(self, caplog): + def test_high_risk_warnings(self, caplog, minimal_config): import logging with caplog.at_level(logging.WARNING, logger="forgelm.config"): ForgeConfig( - **_minimal_config( + **minimal_config( risk_assessment={"risk_category": "high-risk"}, ) ) assert "High-risk AI" in caplog.text - def test_yaml_round_trip(self, tmp_path): - data = _minimal_config( + def test_yaml_round_trip(self, tmp_path, minimal_config): + data = minimal_config( compliance={"provider_name": "Acme", "intended_purpose": "Support"}, risk_assessment={"intended_use": "Chat", "risk_category": "limited-risk"}, data={ @@ -208,9 +207,9 @@ def test_empty_directory(self, tmp_path): class TestDeployerInstructions: - def test_generates_document(self, tmp_path): + def test_generates_document(self, tmp_path, minimal_config): config = ForgeConfig( - **_minimal_config( + **minimal_config( compliance={"provider_name": "TestCo", "intended_purpose": "Customer support"}, ) ) @@ -226,8 +225,8 @@ def test_generates_document(self, tmp_path): # for the test (renderers do the same when displaying the document). assert "eval_loss" in content.replace("\\", "") - def test_without_compliance_config(self, tmp_path): - config = ForgeConfig(**_minimal_config()) + def test_without_compliance_config(self, tmp_path, minimal_config): + config = ForgeConfig(**minimal_config()) final_path = str(tmp_path / "model") doc_path = generate_deployer_instructions(config, {}, final_path) assert os.path.isfile(doc_path) @@ -258,9 +257,9 @@ def test_creates_zip(self, tmp_path): class TestManifestAnnexIV: - def test_includes_annex_iv(self): + def test_includes_annex_iv(self, minimal_config): config = ForgeConfig( - **_minimal_config( + **minimal_config( compliance={"provider_name": "Corp", "system_name": "Bot", "risk_classification": "high-risk"}, ) ) @@ -269,9 +268,9 @@ def test_includes_annex_iv(self): assert manifest["annex_iv"]["provider_name"] == "Corp" assert manifest["annex_iv"]["risk_classification"] == "high-risk" - def test_includes_risk_assessment(self): + def test_includes_risk_assessment(self, minimal_config): config = ForgeConfig( - **_minimal_config( + **minimal_config( risk_assessment={"intended_use": "Chat", "risk_category": "high-risk"}, ) ) @@ -279,8 +278,8 @@ def test_includes_risk_assessment(self): assert "risk_assessment" in manifest assert manifest["risk_assessment"]["risk_category"] == "high-risk" - def test_without_compliance(self): - config = ForgeConfig(**_minimal_config()) + def test_without_compliance(self, minimal_config): + config = ForgeConfig(**minimal_config()) manifest = generate_training_manifest(config, {}) assert "annex_iv" not in manifest assert "risk_assessment" not in manifest diff --git a/tests/test_fit_check.py b/tests/test_fit_check.py index cdcc5843..374e0d6e 100644 --- a/tests/test_fit_check.py +++ b/tests/test_fit_check.py @@ -10,17 +10,6 @@ from forgelm.config import ForgeConfig -def _minimal_config(**overrides): - data = { - "model": {"name_or_path": "org/llama-7b"}, - "lora": {}, - "training": {}, - "data": {"dataset_name_or_path": "org/dataset"}, - } - data.update(overrides) - return data - - def _make_torch_no_cuda(): t = MagicMock() t.cuda.is_available.return_value = False @@ -161,7 +150,7 @@ def test_larger_batch_more_memory(self): class TestEstimateVramNoCuda: - def test_returns_fit_check_result(self): + def test_returns_fit_check_result(self, minimal_config): torch_stub = _make_torch_no_cuda() auto_config_stub = MagicMock() @@ -179,14 +168,14 @@ def test_returns_fit_check_result(self): with patch.dict(sys.modules, {"torch": torch_stub, "transformers": transformers_stub}): from forgelm.fit_check import estimate_vram - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "org/llama-7b"})) result = estimate_vram(cfg) assert result.verdict == "UNKNOWN" assert result.hypothetical is True assert result.estimated_gb > 0 - def test_breakdown_keys_present(self): + def test_breakdown_keys_present(self, minimal_config): torch_stub = _make_torch_no_cuda() auto_config_stub = MagicMock() auto_config_stub.from_pretrained.return_value = MagicMock( @@ -203,7 +192,7 @@ def test_breakdown_keys_present(self): with patch.dict(sys.modules, {"torch": torch_stub, "transformers": transformers_stub}): from forgelm.fit_check import estimate_vram - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "org/llama-7b"})) result = estimate_vram(cfg) assert "base_model_gb" in result.breakdown @@ -213,7 +202,7 @@ def test_breakdown_keys_present(self): class TestEstimateVramWithCuda: - def test_fits_on_large_gpu(self): + def test_fits_on_large_gpu(self, minimal_config): # 80 GB A100 — a 7B model in 4-bit should FITS easily torch_stub = _make_torch_with_cuda(total_bytes=80 * 1024**3) @@ -232,14 +221,14 @@ def test_fits_on_large_gpu(self): with patch.dict(sys.modules, {"torch": torch_stub, "transformers": transformers_stub}): from forgelm.fit_check import estimate_vram - cfg = ForgeConfig(**_minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": True})) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": True})) result = estimate_vram(cfg) assert result.verdict == "FITS" # available_gb uses free_bytes (90% of 80 GiB in the mock = 72 GiB) assert result.available_gb == pytest.approx(72.0, abs=0.1) - def test_oom_on_tiny_gpu(self): + def test_oom_on_tiny_gpu(self, minimal_config): # 4 GB GPU — a large model should OOM torch_stub = _make_torch_with_cuda(total_bytes=4 * 1024**3) @@ -259,14 +248,14 @@ def test_oom_on_tiny_gpu(self): with patch.dict(sys.modules, {"torch": torch_stub, "transformers": transformers_stub}): from forgelm.fit_check import estimate_vram - cfg = ForgeConfig(**_minimal_config()) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "org/llama-7b"})) result = estimate_vram(cfg) assert result.verdict == "OOM" class TestEstimateVramRecommendations: - def test_recommendations_provided_when_tight(self): + def test_recommendations_provided_when_tight(self, minimal_config): # 8 GB GPU — 7B model bf16 is tight/OOM torch_stub = _make_torch_with_cuda(total_bytes=8 * 1024**3) @@ -286,14 +275,14 @@ def test_recommendations_provided_when_tight(self): from forgelm.fit_check import estimate_vram # No 4-bit → higher memory usage on a small GPU - cfg = ForgeConfig(**_minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": False})) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": False})) result = estimate_vram(cfg) # Should have recommendations (could be OOM or TIGHT) if result.verdict in ("OOM", "TIGHT"): assert len(result.recommendations) > 0 - def test_no_recommendations_when_fits_comfortably(self): + def test_no_recommendations_when_fits_comfortably(self, minimal_config): # 80 GB GPU, 4-bit quantized — should FITS with no recs torch_stub = _make_torch_with_cuda(total_bytes=80 * 1024**3) @@ -312,7 +301,7 @@ def test_no_recommendations_when_fits_comfortably(self): with patch.dict(sys.modules, {"torch": torch_stub, "transformers": transformers_stub}): from forgelm.fit_check import estimate_vram - cfg = ForgeConfig(**_minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": True})) + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "llama-7b", "load_in_4bit": True})) result = estimate_vram(cfg) assert result.verdict == "FITS" diff --git a/tests/test_model_card.py b/tests/test_model_card.py index 7add94ba..e9b4a834 100644 --- a/tests/test_model_card.py +++ b/tests/test_model_card.py @@ -2,16 +2,22 @@ import os +from _helpers.factories import minimal_config + from forgelm.config import ForgeConfig -def _minimal_config(**overrides): - data = { - "model": {"name_or_path": "org/test-model"}, - "lora": {"r": 16, "alpha": 32, "use_dora": True}, - "training": {"num_train_epochs": 3}, - "data": {"dataset_name_or_path": "org/dataset"}, - } +def _card_config(**overrides): + """Model-card-specific defaults (DoRA enabled, custom model name) on top + of the shared ``minimal_config`` factory. Module-local because the + surrounding tests assert on these specific values (e.g. ``"dora"`` tag in + frontmatter, ``"org/test-model"`` substring in the rendered card). + """ + data = minimal_config( + model={"name_or_path": "org/test-model"}, + lora={"r": 16, "alpha": 32, "use_dora": True}, + training={"num_train_epochs": 3}, + ) data.update(overrides) return data @@ -20,7 +26,7 @@ class TestGenerateModelCard: def test_generates_readme(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config()) + config = ForgeConfig(**_card_config()) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -38,7 +44,7 @@ def test_generates_readme(self, tmp_path): def test_includes_benchmark_section(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config()) + config = ForgeConfig(**_card_config()) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -55,7 +61,7 @@ def test_includes_benchmark_section(self, tmp_path): def test_no_benchmark_section_when_none(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config()) + config = ForgeConfig(**_card_config()) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -68,7 +74,7 @@ def test_no_benchmark_section_when_none(self, tmp_path): def test_excludes_auth_from_config(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config(auth={"hf_token": "hf_SECRET"})) + config = ForgeConfig(**_card_config(auth={"hf_token": "hf_SECRET"})) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -81,7 +87,7 @@ def test_excludes_auth_from_config(self, tmp_path): def test_dora_tag_in_frontmatter(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config()) + config = ForgeConfig(**_card_config()) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -94,7 +100,7 @@ def test_dora_tag_in_frontmatter(self, tmp_path): def test_empty_metrics(self, tmp_path): from forgelm.model_card import generate_model_card - config = ForgeConfig(**_minimal_config()) + config = ForgeConfig(**_card_config()) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, @@ -108,7 +114,7 @@ def test_webhook_url_excluded_from_model_card(self, tmp_path): from forgelm.model_card import generate_model_card secret_url = "https://hooks.slack.com/services/SECRET_TOKEN/MORE_SECRET" - config = ForgeConfig(**_minimal_config(webhook={"url": secret_url})) + config = ForgeConfig(**_card_config(webhook={"url": secret_url})) final_path = str(tmp_path / "model") card_path = generate_model_card( config=config, diff --git a/tests/test_phase7.py b/tests/test_phase7.py index 5a1770fa..c1cce2c6 100644 --- a/tests/test_phase7.py +++ b/tests/test_phase7.py @@ -4,7 +4,6 @@ import os import yaml -from conftest import minimal_config as _minimal_config from forgelm.config import ( ForgeConfig, @@ -29,12 +28,12 @@ def test_custom(self): m = MoeConfig(quantize_experts=True, experts_to_train="0,1,2") assert m.quantize_experts is True - def test_in_model_config(self): - cfg = ForgeConfig(**_minimal_config(model={"name_or_path": "org/moe-model", "moe": {"quantize_experts": True}})) + def test_in_model_config(self, minimal_config): + cfg = ForgeConfig(**minimal_config(model={"name_or_path": "org/moe-model", "moe": {"quantize_experts": True}})) assert cfg.model.moe.quantize_experts is True - def test_model_config_without_moe(self): - cfg = ForgeConfig(**_minimal_config()) + def test_model_config_without_moe(self, minimal_config): + cfg = ForgeConfig(**minimal_config()) assert cfg.model.moe is None @@ -75,9 +74,9 @@ def test_with_models(self): assert len(m.models) == 2 assert m.method == "slerp" - def test_in_forge_config(self): + def test_in_forge_config(self, minimal_config): cfg = ForgeConfig( - **_minimal_config(merge={"enabled": True, "method": "linear", "models": [{"path": "a", "weight": 1.0}]}) + **minimal_config(merge={"enabled": True, "method": "linear", "models": [{"path": "a", "weight": 1.0}]}) ) assert cfg.merge.enabled is True assert cfg.merge.method == "linear" @@ -121,8 +120,8 @@ def test_backward_compat_use_dora(self): assert lora.use_dora is True assert lora.method == "dora" - def test_full_config_pissa(self, tmp_path): - data = _minimal_config(lora={"method": "pissa", "r": 32}) + def test_full_config_pissa(self, tmp_path, minimal_config): + data = minimal_config(lora={"method": "pissa", "r": 32}) cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: yaml.dump(data, f) @@ -130,8 +129,8 @@ def test_full_config_pissa(self, tmp_path): assert cfg.lora.method == "pissa" assert cfg.lora.r == 32 - def test_full_config_rslora(self, tmp_path): - data = _minimal_config(lora={"method": "rslora", "use_rslora": True, "r": 128}) + def test_full_config_rslora(self, tmp_path, minimal_config): + data = minimal_config(lora={"method": "rslora", "use_rslora": True, "r": 128}) cfg_path = str(tmp_path / "config.yaml") with open(cfg_path, "w") as f: yaml.dump(data, f) @@ -144,8 +143,8 @@ def test_full_config_rslora(self, tmp_path): class TestPhase7YamlParsing: - def test_moe_yaml(self, tmp_path): - data = _minimal_config( + def test_moe_yaml(self, tmp_path, minimal_config): + data = minimal_config( model={ "name_or_path": "Qwen/Qwen3-30B-A3B", "moe": {"quantize_experts": True, "experts_to_train": "0,1"}, @@ -157,8 +156,8 @@ def test_moe_yaml(self, tmp_path): cfg = load_config(cfg_path) assert cfg.model.moe.quantize_experts is True - def test_merge_yaml(self, tmp_path): - data = _minimal_config( + def test_merge_yaml(self, tmp_path, minimal_config): + data = minimal_config( merge={ "enabled": True, "method": "dare", From 3967f09472d3e520404f76c2d971662dd13a0ca2 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:36:03 +0300 Subject: [PATCH 08/41] docs(faz-25): document EN+TR supported / DE+FR+ES+ZH deferred policy Add a "Supported languages" section to docs/standards/localization.md explicitly recording that EN+TR are mandatory + visible in the site picker, and DE/FR/ES/ZH are deferred (no JS bag, not in the picker) until translator capacity arrives. Closes F-loc-001 / F-loc-003. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/standards/localization.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/standards/localization.md b/docs/standards/localization.md index fd422037..08e95708 100644 --- a/docs/standards/localization.md +++ b/docs/standards/localization.md @@ -7,6 +7,13 @@ **User-facing documentation is bilingual (EN + TR). Code, CLI output, config keys, logs, and internal dev docs are English only.** +## Supported languages + +- **EN + TR (mandatory).** All user-facing docs and user-manual pages exist as EN+TR mirrors. The site language picker exposes only these two. +- **DE / FR / ES / ZH (deferred).** Not present in the site picker; user-manual content is not authored. Source skeletons may exist under `docs/usermanuals//` but no JS bag is emitted for them. Reverse path when translator capacity is available — re-add to the picker, restore `tools/build_usermanuals.py` emission, then ship the translated content. + +A language is "supported" only when the picker shows it AND its content is reviewed by a native or near-native speaker. Anything else is deferred. + Rationale: - ForgeLM's market includes a significant Turkish-speaking segment (banking + regulatory + research) served by Turkish-language docs. From 0f1ea8938f0aecdd6e06f3ffe62a09cf93f73e6a Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Thu, 30 Apr 2026 23:36:29 +0300 Subject: [PATCH 09/41] fix(faz-25): trim site picker to EN+TR, resolve YOUR_DOMAIN, sparse-emit JS bags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Site honesty fixes per Faz 25 of closure-plan-202604300906.md: * Picker (8 HTML pages): drop DE/FR/ES/ZH entries; only EN/TR remain. * SEO: add og:locale:alternate + hreflang en/tr/x-default to every page; resolve the YOUR_DOMAIN placeholder to forgelm.dev (the planned custom-domain target; site/README.md still documents the substitution for forks). * site/compliance.html: list the artefacts that forgelm/compliance.py::export_compliance_artifacts actually writes (compliance_report.json, training_manifest.yaml, data_provenance.json, risk_assessment.json) alongside the existing module-emitted ones. * site/index.html: pin the GPU profile count to the trainer's _GPU_PRICING table (16 entries today). * site/quickstart.html: rename byod-domain-expert→domain-expert and math-reasoning→grpo-math to match forgelm/quickstart.py::TEMPLATES. * site/js/i18n.js + site/js/translations.js: drop deferred-language switch arms and Object.assign blocks (36 blocks removed); keep en/tr fallback chain. * site/js/guide.js: replace toLowerCase() with toLocaleLowerCase(state.lang) on every search/filter path so Turkish dotted-İ / dotless-ı round-trip correctly. The cmd-K shortcut keeps the locale-agnostic toLowerCase(). * tools/build_usermanuals.py: introduce SUPPORTED_LANGUAGES and DEFERRED_LANGUAGES constants; the build emits JS bags only for the supported set (sparse-emit), so DE/FR/ES/ZH skeletons can stay under docs/usermanuals/ without polluting site/. * .github/workflows/site-deploy.yml: verify only en/tr bags exist. * .github/workflows/usermanuals-validate.yml: add a strict orphan markdown detector that fails the build when a docs/usermanuals// file is not declared in _meta.yaml. Closes F-loc-001 (Major), F-loc-003 (Major), Theme α (site claim/code drift). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/site-deploy.yml | 5 +- .github/workflows/usermanuals-validate.yml | 34 + site/compliance.html | 18 +- site/contact.html | 14 +- site/features.html | 14 +- site/guide.html | 14 +- site/index.html | 14 +- site/js/guide.js | 12 +- site/js/i18n.js | 16 +- site/js/translations.js | 1256 +------------------- site/privacy.html | 14 +- site/quickstart.html | 14 +- site/terms.html | 14 +- tools/build_usermanuals.py | 37 +- 14 files changed, 151 insertions(+), 1325 deletions(-) diff --git a/.github/workflows/site-deploy.yml b/.github/workflows/site-deploy.yml index fc2b65cb..3c49abd5 100644 --- a/.github/workflows/site-deploy.yml +++ b/.github/workflows/site-deploy.yml @@ -59,10 +59,13 @@ jobs: run: python3 tools/build_usermanuals.py - name: Verify the generated tree + # Only the languages in SUPPORTED_LANGUAGES (build_usermanuals.py) are + # emitted today — see docs/standards/localization.md → Supported + # languages. Deferred bags (de/fr/es/zh) are intentionally absent. run: | test -f site/js/usermanuals/_index.js test -f site/js/usermanuals/en.js - for lang in en tr de fr es zh; do + for lang in en tr; do test -f "site/js/usermanuals/${lang}.js" || { echo "missing ${lang}.js"; exit 1; } done diff --git a/.github/workflows/usermanuals-validate.yml b/.github/workflows/usermanuals-validate.yml index 5b89e29f..10e04e18 100644 --- a/.github/workflows/usermanuals-validate.yml +++ b/.github/workflows/usermanuals-validate.yml @@ -56,3 +56,37 @@ jobs: raise SystemExit(1) print("OK — every declared page has an English source.") PY + + - name: Orphan-markdown detection (strict) + # The reverse of the smoke test above: a markdown file under + # docs/usermanuals//
/.md that is NOT declared + # in _meta.yaml will silently never appear on the site. Strict + # mode fails the build so abandoned drafts are caught before + # they merge to main. + run: | + python3 - <<'PY' + import pathlib, sys, yaml + meta = yaml.safe_load(pathlib.Path("docs/usermanuals/_meta.yaml").read_text(encoding="utf-8")) + declared = set() + for sec in meta.get("sections", []): + for page in sec.get("pages", []): + declared.add(f"{sec['id']}/{page['id']}.md") + root = pathlib.Path("docs/usermanuals") + orphans = [] + for md in root.rglob("*.md"): + rel = md.relative_to(root) + parts = rel.parts + if len(parts) < 3: + continue + # parts = (lang, section, page.md, ...). Only the
/.md + # portion is matched against _meta.yaml. + key = "/".join(parts[1:]) + if key not in declared: + orphans.append(str(rel)) + if orphans: + print("ERROR: markdown files not declared in _meta.yaml (strict orphan check):") + for o in sorted(orphans): + print(f" - {o}") + sys.exit(1) + print("OK — no orphan markdown under docs/usermanuals/.") + PY diff --git a/site/compliance.html b/site/compliance.html index 02fbe6c6..80a95d3d 100644 --- a/site/compliance.html +++ b/site/compliance.html @@ -10,10 +10,14 @@ - - + + - + + + + + @@ -49,10 +53,6 @@
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • -
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +
    • +