testing.md

Testing Standard

Scope: Everything under tests/ + CI workflows under .github/workflows/. Enforced by: .github/workflows/ci.yml (required) + .github/workflows/nightly.yml (compatibility).

Layout

Current structure (one test module per feature area; run ls tests/test_*.py | wc -l for the module count and pytest --collect-only -q for the collected-test count — both grow, so neither is restated here). The tree below is a representative subset — see git ls-files tests/ for the full inventory.

tests/typing/ is deliberately not a test package: it holds public_surface_probe.py, which exists to be type-checked by the mypy --strict public-surface gate and would fail by construction if it were ever executed. It carries no __init__.py and no test_ prefix so pytest does not collect it; tests/test_library_api.py asserts it covers every name in forgelm.__all__ and every public method on those classes.

tests/
├── conftest.py                     # Shared fixtures (minimal_config factory)
├── runtime_smoke.py                # Full-pipeline smoke fixture generator
├── test_smoke.py                   # Basic imports + CLI invocation
├── test_integration.py             # End-to-end dry-run across trainer types
├── test_cli.py                     # CLI argument parsing + exit codes
├── test_cli_subcommands.py         # Subcommand dispatching
├── test_config.py                  # Pydantic schemas + validators
├── test_trainer.py                 # Trainer orchestration logic
├── test_alignment.py               # DPO / SimPO / KTO / ORPO / GRPO
├── test_long_context.py            # RoPE scaling, NEFTune, sample packing
├── test_galore.py                  # GaLore optimizer
├── test_moe_functions.py           # MoE expert quantize + freeze
├── test_phase7.py                  # VLM + merging + PiSSA
├── test_merging_algos.py           # TIES / DARE / SLERP
├── test_synthetic.py               # Teacher → student distillation
├── test_benchmark.py               # lm-eval-harness wrapper
├── test_safety_advanced.py         # Llama Guard + severity + categories
├── test_judge_functions.py         # LLM-as-judge evaluation
├── test_compliance.py              # Audit log + manifests + provenance
├── test_eu_ai_act.py               # Articles 9-15 + Annex IV
├── test_model_card.py              # Model card generation
├── test_cost_estimation.py         # GPU cost heuristics
├── test_webhook.py                 # Slack/Teams notifier
├── test_distributed.py             # DeepSpeed / FSDP config
├── test_data_edge_cases.py         # Malformed datasets, edge cases
├── test_supply_chain_security.py   # Wave 4 / Faz 23 — pip-audit + bandit + SBOM
├── test_check_anchor_resolution.py # Wave 4 / Faz 26 — markdown anchor resolver
├── test_check_bilingual_parity.py  # Bilingual EN/TR mirror parity
├── test_gdpr_erasure.py            # GDPR Article 17 (forgelm purge)
└── …

Rules:

• One test_<module>.py per forgelm/<module>.py where practical.
• Cross-cutting features (EU AI Act, alignment) get their own file that may import multiple modules.
• conftest.py holds shared fixtures only. Domain-specific helpers live in the test files that need them.
• A new feature PR adds the matching test_*.py in the same PR. No "tests in next PR."

Browser wizard regressions

Browser wizard regressions live in tests/site/*.test.cjs. Run node --test tests/site/*.test.cjs with Node 22 or newer and the development Python environment available. The YAML round-trip test uses the active virtual environment, then the checkout's .venv, then python3; set FORGELM_TEST_PYTHON to an explicit interpreter path if needed. CI runs them on Node 24 in the validation job after installing the Python development dependencies. They use Node's built-in test runner and a VM to execute the actual wizard; generated YAML is also parsed by the real ForgeConfig. No npm packages, browser download, GPU, or network calls are needed. The shared harness (tests/site/wizard_harness.cjs) renders every step through a minimal DOM shim and boots the wizard controller for step and focus checks; it does not replace browser or native-language review.

A shim cannot compute accessible names, follow the Tab order or apply CSS. A change to the wizard's markup, focus handling or styles is also checked in a real browser, on desktop and phone widths and in at least English and Turkish, and the pull request records what was checked:

• Every step and conditional branch renders without a console error, including the Compliance step at each risk tier.
• Each field's accessible name, in the browser's accessibility tree, is its visible caption, and clicking a caption focuses its field.
• Tab reaches every control in order, and focus shows a visible ring.
• Toggling a gate, changing the risk tier or choosing an option card keeps focus on that control, and the fields it reveals appear at once.
• Next on an invalid step focuses its first invalid field.

Fixtures

The canonical factory lives in tests/_helpers/factories.py and is re-exported via tests/conftest.py (both as the callable and as a same-named pytest fixture):

def minimal_config(**overrides):
    """Build the smallest valid ForgeConfig dict for testing."""
    # Fresh deepcopy on every call so a caller's mutation cannot leak
    # between tests.
    data = deepcopy(
        {
            "model": {"name_or_path": "org/model"},
            "lora": {},
            "training": {},
            "data": {"dataset_name_or_path": "org/dataset"},
        }
    )
    data.update(overrides)
    return data

Rules:

1. Factory functions over static fixtures. minimal_config(training={"trainer_type": "dpo"}) is better than 50 parametrized fixtures.
2. No GPU in unit tests. Mock torch.cuda.is_available() or use CPU-only paths. Unit tests must run on a laptop with no GPU.
3. No network in unit tests. Mock requests.post, huggingface_hub.snapshot_download, etc. Integration tests may hit localhost but never external services. This is mechanically enforced — see "No-network guard" below.
4. Deterministic. random.seed(42), torch.manual_seed(42) in any test that touches RNG. Flaky test = broken test.

No-network guard

Rule 3 above is not prose-only: tests/conftest.py registers two autouse=True fixtures that enforce it mechanically for every test in the suite, no opt-in required.

• _block_network monkeypatches socket.socket.connect / connect_ex — the TCP-connect surface requests/urllib3/httpx all funnel through. A real connection attempt to anything other than a loopback address (127.0.0.0/8, ::1, localhost) or an AF_UNIX socket raises RuntimeError immediately, naming the offending address and the failing test's node ID. It is a fail-fast tripwire, not a full egress sandbox: bare DNS resolution (getaddrinfo), raw UDP, and subprocess network calls are not intercepted.
• _stub_hf_dataset_fingerprint stubs forgelm.compliance's two Hub-fingerprint helpers (_fingerprint_hf_metadata, _fingerprint_hf_revision) to no-ops by default. The shared minimal_config factory uses a hub-style org/dataset id, so without this stub every test that builds a training manifest would otherwise reach out to the HF Hub — now blocked by _block_network anyway, but this fixture keeps the manifest-building code path itself offline rather than merely converting the failure into a RuntimeError.

Escape hatches (two markers, both registered in conftest.py's pytest_configure):

• @pytest.mark.allow_network — opts a test out of _block_network entirely. Use only when a test genuinely needs a live connection (e.g. spinning up a real local server on a non-loopback-looking address); loopback traffic never needs this marker.
• @pytest.mark.real_fingerprint — opts a test out of the default fingerprint stub so it exercises the real _fingerprint_hf_metadata / _fingerprint_hf_revision code paths. The test itself is responsible for stubbing the underlying network call (e.g. mock huggingface_hub) since _block_network still applies unless also marked allow_network.

Test categories

Category	Scope	Speed	Runs in CI?
Smoke (test_smoke.py)	Import + CLI --help works	< 5s	Every push
Unit (test_<module>.py)	One function / method at a time, heavy mocking	< 60s total	Every push
Integration smoke (test_integration.py)	Full pipeline dry-run, no GPU, mocked HF	< 5min	Every push
Distributed (test_distributed.py)	DeepSpeed/FSDP config generation (no actual multi-GPU)	< 30s	Every push
Compatibility (via nightly.yml)	Upstream dep upgrades — latest TRL, PEFT, Unsloth	~10min	Nightly only
Cross-OS release-tag matrix (via publish.yml)	Wheel install + pytest on 3 OS × 4 Python = 12 combos. Linux-only extras (qlora, unsloth) gated to the Linux runners.	~25-40 min	Release tag push only
Supply-chain (via nightly.yml + on-tag)	pip-audit (CVE feed) + bandit (Python SAST) + CycloneDX SBOM emission per combo. [security] extra.	~5min	Nightly + every release tag

Never write a test that requires an actual GPU. The fixture runtime_smoke.py exists so "full pipeline" checks are dry-runs. If you genuinely need GPU validation, document it as a manual release-gate check in release.md, not a CI test.

Mocking

Preferred libraries: unittest.mock.patch, pytest.monkeypatch, requests_mock.

What to mock:

• Network: requests.post, huggingface_hub.* downloads, OpenAI/Anthropic API calls
• GPU: torch.cuda.is_available, torch.cuda.get_device_name
• Time: time.sleep, datetime patterns where tests need determinism
• Third-party heavy imports: unsloth, bitsandbytes, deepspeed, lm_eval (when unavailable)

What NOT to mock:

• Pydantic validation (use real config objects)
• File I/O under tmp_path (use the real filesystem via pytest fixture)
• YAML parsing
• Anything forgelm-internal that has a fast real implementation

Coverage

From pyproject.toml:

[tool.coverage.report]
fail_under = 40

40% is the floor, not the target. Current repo sits well above it. The floor was raised from 25 to 40 during Phase 11/11.5 review cycles once the audit / ingest module suite landed; the standard is now in lock-step with the toml. Rules:

1. Every new module starts at or above the overall floor.
2. Public API (non-underscore functions) has coverage.
3. Error paths (every raise and every sys.exit(!=0)) must have at least one test that triggers them.
4. pragma: no cover is allowed only for:
   • if __name__ == "__main__": blocks
   • except ImportError: fallbacks for optional deps
   • Explicit "not implemented on this platform" branches

If you need to exempt more, file an issue first.

CI gates

From .github/workflows/ci.yml:

1. Lint — ruff check + ruff format --check on entire repo. Failure = PR blocked.

2. Test matrix — Python 3.10, 3.11, 3.12, 3.13 on ubuntu-latest.

3. Coverage — pytest --cov=forgelm --cov-fail-under=40 (enforced via addopts in pyproject.toml's [tool.pytest.ini_options], kept in lock-step with [tool.coverage.report].fail_under).

4. Dry-run validation — python3 -m forgelm --config config_template.yaml --dry-run must succeed.

5. Doc CI guards (Wave 3 / Wave 4 / Wave 5):

   • python3 tools/check_bilingual_parity.py --strict — H2/H3/H4 spine sync between EN and TR mirrors (every registered pair plus the auto-discovered docs/usermanuals/ pairs; the count self-updates as mirrors are added).
   • python3 tools/check_anchor_resolution.py --strict — every relative markdown link with a #anchor fragment resolves to a real heading.
   • python3 tools/check_cli_help_consistency.py --strict — CLI --help output ↔ docs/usermanuals/{en,tr}/reference/cli.md parity.

   This list is a representative subset. The full doc-guard surface expanded during the full-project-review W1/H11 sweep — ci.yml's validate job is the authoritative inventory, and tests/test_guard_wiring.py fails CI if any tools/check_*.py guard is left unwired without an allowlisted rationale.

From .github/workflows/nightly.yml:

• Unbounded upstream versions (latest TRL/PEFT/Unsloth) to catch breaking changes early.
• Supply-chain pass: pip-audit (CVE feed) + bandit (Python SAST). Provided by the [security] optional extra.
• Failure does not block PRs but triggers an issue.

From .github/workflows/publish.yml (release-tag trigger only):

• Cross-OS matrix: 3 OS × 4 Python = 12 combos installing the packaged wheel + running pytest + emitting a per-combo CycloneDX 1.5 SBOM. Every combo must pass before PyPI publish runs. No fail-fast — all 12 combos run to completion so the failure surface is visible.

|| true discipline. Bare <command> || true in CI is forbidden — it converts non-zero exits into success and creates a fake-green status. The only sanctioned exception is the scanner + severity-tiering helper pattern:

- run: |
    pip-audit --format json > pip-audit.json || true
    python3 tools/check_pip_audit.py pip-audit.json

In this shape, || true only captures the scanner's exit code so the helper can read its JSON output; the helper itself enforces the actual severity gate. Allowed only when:

1. The || true is on the immediately preceding scanner line (not on a pytest, ruff, or test command).
2. The next line in the same run: block runs that scanner's own severity helper — tools/check_bandit.py for bandit, tools/check_pip_audit.py for pip-audit — which exits non-zero on the severities it gates. A new scanner needs its own helper and an entry in the guard's _SCANNER_HELPERS table.
3. Both lines live in the same step (so the helper truly gates the scanner output).
4. The helper is given the report that scanner wrote — its -o / --output path or > target — as one of its arguments.

The two helpers are tools/check_pip_audit.py (CVE severity tier) and tools/check_bandit.py (issue-severity tier). Any other || true in CI must be replaced with continue-on-error: true at the YAML step level, and that step must explicitly document the rationale in a YAML # comment:. Never use it on a step that runs a required command: pytest, node --test, ruff, mypy, the forgelm CLI or a tools/ guard.

tools/check_ci_failure_masking.py enforces these rules on every workflow run: block and composite action, on the shell scripts they run or source from the repository, and on the documented gauntlets. Its shell parser is tools/_shell_syntax.py.

• Fallbacks. || : and || exit 0 count as || true, inside command substitutions and trap actions too.
• Lost failures it rejects:
  • an || fallback that can exit 0, a pipe without pipefail, set +e, !, & and a command substitution;
  • a failure errexit ignores, inside an && / || list or a { … } group, when later commands follow;
  • a required command inside a case, an if branch, or a loop that may not run it (a while, or a for over a variable or with a continue / break before it);
  • continue-on-error, if: false, if: failure() or if: cancelled() on a step or job that runs one, also through local actions, and ruff --exit-zero or --fix-only;
  • an EXIT/ERR trap that can exit 0 or runs exec, an ERR trap that turns errexit off, and a DEBUG or RETURN trap that can exit or turn errexit off;
  • a function, alias or hash -p that takes over a required command;
  • an exit, exec or set -n that bash always reaches first and that keeps it from running: also behind ! or x=1 &&, in a { … } group or a function, and in a sourced or eval'd script;
  • a step shell: other than bash, sh or dash, and a repository script it cannot find (one named from $(dirname "$0") too) or that runs itself.
• Forms it accepts. It accepts only the shapes it models, so keep required commands plain. A fallback that must report before failing ends with exit 1, or captures the status first: || { rc=$?; echo …; exit "$rc"; }, with nothing before that which can leave the group (eval 'exit 0', a function call). An if that reports a failure ends the same way, with nothing else in it that can leave early: if ! cmd; then echo …; exit 1; fi. Re-raising a status captured with set +e, || rc=$? or ${PIPESTATUS[0]} later in the step is rejected. A bare exit $? after echo exits with echo's status, in a fallback or a trap, and so does a bare exit in an ERR trap.
• Scope. It catches accidental masking, not deliberate evasion. Beyond it are a required command run under another name ($CMD), by another program (make, tox) or on another machine or container, whose script it never reads (docker exec c sh -c '…', ssh host '…'); a script generated or downloaded while the step runs; a program replaced on disk or put first on PATH; a captured status changed by a script it cannot follow; a DEBUG trap that skips commands under extdebug; a remote action; any other condition, such as if: always() on a job that needs the tests, ${{ 1 == 2 }}, or an if: on a job that calls a reusable workflow; and options other than --exit-zero and --fix-only that empty a check (pytest --collect-only).

The # comment: rationale that continue-on-error needs is a review rule; the guard does not check it.

Writing a new test

Template for a new module's test file:

"""Tests for forgelm.<module>."""
import pytest
from forgelm.<module> import <public_api>


class TestPublicFunction:
    def test_happy_path(self, tmp_path):
        result = <public_api>(...)
        assert result.status == "ok"

    def test_raises_on_invalid_input(self):
        with pytest.raises(ConfigError, match="trainer_type"):
            <public_api>(bad_input)

    @pytest.mark.parametrize("value,expected", [
        ("sft", "SFTTrainer"),
        ("dpo", "DPOTrainer"),
    ])
    def test_trainer_class_selection(self, value, expected):
        assert select_trainer(value).__name__ == expected

Anti-patterns

Anti-pattern	Why rejected	Correct form
pytest.skip("not yet implemented")	Tracks zero actual behaviour	Use pytest.mark.xfail(reason="...") with an issue link
Test that just calls the function and asserts no exception	Proves nothing	Assert specific returns/side effects
sleep(2); assert something	Flaky by construction	Use monkeypatch for time, or event-based waits
print() to debug tests	Noisy output	Use caplog fixture for log assertions
@pytest.fixture(scope="session") for mutable state	Cross-test pollution	Scope to function unless immutable

Quick checklist before opening PR

• pytest tests/ passes locally
• ruff check . && ruff format --check . passes
• python3 -m forgelm --config config_template.yaml --dry-run succeeds if you touched CLI or trainer
• If docs touched: python3 tools/check_bilingual_parity.py --strict, python3 tools/check_anchor_resolution.py --strict, python3 tools/check_cli_help_consistency.py --strict
• New public function or class has tests covering happy path + one error path
• Any new exit code or exception is tested
• No GPU or network required for new unit tests