diff --git a/CHANGELOG.md b/CHANGELOG.md index 61781716..40ee7bc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,41 @@ All notable changes to ForgeLM are documented here. > Per-PR CHANGELOG entries below collapse into the v0.5.5 release > notes at tag time. +### Added — Wave 2a / Phase 34 — `forgelm doctor` env-check subcommand + +- **`forgelm doctor`** — the first command an operator should run after + installation. Probes Python version, torch + CUDA, GPU inventory, + every optional extra advertised in `pyproject.toml`, HuggingFace Hub + reachability, workspace disk space, and the `FORGELM_OPERATOR` + audit-identity hint. Tabular text report or a structured JSON + envelope (`--output-format json`). +- **`forgelm doctor --offline`** skips the HF Hub network probe and + inspects the local cache (`HF_HOME` / `~/.cache/huggingface/hub`). + Useful for air-gap deployments. +- **Honest pass / warn / fail:** warn = operator-actionable but does + not block (missing optional extra, CPU-only torch); fail = ForgeLM + cannot work this way (Python <3.10, no torch). Exit codes follow + the public contract — 0 every check passed (warns OK), 1 at least + one fail (config-error class), 2 if a probe itself crashed + (runtime-error class). +- **Self-contained probes.** Heavy deps (torch, huggingface_hub) are + imported lazily inside individual check functions so `forgelm doctor` + can run on a brand-new machine where torch is not yet installed + without crashing. One crashing probe does not abort the rest of + the report. +- Closes the `ghost-features-analysis-20260502` GH-001 onboarding + bloker (30 doc references across 8 files in `docs/usermanuals/` no + longer point at a non-existent command). +- New module `forgelm/cli/subcommands/_doctor.py` (~430 lines). +- 38 new tests in `tests/test_doctor.py` covering per-probe behaviour + (Python 3.9/3.10/3.11/3.12, torch presence + CUDA, GPU inventory, + optional-extra detection, HF Hub HEAD, HF cache populated/empty, + disk-space thresholds, operator identity), exit-code mapping, + text/JSON renderers, probe-crash isolation, plan composition, + CLI subprocess smoke, facade re-exports. +- Operator docs updated: `first-run.md` (en + tr) sample output now + matches the shipped table format. + ### Added — Wave 1 closure (Faz 9, 11, 12, 13, 25, 31, 32 — see PR description) - **Article 14 staging directory + `forgelm approve` / `forgelm reject` (Faz 9)** — diff --git a/docs/usermanuals/en/getting-started/first-run.md b/docs/usermanuals/en/getting-started/first-run.md index ff321591..9ef6cc4e 100644 --- a/docs/usermanuals/en/getting-started/first-run.md +++ b/docs/usermanuals/en/getting-started/first-run.md @@ -30,13 +30,23 @@ Before anything else, run `forgelm doctor` to check that Python, PyTorch, CUDA, ```shell $ forgelm doctor -ForgeLM 0.5.2 -Python 3.11.4 · PyTorch 2.4.0 -CUDA available · driver 535.171 · NVIDIA RTX 4090 · 24 GB -bitsandbytes ✓ (installed) -Unsloth ✓ (installed) +forgelm doctor — environment check + + [✓ pass] python.version Python 3.11.4 (CPython). + [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. + [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB). + [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training). + [✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)). + [! warn] extras.deepspeed Optional extra missing — install with: pip install 'forgelm[deepspeed]' (purpose: DeepSpeed ZeRO + offload distributed training). + [✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200). + [✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs. + +Summary: 7 pass, 2 warn, 0 fail. ``` +`--output-format json` returns a structured envelope (`{"success": bool, "checks": [...], "summary": {...}}`) so CI can filter on individual probe results without parsing the table. Pass `--offline` to skip the HF Hub network probe and inspect the local cache instead — useful for air-gapped deployments. + :::tip If `forgelm doctor` reports a problem (missing CUDA, version mismatch, no GPU), fix that first. Every other ForgeLM command will fail in confusing ways otherwise. See [Troubleshooting](#/operations/troubleshooting). ::: diff --git a/docs/usermanuals/en/getting-started/installation.md b/docs/usermanuals/en/getting-started/installation.md index b049e067..1e41ea67 100644 --- a/docs/usermanuals/en/getting-started/installation.md +++ b/docs/usermanuals/en/getting-started/installation.md @@ -100,12 +100,16 @@ If you've installed ForgeLM for GPU training, confirm CUDA is wired up correctly $ forgelm doctor ``` -`forgelm doctor` reports: -- Python and PyTorch versions -- CUDA availability and driver version -- Detected GPU model and VRAM -- Available compute capability -- bitsandbytes / Unsloth detection (if installed) +`forgelm doctor` reports a tabular pass / warn / fail diagnostic over: +- Python version (>=3.10 floor; >=3.11 recommended) +- torch + CUDA availability (CPU-only is a `warn`, not a `fail`) +- GPU inventory (per-device VRAM in GiB) +- Optional extras: `qlora`, `unsloth`, `distributed`, `eval`, `tracking`, `merging`, `export`, `ingestion`, `ingestion-pii-ml`, `ingestion-scale` — missing extras `warn` with the exact `pip install 'forgelm[]'` hint +- HuggingFace Hub reachability (via `HF_ENDPOINT` if set; skipped under `--offline`) +- Workspace disk space (<10 GiB → `fail`, <50 GiB → `warn`) +- `FORGELM_OPERATOR` audit-identity hint (Article 12) + +Pass `--output-format json` for a structured envelope (`{"success": bool, "checks": [...], "summary": {pass, warn, fail}}`); pass `--offline` for air-gap mode (skips the network probe and instead inspects the local HF cache). :::tip Run `forgelm doctor` *before* `forgelm --config ...` for any new environment. It catches missing CUDA libraries, version mismatches, and GPU-not-found errors in two seconds rather than two hours into training. diff --git a/docs/usermanuals/en/operations/air-gap.md b/docs/usermanuals/en/operations/air-gap.md index 326f3c30..0e9a1adb 100644 --- a/docs/usermanuals/en/operations/air-gap.md +++ b/docs/usermanuals/en/operations/air-gap.md @@ -113,15 +113,22 @@ evaluation: ```shell $ forgelm doctor --offline -ForgeLM 0.5.2 (offline mode verified) -✓ HF_HUB_OFFLINE set -✓ TRANSFORMERS_OFFLINE set -✓ Local cache: airgap-bundle/huggingface (3.4 GB cached models) -✓ Llama Guard 3 8B available locally -✓ lm-evaluation-harness: 4 tasks cached +forgelm doctor — environment check + + [✓ pass] python.version Python 3.11.4 (CPython). + [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. + [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA A100 (80.0 GiB). + [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training). + [✓ pass] extras.eval Installed (module lm_eval, purpose: lm-evaluation-harness benchmark scoring). + [! warn] extras.tracking Optional extra missing — install with: pip install 'forgelm[tracking]' (purpose: Weights & Biases experiment tracking). + [✓ pass] hf_hub.offline_cache HF cache at /opt/airgap/huggingface/hub: 3.4 GiB across 47 file(s). HF_HUB_OFFLINE=1. + [✓ pass] disk.workspace Workspace /opt/airgap — 412.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'airgap-op@nodeA'. Pin FORGELM_OPERATOR= for CI / pipeline runs. + +Summary: 7 pass, 2 warn, 0 fail. ``` -If `forgelm doctor --offline` reports anything unavailable, fix that *before* the air-gapped operator wastes their time. +If `forgelm doctor --offline` reports anything as `fail`, fix that *before* the air-gapped operator wastes their time. The HF cache scan honours `HF_HUB_CACHE` first, then `HF_HOME/hub`, then the default `~/.cache/huggingface/hub` — point `HF_HUB_CACHE` at your bundled cache to make the scan deterministic. ## Bundle size estimate diff --git a/docs/usermanuals/tr/getting-started/first-run.md b/docs/usermanuals/tr/getting-started/first-run.md index 99ad1e83..ede4d998 100644 --- a/docs/usermanuals/tr/getting-started/first-run.md +++ b/docs/usermanuals/tr/getting-started/first-run.md @@ -30,13 +30,23 @@ Her şeyden önce Python, PyTorch, CUDA ve opsiyonel bağımlılıkların doğru ```shell $ forgelm doctor -ForgeLM 0.5.2 -Python 3.11.4 · PyTorch 2.4.0 -CUDA available · driver 535.171 · NVIDIA RTX 4090 · 24 GB -bitsandbytes ✓ (installed) -Unsloth ✓ (installed) +forgelm doctor — environment check + + [✓ pass] python.version Python 3.11.4 (CPython). + [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. + [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB). + [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training). + [✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)). + [! warn] extras.deepspeed Optional extra missing — install with: pip install 'forgelm[deepspeed]' (purpose: DeepSpeed ZeRO + offload distributed training). + [✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200). + [✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR= for CI / pipeline runs. + +Summary: 7 pass, 2 warn, 0 fail. ``` +`--output-format json` yapısal bir zarf döner (`{"success": bool, "checks": [...], "summary": {...}}`); CI ayrı ayrı probe sonuçlarını tabloyu parse etmeden filtreleyebilir. `--offline` HF Hub ağ probe'unu atlar ve onun yerine yerel cache'i inceler — air-gap dağıtımları için faydalı. + :::tip `forgelm doctor` bir sorun raporlarsa (eksik CUDA, sürüm uyumsuzluğu, GPU yok), önce onu düzeltin. Diğer her ForgeLM komutu kafa karıştırıcı şekillerde patlar. Bkz. [Sorun Giderme](#/operations/troubleshooting). ::: diff --git a/docs/usermanuals/tr/getting-started/installation.md b/docs/usermanuals/tr/getting-started/installation.md index 19719ecf..a52a9295 100644 --- a/docs/usermanuals/tr/getting-started/installation.md +++ b/docs/usermanuals/tr/getting-started/installation.md @@ -100,12 +100,16 @@ GPU eğitimi için kurduysanız CUDA'nın doğru bağlandığını teyit edin: $ forgelm doctor ``` -`forgelm doctor` şunları raporlar: -- Python ve PyTorch sürümleri -- CUDA varlığı ve sürücü sürümü -- Algılanan GPU modeli ve VRAM -- Mevcut compute capability -- bitsandbytes / Unsloth tespiti (kuruluysa) +`forgelm doctor` tabular bir pass / warn / fail teşhis raporu üretir: +- Python sürümü (>=3.10 zorunlu, >=3.11 önerilen) +- torch + CUDA varlığı (CPU-only `warn`'dur, `fail` değil) +- GPU envanteri (cihaz başına VRAM, GiB) +- Opsiyonel extras: `qlora`, `unsloth`, `distributed`, `eval`, `tracking`, `merging`, `export`, `ingestion`, `ingestion-pii-ml`, `ingestion-scale` — eksik extra'lar tam `pip install 'forgelm[]'` ipucu ile `warn`'lanır +- HuggingFace Hub erişimi (varsa `HF_ENDPOINT` üzerinden; `--offline` ile atlanır) +- Workspace disk alanı (<10 GiB → `fail`, <50 GiB → `warn`) +- `FORGELM_OPERATOR` audit kimliği ipucu (Madde 12) + +`--output-format json` yapısal zarf döner (`{"success": bool, "checks": [...], "summary": {pass, warn, fail}}`); `--offline` air-gap modu (ağ probe'unu atlar, yerel HF cache'i inceler). :::tip Yeni bir ortamda `forgelm --config ...` koşmadan *önce* `forgelm doctor` çalıştırın. Eksik CUDA kütüphanelerini, sürüm uyumsuzluklarını ve "GPU bulunamadı" hatalarını eğitime saatler kala değil saniyeler içinde yakalar. diff --git a/docs/usermanuals/tr/operations/air-gap.md b/docs/usermanuals/tr/operations/air-gap.md index 15b2c21d..20e5f230 100644 --- a/docs/usermanuals/tr/operations/air-gap.md +++ b/docs/usermanuals/tr/operations/air-gap.md @@ -113,15 +113,22 @@ evaluation: ```shell $ forgelm doctor --offline -ForgeLM 0.5.2 (offline mod doğrulandı) -✓ HF_HUB_OFFLINE ayarlı -✓ TRANSFORMERS_OFFLINE ayarlı -✓ Yerel cache: airgap-bundle/huggingface (3.4 GB cache'li model) -✓ Llama Guard 3 8B yerel olarak mevcut -✓ lm-evaluation-harness: 4 görev cache'li +forgelm doctor — environment check + + [✓ pass] python.version Python 3.11.4 (CPython). + [✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4. + [✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA A100 (80.0 GiB). + [✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training). + [✓ pass] extras.eval Installed (module lm_eval, purpose: lm-evaluation-harness benchmark scoring). + [! warn] extras.tracking Optional extra missing — install with: pip install 'forgelm[tracking]' (purpose: Weights & Biases experiment tracking). + [✓ pass] hf_hub.offline_cache HF cache at /opt/airgap/huggingface/hub: 3.4 GiB across 47 file(s). HF_HUB_OFFLINE=1. + [✓ pass] disk.workspace Workspace /opt/airgap — 412.0 GiB free of 500.0 GiB. + [! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'airgap-op@nodeA'. Pin FORGELM_OPERATOR= for CI / pipeline runs. + +Summary: 7 pass, 2 warn, 0 fail. ``` -`forgelm doctor --offline` herhangi bir şeyin erişilemediğini raporlarsa, air-gap operatörünün vaktini boşa harcamadan *önce* düzeltin. +`forgelm doctor --offline` herhangi bir şeyi `fail` olarak raporlarsa, air-gap operatörünün vaktini boşa harcamadan *önce* düzeltin. HF cache taraması önce `HF_HUB_CACHE`'i, sonra `HF_HOME/hub`'ı, en son varsayılan `~/.cache/huggingface/hub`'ı dener — taramayı deterministik yapmak için `HF_HUB_CACHE`'i pakete dahil ettiğiniz cache'e işaret ettirin. ## Paket boyutu tahmini diff --git a/forgelm/cli/__init__.py b/forgelm/cli/__init__.py index 08007312..f71721c0 100644 --- a/forgelm/cli/__init__.py +++ b/forgelm/cli/__init__.py @@ -84,6 +84,7 @@ _add_audit_subcommand, # noqa: F401 — re-export for tests _add_chat_subcommand, # noqa: F401 — re-export for tests _add_deploy_subcommand, # noqa: F401 — re-export for tests + _add_doctor_subcommand, # noqa: F401 — re-export for tests _add_export_subcommand, # noqa: F401 — re-export for tests _add_ingest_subcommand, # noqa: F401 — re-export for tests _add_quickstart_subcommand, # noqa: F401 — re-export for tests @@ -134,6 +135,23 @@ # Chat / export / deploy / ingest subcommand dispatchers. from .subcommands._chat import _run_chat_cmd # noqa: F401 — re-export for tests from .subcommands._deploy import _run_deploy_cmd # noqa: F401 — re-export for tests + +# Doctor subcommand (Phase 34 environment diagnostics). +from .subcommands._doctor import ( + _check_disk_space, # noqa: F401 — re-export for tests + _check_gpu_inventory, # noqa: F401 — re-export for tests + _check_hf_cache_offline, # noqa: F401 — re-export for tests + _check_hf_hub_reachable, # noqa: F401 — re-export for tests + _check_operator_identity, # noqa: F401 — re-export for tests + _check_optional_extra, # noqa: F401 — re-export for tests + _check_python_version, # noqa: F401 — re-export for tests + _check_torch_cuda, # noqa: F401 — re-export for tests + _render_json, # noqa: F401 — re-export for tests + _render_text, # noqa: F401 — re-export for tests + _resolve_exit_code, # noqa: F401 — re-export for tests + _run_all_checks, # noqa: F401 — re-export for tests + _run_doctor_cmd, # noqa: F401 — re-export for tests +) from .subcommands._export import _run_export_cmd # noqa: F401 — re-export for tests from .subcommands._ingest import _run_ingest_cmd # noqa: F401 — re-export for tests diff --git a/forgelm/cli/_dispatch.py b/forgelm/cli/_dispatch.py index 4a7bad3f..47703968 100644 --- a/forgelm/cli/_dispatch.py +++ b/forgelm/cli/_dispatch.py @@ -62,6 +62,14 @@ def _dispatch_subcommand(command: str, args) -> None: elif command == "audit": _cli_facade._run_audit_cmd(args, getattr(args, "output_format", "text")) sys.exit(EXIT_SUCCESS) + elif command == "doctor": + # _run_doctor_cmd does its own sys.exit with a result-mapped exit + # code (0/1/2) so we don't fall through to a blanket EXIT_SUCCESS. + _cli_facade._run_doctor_cmd(args, getattr(args, "output_format", "text")) + # Defensive: _run_doctor_cmd always exits. This line is unreachable + # in practice but keeps the dispatch table exhaustive against a + # future doctor refactor that might forget the sys.exit. + sys.exit(EXIT_SUCCESS) elif command == "verify-audit": sys.exit(_cli_facade._run_verify_audit_cmd(args)) elif command == "approve": diff --git a/forgelm/cli/_parser.py b/forgelm/cli/_parser.py index 55bc2098..48b6dd10 100644 --- a/forgelm/cli/_parser.py +++ b/forgelm/cli/_parser.py @@ -291,6 +291,40 @@ def _add_ingest_subcommand(subparsers) -> None: _add_common_subparser_flags(p, include_output_format=True) +def _add_doctor_subcommand(subparsers) -> None: + """Phase 34: environment check (`forgelm doctor`). + + The first command an operator should run after installation. Probes + Python version, torch / CUDA, GPU inventory, optional extras, HF Hub + reachability (or local cache when ``--offline``), workspace disk + space, and the FORGELM_OPERATOR audit-identity hint. + + Exit codes follow the public contract: 0 = all pass, 1 = at least + one check failed (config-error class), 2 = probe crashed. + """ + p = subparsers.add_parser( + "doctor", + help="Run environment + dependency diagnostics (the first command after install).", + description=( + "Probe Python, torch + CUDA, GPU inventory, optional ForgeLM extras, " + "HuggingFace Hub reachability, workspace disk space, and the " + "FORGELM_OPERATOR audit-identity hint. Emits a tabular text report or " + "a structured JSON envelope (`--output-format json`). Pass `--offline` " + "to skip the HF Hub network probe and instead inspect the local cache." + ), + ) + p.add_argument( + "--offline", + action="store_true", + help=( + "Skip the HuggingFace Hub network probe. Inspects the local HF cache " + "(HF_HOME or ~/.cache/huggingface/hub) instead — useful for air-gapped " + "deployments where the network probe would always fail." + ), + ) + _add_common_subparser_flags(p, include_output_format=True) + + def _add_audit_subcommand(subparsers) -> None: p = subparsers.add_parser( "audit", @@ -560,6 +594,7 @@ def parse_args(): _add_quickstart_subcommand(subparsers) _add_ingest_subcommand(subparsers) _add_audit_subcommand(subparsers) + _add_doctor_subcommand(subparsers) _add_verify_audit_subcommand(subparsers) _add_approve_subcommand(subparsers) _add_reject_subcommand(subparsers) diff --git a/forgelm/cli/subcommands/_doctor.py b/forgelm/cli/subcommands/_doctor.py new file mode 100644 index 00000000..c8597608 --- /dev/null +++ b/forgelm/cli/subcommands/_doctor.py @@ -0,0 +1,793 @@ +"""``forgelm doctor`` env-check subcommand (Phase 34). + +The first command an operator should run after installation. Probes the +environment ForgeLM expects (Python, torch, CUDA, GPU, optional extras, +HF Hub reachability, disk space, audit-log identity) and emits a +structured pass / warn / fail report — text by default, JSON envelope +when ``--output-format json`` is set. + +Design goals (closure plan §9.5 Phase 34): + +- **Self-contained.** Probes use stdlib only at module load. Heavy + deps (torch, huggingface_hub) are imported lazily inside the + individual check functions so ``forgelm doctor`` can run on a + brand-new machine where torch is not yet installed without crashing. +- **Honest pass / warn / fail.** ``warn`` is reserved for "operator + may want to address this" (e.g. optional extra missing); ``fail`` + is reserved for "ForgeLM cannot work this way" (e.g. Python version + too old). Air-gap (``--offline``) skips the HF Hub network probe + rather than reporting it as a failure. +- **Public exit-code contract.** 0 when every check passes, 1 when at + least one fails (config-error class), 2 when a probe itself crashed + (runtime-error class — operator-actionable bug, not config). +""" + +from __future__ import annotations + +import json +import os +import platform +import shutil +import socket +import sys +from dataclasses import dataclass, field +from typing import Any, Callable, Dict, List, Optional, Tuple + +from .._exit_codes import EXIT_CONFIG_ERROR, EXIT_SUCCESS, EXIT_TRAINING_ERROR +from .._logging import logger + +# Status vocabulary. Centralised so a future rename (e.g. "warn" -> +# "warning") cannot drift across the renderers and the JSON contract. +_STATUS_PASS = "pass" +_STATUS_WARN = "warn" +_STATUS_FAIL = "fail" + +# Optional extras advertised in pyproject.toml [project.optional-dependencies]. +# Each entry is ``(extra_name, importable_module, human_purpose_blurb)``. +# ``importable_module`` is the package the extra installs that we probe via +# ``importlib.util.find_spec`` — a successful spec resolve means the extra is +# present. Doctor never *imports* the module (avoids triggering torch / +# transformers / spacy heavy load on a healthy probe). +# +# Wave 2a Round-1 review (qodo bot): extras names audited 2026-05-02 against +# pyproject.toml. The original list used aspirational names ("deepspeed", +# "evaluation", "wandb", "mergekit") that did not match the published extras +# ("distributed", "eval", "tracking", "merging") — the install-hint a +# missing-extra warn produced was unactionable because the suggested +# ``pip install 'forgelm[deepspeed]'`` would 404. Now aligned with the +# actual pyproject names. +_OPTIONAL_EXTRAS: Tuple[Tuple[str, str, str], ...] = ( + ("qlora", "bitsandbytes", "4-bit / 8-bit QLoRA training"), + ("unsloth", "unsloth", "Unsloth-accelerated training (Linux GPUs only)"), + ("distributed", "deepspeed", "DeepSpeed ZeRO + offload distributed training"), + ("eval", "lm_eval", "lm-evaluation-harness benchmark scoring"), + ("tracking", "wandb", "Weights & Biases experiment tracking"), + ("merging", "mergekit", "Model-merge backend for the merge mode"), + ("export", "llama_cpp", "GGUF export via llama-cpp-python (Linux + macOS only)"), + ("ingestion", "pypdf", "PDF / DOCX / EPUB ingestion"), + ("ingestion-pii-ml", "presidio_analyzer", "Presidio ML-NER PII detection in audit"), + ("ingestion-scale", "datasketch", "MinHash LSH for >50K-row dedup"), +) + +# Disk-space thresholds for the workspace check. Tuned for "enough room +# to download a 7B-parameter model + run one fine-tune" — ~50 GB safe, +# 10-50 GB warn, <10 GB fail. +_DISK_FAIL_GB = 10.0 +_DISK_WARN_GB = 50.0 + + +@dataclass +class _CheckResult: + """Outcome of one diagnostic probe. + + ``status`` is one of pass / warn / fail. ``name`` is operator- + facing; ``detail`` carries the actual finding (Python version + string, GPU count, missing extra name, etc.) so the operator can + act without re-running the probe themselves. + """ + + name: str + status: str + detail: str + # Optional structured fields that downstream tooling may want + # (CI gate that filters by category, dashboard widget, etc.). + extras: Dict[str, Any] = field(default_factory=dict) + + +# --------------------------------------------------------------------------- +# Individual checks +# --------------------------------------------------------------------------- + + +def _check_python_version() -> _CheckResult: + """Pin Python to the supported window. + + ForgeLM declares ``python_requires=">=3.10"`` in pyproject.toml. We + treat 3.10 as the floor (fail below), 3.11+ as the recommended + surface (pass above), 3.10.x as warn-with-explanation (still works + but operators should be tracking 3.11 / 3.12). + """ + version = sys.version_info + label = f"{version.major}.{version.minor}.{version.micro}" + if version < (3, 10): + return _CheckResult( + name="python.version", + status=_STATUS_FAIL, + detail=f"Python {label} is below the supported floor (>=3.10). Upgrade Python to >=3.11.", + extras={"version": label, "minimum": "3.10", "recommended": "3.11"}, + ) + if version < (3, 11): + return _CheckResult( + name="python.version", + status=_STATUS_WARN, + detail=f"Python {label} is supported but >=3.11 is recommended for performance + typing improvements.", + extras={"version": label, "minimum": "3.10", "recommended": "3.11"}, + ) + return _CheckResult( + name="python.version", + status=_STATUS_PASS, + detail=f"Python {label} ({platform.python_implementation()}).", + extras={"version": label, "implementation": platform.python_implementation()}, + ) + + +def _check_torch_cuda() -> _CheckResult: + """Detect torch + CUDA availability. + + CPU-only is a *legitimate* setup (small experiments, CI smoke, + inference-only deployments) so a missing CUDA does not fail — + it warns so the operator notices that a GPU run will fail later. + """ + try: + import torch + except ImportError: + return _CheckResult( + name="torch.installed", + status=_STATUS_FAIL, + detail="torch is not installed. Install with: pip install 'forgelm'.", + extras={"installed": False}, + ) + cuda_available = bool(torch.cuda.is_available()) + cuda_version = getattr(torch.version, "cuda", None) if cuda_available else None + if not cuda_available: + return _CheckResult( + name="torch.cuda", + status=_STATUS_WARN, + detail=( + f"torch {torch.__version__} installed but CUDA is unavailable. " + "CPU-only runs are supported but training will be very slow; " + "expect to use this only for tiny experiments or smoke tests." + ), + extras={ + "torch_version": torch.__version__, + "cuda_available": False, + }, + ) + return _CheckResult( + name="torch.cuda", + status=_STATUS_PASS, + detail=f"torch {torch.__version__} with CUDA {cuda_version}.", + extras={ + "torch_version": torch.__version__, + "cuda_available": True, + "cuda_version": cuda_version, + }, + ) + + +def _check_gpu_inventory() -> _CheckResult: + """Enumerate visible GPUs + per-device VRAM. + + Skipped (returns ``warn``) when CUDA is unavailable — no point + enumerating zero devices. + """ + try: + import torch + except ImportError: + return _CheckResult( + name="gpu.inventory", + status=_STATUS_FAIL, + detail="torch is not installed.", + extras={"installed": False}, + ) + if not torch.cuda.is_available(): + return _CheckResult( + name="gpu.inventory", + status=_STATUS_WARN, + detail="No CUDA devices visible; CPU-only mode.", + extras={"device_count": 0}, + ) + count = torch.cuda.device_count() + devices: List[Dict[str, Any]] = [] + for idx in range(count): + try: + props = torch.cuda.get_device_properties(idx) + except (RuntimeError, AssertionError) as exc: # pragma: no cover — defensive + devices.append( + { + "index": idx, + "error": f"could not query device {idx}: {exc}", + } + ) + continue + # ``total_memory`` is bytes; convert to GiB for the operator- + # facing label and keep raw bytes in the structured payload. + vram_gib = round(props.total_memory / (1024**3), 1) + devices.append( + { + "index": idx, + "name": props.name, + "vram_gib": vram_gib, + "vram_bytes": props.total_memory, + } + ) + if count == 0: + return _CheckResult( + name="gpu.inventory", + status=_STATUS_WARN, + detail="CUDA reports no visible devices.", + extras={"device_count": 0}, + ) + summary = ", ".join(f"GPU{d['index']}: {d.get('name', '?')} ({d.get('vram_gib', '?')} GiB)" for d in devices) + return _CheckResult( + name="gpu.inventory", + status=_STATUS_PASS, + detail=f"{count} GPU(s) — {summary}.", + extras={"device_count": count, "devices": devices}, + ) + + +def _check_optional_extra(extra: str, module: str, purpose: str) -> _CheckResult: + """Probe one optional extra without triggering its heavy import side effects.""" + import importlib.util + + spec = importlib.util.find_spec(module) + if spec is None: + return _CheckResult( + name=f"extras.{extra}", + status=_STATUS_WARN, + detail=f"Optional extra missing — install with: pip install 'forgelm[{extra}]' (purpose: {purpose}).", + extras={"extra": extra, "module": module, "installed": False}, + ) + return _CheckResult( + name=f"extras.{extra}", + status=_STATUS_PASS, + detail=f"Installed (module {module}, purpose: {purpose}).", + extras={"extra": extra, "module": module, "installed": True}, + ) + + +# Default Hub endpoint; overridable via the standard HF_ENDPOINT env var. +# Mirrors huggingface_hub's own resolution path so an operator with a +# self-hosted mirror gets the right probe target. +_DEFAULT_HF_ENDPOINT = "https://huggingface.co" + + +def _resolve_hf_endpoint() -> str: + """Resolve the HuggingFace Hub endpoint via the standard env var. + + huggingface_hub honours ``HF_ENDPOINT`` for self-hosted mirrors and + enterprise installs (e.g. internal Datasaur / KServe-fronted Hub). + Hard-coding ``huggingface.co`` would produce a false warning on + those deployments — the corp proxy might block public huggingface.co + while the internal mirror is happily serving requests. + """ + return (os.environ.get("HF_ENDPOINT") or _DEFAULT_HF_ENDPOINT).rstrip("/") + + +def _check_hf_hub_reachable(timeout_seconds: float = 5.0) -> _CheckResult: + """HEAD ``${HF_ENDPOINT}/api/models`` to verify the Hub is reachable. + + Skipped by the caller in ``--offline`` mode. Treats a connection + failure as a *warning* rather than a fail because a transient outage + (operator's wifi, captive portal, corp proxy down) shouldn't refuse + a doctor run that may exist precisely to surface that fact. + + Wave 2a Round-1 review: + + - F-27-04: dropped the ``socket.timeout`` ``except`` branch. + ``socket.timeout`` is an alias for ``TimeoutError`` since Python + 3.10 (the project floor), and ``urllib.request.urlopen`` wraps a + timeout as ``URLError`` anyway — the branch was dead. + - bot (gemini): now resolves the endpoint via ``_resolve_hf_endpoint`` + so ``HF_ENDPOINT=https://internal-mirror.example`` is respected + (mirrors how ``huggingface_hub`` resolves it). + - F-27-02 partial: routed the request through a single User-Agent + header so server logs identify the probe. Full migration to + ``forgelm._http`` is deferred (no `safe_get` exists yet; adding + it is XPR-02 cross-PR scope). + """ + import urllib.error + import urllib.request + + endpoint = _resolve_hf_endpoint() + probe_url = f"{endpoint}/api/models" + headers = {"User-Agent": "forgelm-doctor/Phase-34"} + + try: + request = urllib.request.Request(probe_url, method="HEAD", headers=headers) + with urllib.request.urlopen(request, timeout=timeout_seconds) as response: + status_code = response.status + except urllib.error.URLError as exc: + # urlopen wraps both DNS failures AND timeouts as URLError, so a + # single branch covers both. reason may be a string or another + # exception object; coerce to str for the operator-facing detail. + return _CheckResult( + name="hf_hub.reachable", + status=_STATUS_WARN, + detail=f"Could not reach HuggingFace Hub at {probe_url}: {exc.reason}. Check network / proxy / HF_ENDPOINT.", + extras={"reachable": False, "endpoint": endpoint, "error": str(exc.reason)}, + ) + except OSError as exc: # pragma: no cover — defensive + return _CheckResult( + name="hf_hub.reachable", + status=_STATUS_WARN, + detail=f"Network error reaching HuggingFace Hub at {probe_url}: {exc}.", + extras={"reachable": False, "endpoint": endpoint, "error": str(exc)}, + ) + + if 200 <= status_code < 400: + return _CheckResult( + name="hf_hub.reachable", + status=_STATUS_PASS, + detail=f"HuggingFace Hub reachable at {endpoint} (HTTP {status_code}).", + extras={"reachable": True, "endpoint": endpoint, "status_code": status_code}, + ) + return _CheckResult( + name="hf_hub.reachable", + status=_STATUS_WARN, + detail=f"HuggingFace Hub at {endpoint} returned HTTP {status_code}.", + extras={"reachable": False, "endpoint": endpoint, "status_code": status_code}, + ) + + +# Cache-walk safety cap. Real HF caches reach 50+ GiB on long-lived +# workstations; an unbounded walk on NFS-mounted caches takes 30+s. +# Phase 34 doesn't need an exact size — "is there *something* cached?" +# is enough. We cap depth + file count and report whether the cap fired. +_HF_CACHE_WALK_DEPTH = 4 +_HF_CACHE_WALK_FILE_LIMIT = 5_000 + + +def _resolve_hf_cache_dir() -> str: + """Resolve the HF Hub cache directory honouring the standard env vars. + + Priority (matches huggingface_hub's own resolution): + + 1. ``HF_HUB_CACHE`` — newer dedicated env var, wins outright. + 2. ``HF_HOME/hub`` — the *hub* sub-directory (huggingface_hub + partitions cache by purpose; ``HF_HOME`` sets the *parent*, not + the hub cache directly). Wave 2a Round-1 review (gemini bot) + fix: the original implementation pointed at ``HF_HOME`` + directly which is wrong — that directory contains the hub + + datasets + spaces sub-trees, not just the hub blobs. + 3. ``~/.cache/huggingface/hub`` — the documented default. + """ + hf_hub_cache = os.environ.get("HF_HUB_CACHE") + if hf_hub_cache: + return hf_hub_cache + hf_home = os.environ.get("HF_HOME") + if hf_home: + return os.path.join(hf_home, "hub") + return os.path.expanduser("~/.cache/huggingface/hub") + + +def _check_hf_cache_offline() -> _CheckResult: + """Inspect the local HF cache (``--offline`` mode replacement for the Hub probe). + + A populated cache means an air-gapped run can satisfy + ``local_files_only=True`` without any network access. An empty cache + is a warning — the operator may need to run ``forgelm cache-models`` + (Phase 35) before training. + + Wave 2a Round-1 review fixes: + + - bot (gemini): cache directory resolution now honours + ``HF_HUB_CACHE`` and the ``hub`` sub-directory of ``HF_HOME`` + (was: pointed at ``HF_HOME`` itself, which is the parent of + multiple cache trees). + - F-27-03: walk is depth-capped + file-capped so an NFS-mounted + 50+ GiB cache doesn't take 30+s; cap-hit is recorded in extras + so downstream tooling can detect partial-scan results. + """ + cache_dir = _resolve_hf_cache_dir() + if not os.path.isdir(cache_dir): + return _CheckResult( + name="hf_hub.offline_cache", + status=_STATUS_WARN, + detail=( + f"HF cache not found at {cache_dir}. Air-gapped runs need pre-cached models — " + "see `forgelm cache-models` (Phase 35) once available, or set HF_HUB_CACHE / " + "HF_HOME (with /hub subdirectory) and pre-populate." + ), + extras={"cache_dir": cache_dir, "exists": False}, + ) + # Bytes used by the cache. Sum regular-file sizes only; symlinks in + # the HF cache point at the blob store and would be double-counted. + # Depth + file caps keep the walk bounded so the doctor probe stays + # snappy on populated caches. + total_bytes = 0 + file_count = 0 + walk_truncated = False + cache_dir_abs = os.path.abspath(cache_dir) + base_depth = cache_dir_abs.rstrip(os.sep).count(os.sep) + for root, dirs, files in os.walk(cache_dir_abs): + depth = root.count(os.sep) - base_depth + if depth > _HF_CACHE_WALK_DEPTH: + # Trim sub-directories so os.walk stops descending. + dirs[:] = [] + walk_truncated = True + continue + for filename in files: + if file_count >= _HF_CACHE_WALK_FILE_LIMIT: + walk_truncated = True + break + full = os.path.join(root, filename) + if os.path.islink(full): + continue + try: + total_bytes += os.path.getsize(full) + file_count += 1 + except OSError: + continue + if file_count >= _HF_CACHE_WALK_FILE_LIMIT: + break + cache_gib = round(total_bytes / (1024**3), 2) + offline_env = os.environ.get("HF_HUB_OFFLINE") + detail = ( + f"HF cache at {cache_dir}: {cache_gib} GiB across {file_count} file(s)" + f"{' (scan truncated at depth/file cap)' if walk_truncated else ''}. " + f"HF_HUB_OFFLINE={offline_env or 'unset'}." + ) + return _CheckResult( + name="hf_hub.offline_cache", + status=_STATUS_PASS if file_count > 0 else _STATUS_WARN, + detail=detail, + extras={ + "cache_dir": cache_dir, + "exists": True, + "size_gib": cache_gib, + "file_count": file_count, + "walk_truncated": walk_truncated, + "hf_hub_offline_env": offline_env, + }, + ) + + +def _check_disk_space(path: str = ".") -> _CheckResult: + """Workspace free-space probe. + + Disk space is a frequent source of late-failing training runs (a + fine-tune that crashes mid-checkpoint because the workspace is + full). ``shutil.disk_usage`` works cross-platform and is in the + stdlib so this probe runs without any optional dep. + """ + try: + usage = shutil.disk_usage(path) + except OSError as exc: # pragma: no cover — defensive (permission denied) + return _CheckResult( + name="disk.workspace", + status=_STATUS_FAIL, + detail=f"Could not query disk usage at {path!r}: {exc}.", + extras={"path": os.path.abspath(path), "error": str(exc)}, + ) + free_gib = round(usage.free / (1024**3), 1) + total_gib = round(usage.total / (1024**3), 1) + if free_gib < _DISK_FAIL_GB: + status = _STATUS_FAIL + elif free_gib < _DISK_WARN_GB: + status = _STATUS_WARN + else: + status = _STATUS_PASS + return _CheckResult( + name="disk.workspace", + status=status, + detail=f"Workspace {os.path.abspath(path)} — {free_gib} GiB free of {total_gib} GiB.", + extras={ + "path": os.path.abspath(path), + "free_gib": free_gib, + "total_gib": total_gib, + "fail_threshold_gib": _DISK_FAIL_GB, + "warn_threshold_gib": _DISK_WARN_GB, + }, + ) + + +# Env-var names that may carry secret material — never echo their *values* +# in the doctor output even when the operator runs --output-format json +# and pipes to a CI log. Pattern matches AuditLogger's own discipline. +_DOCTOR_SECRET_ENV_NAMES: frozenset[str] = frozenset( + { + "FORGELM_AUDIT_SECRET", # HMAC key for audit-log signing + "HF_TOKEN", # HuggingFace Hub auth token + "HUGGING_FACE_HUB_TOKEN", # legacy alias of HF_TOKEN + "FORGELM_RESUME_TOKEN", # API resume token (Phase 13 future) + } +) + + +def _mask_env_value_for_audit(name: str, value: str) -> str: + """Mask values for env vars whose name implies secret material. + + Wave 2a Round-1 review (F-27-05): probes echo env-var values + verbatim into ``detail`` + ``extras`` today, which would leak any + future probe surfacing ``HF_TOKEN`` or ``FORGELM_AUDIT_SECRET``. + Centralised mask so the policy lives in one place. ``FORGELM_OPERATOR`` + is *not* in the mask list — it is operator identity, not a secret — + but the discipline for adding new probes is "if the env-var name + contains TOKEN / SECRET / KEY / PASSWORD, add it to + _DOCTOR_SECRET_ENV_NAMES first, then surface it". + """ + if name in _DOCTOR_SECRET_ENV_NAMES: + # Show length only so the operator can confirm "yes, set", + # without exposing the value. + return f"" + return value + + +def _check_operator_identity() -> _CheckResult: + """Article 12 record-keeping reminder. + + ``FORGELM_OPERATOR`` set explicitly is the recommended path for + CI / pipelines (so audit events carry a meaningful identity). + Unset is not a failure — the AuditLogger falls back to + ``getuser()@host`` which is fine on a developer workstation — + but warn so a CI deployer is reminded to pin it. + + Wave 2a Round-1 review (qodo bot): the unresolved-username branch + now respects the ``FORGELM_ALLOW_ANONYMOUS_OPERATOR=1`` opt-in the + same way ``AuditLogger.__init__`` does. When that env var is set, + the doctor returns ``warn`` (anonymous identity OK by operator + choice) instead of ``fail``; when it is not set, ``fail`` stands + because ``AuditLogger`` itself would refuse to start. + """ + explicit = os.environ.get("FORGELM_OPERATOR") + if explicit: + masked = _mask_env_value_for_audit("FORGELM_OPERATOR", explicit) + return _CheckResult( + name="operator.identity", + status=_STATUS_PASS, + detail=f"FORGELM_OPERATOR set to {masked!r}; audit events will carry this identity.", + extras={"FORGELM_OPERATOR": masked, "source": "env"}, + ) + # Try to resolve the fallback identity AuditLogger would use so + # the operator sees what their audit events would look like. + try: + import getpass + + username = getpass.getuser() + except (KeyError, OSError, ImportError): + username = None + hostname = socket.gethostname() or "unknown-host" + if username: + fallback = f"{username}@{hostname}" + return _CheckResult( + name="operator.identity", + status=_STATUS_WARN, + detail=( + f"FORGELM_OPERATOR not set; audit events will fall back to {fallback!r}. " + "Pin FORGELM_OPERATOR= for CI / pipeline runs so the audit log identifies a stable identity." + ), + extras={"FORGELM_OPERATOR": None, "fallback": fallback, "source": "getpass"}, + ) + # No username AND no FORGELM_OPERATOR: AuditLogger refuses to start + # unless the operator explicitly opted in to anonymous audit + # entries. Mirror that opt-in here so doctor's verdict matches the + # actual training-time behaviour (warn-OK-but-suboptimal vs hard-fail). + allow_anonymous = os.environ.get("FORGELM_ALLOW_ANONYMOUS_OPERATOR") == "1" + if allow_anonymous: + anon = f"anonymous@{hostname}" + return _CheckResult( + name="operator.identity", + status=_STATUS_WARN, + detail=( + f"FORGELM_OPERATOR not set, getpass.getuser() unavailable, but " + f"FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 is set; audit events will record {anon!r}. " + "Not recommended for EU AI Act Article 12 record-keeping; pin FORGELM_OPERATOR= when possible." + ), + extras={ + "FORGELM_OPERATOR": None, + "fallback": anon, + "source": "anonymous_opt_in", + }, + ) + return _CheckResult( + name="operator.identity", + status=_STATUS_FAIL, + detail=( + "FORGELM_OPERATOR not set AND getpass.getuser() could not resolve a username. " + "Set FORGELM_OPERATOR= (or FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 to opt in to " + "anonymous audit entries — not recommended for EU AI Act Article 12)." + ), + extras={"FORGELM_OPERATOR": None, "fallback": None}, + ) + + +# --------------------------------------------------------------------------- +# Orchestration +# --------------------------------------------------------------------------- + + +# Order matters for the text rendering — keep it grouped (env first, +# hardware next, optional extras, then network, then operator identity). +def _build_check_plan(*, offline: bool) -> List[Tuple[str, Callable[[], _CheckResult]]]: + """Return the ordered list of ``(label, callable)`` probe entries. + + The label is only used in error reporting if a probe itself crashes; + each ``_CheckResult`` carries its own ``name`` for the renderer. + """ + plan: List[Tuple[str, Callable[[], _CheckResult]]] = [ + ("python.version", _check_python_version), + ("torch.cuda", _check_torch_cuda), + ("gpu.inventory", _check_gpu_inventory), + ] + for extra_name, module, purpose in _OPTIONAL_EXTRAS: + plan.append((f"extras.{extra_name}", _make_extra_probe(extra_name, module, purpose))) + if offline: + plan.append(("hf_hub.offline_cache", _check_hf_cache_offline)) + else: + plan.append(("hf_hub.reachable", _check_hf_hub_reachable)) + plan.append(("disk.workspace", _check_disk_space)) + plan.append(("operator.identity", _check_operator_identity)) + return plan + + +def _make_extra_probe(extra: str, module: str, purpose: str) -> Callable[[], _CheckResult]: + """Build a zero-arg closure that probes one optional extra. + + Defined as a top-level helper rather than a lambda so the closure + binding is explicit and the probe is easy to unit-test by name. + """ + + def _probe() -> _CheckResult: + return _check_optional_extra(extra, module, purpose) + + return _probe + + +def _run_all_checks(*, offline: bool) -> List[_CheckResult]: + """Execute every probe in order, catching crashes per-probe. + + A probe that raises an unexpected exception is converted into a + ``fail`` result so the rest of the report still renders — partial + diagnostics are more useful than zero diagnostics when something is + broken. + """ + results: List[_CheckResult] = [] + for label, probe in _build_check_plan(offline=offline): + try: + results.append(probe()) + except Exception as exc: # noqa: BLE001 — best-effort: per-probe catch keeps the rest of the report rendering when one diagnostic crashes; converting to fail lets the operator see the broken probe name + traceback class instead of silently dropping it. + logger.exception("Doctor probe %r raised: %s", label, exc) + results.append( + _CheckResult( + name=label, + status=_STATUS_FAIL, + detail=f"Probe crashed: {exc.__class__.__name__}: {exc}", + extras={"crashed": True, "error_class": exc.__class__.__name__}, + ) + ) + return results + + +# --------------------------------------------------------------------------- +# Renderers +# --------------------------------------------------------------------------- + + +_STATUS_GLYPHS: Dict[str, str] = { + _STATUS_PASS: "✓", + _STATUS_WARN: "!", + _STATUS_FAIL: "✗", +} + + +def _render_text(results: List[_CheckResult]) -> str: + """Render a tabular text report. + + Plain ASCII so the output stays readable in CI logs, redirected + files, and SSH sessions without colour support. One line per check + plus a summary footer. + """ + pass_count = sum(1 for r in results if r.status == _STATUS_PASS) + warn_count = sum(1 for r in results if r.status == _STATUS_WARN) + fail_count = sum(1 for r in results if r.status == _STATUS_FAIL) + + lines: List[str] = ["forgelm doctor — environment check", ""] + name_width = max(len(r.name) for r in results) if results else 0 + for result in results: + glyph = _STATUS_GLYPHS.get(result.status, "?") + lines.append(f" [{glyph} {result.status}] {result.name:<{name_width}} {result.detail}") + lines.append("") + lines.append(f"Summary: {pass_count} pass, {warn_count} warn, {fail_count} fail.") + return "\n".join(lines) + + +def _render_json(results: List[_CheckResult]) -> str: + """Render the JSON envelope. + + Shape mirrors other ForgeLM subcommand contracts: + ``{"success": bool, "checks": [...], "summary": {"pass": N, "warn": N, "fail": N}}``. + ``success`` is True iff no check failed (warns are operator-actionable + but do not flip the contract). + """ + payload_checks = [ + { + "name": r.name, + "status": r.status, + "detail": r.detail, + "extras": r.extras, + } + for r in results + ] + summary = { + "pass": sum(1 for r in results if r.status == _STATUS_PASS), + "warn": sum(1 for r in results if r.status == _STATUS_WARN), + "fail": sum(1 for r in results if r.status == _STATUS_FAIL), + } + envelope = { + "success": summary["fail"] == 0, + "checks": payload_checks, + "summary": summary, + } + return json.dumps(envelope, indent=2) + + +def _resolve_exit_code(results: List[_CheckResult]) -> int: + """Map the result list to one of the public exit codes. + + Contract: + - 0 (EXIT_SUCCESS) — every check passed (warn included; warns are + operator-actionable but do not block) + - 1 (EXIT_CONFIG_ERROR) — at least one check returned ``fail`` (a + misconfiguration the operator can correct) + - 2 (EXIT_TRAINING_ERROR) — a probe itself crashed (an unexpected + doctor bug, surfaced as runtime-error class so CI/CD retry logic + treats it the same way as other runtime failures) + """ + crashed = any(r.extras.get("crashed") for r in results) + if crashed: + return EXIT_TRAINING_ERROR + has_fail = any(r.status == _STATUS_FAIL for r in results) + return EXIT_CONFIG_ERROR if has_fail else EXIT_SUCCESS + + +def _run_doctor_cmd(args, output_format: str) -> None: + """Top-level dispatcher for ``forgelm doctor``. + + Reads ``args.offline`` (argparse default ``False``) and emits either + the text report or the JSON envelope. Exits with the public + contract code resolved by :func:`_resolve_exit_code`. + """ + offline = bool(getattr(args, "offline", False)) + results = _run_all_checks(offline=offline) + if output_format == "json": + print(_render_json(results)) + else: + print(_render_text(results)) + sys.exit(_resolve_exit_code(results)) + + +# Re-exports for tests / monkeypatch — kept stable. +__all__ = [ + "_CheckResult", + "_check_python_version", + "_check_torch_cuda", + "_check_gpu_inventory", + "_check_optional_extra", + "_check_hf_hub_reachable", + "_check_hf_cache_offline", + "_check_disk_space", + "_check_operator_identity", + "_build_check_plan", + "_run_all_checks", + "_render_text", + "_render_json", + "_resolve_exit_code", + "_run_doctor_cmd", + "_OPTIONAL_EXTRAS", +] + + +def _maybe_unused() -> Optional[None]: # pragma: no cover — kept for typing import alignment + return None diff --git a/tests/test_doctor.py b/tests/test_doctor.py new file mode 100644 index 00000000..f1c5d264 --- /dev/null +++ b/tests/test_doctor.py @@ -0,0 +1,627 @@ +"""Phase 34: ``forgelm doctor`` env-check subcommand. + +Heavy on lightweight unit tests of individual probes (so a CI runner +without torch / GPU / network access can still exercise the doctor +surface), with one CLI subprocess smoke at the bottom. +""" + +from __future__ import annotations + +import json +import sys +from collections import namedtuple +from unittest.mock import MagicMock, patch + +import pytest + +# ``sys.version_info`` is a named tuple (`.major`/`.minor`/`.micro`/...); +# patching with a plain tuple breaks the attribute access in the probe. +# Build a structurally-equivalent fake we can pass to ``patch``. +_FakeVersionInfo = namedtuple("_FakeVersionInfo", ["major", "minor", "micro", "releaselevel", "serial"]) + + +def _make_version(major: int, minor: int, micro: int = 0) -> _FakeVersionInfo: + return _FakeVersionInfo(major, minor, micro, "final", 0) + + +# ``shutil.disk_usage`` returns a ``_ntuple_diskusage`` named tuple +# (`.total` / `.used` / `.free`). Mirror it for the disk-space tests. +_FakeDiskUsage = namedtuple("_FakeDiskUsage", ["total", "used", "free"]) + +# --------------------------------------------------------------------------- +# Individual probes +# --------------------------------------------------------------------------- + + +class TestPythonVersionCheck: + def test_python_310_warns(self) -> None: + from forgelm.cli.subcommands._doctor import _check_python_version + + with patch("sys.version_info", _make_version(3, 10, 0)): + result = _check_python_version() + assert result.status == "warn" + assert "3.11" in result.detail # recommendation appears + + def test_python_311_passes(self) -> None: + from forgelm.cli.subcommands._doctor import _check_python_version + + with patch("sys.version_info", _make_version(3, 11, 5)): + result = _check_python_version() + assert result.status == "pass" + assert "3.11.5" in result.detail + + def test_python_312_passes(self) -> None: + from forgelm.cli.subcommands._doctor import _check_python_version + + with patch("sys.version_info", _make_version(3, 12, 1)): + result = _check_python_version() + assert result.status == "pass" + + def test_python_39_fails(self) -> None: + from forgelm.cli.subcommands._doctor import _check_python_version + + with patch("sys.version_info", _make_version(3, 9, 7)): + result = _check_python_version() + assert result.status == "fail" + assert "3.10" in result.detail + assert "below" in result.detail.lower() + + +class TestTorchCudaCheck: + def test_no_torch_fails(self) -> None: + """When torch is not importable doctor must surface a clear fail + (not crash).""" + import builtins + + from forgelm.cli.subcommands import _doctor + + original_import = builtins.__import__ + + def _block_torch(name, *args, **kwargs): + if name == "torch": + raise ImportError("No module named 'torch'") + return original_import(name, *args, **kwargs) + + with patch.object(builtins, "__import__", _block_torch): + result = _doctor._check_torch_cuda() + assert result.status == "fail" + assert "torch" in result.detail.lower() + + def test_torch_cpu_only_warns(self) -> None: + """CPU-only torch is supported but warned (training will be slow).""" + from forgelm.cli.subcommands._doctor import _check_torch_cuda + + fake_torch = MagicMock() + fake_torch.__version__ = "2.5.0" + fake_torch.cuda.is_available.return_value = False + fake_torch.version.cuda = None + with patch.dict("sys.modules", {"torch": fake_torch}): + result = _check_torch_cuda() + assert result.status == "warn" + assert "CPU-only" in result.detail or "cpu-only" in result.detail.lower() + + def test_torch_cuda_passes(self) -> None: + from forgelm.cli.subcommands._doctor import _check_torch_cuda + + fake_torch = MagicMock() + fake_torch.__version__ = "2.5.0" + fake_torch.cuda.is_available.return_value = True + fake_torch.version.cuda = "12.4" + with patch.dict("sys.modules", {"torch": fake_torch}): + result = _check_torch_cuda() + assert result.status == "pass" + assert "12.4" in result.detail + + +class TestGpuInventoryCheck: + def test_no_torch_fails(self) -> None: + import builtins + + from forgelm.cli.subcommands import _doctor + + original_import = builtins.__import__ + + def _block_torch(name, *args, **kwargs): + if name == "torch": + raise ImportError("nope") + return original_import(name, *args, **kwargs) + + with patch.object(builtins, "__import__", _block_torch): + result = _doctor._check_gpu_inventory() + assert result.status == "fail" + + def test_no_cuda_warns(self) -> None: + from forgelm.cli.subcommands._doctor import _check_gpu_inventory + + fake_torch = MagicMock() + fake_torch.cuda.is_available.return_value = False + with patch.dict("sys.modules", {"torch": fake_torch}): + result = _check_gpu_inventory() + assert result.status == "warn" + assert result.extras["device_count"] == 0 + + def test_two_gpus_pass_with_inventory(self) -> None: + from forgelm.cli.subcommands._doctor import _check_gpu_inventory + + fake_torch = MagicMock() + fake_torch.cuda.is_available.return_value = True + fake_torch.cuda.device_count.return_value = 2 + # 24 GiB and 80 GiB devices. + props_24 = MagicMock(name="A10", total_memory=24 * (1024**3)) + props_80 = MagicMock(name="A100", total_memory=80 * (1024**3)) + # MagicMock attribute trick: ``name`` is special, set explicitly. + props_24.name = "NVIDIA A10" + props_80.name = "NVIDIA A100" + fake_torch.cuda.get_device_properties.side_effect = [props_24, props_80] + with patch.dict("sys.modules", {"torch": fake_torch}): + result = _check_gpu_inventory() + assert result.status == "pass" + assert result.extras["device_count"] == 2 + assert len(result.extras["devices"]) == 2 + assert result.extras["devices"][0]["vram_gib"] == 24.0 + assert result.extras["devices"][1]["vram_gib"] == 80.0 + + +class TestOptionalExtraCheck: + def test_present_module_passes(self) -> None: + from forgelm.cli.subcommands._doctor import _check_optional_extra + + # ``json`` is always installed in stdlib — use it as the + # "definitely present" probe target. + result = _check_optional_extra("fakextra", "json", "stdlib JSON") + assert result.status == "pass" + assert "json" in result.detail + assert result.extras["installed"] is True + + def test_missing_module_warns_with_install_hint(self) -> None: + from forgelm.cli.subcommands._doctor import _check_optional_extra + + result = _check_optional_extra("ghost", "definitely_not_installed_xyz123", "fake purpose") + assert result.status == "warn" + assert "pip install 'forgelm[ghost]'" in result.detail + assert result.extras["installed"] is False + + +class TestHfHubReachableCheck: + def test_unreachable_warns_not_fails(self) -> None: + """A network outage must NOT flip the gate to fail; doctor exists + precisely to surface that fact.""" + import urllib.error + + from forgelm.cli.subcommands._doctor import _check_hf_hub_reachable + + with patch("urllib.request.urlopen", side_effect=urllib.error.URLError("DNS lookup failed")): + result = _check_hf_hub_reachable(timeout_seconds=0.1) + assert result.status == "warn" + assert result.extras["reachable"] is False + + def test_200_response_passes(self) -> None: + from forgelm.cli.subcommands._doctor import _check_hf_hub_reachable + + fake_response = MagicMock() + fake_response.status = 200 + fake_response.__enter__ = MagicMock(return_value=fake_response) + fake_response.__exit__ = MagicMock(return_value=None) + with patch("urllib.request.urlopen", return_value=fake_response): + result = _check_hf_hub_reachable(timeout_seconds=0.1) + assert result.status == "pass" + assert result.extras["status_code"] == 200 + + +class TestHfCacheOfflineCheck: + """Wave 2a Round-1 review (gemini bot): HF cache resolution honours + HF_HUB_CACHE > HF_HOME/hub > ~/.cache/huggingface/hub. Tests must + set up the *correct* cache dir layout (HF_HOME/hub subdirectory) + or use HF_HUB_CACHE directly.""" + + def test_missing_cache_warns(self, tmp_path, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_hf_cache_offline + + monkeypatch.setenv("HF_HUB_CACHE", str(tmp_path / "nonexistent_cache")) + monkeypatch.delenv("HF_HOME", raising=False) + result = _check_hf_cache_offline() + assert result.status == "warn" + assert result.extras["exists"] is False + + def test_populated_cache_passes_via_hf_hub_cache(self, tmp_path, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_hf_cache_offline + + cache_dir = tmp_path / "hub_cache" + cache_dir.mkdir() + (cache_dir / "model_blob").write_bytes(b"x" * 1024) + monkeypatch.setenv("HF_HUB_CACHE", str(cache_dir)) + monkeypatch.delenv("HF_HOME", raising=False) + result = _check_hf_cache_offline() + assert result.status == "pass" + assert result.extras["file_count"] == 1 + assert result.extras["size_gib"] >= 0 + + def test_populated_cache_passes_via_hf_home_hub_subdir(self, tmp_path, monkeypatch) -> None: + """gemini bot fix: HF_HOME → ``HF_HOME/hub`` (sub-directory).""" + from forgelm.cli.subcommands._doctor import _check_hf_cache_offline + + hf_home = tmp_path / "hf_home" + hub_dir = hf_home / "hub" + hub_dir.mkdir(parents=True) + (hub_dir / "model_blob").write_bytes(b"x" * 1024) + monkeypatch.delenv("HF_HUB_CACHE", raising=False) + monkeypatch.setenv("HF_HOME", str(hf_home)) + result = _check_hf_cache_offline() + assert result.status == "pass" + assert "hub" in result.extras["cache_dir"] + + def test_empty_cache_dir_warns(self, tmp_path, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_hf_cache_offline + + cache_dir = tmp_path / "hf_cache_empty" + cache_dir.mkdir() + monkeypatch.setenv("HF_HUB_CACHE", str(cache_dir)) + monkeypatch.delenv("HF_HOME", raising=False) + result = _check_hf_cache_offline() + assert result.status == "warn" + assert result.extras["file_count"] == 0 + + +class TestHfEndpointResolution: + """Wave 2a Round-1 (gemini bot): HF_ENDPOINT must be respected for + self-hosted mirrors / enterprise installs.""" + + def test_default_endpoint_when_unset(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _resolve_hf_endpoint + + monkeypatch.delenv("HF_ENDPOINT", raising=False) + assert _resolve_hf_endpoint() == "https://huggingface.co" + + def test_env_var_override(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _resolve_hf_endpoint + + monkeypatch.setenv("HF_ENDPOINT", "https://internal-mirror.example/") + assert _resolve_hf_endpoint() == "https://internal-mirror.example" + + +class TestOperatorIdentityAnonymousOptIn: + """Wave 2a Round-1 (qodo bot): respect FORGELM_ALLOW_ANONYMOUS_OPERATOR + like AuditLogger.__init__ does — no-username + opt-in => warn (not fail).""" + + def test_no_username_with_opt_in_warns(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_operator_identity + + monkeypatch.delenv("FORGELM_OPERATOR", raising=False) + monkeypatch.setenv("FORGELM_ALLOW_ANONYMOUS_OPERATOR", "1") + with patch("getpass.getuser", side_effect=OSError("no user")): + result = _check_operator_identity() + assert result.status == "warn" + assert "anonymous" in result.detail.lower() + assert result.extras["source"] == "anonymous_opt_in" + + def test_no_username_without_opt_in_fails(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_operator_identity + + monkeypatch.delenv("FORGELM_OPERATOR", raising=False) + monkeypatch.delenv("FORGELM_ALLOW_ANONYMOUS_OPERATOR", raising=False) + with patch("getpass.getuser", side_effect=OSError("no user")): + result = _check_operator_identity() + assert result.status == "fail" + + +class TestSecretEnvMasking: + """Wave 2a Round-1 (F-27-05): secret env-var values must not echo + into doctor output.""" + + def test_mask_helper_redacts_secret_names(self) -> None: + from forgelm.cli.subcommands._doctor import _mask_env_value_for_audit + + masked = _mask_env_value_for_audit("FORGELM_AUDIT_SECRET", "super-secret-key-32-chars-long-x") + assert "super-secret" not in masked + assert " None: + from forgelm.cli.subcommands._doctor import _mask_env_value_for_audit + + passthrough = _mask_env_value_for_audit("FORGELM_OPERATOR", "alice") + assert passthrough == "alice" + + +class TestDiskSpaceCheck: + def test_plenty_passes(self, tmp_path) -> None: + from forgelm.cli.subcommands._doctor import _check_disk_space + + result = _check_disk_space(str(tmp_path)) + # On a CI runner free space typically > 50 GiB; ensure at least + # one of the three valid statuses comes back. + assert result.status in ("pass", "warn", "fail") + assert result.extras["free_gib"] >= 0 + + def test_low_disk_fails(self, tmp_path) -> None: + from forgelm.cli.subcommands import _doctor + + # Build a fake disk_usage that reports 5 GiB free. + fake_usage = _FakeDiskUsage( + total=1000 * (1024**3), + used=950 * (1024**3), + free=5 * (1024**3), + ) + with patch("shutil.disk_usage", return_value=fake_usage): + result = _doctor._check_disk_space(str(tmp_path)) + assert result.status == "fail" + + def test_warn_threshold(self, tmp_path) -> None: + from forgelm.cli.subcommands import _doctor + + # 30 GiB free → warn (between 10 and 50). + fake_usage = _FakeDiskUsage( + total=1000 * (1024**3), + used=970 * (1024**3), + free=30 * (1024**3), + ) + with patch("shutil.disk_usage", return_value=fake_usage): + result = _doctor._check_disk_space(str(tmp_path)) + assert result.status == "warn" + + +class TestOperatorIdentityCheck: + def test_explicit_env_passes(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_operator_identity + + monkeypatch.setenv("FORGELM_OPERATOR", "ci-pipeline-prod") + result = _check_operator_identity() + assert result.status == "pass" + assert "ci-pipeline-prod" in result.detail + + def test_missing_env_warns_with_fallback(self, monkeypatch) -> None: + from forgelm.cli.subcommands._doctor import _check_operator_identity + + monkeypatch.delenv("FORGELM_OPERATOR", raising=False) + result = _check_operator_identity() + # On a normal dev workstation getpass.getuser() resolves so we + # get warn (not fail). + assert result.status in ("warn", "fail") + assert "FORGELM_OPERATOR" in result.detail + + +# --------------------------------------------------------------------------- +# Renderers + exit-code mapping +# --------------------------------------------------------------------------- + + +def _make_results(*statuses: str): + from forgelm.cli.subcommands._doctor import _CheckResult + + return [_CheckResult(name=f"test.{i}", status=s, detail=f"d{i}") for i, s in enumerate(statuses)] + + +class TestExitCodeMapping: + def test_all_pass_returns_zero(self) -> None: + from forgelm.cli.subcommands._doctor import _resolve_exit_code + + assert _resolve_exit_code(_make_results("pass", "pass", "pass")) == 0 + + def test_warn_only_returns_zero(self) -> None: + from forgelm.cli.subcommands._doctor import _resolve_exit_code + + # Warns are operator-actionable but do not flip the gate. + assert _resolve_exit_code(_make_results("pass", "warn", "warn")) == 0 + + def test_fail_returns_one(self) -> None: + from forgelm.cli.subcommands._doctor import _resolve_exit_code + + assert _resolve_exit_code(_make_results("pass", "fail", "warn")) == 1 + + def test_crashed_probe_returns_two(self) -> None: + from forgelm.cli.subcommands._doctor import _CheckResult, _resolve_exit_code + + crashed = _CheckResult( + name="crash.probe", + status="fail", + detail="boom", + extras={"crashed": True, "error_class": "RuntimeError"}, + ) + assert _resolve_exit_code([crashed]) == 2 + + +class TestRenderers: + def test_text_renders_summary_line(self) -> None: + from forgelm.cli.subcommands._doctor import _render_text + + results = _make_results("pass", "warn", "fail") + out = _render_text(results) + assert "1 pass" in out + assert "1 warn" in out + assert "1 fail" in out + # Each check is rendered. + assert "test.0" in out and "test.1" in out and "test.2" in out + + def test_json_envelope_shape(self) -> None: + from forgelm.cli.subcommands._doctor import _render_json + + results = _make_results("pass", "fail") + payload = json.loads(_render_json(results)) + assert payload["success"] is False # has a fail + assert payload["summary"] == {"pass": 1, "warn": 0, "fail": 1} + assert len(payload["checks"]) == 2 + + def test_json_success_true_when_only_passes_and_warns(self) -> None: + from forgelm.cli.subcommands._doctor import _render_json + + results = _make_results("pass", "warn", "pass") + payload = json.loads(_render_json(results)) + assert payload["success"] is True + + +# --------------------------------------------------------------------------- +# Crash isolation +# --------------------------------------------------------------------------- + + +class TestProbeCrashIsolation: + def test_one_crashing_probe_does_not_abort_the_run(self, monkeypatch) -> None: + """If one probe raises an unexpected exception, the rest must + still execute and the failed probe must be converted to a fail + result with a ``crashed`` marker.""" + from forgelm.cli.subcommands import _doctor + + def _boom() -> _doctor._CheckResult: + raise RuntimeError("synthetic crash") + + # Replace the doctor's check plan with one passing probe + one + # crashing probe so we can observe the isolation. + def _fake_plan(*, offline: bool): + return [ + ("ok.probe", lambda: _doctor._CheckResult(name="ok.probe", status="pass", detail="ok")), + ("crash.probe", _boom), + ] + + monkeypatch.setattr(_doctor, "_build_check_plan", _fake_plan) + results = _doctor._run_all_checks(offline=False) + assert len(results) == 2 + ok_result = next(r for r in results if r.name == "ok.probe") + crash_result = next(r for r in results if r.name == "crash.probe") + assert ok_result.status == "pass" + assert crash_result.status == "fail" + assert crash_result.extras.get("crashed") is True + assert "RuntimeError" in crash_result.detail + + +# --------------------------------------------------------------------------- +# Plan composition +# --------------------------------------------------------------------------- + + +class TestCheckPlan: + def test_offline_uses_cache_probe_not_hub_probe(self) -> None: + from forgelm.cli.subcommands._doctor import _build_check_plan + + plan_offline = _build_check_plan(offline=True) + plan_online = _build_check_plan(offline=False) + names_offline = [name for name, _ in plan_offline] + names_online = [name for name, _ in plan_online] + assert "hf_hub.offline_cache" in names_offline + assert "hf_hub.reachable" not in names_offline + assert "hf_hub.reachable" in names_online + assert "hf_hub.offline_cache" not in names_online + + def test_extras_in_plan(self) -> None: + """All optional extras advertised in pyproject.toml are probed.""" + from forgelm.cli.subcommands._doctor import _OPTIONAL_EXTRAS, _build_check_plan + + plan = _build_check_plan(offline=False) + names = {name for name, _ in plan} + for extra, _module, _purpose in _OPTIONAL_EXTRAS: + assert f"extras.{extra}" in names + + +# --------------------------------------------------------------------------- +# Top-level dispatcher +# --------------------------------------------------------------------------- + + +class TestDispatcher: + def test_text_output_prints_summary(self, capsys) -> None: + from forgelm.cli.subcommands._doctor import _run_doctor_cmd + + args = MagicMock() + args.offline = True # avoids the network probe + with pytest.raises(SystemExit): + _run_doctor_cmd(args, output_format="text") + out = capsys.readouterr().out + assert "Summary:" in out + assert "forgelm doctor" in out + + def test_json_output_emits_envelope(self, capsys) -> None: + from forgelm.cli.subcommands._doctor import _run_doctor_cmd + + args = MagicMock() + args.offline = True + with pytest.raises(SystemExit): + _run_doctor_cmd(args, output_format="json") + payload = json.loads(capsys.readouterr().out) + assert "success" in payload + assert "checks" in payload + assert "summary" in payload + assert all(set(c) >= {"name", "status", "detail", "extras"} for c in payload["checks"]) + + def test_dispatcher_exits_with_resolved_code(self, capsys) -> None: + from forgelm.cli.subcommands import _doctor + + # Force every probe to pass so exit code is 0. + def _all_pass_plan(*, offline: bool): + return [("only.probe", lambda: _doctor._CheckResult(name="only.probe", status="pass", detail="ok"))] + + with patch.object(_doctor, "_build_check_plan", _all_pass_plan): + args = MagicMock() + args.offline = False + with pytest.raises(SystemExit) as ei: + _doctor._run_doctor_cmd(args, output_format="text") + assert ei.value.code == 0 + + +# --------------------------------------------------------------------------- +# CLI subprocess smoke +# --------------------------------------------------------------------------- + + +class TestDoctorCLISmoke: + def test_doctor_subcommand_registered(self) -> None: + """`forgelm doctor --help` exits 0 and advertises --offline.""" + import subprocess + + result = subprocess.run( + [sys.executable, "-m", "forgelm.cli", "doctor", "--help"], + capture_output=True, + text=True, + timeout=30, + ) + assert result.returncode == 0, result.stderr + assert "--offline" in result.stdout + # Common-flag inheritance: --output-format / --quiet / --log-level. + assert "--output-format" in result.stdout + + def test_doctor_offline_runs_end_to_end(self) -> None: + """`forgelm doctor --offline --output-format json` produces a valid + JSON envelope and exits with one of the public exit codes.""" + import subprocess + + result = subprocess.run( + [ + sys.executable, + "-m", + "forgelm.cli", + "doctor", + "--offline", + "--output-format", + "json", + ], + capture_output=True, + text=True, + timeout=120, + ) + # Public exit codes: 0 (all pass) / 1 (some fail) / 2 (probe crashed). + assert result.returncode in (0, 1, 2), result.stderr + # The JSON envelope is on stdout regardless. + payload = json.loads(result.stdout) + assert "success" in payload + assert "checks" in payload + assert isinstance(payload["checks"], list) + + +# --------------------------------------------------------------------------- +# Facade re-exports +# --------------------------------------------------------------------------- + + +class TestFacadeReExports: + def test_doctor_helpers_reachable_via_facade(self) -> None: + """Tests / monkeypatches reach doctor helpers via ``forgelm.cli``.""" + from forgelm import cli as _cli_facade + + for name in ( + "_run_doctor_cmd", + "_run_all_checks", + "_render_json", + "_render_text", + "_resolve_exit_code", + "_check_python_version", + "_check_torch_cuda", + "_check_operator_identity", + ): + assert hasattr(_cli_facade, name), f"forgelm.cli must re-export {name!r}"