Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,41 @@ All notable changes to ForgeLM are documented here.
> Per-PR CHANGELOG entries below collapse into the v0.5.5 release
> notes at tag time.

### Added — Wave 2a / Phase 34 — `forgelm doctor` env-check subcommand

- **`forgelm doctor`** — the first command an operator should run after
installation. Probes Python version, torch + CUDA, GPU inventory,
every optional extra advertised in `pyproject.toml`, HuggingFace Hub
reachability, workspace disk space, and the `FORGELM_OPERATOR`
audit-identity hint. Tabular text report or a structured JSON
envelope (`--output-format json`).
- **`forgelm doctor --offline`** skips the HF Hub network probe and
inspects the local cache (`HF_HOME` / `~/.cache/huggingface/hub`).
Useful for air-gap deployments.
- **Honest pass / warn / fail:** warn = operator-actionable but does
not block (missing optional extra, CPU-only torch); fail = ForgeLM
cannot work this way (Python <3.10, no torch). Exit codes follow
the public contract — 0 every check passed (warns OK), 1 at least
one fail (config-error class), 2 if a probe itself crashed
(runtime-error class).
- **Self-contained probes.** Heavy deps (torch, huggingface_hub) are
imported lazily inside individual check functions so `forgelm doctor`
can run on a brand-new machine where torch is not yet installed
without crashing. One crashing probe does not abort the rest of
the report.
- Closes the `ghost-features-analysis-20260502` GH-001 onboarding
bloker (30 doc references across 8 files in `docs/usermanuals/` no
longer point at a non-existent command).
- New module `forgelm/cli/subcommands/_doctor.py` (~430 lines).
- 38 new tests in `tests/test_doctor.py` covering per-probe behaviour
(Python 3.9/3.10/3.11/3.12, torch presence + CUDA, GPU inventory,
optional-extra detection, HF Hub HEAD, HF cache populated/empty,
disk-space thresholds, operator identity), exit-code mapping,
text/JSON renderers, probe-crash isolation, plan composition,
CLI subprocess smoke, facade re-exports.
- Operator docs updated: `first-run.md` (en + tr) sample output now
matches the shipped table format.

### Added — Wave 1 closure (Faz 9, 11, 12, 13, 25, 31, 32 — see PR description)

- **Article 14 staging directory + `forgelm approve` / `forgelm reject` (Faz 9)** —
Expand Down
20 changes: 15 additions & 5 deletions docs/usermanuals/en/getting-started/first-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,23 @@ Before anything else, run `forgelm doctor` to check that Python, PyTorch, CUDA,

```shell
$ forgelm doctor
ForgeLM 0.5.2
Python 3.11.4 · PyTorch 2.4.0
CUDA available · driver 535.171 · NVIDIA RTX 4090 · 24 GB
bitsandbytes ✓ (installed)
Unsloth ✓ (installed)
forgelm doctor — environment check

[✓ pass] python.version Python 3.11.4 (CPython).
[✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4.
[✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB).
[✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training).
[✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)).
[! warn] extras.deepspeed Optional extra missing — install with: pip install 'forgelm[deepspeed]' (purpose: DeepSpeed ZeRO + offload distributed training).
[✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200).
[✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB.
[! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR=<id> for CI / pipeline runs.

Summary: 7 pass, 2 warn, 0 fail.
```

`--output-format json` returns a structured envelope (`{"success": bool, "checks": [...], "summary": {...}}`) so CI can filter on individual probe results without parsing the table. Pass `--offline` to skip the HF Hub network probe and inspect the local cache instead — useful for air-gapped deployments.

:::tip
If `forgelm doctor` reports a problem (missing CUDA, version mismatch, no GPU), fix that first. Every other ForgeLM command will fail in confusing ways otherwise. See [Troubleshooting](#/operations/troubleshooting).
:::
Expand Down
16 changes: 10 additions & 6 deletions docs/usermanuals/en/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,16 @@ If you've installed ForgeLM for GPU training, confirm CUDA is wired up correctly
$ forgelm doctor
```

`forgelm doctor` reports:
- Python and PyTorch versions
- CUDA availability and driver version
- Detected GPU model and VRAM
- Available compute capability
- bitsandbytes / Unsloth detection (if installed)
`forgelm doctor` reports a tabular pass / warn / fail diagnostic over:
- Python version (>=3.10 floor; >=3.11 recommended)
- torch + CUDA availability (CPU-only is a `warn`, not a `fail`)
- GPU inventory (per-device VRAM in GiB)
- Optional extras: `qlora`, `unsloth`, `distributed`, `eval`, `tracking`, `merging`, `export`, `ingestion`, `ingestion-pii-ml`, `ingestion-scale` — missing extras `warn` with the exact `pip install 'forgelm[<name>]'` hint
- HuggingFace Hub reachability (via `HF_ENDPOINT` if set; skipped under `--offline`)
- Workspace disk space (<10 GiB → `fail`, <50 GiB → `warn`)
- `FORGELM_OPERATOR` audit-identity hint (Article 12)

Pass `--output-format json` for a structured envelope (`{"success": bool, "checks": [...], "summary": {pass, warn, fail}}`); pass `--offline` for air-gap mode (skips the network probe and instead inspects the local HF cache).

:::tip
Run `forgelm doctor` *before* `forgelm --config ...` for any new environment. It catches missing CUDA libraries, version mismatches, and GPU-not-found errors in two seconds rather than two hours into training.
Expand Down
21 changes: 14 additions & 7 deletions docs/usermanuals/en/operations/air-gap.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,22 @@ evaluation:

```shell
$ forgelm doctor --offline
ForgeLM 0.5.2 (offline mode verified)
✓ HF_HUB_OFFLINE set
✓ TRANSFORMERS_OFFLINE set
✓ Local cache: airgap-bundle/huggingface (3.4 GB cached models)
✓ Llama Guard 3 8B available locally
✓ lm-evaluation-harness: 4 tasks cached
forgelm doctor — environment check

[✓ pass] python.version Python 3.11.4 (CPython).
[✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4.
[✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA A100 (80.0 GiB).
[✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training).
[✓ pass] extras.eval Installed (module lm_eval, purpose: lm-evaluation-harness benchmark scoring).
[! warn] extras.tracking Optional extra missing — install with: pip install 'forgelm[tracking]' (purpose: Weights & Biases experiment tracking).
[✓ pass] hf_hub.offline_cache HF cache at /opt/airgap/huggingface/hub: 3.4 GiB across 47 file(s). HF_HUB_OFFLINE=1.
[✓ pass] disk.workspace Workspace /opt/airgap — 412.0 GiB free of 500.0 GiB.
[! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'airgap-op@nodeA'. Pin FORGELM_OPERATOR=<id> for CI / pipeline runs.

Summary: 7 pass, 2 warn, 0 fail.
```

If `forgelm doctor --offline` reports anything unavailable, fix that *before* the air-gapped operator wastes their time.
If `forgelm doctor --offline` reports anything as `fail`, fix that *before* the air-gapped operator wastes their time. The HF cache scan honours `HF_HUB_CACHE` first, then `HF_HOME/hub`, then the default `~/.cache/huggingface/hub` — point `HF_HUB_CACHE` at your bundled cache to make the scan deterministic.

## Bundle size estimate

Expand Down
20 changes: 15 additions & 5 deletions docs/usermanuals/tr/getting-started/first-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,23 @@ Her şeyden önce Python, PyTorch, CUDA ve opsiyonel bağımlılıkların doğru

```shell
$ forgelm doctor
ForgeLM 0.5.2
Python 3.11.4 · PyTorch 2.4.0
CUDA available · driver 535.171 · NVIDIA RTX 4090 · 24 GB
bitsandbytes ✓ (installed)
Unsloth ✓ (installed)
forgelm doctor — environment check

[✓ pass] python.version Python 3.11.4 (CPython).
[✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4.
[✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB).
[✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training).
[✓ pass] extras.unsloth Installed (module unsloth, purpose: Unsloth-accelerated training (Linux GPUs only)).
[! warn] extras.deepspeed Optional extra missing — install with: pip install 'forgelm[deepspeed]' (purpose: DeepSpeed ZeRO + offload distributed training).
[✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200).
[✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB.
[! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'. Pin FORGELM_OPERATOR=<id> for CI / pipeline runs.

Summary: 7 pass, 2 warn, 0 fail.
```

`--output-format json` yapısal bir zarf döner (`{"success": bool, "checks": [...], "summary": {...}}`); CI ayrı ayrı probe sonuçlarını tabloyu parse etmeden filtreleyebilir. `--offline` HF Hub ağ probe'unu atlar ve onun yerine yerel cache'i inceler — air-gap dağıtımları için faydalı.

:::tip
`forgelm doctor` bir sorun raporlarsa (eksik CUDA, sürüm uyumsuzluğu, GPU yok), önce onu düzeltin. Diğer her ForgeLM komutu kafa karıştırıcı şekillerde patlar. Bkz. [Sorun Giderme](#/operations/troubleshooting).
:::
Expand Down
16 changes: 10 additions & 6 deletions docs/usermanuals/tr/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,16 @@ GPU eğitimi için kurduysanız CUDA'nın doğru bağlandığını teyit edin:
$ forgelm doctor
```

`forgelm doctor` şunları raporlar:
- Python ve PyTorch sürümleri
- CUDA varlığı ve sürücü sürümü
- Algılanan GPU modeli ve VRAM
- Mevcut compute capability
- bitsandbytes / Unsloth tespiti (kuruluysa)
`forgelm doctor` tabular bir pass / warn / fail teşhis raporu üretir:
- Python sürümü (>=3.10 zorunlu, >=3.11 önerilen)
- torch + CUDA varlığı (CPU-only `warn`'dur, `fail` değil)
- GPU envanteri (cihaz başına VRAM, GiB)
- Opsiyonel extras: `qlora`, `unsloth`, `distributed`, `eval`, `tracking`, `merging`, `export`, `ingestion`, `ingestion-pii-ml`, `ingestion-scale` — eksik extra'lar tam `pip install 'forgelm[<isim>]'` ipucu ile `warn`'lanır
- HuggingFace Hub erişimi (varsa `HF_ENDPOINT` üzerinden; `--offline` ile atlanır)
- Workspace disk alanı (<10 GiB → `fail`, <50 GiB → `warn`)
- `FORGELM_OPERATOR` audit kimliği ipucu (Madde 12)

`--output-format json` yapısal zarf döner (`{"success": bool, "checks": [...], "summary": {pass, warn, fail}}`); `--offline` air-gap modu (ağ probe'unu atlar, yerel HF cache'i inceler).

:::tip
Yeni bir ortamda `forgelm --config ...` koşmadan *önce* `forgelm doctor` çalıştırın. Eksik CUDA kütüphanelerini, sürüm uyumsuzluklarını ve "GPU bulunamadı" hatalarını eğitime saatler kala değil saniyeler içinde yakalar.
Expand Down
21 changes: 14 additions & 7 deletions docs/usermanuals/tr/operations/air-gap.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,22 @@ evaluation:

```shell
$ forgelm doctor --offline
ForgeLM 0.5.2 (offline mod doğrulandı)
✓ HF_HUB_OFFLINE ayarlı
✓ TRANSFORMERS_OFFLINE ayarlı
✓ Yerel cache: airgap-bundle/huggingface (3.4 GB cache'li model)
✓ Llama Guard 3 8B yerel olarak mevcut
✓ lm-evaluation-harness: 4 görev cache'li
forgelm doctor — environment check

[✓ pass] python.version Python 3.11.4 (CPython).
[✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4.
[✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA A100 (80.0 GiB).
[✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training).
[✓ pass] extras.eval Installed (module lm_eval, purpose: lm-evaluation-harness benchmark scoring).
[! warn] extras.tracking Optional extra missing — install with: pip install 'forgelm[tracking]' (purpose: Weights & Biases experiment tracking).
[✓ pass] hf_hub.offline_cache HF cache at /opt/airgap/huggingface/hub: 3.4 GiB across 47 file(s). HF_HUB_OFFLINE=1.
[✓ pass] disk.workspace Workspace /opt/airgap — 412.0 GiB free of 500.0 GiB.
[! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'airgap-op@nodeA'. Pin FORGELM_OPERATOR=<id> for CI / pipeline runs.

Summary: 7 pass, 2 warn, 0 fail.
```

`forgelm doctor --offline` herhangi bir şeyin erişilemediğini raporlarsa, air-gap operatörünün vaktini boşa harcamadan *önce* düzeltin.
`forgelm doctor --offline` herhangi bir şeyi `fail` olarak raporlarsa, air-gap operatörünün vaktini boşa harcamadan *önce* düzeltin. HF cache taraması önce `HF_HUB_CACHE`'i, sonra `HF_HOME/hub`'ı, en son varsayılan `~/.cache/huggingface/hub`'ı dener — taramayı deterministik yapmak için `HF_HUB_CACHE`'i pakete dahil ettiğiniz cache'e işaret ettirin.

## Paket boyutu tahmini

Expand Down
18 changes: 18 additions & 0 deletions forgelm/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
_add_audit_subcommand, # noqa: F401 — re-export for tests
_add_chat_subcommand, # noqa: F401 — re-export for tests
_add_deploy_subcommand, # noqa: F401 — re-export for tests
_add_doctor_subcommand, # noqa: F401 — re-export for tests
_add_export_subcommand, # noqa: F401 — re-export for tests
_add_ingest_subcommand, # noqa: F401 — re-export for tests
_add_quickstart_subcommand, # noqa: F401 — re-export for tests
Expand Down Expand Up @@ -134,6 +135,23 @@
# Chat / export / deploy / ingest subcommand dispatchers.
from .subcommands._chat import _run_chat_cmd # noqa: F401 — re-export for tests
from .subcommands._deploy import _run_deploy_cmd # noqa: F401 — re-export for tests

# Doctor subcommand (Phase 34 environment diagnostics).
from .subcommands._doctor import (

Check warning on line 140 in forgelm/cli/__init__.py

View check run for this annotation

Codacy Production / Codacy Static Code Analysis

forgelm/cli/__init__.py#L140

'.subcommands._doctor._check_disk_space' imported but unused (F401)
_check_disk_space, # noqa: F401 — re-export for tests
_check_gpu_inventory, # noqa: F401 — re-export for tests
_check_hf_cache_offline, # noqa: F401 — re-export for tests
_check_hf_hub_reachable, # noqa: F401 — re-export for tests
_check_operator_identity, # noqa: F401 — re-export for tests
_check_optional_extra, # noqa: F401 — re-export for tests
_check_python_version, # noqa: F401 — re-export for tests
_check_torch_cuda, # noqa: F401 — re-export for tests
_render_json, # noqa: F401 — re-export for tests
_render_text, # noqa: F401 — re-export for tests
_resolve_exit_code, # noqa: F401 — re-export for tests
_run_all_checks, # noqa: F401 — re-export for tests
_run_doctor_cmd, # noqa: F401 — re-export for tests
)
from .subcommands._export import _run_export_cmd # noqa: F401 — re-export for tests
from .subcommands._ingest import _run_ingest_cmd # noqa: F401 — re-export for tests

Expand Down
8 changes: 8 additions & 0 deletions forgelm/cli/_dispatch.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ def _dispatch_subcommand(command: str, args) -> None:
elif command == "audit":
_cli_facade._run_audit_cmd(args, getattr(args, "output_format", "text"))
sys.exit(EXIT_SUCCESS)
elif command == "doctor":
# _run_doctor_cmd does its own sys.exit with a result-mapped exit
# code (0/1/2) so we don't fall through to a blanket EXIT_SUCCESS.
_cli_facade._run_doctor_cmd(args, getattr(args, "output_format", "text"))
# Defensive: _run_doctor_cmd always exits. This line is unreachable
# in practice but keeps the dispatch table exhaustive against a
# future doctor refactor that might forget the sys.exit.
sys.exit(EXIT_SUCCESS)
elif command == "verify-audit":
sys.exit(_cli_facade._run_verify_audit_cmd(args))
elif command == "approve":
Expand Down
35 changes: 35 additions & 0 deletions forgelm/cli/_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,40 @@ def _add_ingest_subcommand(subparsers) -> None:
_add_common_subparser_flags(p, include_output_format=True)


def _add_doctor_subcommand(subparsers) -> None:
"""Phase 34: environment check (`forgelm doctor`).

The first command an operator should run after installation. Probes
Python version, torch / CUDA, GPU inventory, optional extras, HF Hub
reachability (or local cache when ``--offline``), workspace disk
space, and the FORGELM_OPERATOR audit-identity hint.

Exit codes follow the public contract: 0 = all pass, 1 = at least
one check failed (config-error class), 2 = probe crashed.
"""
p = subparsers.add_parser(
"doctor",
help="Run environment + dependency diagnostics (the first command after install).",
description=(
"Probe Python, torch + CUDA, GPU inventory, optional ForgeLM extras, "
"HuggingFace Hub reachability, workspace disk space, and the "
"FORGELM_OPERATOR audit-identity hint. Emits a tabular text report or "
"a structured JSON envelope (`--output-format json`). Pass `--offline` "
"to skip the HF Hub network probe and instead inspect the local cache."
),
)
p.add_argument(
"--offline",
action="store_true",
help=(
"Skip the HuggingFace Hub network probe. Inspects the local HF cache "
"(HF_HOME or ~/.cache/huggingface/hub) instead — useful for air-gapped "
"deployments where the network probe would always fail."
),
)
_add_common_subparser_flags(p, include_output_format=True)


def _add_audit_subcommand(subparsers) -> None:
p = subparsers.add_parser(
"audit",
Expand Down Expand Up @@ -560,6 +594,7 @@ def parse_args():
_add_quickstart_subcommand(subparsers)
_add_ingest_subcommand(subparsers)
_add_audit_subcommand(subparsers)
_add_doctor_subcommand(subparsers)
_add_verify_audit_subcommand(subparsers)
_add_approve_subcommand(subparsers)
_add_reject_subcommand(subparsers)
Expand Down
Loading