Skip to content

Phase 34: forgelm doctor env-check subcommand - #27

Closed
cemililik wants to merge 2 commits into
developmentfrom
closure/faz34-doctor-subcommand
Closed

cemililik wants to merge 2 commits into
developmentfrom
closure/faz34-doctor-subcommand

Conversation

@cemililik

Copy link
Copy Markdown
Collaborator

Summary

Closes the GH-1 onboarding bloker from `ghost-features-analysis-20260502`: 30 doc references across 8 user-manual files (installation, first-run, troubleshooting, air-gap, CLI reference) used to point at a non-existent command. With this PR they now point at a real working subcommand.

`forgelm doctor` is the first command an operator should run after installation. Probes Python, torch, CUDA, GPU inventory, every optional extra advertised in pyproject.toml, HF Hub reachability (or the local cache when `--offline`), workspace disk space, and the FORGELM_OPERATOR audit-identity hint.

```shell
$ forgelm doctor
forgelm doctor — environment check

[✓ pass] python.version Python 3.11.4 (CPython).
[✓ pass] torch.cuda torch 2.4.0 with CUDA 12.4.
[✓ pass] gpu.inventory 1 GPU(s) — GPU0: NVIDIA RTX 4090 (24.0 GiB).
[✓ pass] extras.qlora Installed (module bitsandbytes, purpose: 4-bit / 8-bit QLoRA training).
[! warn] extras.deepspeed Optional extra missing — install with: pip install 'forgelm[deepspeed]' (purpose: DeepSpeed ZeRO + offload distributed training).
[✓ pass] hf_hub.reachable HuggingFace Hub reachable (HTTP 200).
[✓ pass] disk.workspace Workspace /home/me/forgelm — 387.0 GiB free of 500.0 GiB.
[! warn] operator.identity FORGELM_OPERATOR not set; audit events will fall back to 'me@workstation'.

Summary: 7 pass, 2 warn, 0 fail.
```

`--output-format json` returns a structured envelope; `--offline` skips the network probe and inspects the local HF cache instead.

Status semantics

Status Meaning Exit code impact
`pass` Invariant holds none
`warn` Operator-actionable but doesn't block (missing optional extra, CPU-only torch, FORGELM_OPERATOR unset) none
`fail` ForgeLM cannot work this way (Python <3.10, no torch) flips exit code to 1

Exit codes follow the public contract: 0 (all pass / warns), 1 (any fail), 2 (a probe itself crashed — runtime-error class).

Files

File Change
`forgelm/cli/subcommands/_doctor.py` (new, ~430 lines) Probes, renderers, exit-code mapper, top-level dispatcher. Lazy heavy-dep imports inside check functions so `forgelm doctor` runs without torch installed
`forgelm/cli/_parser.py` `_add_doctor_subcommand` registrar with `--offline` + `--output-format`
`forgelm/cli/_dispatch.py` `doctor` branch
`forgelm/cli/init.py` Facade re-exports for tests / monkeypatch
`tests/test_doctor.py` (new, ~430 lines) 38 tests across 11 test classes
`docs/usermanuals/{en,tr}/getting-started/first-run.md` Sample output matches shipped table format
`CHANGELOG.md` Phase 34 entry

Test plan

  • `ruff format . && ruff check .` clean
  • `pytest tests/ -q` → 1051 passed, 13 skipped (1013 → 1051, +38 new)
  • Coverage 68.59% (67.94 → 68.59, +0.65)
  • `forgelm doctor --help` exits 0 and advertises `--offline` + `--output-format`
  • `forgelm doctor --offline --output-format json` end-to-end smoke (subprocess test)
  • One crashing probe does not abort the rest of the report (regression test)

Risk

Low. Strictly additive new subcommand with no shared mutable state; existing subcommands untouched. Heavy deps imported lazily inside check functions so `forgelm doctor` is safe to run on a brand-new machine where torch is not yet installed.

Closes

🤖 Generated with Claude Code

Closes ghost-features-analysis-20260502 GH-1 (onboarding bloker —
30 doc references across 8 user-manual files no longer point at a
non-existent command).

The first command an operator should run after installation. Probes:

  python.version          Python is in the supported window
  torch.cuda              torch + CUDA availability
  gpu.inventory           visible GPUs + per-device VRAM
  extras.<name>           every optional extra (qlora, unsloth,
                          deepspeed, evaluation, wandb, mergekit,
                          ingestion, ingestion-pii-ml,
                          ingestion-scale)
  hf_hub.reachable        HF Hub HEAD probe (skipped under --offline)
  hf_hub.offline_cache    local HF cache size + populated state
                          (--offline-only replacement for the Hub probe)
  disk.workspace          free space on the workspace filesystem
                          (<10 GiB fail, <50 GiB warn, otherwise pass)
  operator.identity       FORGELM_OPERATOR audit-identity hint

Three statuses:

- pass: invariant holds, nothing to do
- warn: operator-actionable but doesn't block (missing optional
  extra, CPU-only torch, FORGELM_OPERATOR unset)
- fail: ForgeLM cannot work this way (Python <3.10, no torch,
  unresolvable operator identity)

Exit codes follow the public contract:

- 0 (EXIT_SUCCESS): every check passed (warns are OK)
- 1 (EXIT_CONFIG_ERROR): at least one fail
- 2 (EXIT_TRAINING_ERROR): a probe itself crashed (operator-actionable
  bug, surfaced as runtime-error class so CI/CD retry logic treats it
  the same way)

Implementation:

- forgelm/cli/subcommands/_doctor.py (new, ~430 lines): probes,
  renderers, exit-code mapper, top-level dispatcher.  Heavy deps
  (torch, huggingface_hub) are imported lazily inside individual
  check functions so `forgelm doctor` runs on a brand-new machine
  where torch is not yet installed without crashing. One probe
  crashing does not abort the rest of the report.
- forgelm/cli/_parser.py: _add_doctor_subcommand registrar with
  --offline + --output-format flags.
- forgelm/cli/_dispatch.py: doctor dispatch branch.
- forgelm/cli/__init__.py: facade re-exports for tests / monkeypatch.

Tests (tests/test_doctor.py — 38 cases, all torch-free):

  TestPythonVersionCheck (4)
    Python 3.10 → warn, 3.11 → pass, 3.12 → pass, 3.9 → fail
  TestTorchCudaCheck (3)
    no torch → fail; CPU-only torch → warn; torch + CUDA → pass
  TestGpuInventoryCheck (3)
    no torch → fail; no CUDA → warn (count=0); 2 GPUs → pass
    (with VRAM enumeration validated)
  TestOptionalExtraCheck (2)
    present module passes; missing module warns with install hint
  TestHfHubReachableCheck (2)
    network outage warns (not fails); HTTP 200 passes
  TestHfCacheOfflineCheck (3)
    missing cache → warn; populated cache → pass; empty cache → warn
  TestDiskSpaceCheck (3)
    free space query smoke; <10 GiB → fail; 30 GiB → warn
  TestOperatorIdentityCheck (2)
    FORGELM_OPERATOR set → pass; unset (with getuser fallback) → warn
  TestExitCodeMapping (4)
    all-pass → 0; warn-only → 0; any-fail → 1; crashed-probe → 2
  TestRenderers (3)
    text summary line; JSON envelope shape; success=True with no fails
  TestProbeCrashIsolation (1)
    one crashing probe does not abort the run; converted to fail with
    crashed=True marker
  TestCheckPlan (2)
    --offline uses cache probe (not Hub probe); all extras in plan
  TestDispatcher (3)
    text output prints summary; JSON envelope structure; exit code
    resolved from result list
  TestDoctorCLISmoke (2)
    `forgelm doctor --help` advertises --offline + --output-format;
    `forgelm doctor --offline --output-format json` end-to-end smoke
  TestFacadeReExports (1)
    every test-touched helper reachable via forgelm.cli

Documentation:

- docs/usermanuals/en/getting-started/first-run.md sample output
  updated to match the shipped table format.
- TR mirror first-run.md updated in the same shape.
- CHANGELOG.md [Unreleased] Phase 34 entry.

Existing user-manual references (30 across 8 files in installation,
first-run, troubleshooting, air-gap, cli reference) were already in
place; they now point at a real working command instead of a ghost
feature.  The closure plan §10 Faz 30 Task A still has the larger
doc sweep on its list; this PR ships the minimum doc updates that
keep the user-manuals correct in the meantime.

Test count: 1013 -> 1051 (+38). Coverage 67.94% -> 68.59%.
ruff format/check clean.

Closes: closure-plan-202604300906.md §9.5 Phase 34
        ghost-features-analysis-20260502.md GH-1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @cemililik, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented May 2, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 7aa1c449-1f6c-4503-a453-e6732f90f470

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch closure/faz34-doctor-subcommand

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@qodo-code-review

Copy link
Copy Markdown

Review Summary by Qodo

Add forgelm doctor environment check subcommand (Phase 34)

✨ Enhancement

Grey Divider

Walkthroughs

Description
• Implement forgelm doctor subcommand for environment diagnostics
  - Probes Python version, torch/CUDA, GPU inventory, optional extras
  - Checks HF Hub reachability and workspace disk space
  - Validates FORGELM_OPERATOR audit-identity hint
• Support --offline mode for air-gapped deployments
  - Skips network probe, inspects local HF cache instead
• Provide text and JSON output formats with structured exit codes
  - 0 = all pass, 1 = config error, 2 = probe crash
• Add 38 comprehensive tests covering all probe scenarios
Diagram
flowchart LR
  A["CLI Parser"] -->|register| B["doctor subcommand"]
  B -->|dispatch| C["_run_doctor_cmd"]
  C -->|execute| D["Check Plan"]
  D -->|run probes| E["Python/Torch/GPU/Extras/Network/Disk/Identity"]
  E -->|collect| F["CheckResult List"]
  F -->|render| G["Text or JSON Output"]
  G -->|resolve| H["Exit Code 0/1/2"]
  H -->|exit| I["Process Exit"]
Loading

Grey Divider

File Changes

1. forgelm/cli/subcommands/_doctor.py ✨ Enhancement +631/-0

Core doctor subcommand implementation with all probes

• New 631-line module implementing all diagnostic probes and renderers
• Individual check functions for Python, torch, GPU, extras, HF Hub, disk, operator identity
• Lazy imports of heavy dependencies (torch, huggingface_hub) inside probes
• Text and JSON renderers with pass/warn/fail status vocabulary
• Exit code mapper implementing public contract (0/1/2)
• Crash isolation so one failing probe doesn't abort the report

forgelm/cli/subcommands/_doctor.py


2. forgelm/cli/_parser.py ✨ Enhancement +35/-0

Register doctor subcommand with argument parser

• Add _add_doctor_subcommand registrar function
• Register --offline flag to skip HF Hub network probe
• Inherit common flags including --output-format
• Register doctor subcommand in main parser subcommand router

forgelm/cli/_parser.py


3. forgelm/cli/_dispatch.py ✨ Enhancement +8/-0

Route doctor command to dispatcher

• Add dispatch branch for doctor command
• Call _run_doctor_cmd with output format parameter
• Handle custom exit code from doctor (0/1/2) instead of blanket EXIT_SUCCESS
• Include defensive comment about unreachable fallback exit

forgelm/cli/_dispatch.py


View more (5)
4. forgelm/cli/__init__.py ✨ Enhancement +18/-0

Export doctor subcommand facade for testing

• Re-export _add_doctor_subcommand from parser module
• Re-export 13 doctor helper functions for tests and monkeypatching
• Includes check functions, renderers, exit code resolver, and orchestrator

forgelm/cli/init.py


5. tests/test_doctor.py 🧪 Tests +546/-0

Comprehensive test suite for doctor subcommand

• New 546-line test module with 38 tests across 11 test classes
• Unit tests for each probe (Python version, torch/CUDA, GPU, extras, HF Hub, cache, disk, identity)
• Tests for exit code mapping (0/1/2 contract)
• Tests for text and JSON renderers
• Tests for probe crash isolation
• Tests for check plan composition (offline vs online)
• CLI subprocess smoke tests
• Facade re-export validation tests

tests/test_doctor.py


6. docs/usermanuals/en/getting-started/first-run.md 📝 Documentation +15/-5

Update English documentation with actual doctor output

• Update sample forgelm doctor output to match actual implementation
• Show tabular format with pass/warn/fail glyphs and detailed messages
• Document --output-format json and --offline flags
• Add explanation of output format and use cases

docs/usermanuals/en/getting-started/first-run.md


7. docs/usermanuals/tr/getting-started/first-run.md 📝 Documentation +15/-5

Update Turkish documentation with actual doctor output

• Update Turkish translation of sample forgelm doctor output
• Mirror English version with tabular format and glyph indicators
• Document JSON and offline modes in Turkish

docs/usermanuals/tr/getting-started/first-run.md


8. CHANGELOG.md 📝 Documentation +35/-0

Document Phase 34 doctor subcommand in changelog

• Add Phase 34 section documenting new forgelm doctor subcommand
• List all features: probes, output formats, offline mode, exit codes
• Document self-contained design with lazy imports
• Reference closure of GH-001 onboarding blocker
• List new test coverage (38 tests, multiple scenarios)
• Note documentation updates

CHANGELOG.md


Grey Divider

Qodo Logo

@qodo-code-review

qodo-code-review Bot commented May 2, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (2) 📎 Requirement gaps (0)

Grey Divider


Action required

1. Wrong extra names suggested ☑ 🐞 Bug ≡ Correctness
Description
_OPTIONAL_EXTRAS hardcodes extras like deepspeed/evaluation/wandb/mergekit and claims they
mirror pyproject.toml, but the actual extras are distributed/eval/tracking/merging (and others),
so forgelm doctor will recommend invalid pip install 'forgelm[...]' commands and report
non-existent extras.
Code

forgelm/cli/subcommands/_doctor.py[R45-61]

+# Optional extras advertised in pyproject.toml [project.optional-dependencies].
+# Each entry is ``(extra_name, importable_module, human_purpose_blurb)``.
+# ``importable_module`` is the package the extra installs that we probe via
+# ``importlib.util.find_spec`` — a successful spec resolve means the extra is
+# present.  Doctor never *imports* the module (avoids triggering torch /
+# transformers / spacy heavy load on a healthy probe).
+_OPTIONAL_EXTRAS: Tuple[Tuple[str, str, str], ...] = (
+    ("qlora", "bitsandbytes", "4-bit / 8-bit QLoRA training"),
+    ("unsloth", "unsloth", "Unsloth-accelerated training (Linux GPUs only)"),
+    ("deepspeed", "deepspeed", "DeepSpeed ZeRO + offload distributed training"),
+    ("evaluation", "lm_eval", "lm-evaluation-harness benchmark scoring"),
+    ("wandb", "wandb", "Weights & Biases experiment tracking"),
+    ("mergekit", "mergekit", "Model-merge backend for the merge mode"),
+    ("ingestion", "pypdf", "PDF / DOCX / EPUB ingestion"),
+    ("ingestion-pii-ml", "presidio_analyzer", "Presidio ML-NER PII detection in audit"),
+    ("ingestion-scale", "datasketch", "MinHash LSH for >50K-row dedup"),
+)
Evidence
The doctor implementation states these are the extras from pyproject.toml, but its keys don’t
match the extras declared in [project.optional-dependencies]. The user manual sample output also
instructs installing forgelm[deepspeed], which will fail if that extra isn’t declared (and it
isn’t).

forgelm/cli/subcommands/_doctor.py[45-61]
pyproject.toml[47-73]
pyproject.toml[74-127]
docs/usermanuals/en/getting-started/first-run.md[33-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`forgelm doctor` hardcodes a list of optional extras and claims it matches `pyproject.toml`, but several extra *keys* differ (e.g. doctor uses `deepspeed/evaluation/wandb/mergekit` while `pyproject.toml` defines `distributed/eval/tracking/merging`). This makes the reported check names and the suggested install commands (`pip install 'forgelm[<extra>]'`) incorrect.
## Issue Context
Because these strings are surfaced to operators and CI via text/JSON, mismatched extra keys will cause immediate user confusion and failed install attempts.
## Fix Focus Areas
- forgelm/cli/subcommands/_doctor.py[45-61]
- pyproject.toml[47-127]
- docs/usermanuals/en/getting-started/first-run.md[33-43]
- docs/usermanuals/tr/getting-started/first-run.md[33-43]
## Suggested approach
- Update `_OPTIONAL_EXTRAS` so `extra_name` values exactly match `[project.optional-dependencies]` keys (e.g. `distributed`, `eval`, `tracking`, `merging`, plus any other extras you truly want to probe like `export`, `chat`, `dev`).
- Keep `module` values as importable module names.
- Update docs sample output to match the corrected extra names and install hints.
- Optionally add/adjust tests to ensure the extras list stays in sync with expected extra keys (at minimum assert the presence of the corrected names in the plan).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Anonymous operator opt-in ignored ☑ 🐞 Bug ≡ Correctness
Description
_check_operator_identity tells users to set FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 but never checks
it, so it still returns fail (exit code 1) when getpass.getuser() can’t resolve a username even
though the rest of ForgeLM can be configured to allow anonymous audit identity.
Code

forgelm/cli/subcommands/_doctor.py[R404-441]

+    explicit = os.environ.get("FORGELM_OPERATOR")
+    if explicit:
+        return _CheckResult(
+            name="operator.identity",
+            status=_STATUS_PASS,
+            detail=f"FORGELM_OPERATOR set to {explicit!r}; audit events will carry this identity.",
+            extras={"FORGELM_OPERATOR": explicit, "source": "env"},
+        )
+    # Try to resolve the fallback identity AuditLogger would use so
+    # the operator sees what their audit events would look like.
+    try:
+        import getpass
+
+        username = getpass.getuser()
+    except (KeyError, OSError, ImportError):
+        username = None
+    hostname = socket.gethostname() or "unknown-host"
+    if username:
+        fallback = f"{username}@{hostname}"
+        return _CheckResult(
+            name="operator.identity",
+            status=_STATUS_WARN,
+            detail=(
+                f"FORGELM_OPERATOR not set; audit events will fall back to {fallback!r}. "
+                "Pin FORGELM_OPERATOR=<id> for CI / pipeline runs so the audit log identifies a stable identity."
+            ),
+            extras={"FORGELM_OPERATOR": None, "fallback": fallback, "source": "getpass"},
+        )
+    return _CheckResult(
+        name="operator.identity",
+        status=_STATUS_FAIL,
+        detail=(
+            "FORGELM_OPERATOR not set AND getpass.getuser() could not resolve a username. "
+            "Set FORGELM_OPERATOR=<id> (or FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 to opt in to "
+            "anonymous audit entries — not recommended for EU AI Act Article 12)."
+        ),
+        extras={"FORGELM_OPERATOR": None, "fallback": None},
+    )
Evidence
_check_operator_identity references FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 in its failure text but
doesn’t read that env var at all. Meanwhile the AuditLogger explicitly supports this opt-in: if
username resolution fails, it allows anonymous identity when FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 is
set instead of erroring.

forgelm/cli/subcommands/_doctor.py[404-441]
forgelm/compliance.py[61-104]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`_check_operator_identity()` suggests `FORGELM_ALLOW_ANONYMOUS_OPERATOR=1` as an opt-in, but it never checks that environment variable. As a result, in environments where `getpass.getuser()` fails (common in some containers/CI), `forgelm doctor` reports `fail` even when ForgeLM’s `AuditLogger` would proceed with `anonymous@<host>` if the opt-in is set.
## Issue Context
This creates a behavior mismatch between `doctor` and the actual audit-logging policy, and can cause CI checks to fail unnecessarily.
## Fix Focus Areas
- forgelm/cli/subcommands/_doctor.py[404-441]
- forgelm/compliance.py[61-104]
## Suggested approach
- In `_check_operator_identity`, when `username` is unavailable:
- Read `FORGELM_ALLOW_ANONYMOUS_OPERATOR`.
- If it is set to `"1"`, return `warn` (or `pass`, depending on your intended semantics) with detail indicating `anonymous@<hostname>` will be used.
- If not set, keep the current `fail` behavior.
- Include the resolved anonymous identity in `extras` for JSON consumers (e.g. `{fallback: 'anonymous@host', source: 'anonymous_opt_in'}`).
- Add a unit test covering the opt-in path.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Doctor reads HF_HOME env 📘 Rule violation ⚙ Maintainability
Description
The new doctor offline cache probe changes behavior based on non-secret environment variables
(HF_HOME, HF_HUB_OFFLINE) rather than validated YAML configuration. This violates the
config-driven behavior requirement and can make runs non-deterministic across CI/hosts.
Code

forgelm/cli/subcommands/_doctor.py[R310-352]

+    cache_dir = os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface/hub")
+    if not os.path.isdir(cache_dir):
+        return _CheckResult(
+            name="hf_hub.offline_cache",
+            status=_STATUS_WARN,
+            detail=(
+                f"HF cache not found at {cache_dir}. Air-gapped runs need pre-cached models — "
+                "see `forgelm cache-models` (Phase 35) once available, or set HF_HOME and pre-populate."
+            ),
+            extras={"cache_dir": cache_dir, "exists": False},
+        )
+    # Bytes used by the cache.  We sum sizes of regular files only; symlinks
+    # in the HF cache point at the blob store and would be double-counted.
+    total_bytes = 0
+    file_count = 0
+    for root, _dirs, files in os.walk(cache_dir):
+        for filename in files:
+            full = os.path.join(root, filename)
+            if os.path.islink(full):
+                continue
+            try:
+                total_bytes += os.path.getsize(full)
+                file_count += 1
+            except OSError:
+                # Skip files that disappeared between the walk and the size
+                # call (race with concurrent HF download).
+                continue
+    cache_gib = round(total_bytes / (1024**3), 2)
+    offline_env = os.environ.get("HF_HUB_OFFLINE")
+    return _CheckResult(
+        name="hf_hub.offline_cache",
+        status=_STATUS_PASS if file_count > 0 else _STATUS_WARN,
+        detail=(
+            f"HF cache at {cache_dir}: {cache_gib} GiB across {file_count} file(s). "
+            f"HF_HUB_OFFLINE={offline_env or 'unset'}."
+        ),
+        extras={
+            "cache_dir": cache_dir,
+            "exists": True,
+            "size_gib": cache_gib,
+            "file_count": file_count,
+            "hf_hub_offline_env": offline_env,
+        },
Evidence
PR Compliance ID 1 forbids env-var sniffing for behavior toggles unless the values are secrets; the
new offline cache logic reads HF_HOME and HF_HUB_OFFLINE to determine probe behavior/output.

CLAUDE.md
forgelm/cli/subcommands/_doctor.py[310-352]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`forgelm doctor` reads non-secret env vars (`HF_HOME`, `HF_HUB_OFFLINE`) to control probe behavior/output, which violates the config-driven behavior requirement.
## Issue Context
The doctor subcommand is newly added and should avoid feature/behavior decisions based on non-secret environment variables.
## Fix Focus Areas
- forgelm/cli/subcommands/_doctor.py[310-352]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Doctor sniffs FORGELM_OPERATOR env 📘 Rule violation ⚙ Maintainability
Description
The new operator-identity probe changes status and output based on the non-secret FORGELM_OPERATOR
environment variable. This introduces env-var-dependent behavior rather than deriving behavior from
validated YAML configuration.
Code

forgelm/cli/subcommands/_doctor.py[R404-411]

+    explicit = os.environ.get("FORGELM_OPERATOR")
+    if explicit:
+        return _CheckResult(
+            name="operator.identity",
+            status=_STATUS_PASS,
+            detail=f"FORGELM_OPERATOR set to {explicit!r}; audit events will carry this identity.",
+            extras={"FORGELM_OPERATOR": explicit, "source": "env"},
+        )
Evidence
PR Compliance ID 1 disallows using non-secret environment variables to drive behavior; the new check
reads FORGELM_OPERATOR and changes the probe result (pass vs warn/fail) based on its presence.

CLAUDE.md
forgelm/cli/subcommands/_doctor.py[404-411]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The doctor subcommand reads `FORGELM_OPERATOR` (non-secret) from the environment and changes probe results based on it, violating the config-driven behavior rule.
## Issue Context
Operator identity should be derived from validated YAML configuration (or otherwise not alter runtime behavior based on env vars), per the compliance requirement.
## Fix Focus Areas
- forgelm/cli/subcommands/_doctor.py[404-411]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. Torch check name flips 🐞 Bug ⚙ Maintainability
Description
When torch is missing, _check_torch_cuda returns name='torch.installed' while other paths (and
the plan/docs) use torch.cuda, causing unstable check naming that breaks JSON consumers filtering
by check name.
Code

forgelm/cli/subcommands/_doctor.py[R132-140]

+    try:
+        import torch
+    except ImportError:
+        return _CheckResult(
+            name="torch.installed",
+            status=_STATUS_FAIL,
+            detail="torch is not installed. Install with: pip install 'forgelm'.",
+            extras={"installed": False},
+        )
Evidence
The check plan label and documentation use torch.cuda, but the ImportError path changes the
emitted result name to torch.installed, so the same logical check appears under different names
depending on environment state.

forgelm/cli/subcommands/_doctor.py[125-146]
forgelm/cli/subcommands/_doctor.py[457-463]
docs/usermanuals/en/getting-started/first-run.md[35-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`_check_torch_cuda()` returns different `CheckResult.name` values depending on whether torch is importable (`torch.cuda`) or missing (`torch.installed`). This makes the JSON/text output unstable and complicates CI filtering.
## Issue Context
The plan label and docs present this check as `torch.cuda`; consumers will likely key off that name.
## Fix Focus Areas
- forgelm/cli/subcommands/_doctor.py[125-146]
- forgelm/cli/subcommands/_doctor.py[457-463]
## Suggested approach
- Always use `name='torch.cuda'` for this probe, even in the ImportError path.
- Keep the additional information in `extras` (e.g. `installed: False`) to preserve detail without changing the stable name.
- Update/extend tests if any assert on the old `torch.installed` name.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@codacy-production

codacy-production Bot commented May 2, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 2 high · 1 medium · 1 minor

Alerts:
⚠ 4 issues (≤ 0 issues of at least minor severity)

Results:
4 new issues

Category Results
BestPractice 1 minor
ErrorProne 1 high
Security 1 medium
1 high

View in Codacy

🟢 Metrics 148 complexity · 0 duplication

Metric Results
Complexity 148
Duplication 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the forgelm doctor subcommand, a comprehensive environment diagnostic tool that checks Python versions, PyTorch/CUDA availability, GPU inventory, optional extras, disk space, and HuggingFace Hub connectivity. The changes include CLI integration, documentation updates in multiple languages, and extensive unit testing. Feedback suggests enhancing the HuggingFace probes by respecting the HF_ENDPOINT environment variable and correcting the cache directory resolution logic to properly handle HF_HOME subdirectories and path expansion.

Comment thread forgelm/cli/subcommands/_doctor.py Outdated
import urllib.error
import urllib.request

request = urllib.request.Request("https://huggingface.co/api/models", method="HEAD")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The Hugging Face Hub reachability probe uses a hardcoded URL, which ignores the HF_ENDPOINT environment variable. This can lead to false warnings in enterprise or air-gapped environments where a local mirror is configured via HF_ENDPOINT.

Suggested change
request = urllib.request.Request("https://huggingface.co/api/models", method="HEAD")
endpoint = os.environ.get("HF_ENDPOINT") or "https://huggingface.co"
url = f"{endpoint.rstrip('/')}/api/models"
request = urllib.request.Request(url, method="HEAD")

Comment thread forgelm/cli/subcommands/_doctor.py Outdated
is a warning — the operator may need to run ``forgelm cache-models``
(Phase 35) before training.
"""
cache_dir = os.environ.get("HF_HOME") or os.path.expanduser("~/.cache/huggingface/hub")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for resolving the Hugging Face cache directory is incorrect when HF_HOME is set. According to the huggingface_hub standard, the hub cache is located in a hub subdirectory of HF_HOME. The current implementation uses HF_HOME directly, which will result in incorrect size calculations (including datasets and other metadata) and may fail to find the actual model cache if it's nested. Additionally, environment variables like HF_HOME should be expanded for user home directory symbols (~) to ensure os.path.isdir works correctly.

    cache_dir = os.environ.get("HF_HUB_CACHE")
    if not cache_dir:
        hf_home = os.environ.get("HF_HOME")
        if hf_home:
            cache_dir = os.path.join(os.path.expanduser(hf_home), "hub")
        else:
            cache_dir = os.path.expanduser("~/.cache/huggingface/hub")

Comment thread forgelm/cli/subcommands/_doctor.py
Comment thread forgelm/cli/subcommands/_doctor.py
Closes the two HIGH findings + four bot comments.

HIGH fixes:

- F-27-01 (HIGH): GH-1 closure was incomplete — only 2 of 10 doc
  files had been updated to match the shipped output.  This commit
  updates the rest:
  * docs/usermanuals/en/getting-started/installation.md — sample
    "what doctor reports" rewritten to enumerate the actual probe
    set + JSON envelope shape + --offline flag.
  * docs/usermanuals/en/operations/air-gap.md — sample --offline
    output replaced with the real table format (was: aspirational
    text "ForgeLM 0.5.2 (offline mode verified)" + "Llama Guard 3
    8B available locally" — none of which the dispatcher actually
    produces).
  * docs/usermanuals/tr/* mirrors updated in lock-step.
  troubleshooting.md was already accurate (only references the
  command name, no sample output to drift).
- F-27-02 (HIGH): HF Hub probe partial migration toward forgelm._http
  discipline.  HF_ENDPOINT env var is now respected via a new
  _resolve_hf_endpoint() helper (mirrors huggingface_hub's own
  resolution) so self-hosted mirrors / corp-proxy installs no longer
  produce false-warn results.  Probe carries a dedicated
  "forgelm-doctor/Phase-34" User-Agent so server logs identify it.
  Full migration to forgelm._http.safe_get is deferred to XPR-02
  cross-PR work (requires adding safe_get to _http.py).

Bot fixes (gemini, qodo):

- gemini bot: HF cache resolution honours HF_HUB_CACHE > HF_HOME/hub
  > ~/.cache/huggingface/hub.  Was: pointed at HF_HOME directly,
  which is the parent of multiple cache trees (hub + datasets +
  spaces) — wrong directory.
- gemini bot: HF_ENDPOINT respected (covered above with F-27-02).
- qodo bot: _OPTIONAL_EXTRAS list audited against pyproject.toml.
  Original used aspirational names ("deepspeed", "evaluation",
  "wandb", "mergekit") that did not match the published extras
  ("distributed", "eval", "tracking", "merging") — install hints
  were unactionable.  Also added "export" extra to the probe list.
- qodo bot: _check_operator_identity now respects
  FORGELM_ALLOW_ANONYMOUS_OPERATOR=1 the same way AuditLogger does.
  Without the opt-in: fail (matches AuditLogger's hard-refuse).
  With the opt-in: warn (matches AuditLogger's "anonymous@host"
  fallback).  Doctor's verdict now matches actual training-time
  behaviour.

MEDIUM fixes:

- F-27-03 (MEDIUM perf): _check_hf_cache_offline walk is now
  depth-capped (4 levels) + file-capped (5 000 files); cap-hit is
  recorded in extras["walk_truncated"] so downstream tooling can
  detect partial-scan results.  Prevents 30+s scans on NFS-mounted
  50+ GiB caches.
- F-27-04 (MEDIUM correctness): dropped the dead socket.timeout
  except branch.  socket.timeout is an alias for TimeoutError since
  Python 3.10 (project floor) and urllib.request.urlopen wraps
  timeouts as URLError anyway.
- F-27-05 (MEDIUM security): added _DOCTOR_SECRET_ENV_NAMES allow-
  list + _mask_env_value_for_audit helper.  FORGELM_OPERATOR is not
  on the list (operator identity, not secret) but the discipline is
  pinned for any future probe surfacing HF_TOKEN / FORGELM_AUDIT_SECRET.

Tests added (TestHfEndpointResolution, TestOperatorIdentityAnonymousOptIn,
TestSecretEnvMasking, plus updated TestHfCacheOfflineCheck for the
HF_HUB_CACHE > HF_HOME/hub > default precedence).  +7 tests
(38 → 45 in test_doctor.py).

DEFERRED (out of this PR's scope):

- F-27-06 (MEDIUM contract): JSON envelope schema page —
  docs/reference/cli_json_schemas.md is XPR-03 cross-PR concern.
- F-27-07 (MEDIUM): "skip" status for HF Hub probe under --offline —
  current behaviour swaps the probe rather than emitting a "skip"
  row; pinning the swap behaviour explicitly is a separate follow-up.
- F-27-08..15 LOW/NIT: backlog.

Tests: 1051 → 1058 (+7). ruff format/check clean.
Coverage 68.59% → 68.74% (+0.15).

Closes (review IDs): F-27-01, F-27-02, F-27-03, F-27-04, F-27-05
                     + gemini-HF_ENDPOINT, gemini-HF_HOME,
                     + qodo-extras, qodo-anonymous-opt-in

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@cemililik

Copy link
Copy Markdown
Collaborator Author

Wave 2a Round-1 multi-agent review — fixes pushed

All CRITICAL + HIGH findings from the multi-agent review (and the relevant inline bot comments) have been addressed in the latest commit on this branch. See the commit message for the per-finding mapping (review IDs F-27-*, gemini bot IDs, qodo bot IDs, sourcery bot IDs).

Per-PR test counts:

ruff format/check clean across all five PRs. Cross-PR XPR-01 concern (audit-log parser proliferation) closed in PR #25 via the new forgelm/cli/subcommands/_audit_log_reader.py shared module that _approve.py and _approvals.py now both delegate to (Phase 21 forgelm purge will use the same module).

Items deferred to follow-up PRs (with rationale) are listed under DEFERRED in each commit message.

@coderabbitai review

@coderabbitai

coderabbitai Bot commented May 2, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@cemililik Thanks for the detailed Wave 2a Round-1 summary! The fixes look well-organized — particularly glad to see the +7 tests covering HF cache, endpoint, anonymous opt-in, and secret masking, and that the XPR-01 audit-log parser proliferation concern has been resolved cross-PR via the new shared _audit_log_reader.py module in PR #25. Triggering a fresh review of PR #27 now.

[review]

@sonarqubecloud

sonarqubecloud Bot commented May 2, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@cemililik

Copy link
Copy Markdown
Collaborator Author

Superseded by #28 (consolidated Wave 2a integration). All commits from this branch are merged into closure/wave2a-integration and the Wave-2a Round-1 review fixes ride on top there. Closing without merge to keep history single-stream.

@cemililik cemililik closed this May 4, 2026
cemililik added a commit that referenced this pull request May 4, 2026
Wave 2a: Phase 17/18/20/34/37 — consolidated integration (supersedes #23-#27)
@cemililik
cemililik deleted the closure/faz34-doctor-subcommand branch May 10, 2026 11:35
cemililik added a commit that referenced this pull request Jun 15, 2026
…PR#63-review)

_format_user_assistant_row gated System validation on truthiness, so 0/False/0.0/[]/{} bypassed clean_string's loud-fail contract and were silently dropped. Gate on sys_text != '' so only the genuine empty-string absent-system path skips, and every other value (incl. None) fails loudly. CodeRabbit #27.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant