--offline / model.offline only sets HF env vars; webhooks, API judge, API teacher and trackers still reach the network, contrary to the air-gap manual (CR-A01-001) #74

Wave 0 · theme D0-5: Documented controls that are not implemented · highest severity Critical · 5 finding(s) · effort medium

Summary

Air-gapped CI with webhook.url pointing at an external Slack gateway (reachable through a proxy) or an API-backed judge: --offline does not stop the call, contrary to air-gap.md:59-66; the compliance claim 'no outbound traffic' is wrong.

Primary finding

CR-A01-001 — --offline / model.offline only sets HF env vars; webhooks, API judge, API teacher and trackers still reach the network, contrary to the air-gap manual

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. _apply_offline_flag (cli/_config_load.py:50-57) sets config.model.offline and exports HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE/HF_DATASETS_OFFLINE, nothing else. The safe_post/safe_get chokepoint (_http.py) and its callers (WebhookNotifier, judge._call_api_judge, synthetic._call_api_teacher, the pipeline notifier) and report_to wiring never consult model.offline. The air-gap manual (air-gap.md:50,57-66,79, and the TR mirror) says --offline refuses any network call and lists these as disabled, and it names a nonexistent webhook.allow_private field.

Where.

• forgelm/cli/_config_load.py:50-57 (confirmed)
• forgelm/cli/_training.py:136-139
• forgelm/webhook.py (no offline reference)
• forgelm/judge.py (no offline reference)
• docs/usermanuals/en/operations/air-gap.md:79 (the sentence saying the API teacher fails on --offline)
• docs/guides/air_gap_deployment.md:139 (the 'local_files_only=True end-to-end' claim)
• forgelm/webhook.py:429
• forgelm/judge.py:223-262 (confirmed)
• forgelm/synthetic.py:208-245
• docs/usermanuals/en/operations/air-gap.md:57-66 (the table; the webhook row names the nonexistent webhook.allow_private, real field is allow_private_destinations at forgelm/config.py:1488)

Evidence.

• log wh_offline_train: real 1-step tiny SFT with --offline and a webhook configured to a loopback server (allow_private_destinations) -> exit 0 and WEBHOOK_POSTS: 2 ['training.start','training.success']. grep for 'offline' in webhook.py/judge.py/_http.py/benchmark.py/safety/*.py: no hits. grep local_files_only in forgelm/: only comments.
• t4_offline.py: after _apply_offline_flag(cfg, True) (offline=True, HF_HUB_OFFLINE=1), WebhookNotifier(cfg).notify_start('run1') POSTed {"event": "training.start", ...} to a loopback receiver. rg 'offline' on webhook.py/judge.py/_http.py returns no gate. synthetic.py only passes offline to the local-teacher revision pin (line 299).
• offline/ workdir: train_judge_api.yaml plus a webhook url, run with --offline under scripted judge-api-valid. EXIT 0. The call log has 5 'served' judge HTTPS POSTs at the urllib3 transport and 2 'blocked' DNS lookups for hooks.review-offline-probe.example. stderr: 'Offline mode enabled. All HF Hub network calls are disabled.' followed by 'Refusing to post webhook ... DNS resolution failed'. The DNS refusal came from the fixture's socket guard, not from ForgeLM.

Recommended fix. Either correct the table to 'HF libraries only' and list the other egress sources with their own switches, or make model.offline reject/disable webhook, API judge/teacher and tracking at config-validation time (fail closed).

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T09-006 (technology, Next release) — Experiment tracking, webhooks and monitoring: MLflow runtime defaults ForgeLM neither sets nor documents (sqlite default, file-store error, telemetry) and the air-gap guide's tracker/webhook rows (at…
  severity Next release
  Item. MLflow runtime defaults ForgeLM neither sets nor documents (sqlite default, file-store error, telemetry) and the air-gap guide's tracker/webhook rows · verdict: at-risk · urgency: Next release
  State in ForgeLM. ForgeLM never sets MLflow variables (rg finds no MLFLOW_, WANDB_MODE or DO_NOT_TRACK in forgelm/). experiment-tracking.md:46-50 documents only MLFLOW_TRACKING_URI, EXPERIMENT_NAME and credentials. docs/usermanuals/en/operations/air-gap.md:62 (TR :62) says --offline disables W&B/MLflow/Comet 'or local-only for MLflow with file URI', and :64 (TR :64) says webhooks are disabled or 'restricted ... vi…
  State of the ecosystem on 2026-10-01. MLflow docs: 'As of MLflow 3.7.0, the default tracking backend has changed from file-based storage (./mlruns) to SQLite database (sqlite:///mlflow.db)', and 'Starting with version 3.2.0, MLflow collects anonymized usage data by default' (opt out with MLFLOW_DISABLE_TELEMETRY=true or DO_NOT_TRACK=true). MLFLOW_ALLOW_FILE_STORE (default False) is needed 'to opt out of the error raised when instanti…
  Recommendation. Document MLFLOW_TRACKING_URI=sqlite:///<output_dir>/mlflow.db or a server URL (never ./mlruns or file:) and MLFLOW_DISABLE_TELEMETRY=true / WANDB_MODE=offline for air-gapped runs. Correct air-gap.md:62 and :64 in EN and TR. Pin the compose mlflow image and pass --backend-store-uri. Optionally have doctor warn when report_to is mlflow and telemetry is not disabled.
  Sources.

  • MLflow self-hosting overview (default backend change as of 3.7.0) (retrieved 2026-10-01 (page undated))
  • MLflow usage tracking (telemetry default since 3.2.0; opt-out variables) (retrieved 2026-10-01 (page undated))

• DOC-D3B02-002 (docs, Critical) — docs/usermanuals/en/operations/air-gap.md: false claim — "The --offline flag tells ForgeLM to refuse any network call." Under --offline: W&B / MLflow / Comet disabled; OpenAI / Anthropic judges d…
  severity Critical
  Where. docs/usermanuals/en/operations/air-gap.md:50,57-66,79 (Turkish mirror: docs/usermanuals/tr/operations/air-gap.md:50,57-66,79) · type: false claim
  The document says. "The --offline flag tells ForgeLM to refuse any network call." Under --offline: W&B / MLflow / Comet disabled; OpenAI / Anthropic judges disabled; webhooks disabled "(or restricted ... via webhook.allow_private)"; an API-backed teacher "fails on --offline".
  What the code does. _apply_offline_flag only sets model.offline and exports HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE. webhook.py, judge.py (_call_api_judge → safe_post), the synthetic API teacher path and training.report_to have no offline gating. Reproduced: with the offline flag applied, a webhook POST was delivered. Comet is not a supported tracker, and the field is allow_private_destin… **Code references.** [forgelm/cli/_config_load.py:50-57](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/cli/_config_load.py#L50-L57), [forgelm/webhook.py:66-83](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/webhook.py#L66-L83), [forgelm/judge.py:223-253](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/judge.py#L223-L253), [forgelm/synthetic.py:197-248](https://github.com/HodeTech/ForgeLM/blob/cab2947/forgelm/synthetic.py#L197-L248) **Proposed correction.** Code fix (see rationale). Interim warning: "--offlineonly puts the Hugging Face libraries in offline mode (HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE / HF_DATASETS_OFFLINE). It does not block webhooks, API judges, API teachers orreport_to: wandb|mlflow` — remove or disable those in the config for air-gapped runs."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B02-015 (docs, High) — docs/usermanuals/en/operations/air-gap.md: false claim — --offline disables W&B/MLflow/Comet, OpenAI/Anthropic judges, webhooks, runtime pip
  severity High
  Where. docs/usermanuals/en/operations/air-gap.md:59-66 · type: false claim
  The document says. --offline disables W&B/MLflow/Comet, OpenAI/Anthropic judges, webhooks, runtime pip
  What the code does. --offline sets HF_*_OFFLINE env vars, skips HF auth and revision lookups only; webhook delivered under --offline in a real run. The field 'webhook.allow_private' does not exist (real name allow_private_destinations) and allowing private destinations widens rather than restricts.
  Code references. forgelm/cli/_config_load.py:50-57, forgelm/config.py:1488
  Proposed correction. Rewrite the table to the HF-only scope or implement the blocks (F5).
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B02-029 (docs, Medium) — docs/usermanuals/en/operations/air-gap.md: false claim — --offline: W&B/MLflow/Comet Disabled; OpenAI/Anthropic judges Disabled; Webhooks Disabled (or restricted via webhook.allow_private)
  severity Medium
  Where. docs/usermanuals/en/operations/air-gap.md:62-65 · type: false claim
  The document says. --offline: W&B/MLflow/Comet Disabled; OpenAI/Anthropic judges Disabled; Webhooks Disabled (or restricted via webhook.allow_private)
  What the code does. apply_offline_flag only sets the HF*_OFFLINE env vars (forgelm/cli/_config_load.py:50-57). judge.py and webhook.py contain no 'offline' check, and WANDB_MODE is never set. The field is webhook.allow_private_destinations (config.py:1488).
  Turkish mirror. the Turkish page has no such statement

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}