Privacy page and air-gap manual promise 'does not phone home / no telemetry', but every online training run sends TRL usage telemetry to huggingface.co (CR-A01-015) #95

Wave 0 · theme D0-5: Documented controls that are not implemented · highest severity High · 1 finding(s) · effort small

Summary

privacy.body (all six locales, translations.js:4345/4354/4363/4372/4381/4390) says the ForgeLM package 'does not phone home. It does not collect telemetry, does not send usage data anywhere'. The user manual says the same ('Telemetry: None — ForgeLM never phones home regardless', docs/usermanuals/en/operations/air-gap.md:66, TR mirror :66). ForgeLM never sets HF_HUB_DISABLE_TELEMETRY. The only offline handling sets the three *_OFFLINE variables, and only when model.offline is true (cli/_config_load.py:50-57). model.offline defaults to False (config.py:268-271). In the pinned TRL range (trl>=1.0.0,<2.0.0, pyproject.toml), trl 1.7.1 _BaseTrainer.init calls send_telemetry(topic='trl/', user_agent={model_arch, peft, distributed, world_size, device, gpu}) (trl/trainer/base_trainer.py:73-125). It is skipped only when HF_HUB_DISABLE_TELEMETRY, HF_HUB_OFFLINE or CI is set. huggingface_hub 1.22.0 sends this as HEAD {HF_ENDPOINT}/api/telemetry/ on a background thread. Building the user-agent also adds the torch version and an agent-detection field, and triggers a GET of {ENDPOINT}/api/agent-harnesses (huggingface_hub/utils/_detect_agent.py:189-204), cached for 24 h. So a…

Primary finding

CR-A01-015 — Privacy page and air-gap manual promise 'does not phone home / no telemetry', but every online training run sends TRL usage telemetry to huggingface.co

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. privacy.body (all six locales, translations.js:4345/4354/4363/4372/4381/4390) says the ForgeLM package 'does not phone home. It does not collect telemetry, does not send usage data anywhere'. The user manual says the same ('Telemetry: None — ForgeLM never phones home regardless', docs/usermanuals/en/operations/air-gap.md:66, TR mirror :66). ForgeLM never sets HF_HUB_DISABLE_TELEMETRY. The only offline handling sets the three *_OFFLINE variables, and only when model.offline is true (cli/_config_load.py:50-57). model.offline defaults to False (config.py:268-271). In the pinned TRL range (trl>=1.0.0,<2.0.0, pyproject.toml), trl 1.7.1 _BaseTrainer.init calls send_telemetry(topic='trl/', user_agent={model_arch, peft, distributed, world_size, device, gpu}) (trl/trainer/base_trainer.py:73-125). It is skipped only when HF_HUB_DISABLE_TELEMETRY, HF_HUB_OFFLINE or CI is set. huggingface_hub 1.22.0 sends this as HEAD {HF_ENDPOINT}/api/telemetry/ on a background thread. Building the user-agent also adds the torch version and an agent-detection field, and triggers a GET of {ENDPOINT}/api/agent-harnesses (huggingface_hub/utils/_detect_agent.py:189-204), cached for 24 h. So a default run that names only a local model and a local dataset still contacts huggingface.co twice. The payload is coarse (no rows, no paths, no token), but it is usage data and it contradicts the published claim, which regulated deployers rely on. trl 1.0.0, the pyproject floor, has no such call. trl 1.7.1 and 1.14.1 (latest lane) do, so a fresh install within the declared range gets it.

Where.

• site/js/translations.js:4390
• site/js/translations.js:4354
• site/js/translations.js:4363
• site/js/translations.js:4372
• site/js/translations.js:4381
• docs/usermanuals/en/operations/air-gap.md:66
• docs/usermanuals/tr/operations/air-gap.md:66
• forgelm/cli/_config_load.py:50-56
• forgelm/config.py:268-271
• pyproject.toml:55

Evidence.

• Repro run_tel3.py + cfg.yaml (tiny-qwen local path, local sft dataset, model.offline default, fpy offline vars removed in-process, socket and telemetry session faked so nothing left the host). The run exited 0. Captured: (1) socket connect ('huggingface.co',443) from huggingface_hub/utils/_detect_agent.py:196; (2) HEAD https://huggingface.co/api/telemetry/trl/SFTTrainer with user-agent 'trl/1.7.1; hf_hub/1.22.0; python/3.13.14; torch/2.12.1; agent/claude-code; model_arch/qwen2; peft/true; distr…

Recommended fix. Export HF_HUB_DISABLE_TELEMETRY=1 (setdefault, so an operator can opt back in) at CLI and library entry, before TRL trainers are built, and add a test that asserts no send_telemetry call. Or correct privacy.body (6 locales) and air-gap.md (EN/TR) to say upstream HF libraries send usage pings unless HF_HUB_DISABLE_TELEMETRY / offline is set.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.
• The release note names the contract this fix restores.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}