The user-manual air-gap recipe has operators transfer a bundle the target host never reads (wrong HF_HOME layout), so offline model and dataset loads fail (CR-A15-003) #450

Wave 1 · theme D1-8: Docs describing features that do not exist (doc side of code findings) · highest severity High · 1 finding(s) · effort small

Summary

An operator follows the user manual: runs cache-models and cache-tasks with --output ./airgap-bundle/, transfers airgap-bundle/, and exports HF_HOME=airgap-bundle/huggingface plus the offline flags on the target. The base model, Llama Guard and every lm-eval dataset are 'not cached', so training or --benchmark-only fails on the air-gapped host. Recovery needs another trip through the transfer process.

Primary finding

CR-A15-003 — The user-manual air-gap recipe has operators transfer a bundle the target host never reads (wrong HF_HOME layout), so offline model and dataset loads fail

severity High · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. docs/usermanuals/en/operations/air-gap.md:25-32 (and the TR mirror docs/usermanuals/tr/operations/air-gap.md:25-32) stage both caches with --output ./airgap-bundle/. Both resolvers use that path as given: _cache.py:157-158 / :180-181 return os.path.abspath(output_arg), cache-models passes it as snapshot_download(cache_dir=...) (_cache.py:259, :338), and cache-tasks stamps it as HF_DATASETS_CACHE (_cache.py:422, :444). The bundle root therefore holds models--* snapshot dirs and <ns>___<name>/ Arrow dirs side by side, with no huggingface/, hub/ or datasets/ subtree. On the target, :42 exports HF_HOME=$(pwd)/airgap-bundle/huggingface, so huggingface_hub reads airgap-bundle/huggingface/hub and datasets reads airgap-bundle/huggingface/datasets. Neither directory exists in the bundle. Line 102 adds that with HF_HOME + HF_DATASETS_OFFLINE=1 'the cache is picked up automatically'. The second invocation at :99 (forgelm cache-tasks --tasks ... with no --output) writes to the staging host's env-resolved datasets cache, which is not part of airgap-bundle/ at all. Table row :19 points to ~/.cache/lm-evaluation-harness/ as the place lm-eval task definitions are cached. cache-tasks never writes there: it fills the HF datasets cache. The developer guide (docs/guides/air_gap_deployment.md:37,62,98-99) uses a consistent hub/ + datasets/ layout with HF_HUB_CACHE/HF_DATASETS_CACHE on both hosts, so the two operator-facing documents give incompatible recipes and only the guide's works. This is independent of M-A15-006: even when the late HF_DATASETS_CACHE stamp works, the user-manual layout cannot be found on the target.

Where.

• docs/usermanuals/en/operations/air-gap.md:42
• docs/usermanuals/en/operations/air-gap.md:25-32
• docs/usermanuals/en/operations/air-gap.md:102
• docs/usermanuals/tr/operations/air-gap.md:42
• docs/usermanuals/tr/operations/air-gap.md:25-32
• docs/usermanuals/tr/operations/air-gap.md:102
• forgelm/cli/subcommands/_cache.py:157-158
• forgelm/cli/subcommands/_cache.py:180-181
• forgelm/cli/subcommands/_cache.py:259
• forgelm/cli/subcommands/_cache.py:338

Evidence.

• Dynamic run, datasets 5.0.0 / huggingface_hub 1.22.0, loopback mock Hub at 127.0.0.1:18731 serving the dataset repo review-org/tiny_ds (configs easy/hard, parquet). Staging: HF_DATASETS_CACHE=/airgap-bundle (what cache-tasks --output ./airgap-bundle/ stamps), load_dataset('review-org/tiny_ds','easy') -> LOADED train 5 / test 3. The bundle root then contains only review-org___tiny_ds/ plus a .lock. Copied to /airgap-bundle. With HF_HOME=/airgap-bundle/huggingface, HF_DAT…

Recommended fix. Align the user manual with the guide: stage into airgap-bundle/hub and airgap-bundle/datasets (via HF_HUB_CACHE / HF_DATASETS_CACHE or HF_HOME=airgap-bundle with no --output), and export the matching variables on the target. Drop the ~/.cache/lm-evaluation-harness/ row, or replace it with the HF datasets cache path. Make the :99 example use the same env as the transfer. Update both EN and TR.

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}