Wizard defaults to two gated Meta repos (base model and default-on Llama-Guard classifier); no wizard or site copy says an HF token and licence acceptance are needed, and a token-… (CR-A18-089) #535

Wave 2 · theme D2-7: Next-release technology updates · highest severity Low · 2 finding(s) · effort small

Summary

A first-time operator accepts the wizard defaults and runs dry-run (exit 0), then trains. The run aborts at the tokenizer download with exit 2 and a gated-repo traceback. With a non-gated model and test_prompts supplied, hours of training finish before the Llama-Guard download fails.

Primary finding

CR-A18-089 — Wizard defaults to two gated Meta repos (base model and default-on Llama-Guard classifier); no wizard or site copy says an HF token and licence acceptance are needed, and a token-less run fails with…

severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced with stubs

What happens. The wizard's default and 'recommended' model is meta-llama/Llama-3.1-8B-Instruct (wizard.js:139; badge translations.js:1540). evaluation.safety is on by default and always emits classifier meta-llama/Llama-Guard-3-8B (wizard.js:714). Both are gated Hub repos. None of these say a Hugging Face token or licence acceptance is required, in any locale: wizard.model.*.desc (translations.js:1539-1552), wizard.step.model.tutorial (1538), wizard.eval.safety.label/hint/detail (1699-1701) and quickstart.html. The custom hint says only 'must be downloadable from the HuggingFace Hub' (1552). The browser wizard never emits an auth block. The user manual covers it (installation.md:149, troubleshooting.md:33-37), but the site surface that generates the YAML does not. Runtime path: setup_authentication only warns when no token is found (utils.py:42-44). prepare_revision_pin swallows the Hub lookup error (compliance.py:1220). AutoTokenizer.from_pretrained then raises the gated-repo OSError (model.py:391-393). The broad catch maps it to exit 2 'Training pipeline failed.' (cli/_training.py:173-182) before dataset preparation or any artefact; dry-run passes with exit 0 because it probes nothing (_dry_run.py:70-89). For a non-gated preset (the Qwen use cases), training completes and the gated classifier is loaded only in the post-training safety eval. Today the missing safety_prompts.jsonl check (safety/_orchestrator.py:195, already reported as M-A18-002) fails first. Once an operator supplies test_prompts, the token-less run fails at the classifier load after the full training run, with evaluation_completed=False.

Where.

• site/js/wizard.js:139
• site/js/wizard.js:714
• site/js/translations.js:1538-1552
• site/js/translations.js:1699-1701
• forgelm/utils.py:42-44
• forgelm/model.py:391-393
• forgelm/cli/_training.py:173-182
• forgelm/cli/_dry_run.py:70-89

Evidence.

• Default-state wizard YAML: name_or_path meta-llama/Llama-3.1-8B-Instruct; --dry-run EXIT=0. Stubbed run with AutoTokenizer/AutoModelForCausalLM.from_pretrained patched to raise the gated OSError, HOME without a token file: 'WARNING No Hugging Face token found', 'base_model ... loading UNPINNED (unresolved)', 'ERROR Training pipeline failed.', SYSTEMEXIT 2; no output dir. grep of translations.js/quickstart.html/wizard.js for gated|HF_TOKEN|login|licence acceptance: no match in wizard or quicksta…

Recommended fix. Add a short 'gated: needs HF token + accepted licence' note to the Llama card, the Llama-Guard safety hint and the quickstart page, or default to a non-gated model. Consider a dry-run Hub access probe (model_info with token) for online runs.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

TECH-T14-002 — Hard-coded model identifiers and model-merging methods: Wizard / config_template defaults: meta-llama/Llama-3.1-8B-Instruct (default, 'recommended'), mistralai/Mistral-7B-Instruct-v0.3, microsoft/Phi…

severity Next release

Item. Wizard / config_template defaults: meta-llama/Llama-3.1-8B-Instruct (default, 'recommended'), mistralai/Mistral-7B-Instruct-v0.3, microsoft/Phi-3-mini-4k-instruct (forgelm/wizard/_state.py:132-138, config_template.yaml:4, site/js/wizard.js:139-140,1549-1552, site/js/translations.js:1538-1548) · verdict: aging · urgency: Next release

State in ForgeLM. POPULAR_MODELS (forgelm/wizard/_state.py:132-138) lists Llama-3.1-8B-Instruct first. With no use-case preset, the CLI wizard's default choice is index 1 (_orchestrator.py:300-301; _prompt_choice in _io.py:124-145 is 1-based), i.e. the gated Llama. The site wizard's initial state is Llama-3.1-8B (wizard.js:139-140). Its use-case presets (wizard.js:501-536) switch to Qwen, so the Llama default applies to the 'custom' path. translations.js:1538 calls Llama-3.1-8B 'a balanced default', :1540 badges it 'recommended', and :1545 says Mistral is 'Apache 2.0'. config_template.yaml:4 (the canonical CI dry-run example) uses Llama-3.1-8B-Instruct. Gating is documented elsewhere (docs/guides/getting-started.md:16,92-98; docs/reference/usage.md:31; docs/usermanuals/en/getting-started/installation.md:149) but not on these surfaces.

State of the ecosystem on 2026-10-01. All three ids exist under the same names. Llama-3.1-8B-Instruct: gated=manual, licence llama3.1, lastModified 2024-09-25; an unauthenticated file fetch returns 'Access ... is restricted'. Mistral-7B-Instruct-v0.3: gated=False, apache-2.0 (the site claim is correct), lastModified 2025-12-03. Phi-3-mini-4k-instruct: gated=False, MIT, lastModified 2025-12-10 (4k context). Successors: Phi-4-mini-instruct (MIT, Phi3ForCausalLM, drop-in) and Ministral-3-8B-Instruct-2512 (apache-2.0, but Mistral3ForConditionalGeneration, i.e. multimodal). Meta's newest meta-llama repos date from April 2025 (Llama 4 / Guard 4 / Prompt Guard 2) and are all gated; there is no newer 1-8B Llama text model.

Impact. A first-time user hits a gated-repo error if they accept the CLI wizard default, take the site wizard's 'custom' path, or copy config_template.yaml. Running it needs an HF account, an approved Meta licence request and HF_TOKEN, and the 'recommended' badge does not say so. Mistral-7B-v0.3 and Phi-3-mini are 2024 models. No id has disappeared.

Recommendation. Make an ungated Apache/MIT model the default and 'recommended' entry in POPULAR_MODELS, the site wizard initial state and config_template.yaml (e.g. Qwen/Qwen3-8B or Qwen/Qwen3-4B-Instruct-2507). Keep Llama-3.1-8B as an option labelled 'gated: needs Meta approval + HF_TOKEN'. Replace Phi-3-mini-4k with Phi-4-mini-instruct. Replace Mistral-7B-v0.3 only after the multimodal loader path for Ministral-3 is validated. Keep the lockstep with site USE_CASE_PRESETS that _state.py:128-131 requires. Fix the non-resolving example ids (see incorrect_statements).

Sources.

• HF API: meta-llama/Llama-3.1-8B-Instruct (gated=manual, llama3.1) (2024-09-25 (lastModified))
• Unauthenticated fetch of gated repo file: 'Access to model ... is restricted' (2026-10-01)
• HF API: mistralai/Mistral-7B-Instruct-v0.3 (gated=False, apache-2.0) (2025-12-03 (lastModified))
• HF API: microsoft/Phi-3-mini-4k-instruct (gated=False, MIT) (2025-12-10 (lastModified))
• HF API: microsoft/Phi-4-mini-instruct (MIT, Phi3ForCausalLM) (2025-12-10 (lastModified))
• HF API: Ministral-3-8B-Instruct-2512 (apache-2.0, Mistral3ForConditionalGeneration) (2026-07-15 (lastModified))

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}