[Wave 3] Medium — Module sweep: Onboarding: quickstart templates and the CLI wizard (part 1/2) #561

Wave 3 · 20 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A14-005 (code, Medium) — Wizard prompts outside the step machine (prelude, resume, save filename, overwrite, 'Start training now?') let navigation tokens and EOF escape as a traceback with exit 1
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator who finishes all 9 steps types 'q' at 'Save config as' to abandon the run. The terminal shows a raw traceback, and a CI or wrapper script sees exit 1, which reads as a config error, instead of exit 5.
  Where. forgelm/wizard/_orchestrator.py:1086 (confirmed), forgelm/wizard/_orchestrator.py:1092 (confirmed), forgelm/wizard/_orchestrator.py:1135 (confirmed), forgelm/wizard/_orchestrator.py:1067-1079
  Recommended fix. Wrap the save, overwrite and start-training prompts in the same WizardCancel/EOFError/KeyboardInterrupt handling as the step machine. Map a cancel before the save to WizardOutcome(None), which gives exit 5, and a cancel after the save to a deferred outcome, which gives exit 0. Treat back/reset at the save prompt as a return to step 9, or reject them there. Align usage.md:269 and cli.md:300 with w…

• CR-A14-006 (code, Medium) — A --wizard-start-from rerun that lowers the risk tier leaves risk_category and require_human_approval behind, producing a contradictory or schema-invalid YAML with exit 0
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator reclassifies a system from high-risk to limited-risk through the wizard. The wizard saves a YAML that either fails to load (exit 0 anyway) or ships contradictory tier declarations, and the Article 14 approval gate stays on without being shown.
  Where. forgelm/wizard/_orchestrator.py:619-624, forgelm/wizard/_orchestrator.py:677-678, forgelm/wizard/_orchestrator.py:113, forgelm/wizard/_orchestrator.py:1238-1246
  Recommended fix. When compliance is re-collected, always mirror risk_classification into an existing risk_assessment.risk_category, or drop the stale block. Prompt for, or reset, require_human_approval when the tier leaves the strict set. Have the pre-flight read both siblings, and fail validation with a non-zero or distinct outcome instead of exit 0.

• CR-A14-007 (code, Medium) — A start-from rerun cannot turn off RoPE scaling, NEFTune or OOM recovery: an explicit 'no' is silently ignored
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN CLI-09 — docs/roadmap/phase-16-trust-surface-hardening.md:502-503,512-513 (S7 open; unit id inferred from commit-3 ordering)
  Failure scenario. An operator iterating on a config declines RoPE scaling or NEFTune. The next training run still applies them, which changes model behaviour, and nothing in the wizard output says so.
  Where. forgelm/wizard/_orchestrator.py:567-600, forgelm/wizard/_collectors.py:544-576, forgelm/wizard/_collectors.py:605-609
  Recommended fix. On None or False, pop rope_scaling and neftune_noise_alpha and set oom_recovery False. Pass the loaded values as prompt defaults for NEFTune and OOM, as the RoPE collector already does. Show removals in the step diff.

• CR-A14-008 (code, Medium) — _collect_judge emits only four llm_judge fields and maps an empty key-env answer to OPENAI_API_KEY: eval_dataset and judge_api_base are never prompted or preserved, and a local judge cannot be select…
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN CLI-09 — docs/roadmap/phase-16-trust-surface-hardening.md:502-503,512-513 (S7 open; unit id inferred)
  Failure scenario. An operator iterates on a config whose judge uses a self-hosted endpoint, presses Enter through step 9 and trains. After training, ForgeLM sends the internal endpoint's bearer token, eval prompts and generations to api.openai.com.
  Where. forgelm/wizard/_collectors.py:981-983, forgelm/wizard/_collectors.py:983 (confirmed), forgelm/wizard/_orchestrator.py:751-758, forgelm/judge.py:233
  Recommended fix. Start the judge, benchmark and safety rebuild from copy.deepcopy(existing) and overwrite only the prompted keys. Let an empty answer clear judge_api_key_env to null, as the prompt text promises. Before saving, warn when a loaded judge_api_base is about to disappear.

• CR-A14-009 (code, Medium) — Compliance step on --wizard-start-from wipes the Annex IV provider fields and silently downgrades risk_classification to minimal-risk
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs · already tracked: KNOWN-OPEN CLI-09 — docs/roadmap/phase-16-trust-surface-hardening.md:502-503,512-513 (S7 open; unit id inferred)
  Failure scenario. An operator reruns the wizard on a high-risk config to change one field, presses Enter through the compliance step and saves over the original. The compliance bundle then reports minimal-risk with empty provider identity.
  Where. forgelm/wizard/_collectors.py:694-737, forgelm/wizard/_collectors.py:707-713, forgelm/wizard/_collectors.py:740-772, forgelm/wizard/_orchestrator.py:619-620
  Recommended fix. Thread state.config['compliance'] and state.config['risk_assessment'] into the collectors as prompt defaults, including the risk-tier default index, mirroring _collect_data_governance.

• CR-A14-010 (code, Medium) — Layerwise GaLore variants offered by the wizard always crash at trainer construction (gradient_accumulation_steps is fixed at 2)
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Failure scenario. A wizard user picks a layerwise GaLore optimizer; training aborts at optimizer creation with exit 2.
  Where. forgelm/wizard/_collectors.py:619-626, forgelm/wizard/_collectors.py:673-685, forgelm/wizard/_orchestrator.py:585, forgelm/config.py:1926-1935
  Recommended fix. Force gradient_accumulation_steps=1 (with a note) when a layerwise variant is chosen, and add the same cross-field check to ForgeConfig._validate_galore.

• CR-A14-011 (code, Medium) — Every quickstart run of a template trains into the same ./checkpoints//final_model and silently overwrites the previous run's model
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A user reruns quickstart to try a different --model and loses the previously trained adapter.
  Where. forgelm/quickstart.py:329-337, forgelm/quickstart.py:364-366, forgelm/templates/customer-support/config.yaml:23, forgelm/templates/code-assistant/config.yaml:21
  Recommended fix. Have _materialize_config rewrite training.output_dir to a per-run path, such as /checkpoints, or correct the docstring and header claims.

• CR-A14-012 (code, Medium) — Safety collector crashes on a schema-valid start-from YAML (min_safety_score: null) and drops unprompted safety fields on rerun
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs · already tracked: KNOWN-OPEN CLI-09 — docs/roadmap/phase-16-trust-surface-hardening.md:502-503,512-513 (S7 open; unit id inferred)
  Failure scenario. An operator reruns the wizard on a valid config and gets a traceback at step 9, or a rerun silently unpins the safety classifier revision.
  Where. forgelm/wizard/_collectors.py:330-337, forgelm/wizard/_collectors.py:310-329, forgelm/wizard/_orchestrator.py:714-719
  Recommended fix. Use a None-safe default (existing.get(k) if existing.get(k) is not None else d), prompt for or preserve min_safety_score, and start the rebuild from a deepcopy of existing.

• CR-A14-013 (code, Medium) — Auto-chat after quickstart loads the trained adapter on an unquantized base (no --load-in-4bit forwarded), so the 7B primary model does not fit the advertised 12 GB GPU
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. A user on a 12 GB GPU completes quickstart training on the 7B primary; the auto-launched chat offloads most layers to CPU or OOMs.
  Where. forgelm/cli/subcommands/_quickstart.py:169-177, forgelm/inference.py:85-99, forgelm/templates/customer-support/config.yaml:8, forgelm/cli/subcommands/_quickstart.py:14-35
  Recommended fix. Forward --load-in-4bit to the chat child when the generated config has model.load_in_4bit true and CUDA is available.

• CR-A14-015 (code, Medium) — Wizard _prompt_choice silently falls back to the default on invalid input; typing 'high-risk' selects minimal-risk and skips strict-tier coercion
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator who types the tier name believes a high-risk system is configured while the file records minimal-risk.
  Where. forgelm/wizard/_io.py:124-145, forgelm/wizard/_collectors.py:707-713, forgelm/wizard/_collectors.py:52-58, forgelm/wizard/_orchestrator.py:1239-1246
  Recommended fix. Re-prompt on invalid input (or echo the chosen option) and add unit tests for default, out-of-range, non-numeric and navigation tokens.

• CR-A14-018 (code, Medium) — Quickstart training-failure envelope drops the cause: 'Training subprocess failed with exit code N' is the only error text
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A CI consumer branches on the envelope and gets no actionable reason. A typo in --dataset is classed as a runtime error (2) rather than caller input (1), so retry logic built on the exit-code table retries a run that can never succeed.
  Where. forgelm/cli/subcommands/_quickstart.py:142-148, forgelm/cli/subcommands/_quickstart.py:201-215, forgelm/quickstart.py:248-249, docs/usermanuals/en/reference/json-output.md:182
  Recommended fix. Validate that a local --dataset path exists in run_quickstart, raising FileNotFoundError, which the dispatcher already maps to exit 1. In JSON mode, run the child with --output-format json and stdout captured, then nest or relay the child's envelope (its error, reverted and awaiting_approval fields) in the parent's envelope instead of a generic string.

• CR-A14-019 (code, Medium) — Wizard inline ingest never offers secrets masking, so credentials reach the training JSONL (and the wizard proceeds past a CRITICAL secrets audit)
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Operator points the wizard at an engineering wiki or support-ticket directory, accepts the recommended masking, trains. Credentials land in the SFT data and the model.
  Where. forgelm/wizard/_byod.py:109-114 (ingest_path called without secrets_mask; default False at forgelm/ingestion.py:2227), forgelm/wizard/_byod.py:109-114, forgelm/wizard/_byod.py:146-194 (_offer_audit_for_jsonl returns only whether the audit ran, never the verdict), forgelm/ingestion.py:2129-2132
  Recommended fix. Ask a second prompt (or a single 'mask PII + secrets', default yes) and pass secrets_mask=True; print pii/secrets redaction counts and files_skipped in the summary.

• CR-A14-021 (code, Medium) — Wizard can emit monitoring.enabled: true with no endpoint; the manifest builder then writes endpoint '$None' into compliance_report.json and annex_iv_metadata.json
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator enables monitoring in the wizard and either presses Enter at the endpoint prompt or types 'env:datadog_url'. The wizard prints 'left unset' and continues. The saved YAML validates. After training, the Annex IV §6 artefact handed to auditors names the monitoring endpoint '$None', which reads as an env-var reference to a variable called None rather than 'no endpoint configured'.
  Where. forgelm/wizard/_collectors.py:889-917, forgelm/wizard/_collectors.py:97, forgelm/config.py:1279-1298, forgelm/compliance.py:1582-1589
  Recommended fix. In the manifest, emit endpoint only when one is set: f"${endpoint_env}" only when endpoint_env is truthy, otherwise null or a 'not configured' marker. In the wizard, re-prompt on an invalid env: reference (as _parse_webhook_value does) or warn that monitoring has no endpoint. Optionally add a MonitoringConfig validator or warning for enabled with neither field. Add a manifest-level test.

• CR-A14-022 (code, Medium) — Quickstart picks the model for the largest card, but training spreads it over every visible GPU, so a small second card gets most of the 7B model plus the loss and the load or step fails
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. Workstation with a 24 GB compute GPU as cuda:0 and an 8 GB (or 4-6 GB display) GPU as cuda:1, no CUDA_VISIBLE_DEVICES set. forgelm quickstart customer-support reports 'available VRAM 22.4 GB >= primary requirement' and picks Qwen2.5-7B-Instruct. The child loads it with device_map='auto': on the 4 GB card, lm_head goes to CPU and from_pretrained raises the bitsandbytes ValueError (exit 2). On the 6-8 GB card the load succeeds, but the first steps put about 4.75 GiB of weights plus the logits a…
  Where. forgelm/quickstart.py:203-234, forgelm/quickstart.py:181, forgelm/cli/subcommands/_quickstart.py:131, forgelm/cli/subcommands/_quickstart.py:142
  Recommended fix. Make sizing and placement agree. Either (a) have the quickstart-generated config (or _device_map_for when no distributed strategy is set) place the 4-bit model on a single device, e.g. {'': }, or set an explicit max_memory, so the max-card decision is the one that runs; or (b) size on the balanced reality (the smallest card's share, or the sum with the last-device rem…

• CR-A14-023 (code, Medium) — Selecting GaLore forces lora.method='lora' and load_in_4bit=False, so an Enter-only --wizard-start-from rerun of a GaLore+DoRA/QLoRA config silently drops DoRA and 4-bit loading
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs · already tracked: KNOWN-OPEN CLI-09 — docs/roadmap/phase-16-trust-surface-hardening.md:502-503,512-513 (S7 open; unit id inferred)
  Failure scenario. An operator iterates on a GaLore+DoRA QLoRA config with --wizard-start-from and presses Enter at every prompt. The saved YAML now trains plain LoRA on a full-precision base. Memory rises and the run may OOM, and the adapter method changed without any notice.
  Where. forgelm/wizard/_collectors.py:1088-1092, forgelm/wizard/_collectors.py:1113-1117, forgelm/wizard/_orchestrator.py:425, forgelm/model.py:609
  Recommended fix. Have the GaLore choice keep the loaded lora.method and load_in_4bit, or prompt for them, instead of overwriting them. Fix the 'dummy, ignored' comment. Fix the label and tutorial text together with M-X-014.

• CR-A14-024 (code, Medium) — Choosing-a-trainer page claims customer-support/code-assistant quickstart templates run SFT → DPO / SFT → ORPO
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A user picks customer-support expecting a helpful-and-safe preference-aligned bot (SFT → DPO) and ships an SFT-only model.
  Where. docs/usermanuals/en/concepts/choosing-trainer.md:78-79, docs/usermanuals/tr/concepts/choosing-trainer.md:78-79, forgelm/quickstart.py:68-91, forgelm/templates/customer-support/config.yaml:26
  Recommended fix. Change the table's trainer sequence to 'SFT only' for both rows (EN + TR), or ship pipeline variants with preference seed data.

• CR-A14-025 (code, Medium) — gradient_checkpointing cannot be overridden, and fit-check/troubleshooting recommend a config key ForgeConfig rejects
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN CORE-09 — docs/roadmap/phase-16-trust-surface-hardening.md:113,532-534,545 (S8 open; C-9)
  Failure scenario. On a CUDA host, --fit-check reports OOM/TIGHT using a no-checkpointing activation estimate, although training will run with checkpointing on. It tells the operator to add a key that makes the config fail validation (exit 1). An operator following troubleshooting.md gets the same rejection.
  Where. forgelm/templates/LICENSES.md:31-34, forgelm/trainer.py:514, forgelm/fit_check.py:359, forgelm/fit_check.py:450-454
  Recommended fix. In fit_check, derive grad_ckpt from the trainer's real rule (torch.cuda.is_available() or deepspeed) and drop the invalid recommendation. Fix the troubleshooting docs and the LICENSES.md 'override' sentence, or add a real field.

• CR-A14-026 (code, Medium) — GRPO generation_batch_size divisibility by num_generations is checked nowhere (wizard, schema, dry-run, docs), so TRL rejects emitted configs after model load
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A user copies the GRPO quick example from the user manual, or answers the wizard with batch size 1 and the default 4 generations, and runs it on a single GPU. Training aborts with exit 2 and a TRL-internal message. Neither the manual nor the wizard told them the rule.
  Where. docs/usermanuals/en/training/grpo.md:49-57 (the whole example block; ga is not set, so it defaults to 2), docs/usermanuals/tr/training/grpo.md:51,53, forgelm/wizard/_collectors.py:419-426 (grpo_num_generations prompt 2..64, default 4), forgelm/wizard/_orchestrator.py:544-551 (batch size prompt 1..512)
  Recommended fix. State the constraint in the configuration reference row for grpo_num_generations (EN+TR) and in the GRPO manual. Fix the quick example (for example set gradient_accumulation_steps: 8 or 16 to match yaml-templates.md). In the wizard's GRPO branch, re-prompt or auto-adjust gradient_accumulation_steps when bs*ga % ng != 0. Have fit-check flag the GRPO divisibility and model the num_generations gener…

• CR-A14-027 (code, Medium) — Wizard BYOD ingest silently overwrites any existing file at the typed or default output path, including the operator's own JSONL or a source document
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Operator keeps a curated data/train.jsonl, runs the wizard, accepts or types a path equal to it (or the default name from a prior run) and loses it with no warning; or types a path under the input tree that coincides with a source .md/.txt and destroys the source document.
  Where. forgelm/wizard/_byod.py:91-94, forgelm/wizard/_byod.py:108-114, forgelm/wizard/_byod.py:136, forgelm/ingestion.py:2321-2332
  Recommended fix. In the wizard, if out_path exists: print that it will be replaced and ask a default-No confirmation (or choose a new name); refuse an out_path that equals a file inside the input tree. Add tests with a real ingest_path and a pre-existing target.

• CR-A14-028 (code, Medium) — The wizard, the site wizard and quickstart --model offer Phi-3, but the LoRA target presets do not fit its fused projections: the default fails with exit 2 after the model loads, and 'extended'/'full…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator picks Phi-3-mini, offered as the smallest model, in the CLI or site wizard and presses Enter at 'Target modules'. The YAML is written, validation passes and the wizard exits 0. Training then downloads and loads 7.6 GB of weights and dies with exit 2 at PEFT injection. An operator who instead picks 'full (all linear layers)' gets a successful run (exit 0) that adapted only 2 of the 7 named projection types, with no attention-input or gate/up adapters.
  Where. forgelm/wizard/_state.py:132-145, forgelm/wizard/_orchestrator.py:362-382, forgelm/wizard/_orchestrator.py:426, forgelm/model.py:535
  Recommended fix. Resolve target_modules per architecture when the model's config.json is reachable (local dir or cache), in the wizard, quickstart and dry-run. After get_peft_model, compare the requested names with the injected ones and warn, or fail with exit 1, on a partial match. Either drop Phi-3 from POPULAR_MODELS and the site presets or give it a [qkv_proj, o_proj, gate_up_proj, down_proj] preset. Map PEFT…

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

---

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}