[Wave 3] Medium — Module sweep: Training orchestration, multi-stage pipelines, resume & no-train modes (part 1/2) #571

Wave 3 · 21 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A04-011 (code, Medium) — --offline with an uncached model prints a traceback with a 'could not connect' message instead of the documented actionable error
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Operator on an air-gapped host sees a network error and starts debugging connectivity instead of re-staging the cache.
  Where. forgelm/cli/_training.py:156-160 (the broad except Exception that surfaces the raw OSError), docs/usermanuals/en/operations/air-gap.md:50-55 (the fabricated ERROR block; the TR mirror has it at line 53), docs/guides/air_gap_deployment.md:139
  Recommended fix. Catch the offline cache-miss OSError in model loading and raise ConfigError/EXIT_CONFIG_ERROR naming the model and the cache-models command.

• ↳ same root cause: DOC-D3B02-027 (docs, Medium) — docs/usermanuals/en/operations/air-gap.md: broken example — Uncached model under --offline fails with 'ERROR: model ... not in local cache. Pre-cache from a connected machine ...'; --offline enforces…
  severity Medium · already tracked: KNOWN-OPEN CLI-04|CLI-02 docs/roadmap/phase-16-trust-surface-hardening.md:483-489,509-510
  Where. docs/usermanuals/en/operations/air-gap.md:50-55; docs/guides/air_gap_deployment.md:139 · type: broken example
  The document says. Uncached model under --offline fails with 'ERROR: model ... not in local cache. Pre-cache from a connected machine ...'; --offline enforces local_files_only=True
  What the code does. Raw traceback with transformers' 'couldn't connect' text; message string not in the codebase; local_files_only is never passed.
  Code references. forgelm/cli/_training.py:136-160
  Proposed correction. Show the real output or implement the message (F9).
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• CR-A04-012 (code, Medium) — Pipeline --stage seeding and the collision scan hardcode 'final_model', ignoring training.final_model_dir
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Stages set final_model_dir: promoted. After a full run, --stage dpo_stage exits 1 because output_dir/final_model is missing.
  Where. forgelm/cli/_pipeline.py:634, forgelm/cli/_pipeline.py:1010 (confirmed), forgelm/trainer.py:1590-1593, forgelm/cli/_pipeline.py:1664 (confirmed)
  Recommended fix. Use merged.training.final_model_dir in both places.

• CR-A04-014 (code, Medium) — --resume-from re-trains earlier non-skip-eligible stages silently; their new output replaces the artefact a later skipped stage was trained from
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A 3-stage chain completes. The operator deletes stage 1's output to free disk, then runs --resume-from s3_sft. Stage 1 is re-trained (compute wasted; with a gate it halts the resume at exit 4). Stage 2 is skipped. Stage 3 chains from stage 2. The state records stage 2 as chained from the new stage-1 model, which it was not trained on.
  Where. forgelm/cli/_pipeline.py:569-580, forgelm/cli/_pipeline.py:943-956, forgelm/cli/_pipeline.py:943-945, forgelm/cli/_pipeline.py:220-222
  Recommended fix. Refuse with EXIT_CONFIG_ERROR (listing the stages) when a pre-target stage is not skip-eligible, or require an explicit flag to re-run it. Alternatively invalidate (re-run) every later pre-target stage whose input changed. At minimum, log at WARNING which earlier stages will be re-executed, and document the behaviour.

• CR-A04-015 (code, Medium) — --stage bypasses the stale-config guard and the pipeline.force_resume audit event, re-stamping earlier stages' records with the new config hash
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Run v1 to completion. Change stage-1 hyperparameters (v2). --resume-from s2 is correctly refused. --stage s2 under v2 succeeds and rewrites the state hash to v2 with the v1-trained s1 carried as completed. --resume-from s2 under v2 then exits 0 and skips the v1-trained s1. Zero pipeline.force_resume events appear, and the Annex IV manifest attributes the whole chain to v2.
  Where. forgelm/cli/_pipeline.py:445-473, forgelm/cli/_pipeline.py:1305-1337
  Recommended fix. Apply _validate_resume_state's hash check (with the --force-resume escape and audit event) before carrying history in _init_state_preserving, or record each carried stage's original config hash. Also carry started_at, finished_at, duration_seconds and exit_code.

• CR-A04-017 (code, Medium) — Training-time ConfigError is reported in text mode as a bare 'Configuration error.'; the actionable reason is dropped
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An unsloth user with lora.method=pissa gets exit 1 and 'Configuration error.' in the terminal or CI log, with no hint which field is wrong.
  Where. forgelm/cli/_training.py:170, forgelm/cli/_training.py:163-172 (confirmed), forgelm/model.py:348, forgelm/cli/_training.py:21-36 (confirmed)
  Recommended fix. Use log_msg=f'Configuration error: {e}'.

• CR-A04-018 (code, Medium) — --benchmark-only default output dir uses os.path.dirname, so a trailing-slash model path writes results inside the model directory
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Benchmarking ./final_model/ adds a file to the model directory, which changes the artefact set that integrity tooling or later deployment steps see.
  Where. forgelm/cli/_no_train_modes.py:88 (confirmed), docs/usermanuals/en/evaluation/benchmarks.md:135, forgelm/verify/_model_integrity.py:197-205, forgelm/config.py:878-879
  Recommended fix. Normalise with os.path.normpath / rstrip(os.sep) before dirname, and never default into the model dir.

• CR-A04-019 (code, Medium) — Pipeline refusals and aborted runs leave no pipeline audit event, and the catalogued pipeline.failed is never emitted by the orchestrator
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An auditor reconstructing why a chain never resumed, or why it stopped mid-run, finds no pipeline-level record. The state file shows a stage stuck in 'running'.
  Where. forgelm/cli/_pipeline.py:1083-1086, forgelm/cli/_pipeline.py:1292-1315, forgelm/cli/_pipeline.py:1776-1788, docs/reference/audit_event_catalog.md:38
  Recommended fix. Emit pipeline.resume_refused, with a reason field, for every resume refusal. Emit pipeline.failed from the run_pipeline_from_args catch, best-effort, once state exists.

• CR-A04-021 (code, Medium) — --benchmark-only applies config.model.trust_remote_code to whatever base model the evaluated adapter's adapter_config.json names
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator who enabled trust_remote_code for their own base model benchmarks a downloaded adapter whose adapter_config points at a different repo. That repo's code runs.
  Where. forgelm/cli/_no_train_modes.py:48, forgelm/cli/_no_train_modes.py:80-86 (confirmed), forgelm/cli/_no_train_modes.py:80, forgelm/cli/_no_train_modes.py:85
  Recommended fix. When trust_remote_code is true and base_path differs from config.model.name_or_path, refuse or warn loudly. Alternatively, apply trust_remote_code only when the two match.

• CR-A04-023 (code, Medium) — --resume (Trainer checkpoint resume) hard-fails on torch 2.4/2.5, which the declared range allows
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. A user on torch 2.5 (allowed by the floor) trains normally, is interrupted, and runs --resume. It fails every time, and the error text does not mention ForgeLM's declared floor.
  Where. forgelm/cli/_training.py:144-149, forgelm/trainer.py:1231, pyproject.toml:46
  Recommended fix. Raise the torch floor to >=2.6.0 (consistent with transformers' own gate), or add a --resume preflight that checks the torch version and gives an actionable message. Document the constraint.

• CR-A04-024 (code, Medium) — Several gate and export surfaces required by the standard emit no audit event
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Where. forgelm/cli/_no_train_modes.py:60-125, forgelm/cli/_no_train_modes.py:248-262, forgelm/cli/subcommands/_safety_eval.py:216-221, docs/standards/logging-observability.md:110-116
  Recommended fix. Either emit the catalogued *.evaluation_completed / compliance.artifacts_exported events from these surfaces (into --output-dir), or narrow the standard's MUST list to the training path and fix its examples.

• CR-A04-029 (code, Medium) — --stage X chains from /final_model on disk presence alone: a gated, rejected, failed, interrupted or auto-reverted predecessor is consumed silently, its status is rewritten to skipped_by_filter…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Where. forgelm/cli/_pipeline.py:633-642 (isdir-only seed; finding cites 620-643), forgelm/cli/_pipeline.py:445-473, forgelm/cli/_pipeline.py:859-865 (downgrade to skipped_by_filter), forgelm/cli/_pipeline.py:1125-1128
  Recommended fix. In _resolve_stage_filter_setup, when a prior state file exists, gate the seed on stages[X-1]. Refuse without --input-model, with the same audited pipeline.resume_refused-style event, when the status is gated_pending_approval, failed, running, pending or skipped_due_to_prior_revert. Accept a gated stage only with the same proof --resume-from uses, preferably corroborated by the stage audit log's h…

• CR-A04-030 (code, Medium) — Pipeline orchestrator runs once per rank: N pipeline_run_ids per launch, state file and Annex-IV manifest from different ranks, duplicate same-id records on resume
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An auditor reconciles pipeline_manifest.json with pipeline_state.json and the audit log. The state names a pipeline_run_id that the manifest does not. The log shows two complete pipelines for one launch, and after a resume shows each stage started twice under the same id.
  Where. forgelm/cli/_pipeline.py:166-175, forgelm/cli/_pipeline.py:1208-1226, forgelm/cli/_pipeline.py:1384-1420, forgelm/cli/_pipeline.py:1422-1462
  Recommended fix. Run the orchestrator's persistence, audit and webhooks on rank 0 only, with a pipeline_run_id broadcast to every rank. Reconcile pipeline.md's concurrency limitation with the documented multi-GPU launch.

• CR-A04-033 (code, Medium) — A resumed run replaces the previous run's compliance bundle in the shared output_dir: resuming a completed run re-certifies it under a new run_id with distorted metrics, and resuming an approval-pend…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. scenF: run 1 (same cfg, max_steps 4) completes with EXIT=0, run_id fg-defc98395d42, train_loss 11.936, epoch 0.6667. A CI retry, per cicd_pipeline.md:274 'Use --resume', reruns the same config with --resume. The result is EXIT=0 with run_id fg-fa10a99281b5. checkpoint-5 appears (global_step 5, max_steps 4). final_model/adapter_model.safetensors sha256 24835281… is identical to run 1. compliance_report.json, training_manifest.yaml, annex_iv_metadata.json and the model card are rewritten: train_l…
  Where. forgelm/cli/_training.py:146, forgelm/compliance.py:2252-2261, forgelm/compliance.py:2283-2287, forgelm/compliance.py:2178-2191
  Recommended fix. Make resume of a completed checkpoint (global_step >= max_steps, or a pipeline.completed already logged for its lineage) a no-op or a refusal with an explicit message. Write per-run bundles (compliance/<run_id>/ with a pointer file), or archive the previous bundle instead of deleting the .prev backups. Put run_id into training_manifest.yaml. Refuse --resume while the output_dir holds a pending hu…

• CR-A04-034 (code, Medium) — Documented early detection of cross-node batch-size mismatch via --dry-run does not exist
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Where. forgelm/cli/_dry_run.py:40-89, docs/usermanuals/en/training/distributed.md:144-146, docs/usermanuals/tr/training/distributed.md:144-146
  Recommended fix. Rewrite the pitfall so it tells operators to ship one config file to every node. Do not claim a check that ForgeLM does not perform.

• CR-A04-035 (code, Medium) — --merge writes into an existing output_dir with no overlap or emptiness check: it overwrites the base/adapter it merged from, and leaves stale adapter/tokenizer files that change what reloads return
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator writes merge.output_dir: ./models/base (the same local directory as model.name_or_path) to 'update the base'. The original weights are silently replaced, CI sees exit 0, and a re-run or a retried CI job merges the adapters in a second time.
  Where. forgelm/cli/_no_train_modes.py:136-150 (exit mapping at :176-177), forgelm/merging.py:74-77, forgelm/merging.py:98-100 (confirmed), forgelm/config.py:194 (validator at :225-246 does not check output_dir)
  Recommended fix. Before loading anything in _run_merge (or in a MergeConfig/ForgeConfig validator), resolve realpath for output_dir, model.name_or_path and each merge.models[].path whenever they exist locally. Refuse with EXIT_CONFIG_ERROR when output_dir equals, contains or lies inside any of them, or when output_dir already holds model weights, unless an explicit overwrite opt-in is given. Document the rule in…

• CR-A04-036 (code, Medium) — SFT stage's base model (weights + LoRA) survives into the next in-process pipeline stage as cyclic garbage; no teardown/gc.collect between stages
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Single-GPU SFT -> DPO (the documented canonical chain, docs/guides/pipeline.md): stage 1 trains on CUDA. Stage 2 calls get_model_and_tokenizer while the stage-1 base model is still on the GPU, so two base models are resident through stage 2's load, trainer construction and first forward/backward/optimizer step, which is where the stage-2 peak occurs. A 7B bf16 base is ~14-15 GB extra (QLoRA 4-bit ~4-5 GB; a 70B 4-bit base ~35-40 GB). A config that fits when run one stage at a time (--stage) OOM…
  Where. forgelm/cli/_pipeline.py:1564-1567, forgelm/trainer.py:739-743, forgelm/trainer.py:1267-1268, forgelm/safety/_generate.py:143-170
  Recommended fix. At the stage boundary in _invoke_trainer, add a finally that drops trainer, model and dataset, then runs gc.collect() and a guarded torch.cuda.empty_cache() (only when torch.cuda.is_available()), and add the same helper at the end of ForgeTrainer.train() for library callers. Optionally also remove the patched instance attribute (model.__dict__.pop('forward', None) on the base model) after train…

• CR-A04-037 (code, Medium) — A plain re-invocation of a pipeline config ignores the existing pipeline_state.json (completed, gated or in-progress) and overwrites the prior run's state and manifest
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. An operator or a scheduled CI job re-runs forgelm --config pipeline.yaml on a pipeline whose earlier run completed and shipped. Possible reasons: to reproduce it, a retried job, or a 'just run it again' after a crash. With no prompt, the earlier run's chain-level Annex IV manifest is replaced by the new run's. A crash anywhere in the new run leaves a failed manifest beside the old run's intact model, and verify-annex-iv --pipeline reports OK for whatever the newest run was. For an interrupted…
  Where. forgelm/cli/_pipeline.py:408 (existence = self._load_state_file()), forgelm/cli/_pipeline.py:408 (confirmed), forgelm/cli/_pipeline.py:442 (the unconditional _init_state fall-through; 443 is blank), forgelm/cli/_pipeline.py:1208-1226 (_init_state mints a new pipeline_run_id via _generate_run_id())
  Recommended fix. In _prepare_resume_or_init_state, when existing is not None and neither flag is set, refuse with EXIT_CONFIG_ERROR (or require an explicit --new-run / --force-new-run flag) and name the prior run id, its final_status and the suggested --resume-from <stopped_at>. When a new run is allowed, first archive the prior pipeline_state.json and pipeline_manifest.json under compliance/runs/<prior_run_i…

• CR-A04-038 (code, Medium) — Concurrent or overlapping invocations of one pipeline config are neither locked nor detected; the surviving manifest certifies a chain built from another run's stage artefacts and verify-annex-iv --p…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. Two CI jobs, such as a scheduled run and a manual re-trigger, or two operators, start the same pipeline config at overlapping times. Alternatively an operator follows the guide and changes only pipeline.output_dir. Both exit 0. The Annex IV pipeline manifest that survives says stage 1 produced the model stage 2 was trained from, but stage 1's weights and evidence on disk come from the other run, and verify-annex-iv --pipeline confirms it with exit 0.
  Where. forgelm/cli/_pipeline.py:408, forgelm/cli/_pipeline.py:441-443, forgelm/cli/_pipeline.py:213-232, forgelm/cli/_pipeline.py:248-264
  Recommended fix. Take an exclusive lock (flock on <pipeline.output_dir>/.pipeline.lock, as _approve.py:51-89 already does for approvals) for the whole run, and refuse when the state file is in_progress under a live owner (record pid/host/start in the state). Extend the run's collision pre-flight to lock or check every stage's training.output_dir, not only pipeline.output_dir. Bind evidence to the run by recording…

• CR-A04-040 (code, Medium) — A preempted pipeline stage cannot be resumed from its checkpoint: --resume-from restarts the stage at step 0, and the new attempt's checkpoint rotation deletes the interrupted attempt's checkpoints
  severity Medium · reachability default · intent intended-and-acceptable · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A 20-hour DPO stage on spot capacity is reclaimed at hour 19. The operator follows the docs: --resume is ignored, and --resume-from retrains the stage from step 0 while deleting the checkpoints that could have recovered it.
  Where. forgelm/cli/_pipeline.py:1566-1567, forgelm/cli/_pipeline.py:526, docs/guides/pipeline.md:136-139, docs/guides/enterprise_deployment.md:159-171
  Recommended fix. When --resume-from names a stage recorded as 'running' or 'failed' whose output_dir holds checkpoints from the same pipeline_run_id, pass the latest complete checkpoint to trainer.train(resume_from_checkpoint=...), or honour --resume on pipeline configs for the resumed stage. Record the decision in a pipeline.stage_resumed audit event. Otherwise, document that pipeline resume is stage-granular an…

• CR-A04-041 (code, Medium) — Overlap pre-flight compares realpaths for equality only: a stage output_dir equal to or inside an earlier stage's final_model passes, and stages then overwrite and delete each other's artefacts, incl…
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Operator writes stage 2 training.output_dir: ./run/sft/final_model/dpo, or ./run/sft/final_model, to keep the chain together. Dry-run passes and the pipeline exits 0. The SFT deliverable now contains DPO checkpoints, a DPO audit log and a compliance bundle, and its model card has been replaced by TRL's generic card. Later, forgelm --config p.yaml --stage sft after a gate tweak triggers auto-revert, which silently erases the DPO stage's audit log and compliance evidence. The state file sti…
  Where. forgelm/cli/_pipeline.py:1003, forgelm/cli/_pipeline.py:1010, forgelm/cli/_pipeline.py:1083, forgelm/trainer.py:354
  Recommended fix. Extend the scan to containment. Reject (or at least warn on) any stage output_dir that equals or is nested inside another stage's output_dir/<final_model_dir>, and any earlier stage output_dir nested inside a later one's final_model, using commonpath on realpaths. Use each stage's final_model_dir instead of the literal 'final_model'. Document the rule next to the existing equality rule.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}