[Wave 4] Low/info — Module sweep: Training orchestration, multi-stage pipelines, resume & no-train modes (part 2/3) #713

Wave 4 · 20 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A04-063 (code, Low) — Pipeline guide lists 7 pipeline.* events; the orchestrator emits 8 (pipeline.resume_refused undocumented there)
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)
  Where. docs/guides/pipeline.md:204-215, forgelm/cli/_pipeline.py:562, forgelm/cli/_pipeline.py:1389-1392, docs/reference/audit_event_catalog.md:43
  Recommended fix. Add pipeline.resume_refused to the guide's event list (EN and TR) and to the _audit_event docstring.

• CR-A04-064 (code, Low) — Training error envelope and audit record carry an empty error string for message-less exceptions
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end · already tracked: PARTIALLY-FIXED C33-TRAIN-JSON-CONTRACT — docs/roadmap/phase-16-trust-surface-hardening.md:291,308 (S2 done); CHANGELOG.md:102-105
  Failure scenario. The generation-mode safety crash produces {"success": false, "error": ""}, so a CI consumer and an auditor see no cause at all.
  Where. forgelm/cli/_training.py:173-180
  Recommended fix. Use f"{type(e).name}: {e}" in the envelope and in the pipeline.failed audit event.

• CR-A04-065 (code, Low) — Pipeline pre-flight skips per-stage merge/validation errors, so a later stage's invalid config is rejected only after earlier stages train
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced with stubs
  Failure scenario. The operator runs a two-stage high-risk pipeline (SFT then DPO) without first running --dry-run. The DPO stage's evaluation override omits safety. Config-load passes and the SFT stage trains for hours and writes its artefacts. Only then does the run stop with exit 1 on the DPO stage. If SFT instead hits its human-approval gate (exit 4), the DPO stage's defect stays hidden until the operator resumes.
  Where. forgelm/cli/_pipeline.py:1028-1044, forgelm/cli/_pipeline.py:1083-1086, forgelm/cli/_pipeline.py:1503-1512 (confirmed), forgelm/cli/_pipeline.py:1604-1605
  Recommended fix. In PipelineOrchestrator.run, treat a stage merge failure that is a ConfigError as a pre-flight failure, as dry_run() already does: return EXIT_CONFIG_ERROR before pipeline.started is emitted and before any stage trains. Alternatively, reword configuration.md:342/356 to say that per-stage violations are caught by --dry-run or when the stage starts.

• CR-A04-066 (code, Low) — --resume auto treats directories with non-ASCII decimal digits as numbered checkpoints, so they can win as 'latest'
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
  Where. forgelm/cli/_resume.py:44-48
  Recommended fix. Use re.fullmatch(r'[0-9]+', suffix) (or suffix.isascii() and suffix.isdecimal()).

• CR-A04-067 (code, Low) — Pipeline state stores cwd-relative stage paths: --resume-from/--stage from another directory silently re-trains or chains from that directory's models
  severity Low · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A scheduler resumes a multi-GPU-hour pipeline from a different working directory. Completed stages run again from scratch, and the manifests cannot show which ./s1/fm was consumed.
  Where. forgelm/cli/_pipeline.py:569-580 (confirmed), forgelm/cli/_pipeline.py:1644, forgelm/cli/_pipeline.py:226-231 (confirmed), forgelm/cli/_pipeline.py:634-643 (confirmed)
  Recommended fix. Store absolute (realpath) output_model/input_model in pipeline state and manifests. On resume, warn or refuse when a 'completed' stage's recorded output is missing rather than silently re-running it.

• CR-A04-068 (code, Low) — Model-load failures that are operator input errors exit 2 (retry class) with an unclassified envelope and misleading messages; the ImportError for bitsandbytes carries a non-ForgeLM hint
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced (plumbing only)
  Failure scenario. A CI orchestrator that retries exit 2 loops on a misspelled model id; the operator reading the log for a local-path typo is told about repo-id syntax instead of 'path does not exist'.
  Where. forgelm/cli/_training.py:141, forgelm/cli/_training.py:173-180, forgelm/model.py:291-295, forgelm/model.py:454-465
  Recommended fix. Pre-check local-path-looking model.name_or_path (os.path.isabs or ./ prefix) and wrap OSError and ImportError from the load into a ConfigError with the model id, the field name and the forgelm[qlora] extra; raise the qlora lower bound to 0.46.1 or document it; add an error class/stage key to the training error envelope.

• CR-A04-069 (code, Low) — An incompatible checkpoint on resume (lora.r or target_modules changed) exits 2, the retry class, with a raw PyTorch message that names neither --resume nor the changed field
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. scenC, text mode: seed checkpoint trained with r=4, cfg2.yaml with lora.r 8, --resume. Result: EXIT=2. stderr shows '[ERROR] forgelm.trainer: Training pipeline failed.' then 'RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM: size mismatch for base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight: copying a param with shape torch.Size([4, 128]) from checkpoint, the shape in current model is torch.Size([8, 128]).' repeated under '[ERROR] forgelm.cli: Traini…
  Where. forgelm/cli/_training.py:141-149, forgelm/cli/_training.py:173-180, forgelm/trainer.py:1231-1235, forgelm/trainer.py:1693-1709
  Recommended fix. Before model load, compare the checkpoint's adapter_config.json (r, lora_alpha, target_modules, base_model_name_or_path) with config.lora and config.model, and raise ConfigError (exit 1) with a message naming the differing fields and the checkpoint path. This reuses the check proposed in the compatibility finding. As a fallback, catch the state_dict or optimizer-group mismatch around trainer.trai…

• CR-A04-070 (code, Low) — --benchmark-only with an explicit evaluation.benchmark.output_dir equal to the model directory writes benchmark_results.json into the artefact; verify-integrity then exits 6 (extends M-A04-026)
  severity Low · reachability documented-opt-in · intent intended-but-harmful · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. An operator stores baseline scores next to the weights (output_dir: <final_model>). The next CI verify-integrity step raises the exit-6 tamper alarm on an untouched model.
  Where. forgelm/cli/_no_train_modes.py:88, forgelm/benchmark.py:110-113, forgelm/verify/_model_integrity.py:197-206, forgelm/config.py:878-880
  Recommended fix. Refuse or warn when realpath(output_dir) equals or lies inside the evaluated model directory (or inside any directory holding model_integrity.json). Fix the default and its description together with M-A04-026.

• CR-A04-071 (code, Low) — Pipeline docs never say that every pipeline path resolves against the invoking cwd; the cross-directory refusal prints a relative path and says 'Run the pipeline once first'
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end
  Failure scenario. Following the guide, an operator runs forgelm --config /srv/cfg/pipeline.yaml from /srv/job1 and, after a crash, runs forgelm --config /srv/cfg/pipeline.yaml --resume-from dpo_stage from their home directory. They get exit 1 with 'no pipeline_state.json found at ./pipeline_run/pipeline_state.json. Run the pipeline once first.', then follow the advice and re-run the whole chain from the home directory.
  Where. docs/guides/pipeline.md:56-84, docs/guides/pipeline.md:137-140, docs/guides/pipeline.md:157-162, docs/guides/pipeline.md:192
  Recommended fix. Add a Limitations bullet to pipeline.md (and its TR mirror and the user-manual page) saying that relative paths resolve against the invoking cwd, that resume/--stage/approve must run from the same directory, and that absolute paths are recommended. Make the refusal print os.path.abspath(state_file) and os.getcwd(), and drop or qualify 'Run the pipeline once first'.

• CR-A04-072 (code, Low) — Status transitions outside _execute_one_stage leave attempt-scoped fields stale: a chain-broken skipped stage keeps exit_code 0, gate_decision 'passed', metrics and output_model; an approved gated st…
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. Pipeline completes all 3 stages. A later --resume-from grpo_stage re-runs sft_stage, which fails (exit 2). The manifest rows for dpo_stage and grpo_stage now read status=skipped_due_to_prior_revert, exit_code=0, gate_decision=passed, metrics={eval_loss: 1.0}, output_model=./out/stage2_dpo/final_model, together with skipped_reason 'Stage sft_stage failed with a training error; downstream stages did not run.' A reviewer reading the Annex IV manifest gets contradictory per-stage evidence.
  Where. forgelm/cli/_pipeline.py:873-877, forgelm/cli/_pipeline.py:540-541, forgelm/cli/_pipeline.py:744-777
  Recommended fix. Factor the attempt-field reset at 757-776 into a helper. Call it in the chain_broken skip branch (keep only status and skipped_reason), and in the approved-gate flip set exit_code to EXIT_SUCCESS (or None) and record gate_decision as 'approved'. Add a resume test that asserts the fields of skipped stages.

• CR-A04-073 (code, Low) — Gap rows G-A04-01..08: verdicts and fix-pinning (all eight confirmed as real gaps)
  severity Low · reachability unreachable · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
  Where. tests/test_pipeline_orchestrator.py:103-131 (fake trainer; finding cited 95-135), tests/test_pipeline_cli.py:393, tests/test_no_train_modes.py:16-39 (finding cited 12-25 / 33-39), forgelm/cli/_no_train_modes.py:131-177 (_run_merge; finding cited 134)
  Recommended fix. Add per-gap tests with the fix of each finding: real-trainer two-stage test (G-01), JSON envelope on early returns and dry-run (G-02), _run_merge body incl. exit 2 and trust_remote_code forwarding (G-03), benchmark failing gate, default output dir with trailing slash, NaN scores (G-04), --resume and --offline on pipeline configs (G-05), reject-then-removed staging and audit-event based approval (…

• CR-A04-074 (code, Low) — The abi_preflight_crashed envelope (exit 2) is missing from the locked JSON-output contract and uses non-standard field names
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)
  Failure scenario. A corrupted torch whose version access raises makes the preflight emit an error token that json-output.md never documents. A CI consumer written against the documented contract cannot anticipate the token or the exception_class/exception_message keys.
  Where. forgelm/cli/_training.py:75-93, docs/usermanuals/en/reference/json-output.md:111-133, docs/standards/error-handling.md:277-281
  Recommended fix. Document the abi_preflight_crashed envelope next to numpy_torch_abi_mismatch in json-output.md and its TR mirror. Consider renaming exception_class to the standard's error_type, or emitting both.

• CR-A04-075 (code, Low) — Merged model has no provenance or audit record; the only reachable compliance export describes a fictitious SFT/QLoRA training run and never mentions the merge
  severity Low · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced (plumbing only)
  Failure scenario. For a high-risk system, an auditor asks how the deployed model was produced. The audit log holds the specialists' training runs but nothing about the merge that created the deployed weights. The compliance bundle the operator generated for that config states the model is a 3-epoch QLoRA SFT fine-tune of the base, which is false: the deployed weights are a TIES/linear combination of other runs with operator-chosen weights, and they never passed an eval, safety or approval gate.
  Where. forgelm/cli/_no_train_modes.py:128-177, forgelm/cli/_no_train_modes.py:248-261, forgelm/merging.py:98-108, forgelm/compliance.py:1491-1540
  Recommended fix. Have _run_merge create an AuditLogger in merge.output_dir (or its parent) and emit catalogued events (e.g. model.merge_completed / model.merge_failed with method, inputs+weights, hyper-parameters, base, output_dir and an integrity hash), and fire the configured webhooks. Add a merge block to the training manifest when merge.enabled, or make --compliance-export refuse or label a merge-only config.…

• CR-A04-076 (code, Low) — The JSON envelopes of the no-train modes (--merge, --benchmark-only, --generate-data, --compliance-export, --fit-check) are absent from the locked json-output contract page
  severity Low · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN C33-CLI-ERROR-ENVELOPE — docs/roadmap/phase-16-trust-surface-hardening.md:505-506 (S7 open)
  Failure scenario. A CI consumer has no documented schema or exit mapping for forgelm --merge --output-format json. A future rename of num_models, or a fix to the blank failure fields, would ship without the MAJOR-bump discipline the page exists to enforce.
  Where. docs/usermanuals/en/reference/json-output.md:3, docs/usermanuals/en/reference/json-output.md:59, forgelm/cli/_no_train_modes.py:101-113, forgelm/cli/_no_train_modes.py:152-164
  Recommended fix. Add sections (EN and TR) for each no-train mode envelope with its exit mapping, and pin the key sets in tests/test_json_envelope_contract.py.

• CR-A04-077 (code, Low) — Training-crash JSON envelope omits run_id and config_hash although the run's audit log already has them
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. A run crashes mid-training (OOM, dataloader error). CI gets {"success": false, "error": "..."} with no run_id, so it cannot locate the pipeline.failed audit entry or tie the failure to the config that produced it.
  Where. forgelm/cli/_training.py:163-178, forgelm/trainer.py:1686-1710, forgelm/cli/_no_train_modes.py:104-113, forgelm/cli/_pipeline.py:141-142
  Recommended fix. Have the CLI keep the ForgeTrainer reference, or attach run_id/config_hash to the raised exception, and add both keys to the error payload whenever the audit logger was created.

• CR-A04-078 (code, Low) — --benchmark-only stdout example is not what the CLI prints
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · reproduced with stubs
  Failure scenario. An operator pipes stdout into jq to read the baseline and gets empty input.
  Where. docs/usermanuals/en/evaluation/benchmarks.md:82-84, forgelm/cli/_no_train_modes.py:101-120
  Recommended fix. Show the real output in the docs (log lines, or the --output-format json envelope).

• CR-A04-079 (code, Low) — Multi-stage orchestrator emits pipeline.completed for stopped_at_stage runs that exit 2/3, contradicting the catalog and audit-log manual ('returned exit code 0')
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A pipeline stage fails its gate and the CLI exits 3, yet the pipeline-level audit_log.jsonl contains pipeline.completed. A reader who follows the user manual records the run as having exited 0.
  Where. forgelm/cli/_pipeline.py:819-830 (confirmed), docs/usermanuals/en/compliance/audit-log.md:40 (confirmed), docs/reference/audit_event_catalog.md:37 (confirmed), docs/reference/audit_event_catalog.md:42
  Recommended fix. Reword audit-log.md:40 (EN+TR) and catalog row 37: single-stage means exit 0; the multi-stage orchestrator emits it on every terminal state, with final_status distinguishing completed from stopped_at_stage.

• CR-A04-080 (code, Low) — --input-model is never existence-checked: a mistyped local path becomes a Hub lookup (egress, typo-squat surface with inherited trust_remote_code) and exits 2 with a 'could not connect' or 'not a val…
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. Operator types --input-model runs/ckpt_stage1 instead of runs/ckpt-stage1. ForgeLM queries huggingface.co for repo runs/ckpt_stage1, leaking the local path name. If someone has registered that name, the stage trains on their weights, and with root trust_remote_code: true it runs their code. Otherwise CI gets exit 2 and retries.
  Where. forgelm/cli/_pipeline.py:606-607, forgelm/cli/_pipeline.py:1515-1526, forgelm/cli/_pipeline.py:1573-1576, forgelm/model.py:566-585
  Recommended fix. Treat --input-model as a path by default. When it looks like a filesystem path (contains os.sep, starts with '.' or '/', or does not exist and looks path-like), refuse with EXIT_CONFIG_ERROR before stage_started. Allow Hub ids only through an explicit form, or after an existence and shape check, and say so in the help text.

• CR-A04-081 (code, Low) — When a cached HF token file exists, a local-only run authenticates against huggingface.co (whoami) without being configured to
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Where. forgelm/cli/_training.py:136-137, forgelm/utils.py:21-45, forgelm/utils.py:21, forgelm/utils.py:48
  Recommended fix. Do not call login() just because a token file exists. huggingface_hub already picks up the stored token lazily for real Hub downloads. Validate only when auth.hf_token is configured, or skip validation (no whoami) and let the first real Hub request authenticate. Use huggingface_hub.get_token() rather than hard-coded paths if a lookup is kept.

• CR-A04-082 (code, Low) — --merge failure JSON envelope reports method "" and num_models 0 instead of the configured merge
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A CI job's merge fails. Its dashboard records method '' and 0 models, so the operator cannot tell from the envelope which merge was attempted, and an empty output_dir looks like 'nothing configured'.
  Where. forgelm/cli/_no_train_modes.py:152-164, forgelm/merging.py:29-36, forgelm/merging.py:103-112, forgelm/merging.py:66-67
  Recommended fix. Populate method, num_models and output_dir from the request in the failure return, or have _run_merge take these fields from config.merge rather than from the result.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}