[Wave 4] Low/info — Module sweep: Training orchestration, multi-stage pipelines, resume & no-train modes (part 1/3) #712

Wave 4 · 21 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A04-042 (code, Low) — --dry-run and doctor skip trainer-construction-time validation, so deterministic config failures surface only after the model loads
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A CI gate runs --dry-run as the pre-merge check, which goes green. The GPU job then fails with exit 2 after loading the model.
  Where. forgelm/cli/_training.py:113-150, forgelm/trainer.py:685-783 (confirmed), forgelm/cli/_dry_run.py:61 (confirmed), forgelm/trainer.py:591-617 (confirmed)
  Recommended fix. In --dry-run, import and construct the trainer-specific TRL config, and check dataset columns against _validate_trainer_columns and the SFT processor.

• CR-A04-043 (code, Low) — A partial --stage run marks the whole pipeline final_status 'completed'
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced (plumbing only)
  Failure scenario. A reviewer or CI step that reads final_status == 'completed' from pipeline_manifest.json accepts a chain that was never fully executed.
  Where. forgelm/cli/_pipeline.py:819-821, forgelm/cli/_pipeline.py:859-870
  Recommended fix. Use a distinct terminal status (e.g. 'partial') when any stage is skipped_by_filter without a carried-over completed row, or document that final_status reflects only the stages run in that invocation.

• CR-A04-044 (code, Low) — After a cross-version --resume-from, pipeline_manifest.json still names only the version that started the run
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. An auditor reading the Annex IV chain manifest of a resumed pipeline concludes every stage was produced by 0.11.0, but the DPO stage was trained by 0.11.1rc1.
  Where. forgelm/cli/_pipeline.py:323, forgelm/cli/_pipeline.py:1223, forgelm/compliance.py:3031
  Recommended fix. Record forgelm_version per stage in PipelineStageState and the manifest, or keep a list of the versions that touched the run; document the semantics of the top-level field.

• CR-A04-045 (code, Low) — Training loads the full base model before the dataset is loaded or schema-checked, so data errors cost a whole model load
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · not reproduced (code trace)
  Where. forgelm/cli/_training.py:141-142, forgelm/data.py:698-722, forgelm/data.py:731
  Recommended fix. Load the tokenizer first (cheap), run prepare_dataset, then load weights; or run a column pre-flight on the primary dataset before the weight load. Order of gain: minutes per failed launch on large models; nil on successful runs.

• CR-A04-046 (code, Low) — After forgelm reject, resume refuses forever while its error text recommends running reject
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end
  Failure scenario. The reviewer rejects stage 1 and runs --resume-from s2. It exits 1 with the same 'Run forgelm approve ... (or forgelm reject ...)' text, a loop.
  Where. forgelm/cli/_pipeline.py:550-558, forgelm/cli/subcommands/_approve.py:694-700
  Recommended fix. Read the stage audit log. On human_approval.rejected, say that the stage was rejected and that --resume-from <gated stage> re-trains it, and document this in the guide.

• CR-A04-047 (code, Low) — A resumed stage that fails before producing a result keeps the previous attempt's training_manifest pointer
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · reproduced end to end
  Failure scenario. Stage s2 completed earlier. --resume-from s2 --force-resume with a broken dataset path crashes s2 (exit 2). The state shows s2 failed with metrics cleared, yet training_manifest still names the earlier attempt's evidence file (mtime from the earlier run).
  Where. forgelm/cli/_pipeline.py:752-776, forgelm/cli/_pipeline.py:1515, forgelm/cli/_pipeline.py:1644
  Recommended fix. Also reset training_manifest (and input_model) in _execute_one_stage's clearing block.

• CR-A04-048 (code, Low) — Pipeline audit vocabulary gaps: a stage that fails merge or the chain guard has stage_completed without stage_started, failed-stage events omit cause and exit code, and resume decisions are unaudited
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · not reproduced (code trace)
  Where. forgelm/cli/_pipeline.py:1604-1614, forgelm/cli/_pipeline.py:1607, forgelm/cli/_pipeline.py:714-720, forgelm/cli/_pipeline.py:1131-1139
  Recommended fix. Add error/exit_code to failed stage events, emit pipeline.resumed (with skipped and approved-inferred stages) on --resume-from, and emit stage_started before merge or a distinct stage_config_failed event.

• ↳ same root cause: DOC-D2B06-029 (docs, Low) — docs/guides/pipeline.md: wrong detail — pipeline.stage_reverted — stage name, auto-revert reason
  severity Low
  Where. docs/guides/pipeline.md:213 · type: wrong detail
  The document says. pipeline.stage_reverted — stage name, auto-revert reason
  What the code does. The emitted payload is stage_name, gate_decision='failed' and auto_revert_triggered=True; no reason field (forgelm/cli/_pipeline.py:714-720). The reason is only in the state/manifest 'error' field and the per-stage training audit log.
  Code references. forgelm/cli/_pipeline.py:714-720
  Proposed correction. Either emit reason=stage_state.error in pipeline.stage_reverted or change the guide to say the reason is in pipeline_state.json/pipeline_manifest.json 'error'.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• CR-A04-049 (code, Low) — CLI stdout JSON writers outside the strict-JSON guard use json.dumps(allow_nan=True), and non-finite config floats reach them
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end · already tracked: PARTIALLY-FIXED C33-TRAIN-JSON-CONTRACT — docs/roadmap/phase-16-trust-surface-hardening.md:291,308 (S2 done); CHANGELOG.md:91-109
  Failure scenario. An lm-eval task returns NaN. --benchmark-only --output-format json prints "average": NaN, and jq fails in the CI step.
  Where. forgelm/cli/_dry_run.py:77 (confirmed), forgelm/cli/_no_train_modes.py:113 (refuted), forgelm/cli/_no_train_modes.py:154 (refuted), forgelm/cli/_no_train_modes.py:202 (refuted)
  Recommended fix. Switch these writers to dumps_strict, and bound neftune_noise_alpha and the rope factor to finite values in config.

• CR-A04-051 (code, Low) — Pipeline --dry-run returns before stage-name validation, so unknown --stage/--resume-from names exit 0
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A CI pre-flight validates the exact command line it will run, including a misspelled --stage. The dry-run passes and the real invocation fails.
  Where. forgelm/cli/_pipeline.py:1762-1763 (confirmed), docs/guides/pipeline.md:155, forgelm/cli/_pipeline.py:1156-1206 (confirmed), docs/reference/configuration.md:461
  Recommended fix. Run the stage-name checks from _resolve_stage_filter_setup and _resolve_resume_skiplist (the name parts only) inside dry_run, collecting their errors like the others. Fix pipeline.md (EN and TR).

• CR-A04-052 (code, Low) — Pipeline dry-run merge-error path (fail-open if removed) and several config-error branches are untested
  severity Low · reachability default · intent intended-and-acceptable · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A future refactor of _scan_output_dirs or dry_run turns an invalid stage config into 'Pipeline dry-run OK', and CI ships a pipeline that fails at runtime. No test goes red.
  Where. forgelm/cli/_pipeline.py:1183-1185, forgelm/cli/_pipeline.py:1520-1526, forgelm/cli/_pipeline.py:626-633, forgelm/cli/_pipeline.py:1780-1783
  Recommended fix. Add a dry-run test with two stages that each fail merge (for example layerwise GaLore with root distributed.strategy) and assert exit 1 and two reported errors. Add tests for the run-time merge failure, the chain-presence guard and the ConfigError boundary.

• CR-A04-053 (code, Low) — Single-stage training seam: the destructive checkpoint call, --resume pass-through and explicit-path refusal are all unprotected by tests
  severity Low · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A refactor changes the call to action='delete' and every run silently deletes all checkpoint-* dirs (a destructive operation per the threat model's list). Or --resume stops forwarding the checkpoint and every 'resume' restarts from scratch. CI stays green either way.
  Where. forgelm/cli/_training.py:145-152, forgelm/cli/_resume.py:15-18, tests/test_cli_training_seam.py:49-50, tests/test_cli_training_seam.py:76
  Recommended fix. Make the stub record its arguments and assert action == 'keep'. Add a seam test with resume='auto' and a checkpoint dir that asserts train() received the path. Add a test that a missing explicit path exits 1.

• CR-A04-054 (code, Low) — Pipeline audit/webhook wiring and result-envelope blocks are not pinned (surviving mutants)
  severity Low · reachability default · intent intended-and-acceptable · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. The reverted-stage event or webhook is lost in a refactor, or the safety verdict disappears from the JSON envelope that CI parses, and nothing fails.
  Where. forgelm/cli/_pipeline.py:722, forgelm/cli/_pipeline.py:1443 (inside _notify_pipeline; the kind=='started' branch is at :1440, the caller at :1139), forgelm/cli/_pipeline.py:1479-1480 (_resolve_input_source; the return 'cli_override' is on :1480), forgelm/cli/_result.py:44-51
  Recommended fix. Assert the event list for a reverted stage, the notifier kinds called, input_source in a real --input-model run, and the envelope blocks.

• CR-A04-055 (code, Low) — Weak or no-op assertions around resume and dry-run tests
  severity Low · reachability unreachable · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end
  Failure scenario. A regression drops the force-resume warning or the dry-run warning and nothing fails. The resume semantics change without anyone noticing.
  Where. tests/test_pipeline_orchestrator.py:1118-1129, tests/test_cli.py:51-93, tests/test_pipeline_cli.py:381
  Recommended fix. Assert the warning, give every dry-run test a concrete assertion, and add a resume-from-with-earlier-failed-stage test that fixes the intended semantics.

• CR-A04-056 (code, Low) — No test covers the ABI preflight wiring: _preflight_numpy_torch_abi can be removed from _run_training_pipeline with the whole suite green (F-157)
  severity Low · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A refactor drops or reorders the preflight call. Training on torch<2.3 + numpy>=2 then fails mid-run with a cryptic NameError, which is the exact failure the feature exists to prevent, and CI stays green.
  Where. forgelm/cli/_training.py:39, forgelm/cli/_training.py:118, tests/test_cli_training_seam.py:70, tests/test_abi_check.py:171
  Recommended fix. Add a seam test that monkeypatches compute_numpy_torch_abi_status to return ABI_BROKEN, calls _run_training_pipeline, and asserts SystemExit(2) plus the remediation text without the preflight stubbed.

• CR-A04-057 (code, Low) — --generate-data exit contract: zero-yield success guard, JSON 'success' field and disabled-block rejection are untested
  severity Low · reachability default · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
  Where. forgelm/cli/_no_train_modes.py:183-190, forgelm/cli/_no_train_modes.py:197-205, forgelm/cli/_no_train_modes.py:238-247, tests/test_no_train_modes.py:187-290
  Recommended fix. Add: zero successes with min_success_rate=0.0 must exit 2; the json payload 'success' is false when the process exits non-zero; synthetic disabled -> EXIT_CONFIG_ERROR.

• CR-A04-058 (code, Low) — Merge and synthetic blocks in a pipeline config are validated but unreachable
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · not reproduced (code trace)
  Failure scenario. An operator expects a post-chain merge to happen because merge.enabled is set in the pipeline YAML. Nothing merges and nothing warns.
  Where. forgelm/cli/_pipeline.py:1739-1754, forgelm/config.py:2018-2023, docs/guides/pipeline.md:113
  Recommended fix. Warn (or reject) when merge.enabled or synthetic.enabled is set alongside pipeline:.

• CR-A04-059 (code, Low) — Standard's JSON output example and rules do not match the shipped envelope
  severity Low · reachability unreachable · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)
  Failure scenario. A new subcommand follows the standard literally and emits {status, exit_code}, breaking the success-key contract.
  Where. docs/standards/logging-observability.md:61, docs/standards/logging-observability.md:82, docs/standards/error-handling.md:264, forgelm/cli/_result.py:9
  Recommended fix. Rewrite the example and rules to the shipped success/error envelope and scope the run_id/config_hash rule to training results.

• CR-A04-060 (code, Low) — --resume auto picks the highest-numbered checkpoint even when its save was interrupted, so the resume crashes
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A training job is preempted while writing checkpoint-1200; every automatic restart with --resume auto crashes until a human deletes the partial directory.
  Where. forgelm/cli/_resume.py:58-66, forgelm/cli/_resume.py:64
  Recommended fix. In auto mode, skip checkpoint dirs without trainer_state.json, the last file HF writes, log a warning naming the skipped dir, and fall back to the next-highest checkpoint.

• CR-A04-061 (code, Low) — pipeline_state.json and compliance/pipeline_manifest.json persist raw 'Trainer crashed: {exc}' / 'Config merge failed: {exc}' text, including HF_ENDPOINT credentials
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. pipeline_manifest.json is compliance evidence handed to auditors and verified by verify-annex-iv --pipeline, and it carries the mirror credential.
  Where. forgelm/cli/_pipeline.py:1511, forgelm/cli/_pipeline.py:1574, forgelm/cli/_pipeline.py:1371-1382, forgelm/cli/_pipeline.py:1671-1672
  Recommended fix. Redact URL credentials and paths in stage_state.error before persisting it, or store only the exception class plus a redacted message.

• CR-A04-062 (code, Low) — PR, issue and review-finding identifiers are embedded throughout shipped code, comments and one user-facing error message
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · reproduced end to end
  Failure scenario. The comments are meaningless to maintainers without the private notes and drift as notes change. The semantic-chunking error tells users to look up an ID that does not exist in a tracker.
  Where. forgelm/cli/_pipeline.py:292, forgelm/model.py:11, forgelm/trainer.py:12, forgelm/ingestion.py:1519-1527
  Recommended fix. Strip the identifiers, keep the rationale, and put a real issue URL in the NotImplementedError. Extend the no-analysis-refs guard to flag F-[A-Z]+-\d+ and PR #\d+ in forgelm/.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}