[Wave 3] Medium — Module sweep: Config schema & public library API (part 1/2) #545

Wave 3 · 22 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A01-024 (code, Medium) — Float hyper-parameters accept inf/nan or out-of-range values (no allow_inf_nan=False; neftune_noise_alpha unconstrained) and fail late or produce a NaN model
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end · already tracked: PARTIALLY-FIXED C31-NUMERIC-CONFIG — docs/roadmap/phase-16-trust-surface-hardening.md:289-292 (S2 done); CHANGELOG.md:9-20
  Failure scenario. A templated or typo'd config value (.inf learning rate) produces a diverged model that ships with exit 0 when no evaluation block is set. Other values fail only after model load or after the first paid API call, instead of at load with exit 1.
  Where. forgelm/config.py:505-509 (orpo_beta, dpo_beta, simpo_gamma, simpo_beta, kto_beta), forgelm/config.py:561-563, forgelm/config.py:586-589 (sliding_window_attention), forgelm/config.py:609-613
  Recommended fix. Add allow_inf_nan=False to every float hyper-parameter, gt=0 (finite) to neftune_noise_alpha and api_delay, ge=1 to sliding_window_attention, and ge=1 / ge=0 to benchmark.limit / num_fewshot. Consider a NaN-loss check that does not depend on an evaluation block being configured.

• CR-A01-025 (code, Medium) — load_config's yaml.safe_load silently keeps the last duplicate mapping key, so a repeated block can disable gates
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator follows the template and appends a second evaluation: block for human approval. The auto_revert and loss-gate settings in the first block vanish, the dry-run is green, and the trained model is never gated.
  Where. forgelm/config.py:2250-2254 (yaml.safe_load at :2252, no duplicate-key detection anywhere in forgelm/), config_template.yaml:78-85 (two commented lora: blocks), :87-98 and :119-134 (commented training: blocks that duplicate the active top-level key when uncommented), config_template.yaml:119-134, forgelm/config.py:2252
  Recommended fix. Load YAML with a SafeLoader subclass that raises ConfigError on duplicate keys (override construct_mapping) and add a test. Optionally restructure the template so optional fields sit as commented lines inside the single block.

• CR-A01-027 (code, Medium) — eval_steps/save_steps cadence mismatch passes --dry-run and fails only after model load (exit 1, not 2)
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end · already tracked: PARTIALLY-FIXED CORE-14 — docs/roadmap/phase-16-trust-surface-hardening.md:292 (S2 done); CHANGELOG.md:21-25
  Failure scenario. CI's --dry-run gate passes, and the job fails minutes later after model download and load, firing a start webhook and a failure audit event for a pure config defect.
  Where. forgelm/config.py:1731-1748, forgelm/trainer.py:483-494, forgelm/cli/_training.py:163-171, forgelm/trainer.py:458-459
  Recommended fix. In the config validator, warn (or error when evaluation is configured) whenever save_steps % eval_steps != 0, and state that the run will fail if a validation split exists.

• CR-A01-031 (code, Medium) — Pipeline stage-level cross-field validation runs only at stage merge time; a non-dry-run trains earlier stages before a later stage's config error
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A multi-hour SFT stage completes. The DPO stage then fails with EXIT_CONFIG_ERROR on a config defect that was knowable up front, and the GPU hours of stage 1 are wasted.
  Where. forgelm/config.py:2156-2233 (confirmed), forgelm/config.py:1993-2046 (PipelineStage), forgelm/cli/_pipeline.py:1032-1041, forgelm/cli/_pipeline.py:1504-1513
  Recommended fix. Validate every stage via merge_pipeline_stage_config in a PipelineConfig/ForgeConfig model validator, or make the run() pre-flight fail on merge errors the way dry_run() does.

• CR-A01-034 (code, Medium) — evaluation.benchmark.batch_size is typed str: the documented integer example is rejected while typos pass and fail after training
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator pastes the documented evaluation block and the run is rejected at config load.
  Where. forgelm/config.py:876, docs/usermanuals/en/evaluation/benchmarks.md:21 (confirmed), docs/usermanuals/tr/evaluation/benchmarks.md:21, forgelm/config.py:873-877 (confirmed)
  Recommended fix. Accept Union[int, Literal['auto'], str] (or coerce numbers) or quote the example.

• ↳ same root cause: DOC-BYAML-006 (docs, Medium) — docs/usermanuals/en/evaluation/benchmarks.md: broken example — evaluation.benchmark.batch_size: 8 (integer)
  severity Medium · already tracked: KNOWN-OPEN C31-DOC-SCHEMA-PROSE — docs/roadmap/phase-16-trust-surface-hardening.md:773,785-786
  Where. docs/usermanuals/en/evaluation/benchmarks.md:21 (TR benchmarks.md:21) · type: broken example
  The document says. evaluation.benchmark.batch_size: 8 (integer)
  What the code does. BenchmarkConfig.batch_size is a str ('auto' or an integer string); the YAML int 8 fails with 'Input should be a valid string'. configuration.md:143 correctly shows "auto" with 'or an integer string'.
  Code references. forgelm/config.py:876
  Proposed correction. Quote the value in the docs ("8") and consider a pydantic before-validator coercing int to str.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• CR-A01-036 (code, Medium) — judge_api_base set without judge_api_key_env is silently ignored: the judge falls back to loading judge_model locally, and with auto_revert the model is deleted
  severity Medium · reachability documented-opt-in · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator points the judge at an internal unauthenticated vLLM endpoint. Training finishes, the judge tries a Hub load of the endpoint's model name, fails, and auto_revert deletes the model.
  Where. forgelm/config.py:1089-1094 (judge_api_key_env 'None = local judge model'; judge_api_base), forgelm/config.py:1142-1158 (only the revision + key_env mismatch is rejected), forgelm/judge.py:625-633 (is_api_judge = judge_api_key is not None; local load failure -> passed=False), forgelm/trainer.py:384-394 (preflight checks only key_env set but unset)
  Recommended fix. Select the API path when judge_api_base is set (an empty bearer is acceptable for unauthenticated endpoints), or reject judge_api_base without judge_api_key_env at validation.

• CR-A01-039 (code, Medium) — data.add_eos: false has no effect for SFT, because TRL always appends EOS to non-conversational text
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator training continued-pretraining-style text with add_eos:false, to avoid document boundaries, still gets EOS on every sample.
  Where. forgelm/config.py:821-823, forgelm/data.py:348, forgelm/data.py:398-399, forgelm/data.py:429-430
  Recommended fix. Document that TRL appends EOS, or remove the field for SFT, or post-process the tokenized data when add_eos is false.

• CR-A01-040 (code, Medium) — risk_assessment.risk_category is documented as driving auto-revert thresholds and Annex IV gating, but only gates safety.enabled and is echoed into manifests
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator declares high-risk expecting tighter gates and auto-revert. They get the same thresholds, and with auto_revert left false a failed safety gate still produces exit 0.
  Where. docs/reference/configuration.md:352 (confirmed), forgelm/config.py:1835-1841, forgelm/compliance.py:1564-1569 (confirmed), forgelm/config.py:1266-1268
  Recommended fix. Reword configuration.md (+TR) to describe the actual effect: it requires safety.enabled, warns when auto_revert is off, prints the Art. 5 banner, and is copied into Annex IV. Either change the warning to 'is recommended' or enforce auto_revert for strict tiers.

• ↳ same root cause: DOC-D1B01-018 (docs, Medium) — docs/reference/configuration.md: false claim — risk_assessment.risk_category: "Drives auto-revert thresholds and Annex IV gating."
  severity Medium
  Where. docs/reference/configuration.md:352 (Turkish mirror: docs/reference/configuration-tr.md:296) · type: false claim
  The document says. risk_assessment.risk_category: "Drives auto-revert thresholds and Annex IV gating."
  What the code does. No threshold depends on the risk tier. Strict tiers only trigger the config-load hard gate (safety.enabled required) and warnings (a missing auto_revert, the Article 5 banner). The value is otherwise only copied into the manifest.
  Code references. forgelm/config.py:1822-1873, forgelm/compliance.py:1564-1571
  Proposed correction. Owner decision. Either implement tier-dependent thresholds, or change the doc to: "Strict tiers (high-risk, unacceptable) trigger the config-load hard gate below and a warning when auto_revert is off; the value is recorded in the Annex IV bundle. It does not change any evaluation threshold."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• CR-A01-044 (code, Medium) — No test asserts a runtime effect of retention.enforce (warn_on_excess / block_on_excess)
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator sets retention.enforce: block_on_excess expecting a pre-flight abort with exit 3 when raw documents, the audit log or staging exceed their horizons. Training starts anyway with exit 0. No test could fail, because none asserts the documented gate.
  Where. forgelm/config.py:1585, forgelm/cli/subcommands/_purge.py:1153, tests/test_gdpr_erasure.py:311, tests/test_config.py:603
  Recommended fix. Minimal test that would have caught M-X-002. Build a tmp output_dir containing an audit_log.jsonl and a final_model.staging dir with mtime older than the horizon, with retention={staging_ttl_days:1, enforce:'block_on_excess'}. Invoke the trainer or CLI pre-flight, with a stubbed model and trainer, through the cli/_training.py entry. Assert that the process exits with EXIT_EVAL_FAILURE (3) before…

• CR-A01-046 (code, Medium) — Docs never state what a relative path is relative to, and point at path values that do not resolve for wheel installs; benchmark.output_dir default is misdescribed
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Failure scenario. A pip-installed user copies test_prompts: configs/safety_prompts/general_safety.jsonl from the template. The safety gate finds no file and fails after training (evaluation_completed=False). A user looking for benchmark_results.json in output_dir does not find it.
  Where. docs/reference/configuration.md:211, config_template.yaml:200, docs/reference/configuration.md:198, forgelm/config.py:878-880
  Recommended fix. Add one sentence per path field (or a section header note) that relative paths resolve against the process working directory, or implement config-dir resolution. Ship the safety sets as package data with a documented keyword, or reword the docs. Fix the benchmark.output_dir description.

• CR-A01-047 (code, Medium) — Strict risk tiers do not require evaluation.require_human_approval (or Annex IV) at runtime; only the wizards enforce it, contrary to the deployer guide and site
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A deployer follows the checklist expecting ConfigError as a backstop and forgets require_human_approval. The high-risk model is promoted to final_model/ with no Article 14 human oversight, and the auditor is told the control is enforced.
  Where. forgelm/config.py:1822-1847, forgelm/config.py:1849-1876, forgelm/config.py:1220-1223, docs/guides/iso_soc2_deployer_guide.md:64-66
  Recommended fix. Either add the ConfigError for strict tiers when require_human_approval is false (a behaviour change that needs a CHANGELOG entry), or correct the EN and TR guides to say that F-compliance-110 enforces safety.enabled only.

• CR-A01-048 (code, Medium) — staging_ttl_days field says 'after a forgelm reject decision', but --check-policy ages and flags pending (undecided) staging dirs too
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator purging every --check-policy violation also erases staged models that a reviewer has not yet decided on. Alternatively, an auditor reads the TTL as 'post-reject' when it is not.
  Where. forgelm/config.py:1558-1566, config_template.yaml:271, forgelm/cli/subcommands/_purge.py:1261-1289, forgelm/cli/subcommands/_purge.py:1372-1395 (_resolve_artefact_age body ends at 1395)
  Recommended fix. Decide on one meaning. Either age from the human_approval.rejected event and skip runs without a terminal decision, or redescribe the field as 'staging age from run start, whatever the decision' in the config, template and docs.

• CR-A01-049 (code, Medium) — synthetic.api_base is required at runtime but not validated at config time; the cli.md teacher snippet omits it
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. Config with synthetic.enabled true, teacher_backend api and no api_base: --dry-run exit 0; --generate-data fails all 4 prompts with 'synthetic.api_base is required', exit 2.
  Where. forgelm/config.py:1343, forgelm/config.py:1419-1441 (validator decorator at 1419, def at 1420), forgelm/synthetic.py:218-222, docs/usermanuals/en/reference/cli.md:282-289
  Recommended fix. Add an api_base check to _validate_synthetic_payload when enabled and teacher_backend == 'api', and add api_base to the cli.md snippet.

• CR-A01-050 (code, Medium) — data.mix_ratio omitted is documented as 'uniform' but is plain size-proportional concatenation
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN CORE-11 — docs/roadmap/phase-16-trust-surface-hardening.md:588-589,605-606 (S10 open)
  Failure scenario. An operator adds a small domain dataset as an extra without a mix_ratio, trusting 'uniform if omitted'. The small dataset makes up about 1% of training.
  Where. forgelm/config.py:813-816, config_template.yaml:64, docs/usermanuals/en/reference/configuration.md:77, forgelm/data.py:545-548
  Recommended fix. Either implement uniform weighting when None, or document that omitting mix_ratio concatenates the datasets in proportion to their sizes.

• CR-A01-051 (code, Medium) — GaLore low-rank projection cannot reduce optimizer state for LoRA tensors when galore_rank >= lora.r (always true at defaults)
  severity Medium · reachability documented-opt-in · intent contract-violation · confidence confirmed · independently confirmed · not reproduced (code trace)
  Failure scenario. An operator enables GaLore (non-8-bit galore_adamw, default rank 128) on top of LoRA r=8 to save optimizer memory and gets no reduction, plus per-gap SVD cost.
  Where. notebooks/galore_memory_optimization.ipynb:29-31, forgelm/config.py:539-541, forgelm/config.py:347-352, forgelm/trainer.py:544-562
  Recommended fix. State in the notebook and docs that projection only helps when galore_rank < lora.r, and that the 8-bit variant is the source of savings in the LoRA setup. Optionally warn at config time when galore_rank >= lora.r.

• CR-A01-052 (code, Medium) — Product strategy says automatic EU AI Act Art. 9-17 artefacts come from a normal run; Art. 9/11/14/15 need opt-in config and Art. 16/17 have no artefact
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A compliance lead reads the strategy doc and assumes every run's output dir holds Art. 9 and Art. 11 evidence. After a default-config run, the Annex IV and risk files are missing, and no safety evaluation ran.
  Where. docs/product_strategy.md:30-37, docs/product_strategy.md:28, docs/product_strategy.md:115, forgelm/config.py:1625-1632
  Recommended fix. Reword the doc to say which artefacts are always produced (Art. 10, 12, 13, integrity) and which need which config block (Art. 9 risk_assessment, Art. 11 compliance, Art. 14 require_human_approval, Art. 15 evaluation.safety). Either drop '9-17' or add Art. 16/17 mappings (for example the QMS templates).

• CR-A01-053 (code, Medium) — Annex IV / pipeline manifest digest helper and strict writer disagree on non-finite values: the stamped hash is computed over a payload the writer then sanitises differently or refuses
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end · already tracked: PARTIALLY-FIXED TRUST-15 — docs/roadmap/phase-16-trust-surface-hardening.md:9-11,339-343 (S2 done); CHANGELOG.md:91-101
  Failure scenario. An operator sets training.gpu_cost_per_hour: .inf and has a compliance: block. The run completes and writes compliance/annex_iv_metadata.json. computational_resources on disk is {'gpu_cost_per_hour_usd': 'inf', 'estimated_cost_usd': 'inf', ...}. forgelm verify-annex-iv <file> prints 'FAIL ... Manifest hash mismatch — artifact may have been modified after generation.' and exits 6, though nothing was modified. The same happens for a NaN train_loss (a diverged run with no validation split, where…
  Where. forgelm/config.py:609-613, forgelm/trainer.py:1891-1896, forgelm/compliance.py:1618-1619, forgelm/compliance.py:1843-1847
  Recommended fix. Apply sanitize_non_finite inside compute_annex_iv_manifest_hash before canonicalising, so the writer and the verifier hash the same structure. This covers the single-stage and the pipeline manifest. Add allow_inf_nan=False to gpu_cost_per_hour. Add a regression test that writes a non-finite metric and cost, then verifies the artefact round trip with the real verifier.

• CR-A01-054 (code, Medium) — benchmark.output_dir default is documented as the training output_dir but resolves to <output_dir>/benchmark (in-run) or dirname(model_path)/benchmark (--benchmark-only)
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. CI reads <training.output_dir>/benchmark_results.json per the docs. The file is at <training.output_dir>/benchmark/benchmark_results.json.
  Where. forgelm/config.py:878-880 (confirmed), forgelm/trainer.py:1776 (confirmed), forgelm/cli/_no_train_modes.py:88 (confirmed), docs/usermanuals/en/evaluation/benchmarks.md:135
  Recommended fix. Fix the field description and the docs table to name the benchmark/ subdirectory and the --benchmark-only rule.

• CR-A01-055 (code, Medium) — ISO 27001 mapping says config_hash is bound into training_manifest.yaml, but the YAML sidecar carries no config_hash
  severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced with stubs
  Failure scenario. An auditor following the ISO A.8.9 row opens compliance/training_manifest.yaml to find the config anchor and finds none.
  Where. docs/reference/iso27001_control_mapping.md:110, docs/guides/iso_soc2_deployer_guide.md:112-116, forgelm/compliance.py:2178-2190, docs/reference/iso27001_control_mapping-tr.md:109
  Recommended fix. Change the A.8.9 row (EN and TR) to name compliance_report.json.

• CR-A01-056 (code, Medium) — No strict typing on ForgeConfig: YAML 1.1 booleans and octal/sexagesimal literals silently become numeric hyper-parameters and gate thresholds
  severity Medium · reachability default · intent intended-but-harmful · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. An operator writes max_acceptable_loss: off with auto_revert: true. The config validates and the run trains, then deletes the final model and exits 3. Similarly, learning_rate: yes trains at LR 1.0, save_steps: 0500 checkpoints every 320 steps, and llm_judge.min_score: true makes the judge gate pass every model. None of these produces a warning.
  Where. forgelm/config.py:122, forgelm/config.py:465-509, forgelm/config.py:586-589, forgelm/config.py:663-666
  Recommended fix. Set strict=True on numeric fields: use StrictInt/StrictFloat, with floats still accepting ints, or add one shared mode='before' validator that rejects bool for every int/float field, as rope_scaling already does. Consider a YAML 1.2-style loader, or reject leading-zero and base-60 ints. Document the type rules and quoting advice in configuration.md (EN+TR).

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}