[Wave 3] Medium — Docs: notebooks/ #621

Wave 3 · 12 finding(s) in 12 fix package(s). Documents with fewer than five findings each in this wave, grouped by directory.

Findings

• DOC-BLEGAL-151 (docs, Medium) — notebooks/dpo_alignment.ipynb: broken example — dataset_name_or_path: "argilla/ultrafeedback-binarized-preferences"
  severity Medium
  Where. notebooks/dpo_alignment.ipynb cell 3 · type: broken example
  The document says. dataset_name_or_path: "argilla/ultrafeedback-binarized-preferences"
  What the code does. The dataset's columns are source, instruction, chosen_response, rejected_response, chosen_avg_rating, rejected_avg_rating and chosen_model, with no prompt, chosen or rejected. ForgeLM's _detect_dataset_format returns 'unknown format', and _validate_trainer_columns('dpo') raises KeyError. The config-only --dry-run in cell 4 does not catch this.
  Proposed correction. Use a dataset with prompt/chosen/rejected columns (e.g. a binarized variant with those names), or add a column-mapping step before training.

• DOC-BLEGAL-153 (docs, Medium) — notebooks/post_training_workflow.ipynb: wrong detail — This file [model_integrity.json] is part of the audit trail required by EU AI Act Article 15
  severity Medium
  Where. notebooks/post_training_workflow.ipynb cell 13 · type: wrong detail
  The document says. This file [model_integrity.json] is part of the audit trail required by EU AI Act Article 15
  What the code does. Art. 15 requires appropriate accuracy, robustness and cybersecurity (15(1)-(5)) and never mentions a manifest, hash or checksum. A SHA-256 manifest is a legitimate supporting measure for 15(1)/(4) consistency, and Annex IV 2(h) asks for a description of cybersecurity measures, but Art. 15 does not require it.
  Proposed correction. 'This file is an integrity measure that supports Art. 15 robustness/cybersecurity evidence (Annex IV 2(h)) — deployers can verify …'.

• DOC-D4B02-012 (docs, Medium) — notebooks/safety_evaluation.ipynb: misleading — "measure safety degradation"; "4. Auto-revert if safety degradation exceeds threshold"
  severity Medium
  Where. notebooks/safety_evaluation.ipynb:9 (cell 0) and :303 (cell 8) · type: misleading
  The document says. "measure safety degradation"; "4. Auto-revert if safety degradation exceeds threshold"
  What the code does. The gate compares the fine-tuned model's absolute unsafe-response ratio with max_safety_regression. No base-model run happens and no degradation or delta is computed (code comment: "Despite the field name, max_safety_regression is NOT a ..." relative bound).
  Code references. forgelm/safety/_gates.py:257-261, forgelm/cli/_parser.py (safety-eval --max-safety-regression help: "Absolute bound, not baseline-relative")
  Proposed correction. "measure the fine-tuned model's unsafe-response ratio"; "# 4. Auto-revert if the unsafe-response ratio exceeds the threshold (absolute, not relative to the base model)"

• DOC-D4B05-004 (docs, Medium) — notebooks/galore_memory_optimization.ipynb: misleading — The layerwise variant's only restriction is single GPU / no DDP; table lists 4 of the 6 variants
  severity Medium
  Where. notebooks/galore_memory_optimization.ipynb:262 (variants table: galore_adamw_layerwise | Lowest memory - single GPU only, no DDP) · type: misleading
  The document says. The layerwise variant's only restriction is single GPU / no DDP; table lists 4 of the 6 variants
  What the code does. Transformers' low-rank optimizer setup raises ValueError: Layerwise ... does not support gradient accumulation! unless gradient_accumulation_steps == 1 (transformers.trainer_optimizer _setup_low_rank_optimizer, installed 5.13.0). ForgeLM's default is gradient_accumulation_steps: 2 (config.py:480-484) and _validate_galore (config.py:1920-1935) only checks the distributed restriction, so a re…
  Code references. forgelm/config.py:542-549, forgelm/config.py:1931-1935
  Proposed correction. Write: 'galore_adamw_layerwise | Lowest optimizer memory - single GPU only (no DDP) and requires gradient_accumulation_steps: 1; galore_adamw_8bit_layerwise and galore_adafactor_layerwise follow the same rule.'

• DOC-D4B05-005 (docs, Medium) — notebooks/multi_dataset.ipynb: false claim — mix_ratio: [0.7, 0.3] # 70% general, 30% coding and 'Fine-tuned model learned from BOTH datasets (70% general + 30% coding).'
  severity Medium · already tracked: KNOWN-OPEN CORE-11|CORE-08 docs/roadmap/phase-16-trust-surface-hardening.md:587-588
  Where. notebooks/multi_dataset.ipynb:153 and :252 (mix_ratio comment and final print) · type: false claim
  The document says. mix_ratio: [0.7, 0.3] # 70% general, 30% coding and 'Fine-tuned model learned from BOTH datasets (70% general + 30% coding).'
  What the code does. With the notebook's data (4 general + 4 coding rows) the mix is n_i = min(int(max_dataset_size * ratio_i), len(ds_i)) (data.py:496-510): int(40.7)=2 general and int(40.3)=1 coding rows, i.e. 3 rows, 67%/33%. _ensure_validation_split then carves 1 of the 3 rows into validation. I ran prepare_dataset on the notebook's files: train = 2 rows (both from general.jsonl: 'What is NLP?', 'What is de…
  Code references. forgelm/data.py:496-510, forgelm/data.py:549-580, forgelm/data.py:700-710
  Proposed correction. Generate at least ~50 rows per dataset in Step 3 (for example repeat or template the examples) so the 70/30 sampling and the 10% validation split behave as described. Replace the final print with: 'Training data was sampled from both datasets in roughly a 70/30 ratio.' Add one line: 'mix_ratio weights are applied relative to the largest dataset; t…

• DOC-D4B05-006 (docs, Medium) — notebooks/quickstart_sft.ipynb: misleading — export and deploy can be pointed at ./checkpoints/final_model, which is the adapter directory the notebook just produced
  severity Medium
  Where. notebooks/quickstart_sft.ipynb:270 (Turkish mirror: notebooks/post_training_workflow.ipynb (out of batch)) · type: misleading
  The document says. export and deploy can be pointed at ./checkpoints/final_model, which is the adapter directory the notebook just produced
  What the code does. The notebook's own Step 6 and Step 8 say final_model is a PEFT adapter (merge_adapters defaults to false, config.py:457). A real run lists only README.md, adapter_config.json, adapter_model.safetensors, chat_template.jinja, deployer_instructions.md, model_integrity.json, tokenizer.json and tokenizer_config.json, with no config.json or full weights. forgelm export passes model_path unchanged to…
  Code references. forgelm/export.py:424-443, forgelm/deploy.py:127-150, forgelm/config.py:457 (merge_adapters)
  Proposed correction. Replace the export row with forgelm export HuggingFaceTB/SmolLM2-135M-Instruct --adapter ./checkpoints/final_model --output model.gguf --quant q8_0 (merges the adapter first), or add training.merge_adapters: true to the Step 3 YAML. Add: 'final_model is a LoRA adapter; export and deploy need a merged model (--adapter on export, or merge_adapte…

• DOC-D4B06-004 (docs, Medium) — notebooks/data_curation.ipynb: broken example — Handbook v2 is 'A near-duplicate paragraph (only one word changed) — the audit should flag it.'
  severity Medium
  Where. notebooks/data_curation.ipynb cell 3 (code comment) and cell 2 (markdown) · type: broken example
  The document says. Handbook v2 is 'A near-duplicate paragraph (only one word changed) — the audit should flag it.'
  What the code does. Running the notebook's own cells (ingest then audit): near_duplicate_summary.pairs_per_split = {train: 0}. After markdown chunking at 600 chars the v2 'Vacation' section is packed into a chunk with different sections and a different breadcrumb ('# Employee Handbook v2'); simhash Hamming distance between the two closest chunks is 8, above the default threshold 3 (forgelm/data_audit/_types.py DEFAU…
  Code references. forgelm/data_audit/_aggregator.py, forgelm/ingestion.py:1757-1770
  Proposed correction. Either rework the sample so the near-duplicate survives chunking (identical heading and an otherwise identical short document), or drop the claim and say the audit may report zero pairs here.

• DOC-D4B06-005 (docs, Medium) — notebooks/data_curation.ipynb: wrong detail — Optional PII + secrets masking, four chunking strategies, token-aware sizing.
  severity Medium
  Where. notebooks/data_curation.ipynb cell 0 (markdown intro) · type: wrong detail
  The document says. Optional PII + secrets masking, four chunking strategies, token-aware sizing.
  What the code does. The CLI exposes three strategies (sliding, paragraph, markdown): forgelm/cli/_parser.py:236-239 choices; the fourth, 'semantic', raises NotImplementedError and is hidden (forgelm/ingestion.py:1510-1525, :1866-1867; describe_strategies at :2018-2030).
  Code references. forgelm/cli/_parser.py:236-239, forgelm/ingestion.py:92, forgelm/ingestion.py:1510-1525
  Proposed correction. 'three chunking strategies (sliding, paragraph, markdown)'.

• DOC-D4B06-006 (docs, Medium) — notebooks/data_curation.ipynb: false claim — --strategy markdown ... Falls back to paragraph mode for non-markdown inputs (.txt here); 'the .txt file fell back to the paragraph chunker (no headin…
  severity Medium
  Where. notebooks/data_curation.ipynb cell 4 (markdown) and cell 6 (code comment) · type: false claim
  The document says. --strategy markdown ... Falls back to paragraph mode for non-markdown inputs (.txt here); 'the .txt file fell back to the paragraph chunker (no headings to split on)'.
  What the code does. There is no paragraph fallback. _chunk_markdown (forgelm/ingestion.py:1722-1757) treats headingless text as one section and emits it whole even when larger than --chunk-size. Measured: a 10-paragraph (~2000 char) .txt with --strategy markdown --chunk-size 300 yields 1 chunk; --strategy paragraph yields 10. It only looks like paragraph mode in the notebook because the runbook is shorter than 600 c…
  Code references. forgelm/ingestion.py:1722-1757, forgelm/ingestion.py:1860-1867
  Proposed correction. Change to: 'Files without headings (such as the .txt here) become a single section; the markdown strategy does not fall back to paragraph chunking, so a long heading-less file is emitted as one chunk above --chunk-size. Use --strategy paragraph for plain-text corpora.'

• DOC-D4B06-007 (docs, Medium) — notebooks/post_training_workflow.ipynb: misleading — Download config_template.yaml from the v0.9.0 tag ... so the YAML matches the wheel above (forgelm==0.11.0).
  severity Medium · already tracked: KNOWN-OPEN DOCS-13|DOCS-17 docs/roadmap/phase-16-trust-surface-hardening.md:815-817
  Where. notebooks/post_training_workflow.ipynb cell 1 (code comment) · type: misleading
  The document says. Download config_template.yaml from the v0.9.0 tag ... so the YAML matches the wheel above (forgelm==0.11.0).
  What the code does. The pinned wheel is 0.11.0 but the template is fetched from the v0.9.0 tag, so by the notebook's own rationale it does not match the wheel. The snapshot (0.11.1rc1) template passes --fit-check and --dry-run; the v0.9.0 file could not be inspected offline.
  Code references. notebooks/post_training_workflow.ipynb cell 1, pyproject.toml:19
  Proposed correction. Fetch config_template.yaml from the tag matching the pinned wheel (v0.11.0) and bump both together; update the comment accordingly.

• DOC-D4B06-008 (docs, Medium) — notebooks/post_training_workflow.ipynb: wrong detail — Every export updates model_integrity.json with the new artifact's path, format, quant, SHA-256, and size in bytes ... deployers can verify the…
  severity Medium · already tracked: KNOWN-OPEN CLI-03 docs/roadmap/phase-16-trust-surface-hardening.md:426-429
  Where. notebooks/post_training_workflow.ipynb cell 13 (markdown, Compliance Artifacts) · type: wrong detail
  The document says. Every export updates model_integrity.json with the new artifact's path, format, quant, SHA-256, and size in bytes ... deployers can verify the GGUF they pulled matches the one ForgeLM signed off on.
  What the code does. _update_integrity_manifest (forgelm/export.py:207-239) returns silently when model_integrity.json is absent (it is written only into final_model/ by a ForgeLM training run: trainer.py:2157-2166), logs failures at DEBUG only, writes the record under exported_artifacts with key file (basename, not path), and for K-quants records only the intermediate .f16.gguf (the llama-quantize output is neve…
  Code references. forgelm/export.py:207-239, forgelm/trainer.py:2157-2166, forgelm/compliance.py:760-815, forgelm/verify/_gguf.py:1-60
  Proposed correction. 'When the model directory already holds a model_integrity.json (written by ForgeLM training), export appends the produced file (file name, format, quant, sha256, size_bytes) under exported_artifacts; for K-quants this is the intermediate .f16.gguf. The record is an unsigned checksum for manual comparison.'

• DOC-D4B06-009 (docs, Medium) — notebooks/post_training_workflow.ipynb: broken example — forgelm export ./checkpoints/final_model --quant q8_0 and forgelm deploy ./checkpoints/final_model --target ollama/vllm work on 'a model f…
  severity Medium
  Where. notebooks/post_training_workflow.ipynb cell 10 (export), cell 12 (deploy), prerequisites in cell 0 · type: broken example
  The document says. forgelm export ./checkpoints/final_model --quant q8_0 and forgelm deploy ./checkpoints/final_model --target ollama/vllm work on 'a model fine-tuned with ForgeLM (see quickstart_sft.ipynb) under ./checkpoints/final_model'.
  What the code does. quickstart_sft leaves an adapter-only directory at ./checkpoints/final_model (adapter_config.json present; training.merge_adapters defaults to False; forgelm/trainer.py:1714-1731, config.py:457). forgelm export converts the given directory as-is and only merges when --adapter is passed (export.py:372-419); an adapter-only dir has no base weights/config for convert_hf_to_gguf.py. Ollama FROM and v…
  Code references. forgelm/trainer.py:1714-1731, forgelm/config.py:457-460, forgelm/export.py:372-419, forgelm/deploy.py:240-260
  Proposed correction. Tell the reader to either train with training.merge_adapters: true, or run forgelm export ./base_model --adapter ./checkpoints/final_model ... (and deploy from the merged directory), and make that the primary example.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}