[Wave 3] Medium — Docs: docs/guides/performance.md #578

Wave 3 · 9 finding(s) in 9 fix package(s). One pass over this document; keep the Turkish mirror in sync.

Findings

• DOC-D2B04-019 (docs, Medium) — docs/guides/performance.md: misleading — The Llama Guard evaluator's classification generation is batched at evaluation.safety.batch_size. The VRAM/throughput table is for 'Llama Guard 3 8B, fp16' by…
  severity Medium
  Where. docs/guides/performance.md:38-75, docs/guides/performance.md:169-171 (Turkish mirror: docs/guides/performance-tr.md:38-75, docs/guides/performance-tr.md:169-171) · type: misleading
  The document says. The Llama Guard evaluator's classification generation is batched at evaluation.safety.batch_size. The VRAM/throughput table is for 'Llama Guard 3 8B, fp16' by batch_size. 'The safety classifier holds activations for the full batch.'
  What the code does. batch_size batches the fine-tuned model's response generation. The model is then released from the GPU, and Llama Guard (default generative mode) classifies sequentially at batch size 1. batch_size therefore has no effect on Llama Guard VRAM or throughput.
  Code references. forgelm/safety/_generate.py:106-138, forgelm/safety/_orchestrator.py:238-239, forgelm/safety/_score_generation.py:261-270, forgelm/safety/_score_generation.py:286
  Proposed correction. Write: 'evaluation.safety.batch_size batches the fine-tuned model's response generation (pad-longest). The model is then unloaded and Llama Guard scores each response one at a time.' Re-label the table as the trained model's generation memory, or drop it.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-020 (docs, Medium) — docs/guides/performance.md: wrong detail — test_prompts: "configs/safety_prompts/general_safety.jsonl" — the 'bundled' general_safety.jsonl.
  severity Medium
  Where. docs/guides/performance.md:49, docs/guides/performance.md:64 (Turkish mirror: docs/guides/performance-tr.md:49, docs/guides/performance-tr.md:64) · type: wrong detail
  The document says. test_prompts: "configs/safety_prompts/general_safety.jsonl" — the 'bundled' general_safety.jsonl.
  What the code does. configs/safety_prompts/ is repo-relative and not in the wheel (packages.find include forgelm*), and the code itself calls that path 'broken on pip install'. The bundled set is forgelm/safety_prompts/default_probes.jsonl.
  Code references. forgelm/wizard/_collectors.py:61-66, pyproject.toml:250-252, forgelm/safety_prompts/default_probes.jsonl
  Proposed correction. Say the path exists only in a source checkout, or reference the packaged default probe set (forgelm/safety_prompts/default_probes.jsonl) as the bundled option.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-021 (docs, Medium) — docs/guides/performance.md: wrong detail — The LLM-as-judge evaluator batches local-model judging. For API-backed judging the parameter is irrelevant. With a bigger judge (e.g. Qwen2.5-72B), drop bat…
  severity Medium
  Where. docs/guides/performance.md:87-108 (Turkish mirror: docs/guides/performance-tr.md:87-108) · type: wrong detail
  The document says. The LLM-as-judge evaluator batches local-model judging. For API-backed judging the parameter is irrelevant. With a bigger judge (e.g. Qwen2.5-72B), drop batch_size to 1-2.
  What the code does. evaluation.llm_judge.batch_size batches the fine-tuned model's response generation for both API and local judges. The judge call itself, local or API, is per-prompt and unbatched.
  Code references. forgelm/judge.py:549-596, forgelm/judge.py:702-714
  Proposed correction. Write: 'batch_size batches the fine-tuned model's answer generation for every judge type; judge scoring is one call per prompt. Size it to the fine-tuned model, not the judge.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-022 (docs, Medium) — docs/guides/performance.md: broken example — The shown multi-line forgelm ingest INPUT_PATH --output ... --strategy paragraph --chunk-size 1024 --overlap 128 command, with the strategy comment 'sli…
  severity Medium
  Where. docs/guides/performance.md:116-122 (Turkish mirror: docs/guides/performance-tr.md:116-122) · type: broken example
  The document says. The shown multi-line forgelm ingest INPUT_PATH --output ... --strategy paragraph --chunk-size 1024 --overlap 128 command, with the strategy comment 'sliding | paragraph | markdown | semantic'.
  What the code does. L119 has no trailing backslash. The shell runs forgelm ingest INPUT_PATH --output data/policies.jsonl --strategy paragraph with the default chunk size (2048), then tries to execute --chunk-size 1024 and --overlap 128 as separate commands. --strategy semantic is rejected by argparse: invalid choice, exit 2 (runtime).
  Code references. forgelm/cli/_parser.py:236-239
  Proposed correction. Add the missing backslashes, move the comments out of the continuation lines, and list choices as 'sliding | paragraph | markdown'.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-023 (docs, Medium) — docs/guides/performance.md: wrong detail — --strategy paragraph # sliding | paragraph | markdown | semantic
  severity Medium
  Where. docs/guides/performance.md:119 · type: wrong detail
  The document says. --strategy paragraph # sliding | paragraph | markdown | semantic
  What the code does. CLI choices exclude semantic (_parser.py:239); passing it fails argparse validation.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-024 (docs, Medium) — docs/guides/performance.md: false claim — Sample packing: verify that attention masking is reliable with forgelm audit.
  severity Medium
  Where. docs/guides/performance.md:157 (Turkish mirror: docs/guides/performance-tr.md:157) · type: false claim
  The document says. Sample packing: verify that attention masking is reliable with forgelm audit.
  What the code does. forgelm audit computes length/language stats, near-duplicates, PII, secrets, quality and cross-split leakage over JSONL rows. It has no packing or attention-mask check; nothing under forgelm/data_audit or cli/subcommands/_audit.py references either.
  Code references. forgelm/data_audit/_orchestrator.py:233-320, forgelm/cli/subcommands/_audit.py
  Proposed correction. Remove 'verify with forgelm audit' and say: 'ForgeLM does not verify packed attention masks; confirm with your TRL version's packing documentation.'
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-025 (docs, Medium) — docs/guides/performance.md: unverifiable offline — Throughput multipliers (1x/3.2x/5.5x/9x/14x) and VRAM figures were 'measured on Llama Guard 3 8B with the bundled general_safety.jsonl'
  severity Medium
  Where. docs/guides/performance.md:56-64 (Turkish mirror: docs/guides/performance-tr.md) · type: unverifiable offline
  The document says. Throughput multipliers (1x/3.2x/5.5x/9x/14x) and VRAM figures were 'measured on Llama Guard 3 8B with the bundled general_safety.jsonl'
  What the code does. No benchmark in the repo produces these numbers; the measured subject is stated wrongly (batch_size does not touch Llama Guard). general_safety.jsonl lives in configs/safety_prompts/ (repo checkout), not in the installed package (package data is forgelm/safety_prompts/default_probes.jsonl).
  Code references. forgelm/safety/_generate.py:108-137
  Proposed correction. Delete the numeric table or replace with qualitative guidance ('larger batches use more VRAM for activations and KV cache of the fine-tuned model; measure on your hardware'); do not claim measurement.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-026 (docs, Medium) — docs/guides/performance.md: wrong detail — Performance table: paragraph memory O(longest_paragraph), markdown memory O(longest_section), markdown 'one chunk per heading', sliding O(chunk_size)
  severity Medium
  Where. docs/guides/performance.md:126-131 (Turkish mirror: docs/guides/performance-tr.md) · type: wrong detail
  The document says. Performance table: paragraph memory O(longest_paragraph), markdown memory O(longest_section), markdown 'one chunk per heading', sliding O(chunk_size)
  What the code does. All three operate on the whole extracted document text: paragraph builds the full list of paragraphs (text.split('\n\n'), ingestion.py:1491) and markdown builds all sections before packing (ingestion.py:1752), so memory is O(document), not O(longest unit). Markdown packs sections greedily up to chunk_size (ingestion.py:1757-1768), so a chunk holds one or MORE sections, not one per heading. Slidin…
  Code references. forgelm/ingestion.py:1482-1507, forgelm/ingestion.py:1722-1771
  Proposed correction. Drop the O() memory column (or state 'proportional to document text') and describe markdown as 'heading-aware: whole sections packed greedily up to chunk_size, each chunk prefixed with a heading breadcrumb; a section larger than chunk_size is emitted whole'.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D2B04-027 (docs, Medium) — docs/guides/performance.md: misleading — Packing is safe 'if attention masks are correct'; verify with forgelm audit
  severity Medium
  Where. docs/guides/performance.md:157 (Turkish mirror: docs/guides/performance-tr.md) · type: misleading
  The document says. Packing is safe 'if attention masks are correct'; verify with forgelm audit
  What the code does. forgelm audit (data_audit package) analyses length, language, duplicates, PII, secrets, quality; it has no attention-mask or packing check. Packing is delegated to TRL SFTConfig(packing=...).
  Code references. forgelm/trainer.py:693, forgelm/cli/_parser.py:496-512
  Proposed correction. Remove 'verify with forgelm audit'; say packing is delegated to TRL and recommend comparing eval loss with and without packing.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}