[Wave 4] Low/info — Module sweep: Post-training: inference, chat, GGUF export, deploy, merging (part 1/3) #705

Wave 4 · 21 finding(s) in 20 fix package(s). Fix these whenever a pull request touches this module ("clean the module you touch").

Findings

• CR-A13-034 (code, Low) — Chat REPL passes non-model text through rich markup unescaped: /help loses bracketed arguments, and stray closing tags raise MarkupError and end the session
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end
  Failure scenario. Users cannot tell that /save and /system take an argument.
  Where. forgelm/chat.py:227, forgelm/chat.py:229, forgelm/chat.py:218, forgelm/chat.py:221-236
  Recommended fix. Escape with rich.markup.escape or print help via the plain path.

• CR-A13-035 (code, Low) — Merge does not check adapter base-model compatibility
  severity Low · reachability documented-opt-in · intent intended-but-harmful · confidence likely · single reviewer · reproduced (plumbing only)
  Failure scenario. Same-shape adapters from different bases merge into a meaningless model with a success exit, contradicting the documented rejection.
  Where. forgelm/merging.py:64-112, forgelm/merging.py:135
  Recommended fix. Compare adapter_config base_model_name_or_path with model.name_or_path (warn/fail) and map shape errors to a clear message.

• ↳ same root cause: DOC-D3B08-069 (docs, Low) — docs/usermanuals/en/deployment/model-merging.md: misleading — slerp requires exactly two entries; adapters of a different base are rejected with a clear error
  severity Low
  Where. docs/usermanuals/en/deployment/model-merging.md:77,150 · type: misleading
  The document says. slerp requires exactly two entries; adapters of a different base are rejected with a clear error
  What the code does. 3 slerp entries silently become linear; only a shape mismatch raises a raw PEFT RuntimeError (exit 2); base-name mismatch passes.
  Code references. forgelm/merging.py:183-185, forgelm/merging.py:64-112
  Proposed correction. Fix F7/F8 or soften docs.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• CR-A13-036 (code, Low) — Merge entries must be PEFT adapters; full fine-tuned models, promised by the module docstring, user manual and field description, fail with exit 2
  severity Low · reachability default · intent contract-violation · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. An operator following the user manual lists two fully fine-tuned checkpoints and gets a training-error exit with a PEFT error message.
  Where. forgelm/merging.py:3, forgelm/merging.py:135, forgelm/merging.py:304, forgelm/config.py:166
  Recommended fix. Either narrow the docs and field description to PEFT adapters, or load full-model entries with AutoModelForCausalLM and take deltas against the base.

• CR-A13-037 (code, Low) — load_model silently ignores --load-in-4bit/--load-in-8bit without CUDA and loads full precision
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. An operator on a 16 GB Mac runs forgelm chat <7B> --load-in-4bit. The process loads around 28 GB of fp32 weights and is OOM-killed with no hint that quantisation was skipped.
  Where. forgelm/inference.py:91-104, forgelm/cli/subcommands/_safety_eval.py:156 (refuted), forgelm/cli/_no_train_modes.py:80-84, forgelm/inference.py:69-71
  Recommended fix. Log a WARNING, or raise, when a quantisation flag is set and CUDA is unavailable.

• CR-A13-038 (code, Low) — The export merge loads the base tokenizer instead of the adapter's, with no revision pin, unlike inference._pick_tokenizer_source
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · not reproduced (code trace)
  Failure scenario. An adapter trained with a custom ChatML template on a base without one is exported. The GGUF has no or the wrong template, and Ollama or llama.cpp formats prompts differently from what forgelm chat showed.
  Where. forgelm/export.py:155-185, forgelm/inference.py:23-38
  Recommended fix. Reuse _pick_tokenizer_source(model_path, adapter_path), and accept or propagate a revision for Hub bases.

• CR-A13-040 (code, Low) — The Unsloth backend warns that it is 'preferring 4-bit' when both flags are set, but passes both flags unchanged
  severity Low · reachability documented-opt-in · intent contract-violation · confidence likely · independently confirmed · not reproduced (code trace)
  Failure scenario. The operator sees 'preferring 4-bit', and then Unsloth either errors on the conflicting flags or loads in a mode the log did not announce.
  Where. forgelm/inference.py:69-78, forgelm/inference.py:147-157
  Recommended fix. Set load_in_8bit=False when load_in_4bit is set before calling _load_unsloth.

• CR-A13-041 (code, Low) — The CLI export cannot raise the fixed 3600 s converter timeout, yet the timeout error tells the user to pass timeout_seconds
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · not reproduced (code trace)
  Failure scenario. forgelm export <70B> --quant f16 is killed after one hour with exit 2, and the operator has no CLI way to extend the limit.
  Where. forgelm/cli/subcommands/_export.py:16-23 (confirmed), forgelm/export.py:329-340 (confirmed), forgelm/export.py:400-402, forgelm/export.py:382
  Recommended fix. Add --timeout-seconds to the export subparser and mention it in the error text.

• CR-A13-042 (code, Low) — Forgelm chat routes every failure, operator-input errors included, to exit 2 through a catch-all Exception at CLI level, with tracebacks and non-actionable messages
  severity Low · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. forgelm chat ./typo-dir exits 2 with a traceback and an HF repo-id validation message, and CI retries it as a transient runtime failure. forgelm deploy ./typo-dir --target tgi exits 1 for the same mistake.
  Where. forgelm/cli/subcommands/_chat.py:13-34, forgelm/inference.py:133-138, forgelm/export.py:414-433, forgelm/merging.py:110-112
  Recommended fix. In _run_chat_cmd, catch (ValueError, FileNotFoundError, OSError path-not-found, ImportError) as input/environment errors and exit 1 with an actionable message, without a traceback. Let unknown errors propagate or exit 2. Pre-check that a local-looking model_path exists. Apply the same split to the export adapter/model path checks and to merge source paths.

• CR-A13-043 (code, Low) — Converter failure message in the JSON envelope keeps the first 500 characters of stderr, dropping the actual exception
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. Exporting an unsupported architecture fails mid-conversion. The JSON envelope's error is 500 characters of 'INFO:hf-to-gguf:Loading model part ...' lines with no 'NotImplementedError: Architecture ... not supported'.
  Where. forgelm/export.py:347-360
  Recommended fix. Keep the tail (error_detail[-500:]) or extract the last non-empty line(s) of stderr.

• CR-A13-044 (code, Low) — JSON-output contract page for export/deploy disagrees with the shipped envelopes and exit codes
  severity Low · reachability default · intent contract-violation · confidence likely · single reviewer · not reproduced (code trace)
  Failure scenario. A CI consumer written to the contract treats exit 1 from export as impossible, and treats quant as the requested level. It ships an intermediate f16 GGUF believing it is q4_k_m, because it never reads manual_step_required.
  Where. docs/usermanuals/en/reference/json-output.md:863-890, forgelm/cli/subcommands/_export.py:25-70, forgelm/export.py:266-292, forgelm/deploy.py:316-330
  Recommended fix. Update the EN and TR contract pages: exit 1 for config errors, the three K-quant keys, a K-quant example showing quant f16 plus manual_step_required, and the deploy render-error code. Document the --merge envelope.

• CR-A13-045 (code, Low) — Merge arithmetic runs in the checkpoint dtype (bf16/fp16): task vectors are recovered as round(W+dW)-W and linear accumulation is done in low precision
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced with stubs
  Failure scenario. TIES trim and sign election operate on noisy or zeroed deltas for very small LoRA updates, giving a slightly different merge than a float32 computation.
  Where. forgelm/merging.py:306, forgelm/merging.py:139-143, forgelm/merging.py:360
  Recommended fix. Accumulate in float32 per key (streaming, as proposed above) and cast back once; or compute the delta from the LoRA factors. Gain: fidelity only, no extra peak memory when done per key.

• CR-A13-046 (code, Low) — generate_stream has no behavioural test; its only test greps the function source for a type annotation
  severity Low · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end
  Failure scenario. A regression leaks the prompt into the streamed reply, swallows a worker-thread OOM so the REPL shows an empty answer as success, or changes the timeout behaviour without any test failing.
  Where. forgelm/inference.py:259-339, forgelm/inference.py:310, forgelm/inference.py:325, tests/test_inference.py:616-639
  Recommended fix. Add real-model tests on tiny-qwen (or a stub model whose generate pushes tokens into the streamer): stream equals non-stream under greedy decoding, the prompt is excluded, a raising generate re-raises the original exception, and the timeout path raises TimeoutError. Replace the source-grep test.

• CR-A13-047 (code, Low) — A model.generate() failure inside generate_stream surfaces only after a fixed 60 s stall (the chat REPL freezes)
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced end to end · already tracked: KNOWN-OPEN CLI-08 — docs/roadmap/phase-16-trust-surface-hardening.md:475-476 (S6 open; unit id inferred)
  Failure scenario. In forgelm chat, a CUDA OOM on the first long prompt leaves the terminal frozen for a minute before '[Generation error: ...]' is printed. Slow generation that legitimately pauses more than 60 s (large CPU model) is reported as a timeout.
  Where. forgelm/inference.py:286-296, forgelm/inference.py:308-339, forgelm/inference.py:325
  Recommended fix. On worker failure push the stop signal into the streamer (streamer.end() in a finally) or poll thread liveness with a short timeout; add a test with a tiny timeout.

• CR-A13-048 (code, Low) — The REPL loop, run_chat wiring and the chat dispatcher's failure exit are untested (F-130)
  severity Low · reachability default · intent intended-and-acceptable · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. Chat could loop forever on EOF (a piped or CI invocation) or ignore the --adapter flag without a failing test.
  Where. forgelm/chat.py:119, forgelm/chat.py:358, forgelm/cli/subcommands/_chat.py:20, forgelm/cli/subcommands/_chat.py:32
  Recommended fix. Add scripted-input tests for run() using input_fn (text, /exit, EOF, KeyboardInterrupt, empty line), a run_chat test asserting the exact load_model kwargs, and _run_chat_cmd tests for argument mapping and the exit-2 failure path.

• CR-A13-049 (code, Low) — export/deploy dispatcher flag wiring is unpinned (--no-integrity-update, --vendor, --max-length, --port, --gpu-memory-utilization)
  severity Low · reachability unreachable · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
  Failure scenario. --no-integrity-update stops working, or a --vendor azure request silently produces an aws endpoint spec, with a green suite.
  Where. forgelm/cli/subcommands/_export.py:22, forgelm/cli/subcommands/_deploy.py:25, tests/test_export.py:600, tests/test_cli_phase10.py:712
  Recommended fix. Add CLI round-trip tests that read the output file or assert export_model kwargs for each flag, including a --no-integrity-update run that leaves model_integrity.json unchanged.

• CR-A13-050 (code, Low) — Export failure paths and the Modelfile SYSTEM escaping are weakly pinned
  severity Low · reachability default · intent intended-and-acceptable · confidence likely · single reviewer · reproduced end to end
  Failure scenario. A regression in the error mapping or the escaping goes unnoticed.
  Where. forgelm/export.py:329, forgelm/export.py:344, forgelm/export.py:453, forgelm/deploy.py:114
  Recommended fix. Assert the exact error string and error_kind for nonzero exit, timeout and missing output; add a test for each escape (newline, CR, backslash) round-tripping the SYSTEM directive.

• CR-A13-051 (code, Low) — inference.py sampling and load paths are tested through hand-rolled torch stubs with near-vacuous assertions
  severity Low · reachability unreachable · intent intended-and-acceptable · confidence likely · independently confirmed · reproduced end to end
  Failure scenario. A wrong sign or condition in the sampler or in the quantisation config ships without a test failure.
  Where. forgelm/inference.py:91-104, forgelm/inference.py:408-427, forgelm/inference.py:415, forgelm/inference.py:422
  Recommended fix. Use real torch for logit_stats/adaptive_sample with seeded generators and assert distributions/membership; mock only BitsAndBytesConfig and from_pretrained for the quantisation branches and assert their kwargs. Log a warning when a quantisation flag is ignored.

• CR-A13-053 (code, Low) — A failed or timed-out GGUF conversion leaves a partial file at the final output path
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced with stubs
  Failure scenario. A 70B conversion hits timeout_seconds, subprocess.run kills the child, and a truncated model.f16.gguf stays on disk. The next tool, a deploy Modelfile or a manual ollama create, picks it up because the name looks valid. If an earlier export existed at that path, the new failure destroys it. model_integrity.json still lists the previous sha256 for the same file name, so the manifest and file silently disagree. Reproduced: a converter that writes 12 bytes and exits 1 leaves the 12-byte file after…
  Where. forgelm/export.py:313-360, forgelm/export.py:444-448
  Recommended fix. Have the converter write to a temp name and rename on success, or unlink actual_output_path when the converter fails or times out.

• CR-A13-054 (code, Low) — Deploy accepts a nonexistent local-looking model path for vllm and hf-endpoints and emits a config that cannot work
  severity Low · reachability default · intent intended-but-harmful · confidence likely · single reviewer · reproduced end to end · already tracked: KNOWN-OPEN CLI-15 — docs/roadmap/phase-16-trust-surface-hardening.md:627-628 (S11 open; unit id inferred)
  Failure scenario. forgelm deploy ./outputs/final_modle --target hf-endpoints --output hf.json exits 0 and writes repository './outputs/final_modle'. The failure appears only when the endpoint is created. Reproduced: './nope' with hf-endpoints exited 0. The property about input errors mapping to exit 1 and OSError to exit 2 did hold: ./nope with tgi or ollama gave exit 1, and a directory or /dev/null/x output gave exit 2.
  Where. forgelm/deploy.py:261-274, forgelm/deploy.py:131-155
  Recommended fix. For vllm and hf-endpoints, warn or fail when model_path looks like a filesystem path (starts with ./, ../, / or ~, or contains a separator and no '/' Hub-ID shape) and does not exist.

• CR-A13-055 (code, Low) — export_model's adapter path has no test for a pre-existing '_merged_for_export', a partial _merge_adapter failure, or an HF-ID model_path
  severity Low · reachability default · intent intended-but-harmful · confidence likely · independently confirmed · reproduced (plumbing only) · already tracked: KNOWN-OPEN C33-EXPORT-TEMP-OWNERSHIP — docs/roadmap/phase-16-trust-surface-hardening.md:468-470 (S6 open)
  Failure scenario. A user who has a directory named '_merged_for_export' loses it on export --adapter. A disk-full merge leaves a multi-GB partial dir behind. The test suite passes in both cases.
  Where. tests/test_export.py:449-579, tests/test_export.py:549-584, forgelm/export.py:447-450, forgelm/export.py:362-369
  Recommended fix. Add four tests: a pre-existing merged_dir; a merge that writes then raises (assert no leftover dir); an HF-ID model_path; and a failure between merge and converter. Fix M-A13-008 (unique tempfile.mkdtemp directory with try/finally spanning the merge) so these tests pass.

Done when

• Every finding above is fixed or closed with a reason (tick it here).
• Code fixes carry a test; doc fixes update the Turkish mirror in the same PR.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}