--merge accepts degenerate inputs (slerp with !=2 models, dare_drop_rate 1.0) that merging.py silently degrades, then reports success under the requested method (CR-A01-041) #363

Wave 1 · theme D1-11: Other high-severity packages · highest severity High · 7 finding(s) · effort small

Summary

An operator who raises dare_drop_rate toward the paper's 0.9+ regime and sets 1.0 gets the untouched base model published as a merge, with exit 0.

Primary finding

CR-A01-041 — --merge accepts degenerate inputs (slerp with !=2 models, dare_drop_rate 1.0) that merging.py silently degrades, then reports success under the requested method

severity Medium · reachability default · intent contract-violation · confidence confirmed · independently confirmed · reproduced end to end

What happens. MergeConfig._validate_merge_inputs only requires >=2 models for any method and allows dare_drop_rate up to 1.0 (forgelm/config.py:209-246). In forgelm/merging.py, _slerp_merge falls back to _linear_merge with only a WARNING when len(adapters)!=2 (181-185), and _dare_merge_tensor returns zeros for drop_rate>=1.0 (436-437), leaving the base unchanged. merge_peft_adapters still returns MergeResult(success=True, method=) (103-108), which the --merge JSON envelope echoes with exit 0 (forgelm/cli/_no_train_modes.py:152-172).

Where.

• forgelm/config.py:225-246 (_validate_merge_inputs; arity check at line 237)
• forgelm/merging.py:436-437 (confirmed)
• docs/reference/configuration.md:313
• forgelm/config.py:209-213
• forgelm/merging.py:103-108 (success=True at line 104)
• forgelm/merging.py:181-185 (fallback warning at line 184)
• forgelm/config.py:225-246 (confirmed)
• forgelm/merging.py:183-185 (confirmed)
• forgelm/cli/_no_train_modes.py:152-172 (JSON echoes result.method; exit 0)
• docs/reference/configuration.md:304-317

Evidence.

• work/dare1.yaml via fpy -m forgelm --config ... --merge --output-format json: {success: true, method: dare, num_models: 2}, EXIT=0; out_dare1/model.safetensors: all 26 tensors torch.equal to env/tiny-qwen/model.safetensors.
• t4/dare.py: MergeConfig(dare_drop_rate=1.0) was accepted. merge_peft_adapters(tiny-qwen, [sft, dpo], method='dare', dare_drop_rate=1.0) returned MergeResult(success=True, method='dare', num_models=2). Reloading the output, max |merged - base| = 0.0.
• merge/cfg.yaml with method slerp and three env/artifacts adapters (sft, dpo, kto): EXIT=0, JSON {success:true, method:'slerp', num_models:3}; log 'SLERP requires exactly 2 models. Got 3. Falling back to linear.'

Recommended fix. Make dare_drop_rate lt=1.0 to match ties_trim_fraction, or raise instead of returning zeros.

Other findings with the same root cause

These close with the same fix; update them in the same pull request.

• TECH-T14-004 (technology, Next release) — Hard-coded model identifiers and model-merging methods: Merge defaults: TIES ties_trim_fraction 0.2 and DARE dare_drop_rate 0.3, with the stated rationale 'the DARE paper recommends 0.9+' (forgelm/me…
  severity Next release
  Item. Merge defaults: TIES ties_trim_fraction 0.2 and DARE dare_drop_rate 0.3, with the stated rationale 'the DARE paper recommends 0.9+' (forgelm/merging.py:16-25,277-282; forgelm/config.py:195-219; docs/reference/configuration.md:318-326 + TR mirror) · verdict: incorrect-claim · urgency: Next release
  State in ForgeLM. merging.py:23 trims the bottom 20% (keeps 80%). merging.py:24 sets drop 0.3 with the comment '(paper recommends 0.9+ for FT deltas)'. The docstring at merging.py:277-282 and config.py:200-202 / :216 call both values deliberate, conservative departures from the papers. DARE is implemented as DARE-linear (random Bernoulli mask, 1/(1-p) rescale, no sign election; merging.py:432-452). configuration.m…
  State of the ecosystem on 2026-10-01. TIES paper (arXiv 2306.01708, NeurIPS 2023): the validation-free generic recipe keeps the top-20% of the task vector with lambda=1, chosen from k in {10,20,30} on PEFT ((IA)3) experiments. ForgeLM's description of TIES is accurate. The DARE paper (arXiv 2311.03099, ICML 2024) gives no default. It says DARE 'can effectively remove 90% delta parameters without significantly decreasing the performan…
  Recommendation. Reword merging.py:24 and :281-282, config.py:216 and configuration.md:321-322 (+ configuration-tr.md:381-382). The new text should say that the DARE paper reports tolerance up to ~0.9 (0.99 only at 70B), selects p by validation, used 0.3 and 0.5 for its own 7B merges, and needs small delta magnitudes. It should add that mergekit defaults to density 1.0 and PEFT's example uses density 0.2. Keep 0.…
  Sources.

  • Language Models are Super Mario (DARE), ICML 2024 - incl. A.4 grid [0.1..0.9] and A.5 drop rate 0.3 (supermario_v1) / 0… (2024-06-13 (v3))
  • DARE paper PDF v3 (ICML 2024) (2024-06-13 (v3))

• DOC-D3B08-016 (docs, High) — docs/usermanuals/en/deployment/model-merging.md: broken example — A config containing only model: + merge: (TIES example, also the Linear example at 57-68 and the Configuration section at 112-126…
  severity High
  Where. docs/usermanuals/en/deployment/model-merging.md:31-53 (Turkish mirror: docs/usermanuals/tr/deployment/model-merging.md:50) · type: broken example
  The document says. A config containing only model: + merge: (TIES example, also the Linear example at 57-68 and the Configuration section at 112-126) runs with forgelm --merge --config configs/merge.yaml. The eval config at 85-90 likewise has only evaluation:.
  What the code does. ForgeConfig requires lora, training and data blocks (config.py:1607-1612), and --merge validates the full config first (cli/_dispatch.py:242). Reproduced: the TIES file as shown exits 1 with 3 'Field required' errors. With base blocks added, the same merge keys validate.
  Code references. forgelm/config.py:1604-1612, forgelm/cli/_dispatch.py:242, forgelm/cli/_no_train_modes.py:128-135
  Proposed correction. Add the required placeholder blocks to each example (e.g. lora: {r: 8}, training: {output_dir: ./checkpoints/merge-run}, data: {dataset_name_or_path: ./data/train.jsonl}) and note they are required by the schema even for --merge / --benchmark-only.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-098 (docs, Medium) — docs/reference/configuration.md: wrong detail — These defaults are intentionally more conservative than the published TIES (keep top ~20%) and DARE (drop_rate 0.9+) defaults
  severity Medium
  Where. docs/reference/configuration.md:321-322 (Turkish mirror: docs/reference/configuration-tr.md:382) · type: wrong detail
  The document says. These defaults are intentionally more conservative than the published TIES (keep top ~20%) and DARE (drop_rate 0.9+) defaults
  What the code does. The TIES half is correct: the fixed recipe keeps the top 20% of each task vector, k=20 (arXiv 2306.01708). The DARE paper (arXiv 2311.03099, ICML 2024) publishes no default drop rate. It shows that 90-99% of the deltas of a single SFT model can be dropped, that tolerance grows with model size (WizardLM-70B already degrades at p=0.9 on AlpacaEval), and that the authors grid-search p in [0.1..0.9]…
  Proposed correction. These defaults retain more signal than the TIES paper's recipe (keep top ~20%). The DARE paper publishes no default drop rate: it shows that up to 90-99% of a single model's delta parameters can be dropped (tolerance grows with model size), and its own merges use drop rates as low as 0.3.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-044 (docs, Medium) — docs/usermanuals/en/deployment/model-merging.md: broken example — merge YAML with only model: and merge: blocks runs via forgelm --merge --config
  severity Medium
  Where. docs/usermanuals/en/deployment/model-merging.md:31-47,57-68,112-126 · type: broken example
  The document says. merge YAML with only model: and merge: blocks runs via forgelm --merge --config
  What the code does. ForgeConfig requires lora, training and data; the example exits 1 with 3 validation errors.
  Code references. forgelm/config.py:1607-1612
  Proposed correction. Add minimal lora/training/data blocks or state they are required.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-BLEGAL-206 (docs, Low) — docs/reference/configuration.md: wrong detail — [IS-T14-04] 'more conservative than the published TIES (keep top ~20%) and DARE (drop_rate 0.9+) defaults'
  severity Low
  Where. docs/reference/configuration.md:321-322 (Turkish mirror: docs/reference/configuration-tr.md:381-382) · type: wrong detail
  The document says. [IS-T14-04] 'more conservative than the published TIES (keep top ~20%) and DARE (drop_rate 0.9+) defaults'
  What the code does. The TIES half is correct (top-20% generic recipe). The DARE paper (arXiv 2311.03099, ICML 2024) publishes no default drop rate. It shows 90-99% of SFT deltas can be dropped from a single model, with tolerance growing with size (WizardLM-70B already degrades at 0.9). It grid-searches p in [0.1..0.9] for merges, and its own 7B leaderboard merges used p=0.3/0.5. mergekit's density default is 1.0.
  Proposed correction. These defaults retain more signal than the TIES paper's generic recipe (keep top ~20%); the DARE paper publishes no default drop rate (it tunes p, and used 0.3–0.5 for its own 7B merges), so drop_rate=0.3 matches its merge practice.
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

• DOC-D3B08-068 (docs, Low) — docs/usermanuals/en/deployment/model-merging.md: misleading — DARE 'pairs well with TIES (DARE-TIES)'.
  severity Low
  Where. docs/usermanuals/en/deployment/model-merging.md:27 (Turkish mirror: docs/usermanuals/tr/deployment/model-merging.md:27) · type: misleading
  The document says. DARE 'pairs well with TIES (DARE-TIES)'.
  What the code does. There is no DARE-TIES method: merge.method is Literal ties/dare/slerp/linear (config.py:186-189). dare does random drop and rescale, then a weighted sum, with no TIES trim or sign election (merging.py _dare_merge_tensor).
  Code references. forgelm/config.py:186-189, forgelm/merging.py:350-357
  Proposed correction. "Mitigates interference. ForgeLM's dare is plain drop-and-rescale; it does not combine with TIES (no DARE-TIES mode)."
  Turkish mirror. the Turkish page has the same error; fix both in the same PR

Acceptance criteria

• A regression test reproduces the failure scenario and asserts the corrected behaviour.
• Documentation listed above is corrected in English and in its Turkish mirror in the same PR.
• Every finding in this issue is resolved or explicitly closed with a reason.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}