[Strategy] Approaches to retire or re-scope (technology review, October 2026) #839

Wave 2 · 9 approach-level verdict(s) from the October 2026 technology review (state of the ecosystem on 2026-10-01). Each is a product decision; the concrete defects beneath an approach are tracked by the findings it lists.

Approaches

GaLore optimizer path advertised as full-parameter training at LoRA memory

Verdict retire · confidence medium · tracked findings: TECH-T03-008, TECH-T03-009

Why. What runs is not what is advertised. get_peft_model is always applied (forgelm/model.py:609), so GaLore only projects the gradients of the LoRA adapter parameters and the base weights stay frozen (T03[7]). The docs, the wizard and fit_check all describe full-parameter learning. The dependency is also at risk. galore_torch is an unmaintained 2024 wheel that transformers itself does not recommend, and its 8-bit variants silently change optimizer arguments under bitsandbytes>=0.50 (T03[6]). transformers 5.18 ships APOLLO (apollo_adamw) as a maintained alternative.

Recommendation. Either implement a real no-adapter path when galore_enabled is set, or rename the feature everywhere to 'LoRA + GaLore optimizer' and fix fit_check. Add a [galore] extra pinned to a tested combination, refuse 8-bit GaLore with bnb>=0.50, and evaluate APOLLO as the replacement.

Optional backend and architecture extensions: Unsloth backend and MoE expert controls (model.moe.experts_to_train, quantize_experts)

Verdict retire · confidence medium · tracked findings: TECH-T03-001, TECH-T03-002, TECH-T03-003, TECH-T03-011

Why. Neither currently engages. Every current unsloth requires trl<=0.24, datasets<4.4 and transformers<=5.5, so pip install forgelm[unsloth] cannot install any unsloth built for ForgeLM's core pins (T03[9]). Under transformers 5, Mixtral, Qwen2/3-MoE, DeepSeek-V3, gpt-oss and Llama-4 hold experts as fused 3-D parameters. ForgeLM's per-expert name regexes therefore match nothing, and experts_to_train and quantize_experts become silent no-ops (INFO/WARNING only). README.md:156 'MoE-aware (Qwen3, Mixtral, DeepSeek)' overstates this (T03[12]). A silent no-op on an operator config is a footgun.

Recommendation. Withdraw both from the advertised feature set until they are reimplemented. Raise a ConfigError on fused-expert models instead of a silent no-op, or reimplement through peft target_parameters plus gradient masking. Document [unsloth] as a separate-environment backend, and add a pip install --dry-run 'forgelm[unsloth]' resolution check to CI.

Distributed training wiring: FSDP through the legacy TrainingArguments string API (FSDP1 semantics), plus DeepSpeed ZeRO JSON presets

Verdict retire · confidence high · tracked findings: TECH-T02-004, TECH-T02-009, TECH-T04-002, TECH-T04-003, TECH-T04-005, TECH-T04-009

Why. transformers made FSDP2 the default between 5.8 and 5.10, and a string or list fsdp value now logs that it 'will be removed in Transformers v5.20'. On transformers >=5.10, no_shard and hybrid_shard crash at TrainingArguments construction, fsdp_backward_prefetch is silently ignored, and no FSDP run can start on the floor pair (T02[4], T04[4], T02[6]). DeepSpeed ZeRO itself remains valid, but the 'built-in' zero2/zero3/zero3_offload presets are not in the wheel (T04[3]), and the ZeRO-3 load-order claims are wrong (T04[9]). Long-context parallelism (DeepSpeed-Ulysses/ALST, context parallelism) is not offered (T04[8]).

Recommendation. Retire the FSDP1 string form. Emit fsdp=True with fsdp_config{version:2, reshard_after_forward, cpu_offload, state_dict_type, auto_wrap_policy}, and reject no_shard, hybrid_shard and backward_prefetch at config time. Cap transformers<5.20 until the migration lands, and raise accelerate to >=1.7. Ship the DeepSpeed presets as package data and test them.

Unsigned, self-referential integrity manifests as tamper evidence: model_integrity.json (SHA-256 file list stored inside the model directory; verify-integrity) and the Annex IV metadata.manifest_hash…

Verdict retire · confidence high · tracked findings: TECH-T10-006

Why. Under the project's threat model, the producer of an artefact handed to verify-* is the adversary. Both manifests are unkeyed hashes stored with the thing they protect, so whoever can edit the artefact can recompute them. generate_model_integrity writes plain SHA-256 entries (forgelm/compliance.py:760-817; written at forgelm/trainer.py:2157-2168), and verify_integrity re-hashes against the copy in the same directory (forgelm/verify/_model_integrity.py:81-86). No HMAC or signature is ever applied to it: HMAC appears only in the audit logger (forgelm/compliance.py:575-579, 2434). (Part of this assessment concerns a vulnerability tracked in a private security advisory until a fix is released; see SECURITY.md.)

Recommendation. Keep the SHA-256 walk, but present it only as a corruption check. Add an optional [signing] extra built on model_signing (OMS): sign final_model, adapters, merged models and GGUF exports at the end of the run, with key or certificate for air-gapped sites and Sigstore keyless in CI. verify-integrity should verify the OMS bundle against an identity or key policy and return exit 6 on a signature mismatch. Add a --require-hash or --require-signature mode to verify-annex-iv that fails when no hash or signature is present. (Part of this assessment concerns a vulnerability tracked in a private security advisory until a fix is released; see SECURITY.md.)

Treating the data audit (PII, secrets and quality scans) plus a dataset SHA-256 fingerprint as the training-data-poisoning control (RTP R-01, residual rated Med x Low)

Verdict retire · confidence high · tracked findings: TECH-T10-012

Why. None of these scans detects poisoning or backdoors. The risk treatment plan nonetheless lists them as the R-01 treatment and lowers the residual risk to MED (docs/qms/risk_treatment_plan.md:42-51). No poisoning, backdoor or trigger detection exists anywhere in forgelm/ (rg over the snapshot). 2026 evidence makes the gap material. Souly et al. (arXiv 2510.07192) found that 250 poisoned documents compromise models from 600M to 13B parameters regardless of corpus size, and showed 'the same dynamics for poisoning during fine-tuning'. Qi et al. (arXiv 2310.03693) jailbroke a model by fine-tuning on 10 examples. OWASP LLM05:2026 now 'absorbs fine-tuning subversion'. Its mitigations include lineage tracking with signing (1), anomaly detection over training data and loss (5), and 'Dedicated trigger-probing ... after every alignment cycle' (10). LLM06:2026 describes output explosion caused by a…

Recommendation. Re-rate R-01 honestly and state in the docs that ForgeLM does not detect poisoning. Add opt-in controls: a post-training trigger and backdoor probe stage that compares the base model with the fine-tuned model; corpus anomaly signals such as over-represented rare n-grams and inconsistent chosen/rejected pairs; a generation-length/EOS check after training; and a policy that requires pinned dataset revisions plus signed dataset manifests. Map these to Art. 15(5) as evidence of 'measures', not as guarantees.

GGUF export that relies on a convert_hf_to_gguf.py 'bundled' with llama-cpp-python (the [export] extra)

Verdict retire · confidence high · tracked findings: TECH-T08-001, TECH-T08-002, TECH-T08-007

Why. The mechanism never worked from a pip install. llama-cpp-python has never shipped a wheel on PyPI (209 releases, 0 wheels), and its sdist carries the converter only under vendor/llama.cpp, so no installed llama-cpp-python contains it. forgelm export therefore fails on every [export] install unless FORGELM_GGUF_CONVERTER points into a full llama.cpp checkout (T08[0], incorrect-claim/now). The quant menu is valid but stale. Only f16/q8_0 come from the converter directly, the forced f16 intermediate loses bf16 fidelity, and K-quants need llama-quantize (T08[2]). verify-gguf skips the metadata parse because gguf is undeclared (T08[1]). OWASP LLM04:2026 Risk 5 calls conversion and quantization workflows high-risk promotion points, so GGUF outputs should fall under the signing proposed above.

Recommendation. Document that a llama.cpp checkout or converter is required and make doctor test for an actual converter. Implement discovery (PATH, an explicit path, llama.cpp checkout). Use --outtype auto or bf16 and run llama-quantize when it is found. Declare gguf>=0.19. Sign the exported GGUF with the model's OMS bundle.

Notifications and monitoring: a single generic-JSON webhook payload presented as Slack/Teams/Discord-compatible, and a declarative monitoring.metrics_export (prometheus/datadog/custom_webhook) with n…

Verdict retire · confidence high · tracked findings: TECH-T09-001, TECH-T09-002, TECH-T09-003, TECH-T09-010

Why. Office 365 Connectors were retired in May 2026 (rollout completed 2026-05-22). A Teams Workflows URL accepts ForgeLM's generic JSON with a 2xx response and then shows nothing. ForgeLM logs success, and the approval-required notice can therefore go unseen (T09[0], incorrect-claim/now). Discord likewise needs content or embeds (T09[1]). metrics_export exports nothing and gives no warning, yet the compliance manifest and the Annex IV artefact record a post-market-monitoring setup that never runs (T09[3]). Experiment tracking through report_to (tensorboard, wandb, mlflow) is current (T09[9]).

Recommendation. Fix the docs now. Add an opt-in webhook.format selector (generic, teams_adaptive, discord) that builds valid cards. Either implement a Pushgateway or OTLP exporter or relabel metrics_export as declarative metadata with a config-load warning, and stop recording it as active monitoring in the Annex IV artefact.

Resource planning heuristics: fit-check VRAM model (AdamW 8 B/param, activations 12sbhL, sqrt(L) checkpointing factor, ±15% claim) and a built-in GPU price table

Verdict retire · confidence medium · tracked findings: TECH-T13-002, TECH-T13-003

Why. The activation term is below the published per-layer estimate (Korthikanti et al.: 34sbh plus an attention term). HF gradient checkpointing does not follow the sqrt(L) rule. fit_check also models a no-checkpointing run that ForgeLM never trains, so the ±15% claim is unsupported (T13[10], superseded-method). The 'mid-2026 cloud averages' price table and its name matcher are wrong in places (T13[9]).

Recommendation. Replace the formula with about 34sbh per layer, per-layer checkpointing that matches the trainer, and explicit logits and embedding terms. Drop the accuracy claim or calibrate it against measured runs. Make prices operator-supplied rather than built in.

Platform baseline: Python 3.10 floor, CUDA-or-CPU device model (Apple Silicon forced to CPU, no ROCm/XPU statement), CUDA 12.4 Docker base

Verdict retire · confidence high · tracked findings: TECH-T02-008, TECH-T13-001, TECH-T13-004, TECH-T13-005, TECH-T13-006, TECH-T13-009

Why. Python 3.10 reached end of life on 2026-10-01 (PEP 619; 3.10.22 was the final security release), yet a compliance-oriented toolkit still declares it supported (T13[0]). Python 3.14 is absent from the classifiers and CI (T13[1]). With use_cpu=not torch.cuda.is_available(), Apple Silicon always trains on CPU, although transformers picks MPS automatically (T02[5], T13[7]). The nvidia/cuda:12.4.1 base image is end-of-life (T13[3]), and the docs say nothing about ROCm or XPU (T13[11]).

Recommendation. Announce the deprecation now and move to requires-python>=3.11 at the next minor, adding a 3.14 lane. Let transformers pick the device, or add an explicit device switch, with MPS behind a smoke test. Rebase the Docker image on a supported CUDA base. State the ROCm/XPU position.

Independent counter-reading

A second reviewer re-assessed the approaches without seeing these verdicts first. Where it disagreed:

• Training audit log, rated adopt/medium in the synthesis: I rate it watch. I established it end to end. That is a fail-open of the stated guarantee (hook 3). The default must change before 'adopt' is justified, and th… (Part of this assessment concerns a vulnerability tracked in a private security advisory until a fix is released; see SECURITY.md.)

• Post-training safety gate: the synthesis does not record that exported quantized/GGUF artefacts are never safety-gated (forgelm/export.py has no evaluation call). OWASP LLM04:2026 line 32 says 'full-precision assurances do not transfer to the deployed quantized artifact'. The synthesis's GGUF item proposes signing only, and signing proves integrity, not safety (LLM04 line 48).

• Release supply chain, rated adopt/high in the synthesis: the PEP 740 part merits adopt. The SBOM part is an urgency-now incorrect claim: never attached to a release, inventories the test environment, no hashes. It should be watch rather than folded into a high-confidence adopt.

• Unsloth/MoE, rated retire in the synthesis: retiring the Unsloth backend conflates a packaging/pin incompatibility with the approach. Unsloth remains an active, widely used backend. I would withdraw the advertised claim and rate it watch. Retiring the silent MoE no-op claim is fine.

• Minor legal omission: Reg. (EU) 2026/1744 adds Art. 42(3) to Reg. 2024/1689. High-risk systems within the CRA that meet CRA Art. 12(1) are deemed to comply with the Art. 15 cybersecurity requirements (verified in my cached OJ text). The synthesis's poisoning, Art. 15 and CRA items do not mention it.

Done when

• A decision is recorded for each approach (adopt the recommendation, re-scope, or keep with a reason), and the related findings are updated or closed accordingly.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}