[Strategy] Approaches to watch (technology review, October 2026) #840

Wave 4 · 7 approach-level verdict(s) from the October 2026 technology review (state of the ecosystem on 2026-10-01). Each is a product decision; the concrete defects beneath an approach are tracked by the findings it lists.

Approaches

ORPO and SimPO (SimPO through CPOTrainer, loss_type='simpo') offered as first-class trainers next to SFT/DPO/KTO/GRPO

Verdict watch · confidence high · tracked findings: TECH-T01-001, TECH-T01-003, TECH-T01-004, TECH-T01-006

Why. Both paths fail before training on every trl the range admits. ForgeLM imports ORPOConfig/ORPOTrainer/CPOConfig/CPOTrainer from the top level of trl (forgelm/trainer.py:723, 737, 1037, 1047). TRL moved them to trl.experimental in v0.26.0 and removed the top-level shims in v0.29.0, so each import raises ImportError (T01[0], T01[1]). Upstream they are experimental, and TRL says such code 'may change or be removed in any release'. The method evidence is mixed. Tülu 3 put SimPO below its SFT base, and a 2026 controlled study found no DPO variant that significantly beats vanilla DPO (T01[10], T01 Q8). No source names a method that replaced ORPO, so it is aging and niche rather than retired. README.md:153 still lists them on equal footing ('6 types: SFT, DPO, SimPO, KTO, ORPO, GRPO').

Recommendation. Import through a resolver that tries the top level, then trl.experimental.orpo/cpo, then fails with an actionable ConfigError. Label both 'experimental upstream' in the README and the manuals, and remove them from the headline trainer count. Recommend DPO, or DPO with loss_type='sigmoid_norm', as the default preference method. Make a missing advertised class fail CI instead of skipping.

Bundled static probe sets as the safety evidence base (51 default probes in the wheel, 140 category probes repo-only)

Verdict watch · confidence high · tracked findings: TECH-T05-011, TECH-T05-013

Why. The sets are honest smoke tests. The 51-prompt default is described as a smoke-test seed, and a 5% gate on 51 prompts moves in roughly 2% steps (T05[5]). The 140-prompt library is not in the wheel. 2026 practice treats a static probe pass as weak evidence. OWASP LLM01:2026 mitigation 11 says to 'Test against adaptive attackers who have read the deployed defense, and reject static-only attack-success claims'. LLM05:2026 mitigation 10 requires 'Dedicated trigger-probing ... after every alignment cycle'. The default set has only 5 jailbreak prompts (forgelm/safety_prompts/default_probes.jsonl, category counts) and no backdoor-trigger probes. AILuminate v1.0 offers a public 12-hazard benchmark family (T05[11]).

Recommendation. Keep the sets as a regression smoke layer and say in the docs that passing is not robustness evidence. Package the 140-prompt library as package data, or label it repo-only. Map categories to AILuminate hazard names. Add a hook for operator-supplied adaptive red-team sets and a trigger-probe set (see the poisoning item).

lm-evaluation-harness benchmark gate with arc_easy/hellaswag/mmlu/truthfulqa as the documented tasks and an unweighted mean accuracy

Verdict watch · confidence high · tracked findings: TECH-T05-003, TECH-T05-004, TECH-T05-010

Why. As a cheap before/after regression probe on small fine-tunes, these tasks still give signal. For comparing models they are saturated or contaminated, per HF's Open LLM Leaderboard v2 post, and HF has since retired that leaderboard (T05[10]). The aggregation is unsound. benchmark.py:195 averages every result row, so mmlu's subject rows dominate. 'truthfulqa' is a tag with no aggregate row, so its gen task is scored through a bleu_acc fallback. The mmlu_tr/belebele_tr templates the docs mention do not exist. Dependency issues: the lm-eval floor (0.4.4) admits releases that cannot import under transformers 5 (T05[9]), and cache-tasks uses the deprecated get_task_dict and mishandles group tasks (T05[8]).

Recommendation. Keep the gate as an explicitly labelled regression smoke set. Average only the requested tasks or group aggregates. Use truthfulqa_mc2. Offer an optional harder preset (IFEval, MMLU-Pro, GPQA, BBH) with per-task metrics. Raise the floor to lm-eval>=0.4.10, move cache-tasks to TaskManager.load, and record the lm_eval version in the audit event.

Provenance of trained artefacts recorded in ForgeLM-specific JSON (training manifest, Annex IV artefact, data_audit_report, Croissant card) with no standard AI-BOM or provenance attestation

Verdict watch · confidence medium · tracked findings: TECH-T06-017, TECH-T12-003

Why. ForgeLM already gathers most of an AI-BOM: base-model revision pins, dataset fingerprints, hyperparameters, metrics, safety results and risk fields. It does so only in its own formats. 2026 practice expects standard, signed inventories. OWASP LLM04:2026 mitigation 4 asks for 'an up-to-date, signed inventory of components using a Software Bill of Materials (SBOM), extended to models, adapters, and datasets with AI BOMs (AIBOMs) and ML SBOMs', naming the CycloneDX ML-BOM. LLM05:2026 mitigation 1 asks for lineage tracking via SBOM/ML-BOM with signing and verification. CycloneDX's AI/ML-BOM covers model and dataset provenance and training methodology. The SPDX 3.0.1 AI profile defines finetuningEnergyConsumption, hyperparameter, metric, safetyRiskAssessment, standardCompliance and useSensitivePersonalInformation, fields ForgeLM's manifest already holds. SLSA v1.2 (status Approved) defines b…

Recommendation. Add an opt-in export of a CycloneDX (>=1.6) ML-BOM, with a modelCard component per trained model or adapter referencing the base-model revision, dataset fingerprints and evaluation results, and/or an SPDX 3.0.1 AI-profile document. Embed it as an in-toto predicate inside the OMS signature from the previous item. Do not claim a SLSA level.

Secrets scanning of training corpora with an in-house nine-family regex set (always-on audit gate and ingest --secrets-mask)

Verdict watch · confidence high · tracked findings: TECH-T06-001, TECH-T12-005

Why. Scanning corpora for credentials is the right control; OWASP LLM02:2026 lists API keys and credentials as protected information. The hand-maintained rule set is the weak part. gitleaks v8.30.1 ships 222 rules, trufflehog has 886 detectors, and GitHub secret scanning covers 526 types. ForgeLM misses hf_/api_org_, PKCS#8 'BEGIN PRIVATE KEY' (the GCP service-account format), GitLab, Stripe, npm, PyPI, Slack xapp-/xoxe-, and AWS ABIA/ACCA keys. It labels Anthropic keys as OpenAI and only partly masks ghs_ JWTs (T06[13], outdated/now). The docstring's claim that these are 'the families mainstream scanners agree on' no longer holds.

Recommendation. Keep the gate. Widen the patterns as T06[13] lists, or vendor a maintained upstream rule corpus with a pinned version and a refresh cadence, and record the rule-set version in the audit report. Note that gitleaks upstream is now security-patch-only (T12 Q6).

Heuristic quality filters (Gopher/C4-style thresholds) and langdetect-based language identification in the data audit

Verdict watch · confidence medium · tracked findings: TECH-T06-005, TECH-T06-006, TECH-T07-007

Why. Rule-based quality flags are still a cheap first pass. ForgeLM's thresholds are its own adaptations and differ from Gopher's: character-level alpha 0.70 against word-level 0.8, and word length 3-12 against 3-10. The user manual documents checks that do not exist (T06[14]). Language identification rests on langdetect, whose last release was in 2021, with 2014 profiles and 55 languages. It is misdescribed as a CLD2 port. DataTrove defaults to fastText lid.176 and offers GlotLID (T06[15]). This language-distribution evidence is inlined into the Article 10 governance artefact.

Recommendation. Keep the heuristics as advisory flags. Rewrite quality-filter.md from _quality.py, say where the thresholds differ from Gopher, and cite arXiv 2112.11446. Retire langdetect as the only backend: add a pluggable GlotLID or fastText backend loaded from an offline model file, and record the backend in the report.

Sensitive-information exposure of fine-tuned weights (memorisation and extraction) handled only by ingest-time masking plus a memorisation warning on purge

Verdict watch · confidence medium · tracked findings: TECH-T11-001

Why. ForgeLM masks before training and warns that purge does not unmemorise weights. It has no memorisation, extraction or membership-inference test and no DP-SGD option (rg over forgelm/ finds none). OWASP LLM02:2026 reports that fine-tuned models and their LoRA adapters are more extractable than base models of the same scale, that deduplication reduces memorisation without eliminating it, and that DP-SGD paired with detection is the training-time mitigation. EDPB Opinion 28/2024 requires a case-by-case demonstration that extraction is insignificant before a model can be called anonymous (T11[2]). ForgeLM's manual instead implies masking makes the model free of personal data.

Recommendation. Fix the anonymity wording now (T11[2]). Add an optional extraction probe to the evaluation stage, for example verbatim-continuation tests on held-out training prefixes or canary strings, and record its result as evidence without claiming anonymity. Document that DP training is not supported.

Done when

• A decision is recorded for each approach (adopt the recommendation, re-scope, or keep with a reason), and the related findings are updated or closed accordingly.

_{Found by the October 2026 ForgeLM code and documentation review against commit cab2947. Code links point at that commit; line numbers may have moved since. Finding IDs (CR-…, DOC-…, TECH-…) are stable references for this backlog.}