Repository navigation
fix: rope_scaling on transformers 5, hub telemetry, reserved retention.enforce, transformers CVE floor, FSDP strategies, purge output dir - #999
Merged
Conversation
…is set `retention.enforce` is documented, validated and offered by both wizards as a gate: `warn_on_excess` "adds a structured warning to stderr" and `block_on_excess` "aborts the trainer pre-flight with EXIT_EVAL_FAILURE (3)" so "a regulated CI gate does not silently extend the retention horizon". No command reads the field. The only consumer of the retention block is `forgelm purge --check-policy`, which reports artefacts past their horizon, always exits 0 and deletes nothing. A regulated team that set `block_on_excess` believed a gate existed. The field description, the RetentionConfig docstring, the wizard prompt, the configuration reference, the GDPR erasure guide, the user-manual reference and `config_template.yaml` now say that the horizons are reported and that `enforce` is reserved (English and Turkish). Loading a config that sets `warn_on_excess` or `block_on_excess` logs one warning saying nothing acts on it, instead of accepting it silently. Implementing the gate (a trainer pre-flight, an audit event and an exit-code decision) stays open on the issue. Refs #92, #98 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rmers 5
`training.rope_scaling` (documented, offered by the wizard for contexts above 4096, and in config_template.yaml) was
passed to `from_pretrained(rope_scaling=...)`. On transformers 5 the RoPE settings live in `config.rope_parameters`:
the documented `{type: linear, factor: 4}` raises `KeyError: 'rope_type'` while the model is built, and
`{rope_type: ..., factor: ...}` replaces the whole dictionary so `rope_theta` is lost and the frequency computation
raises `TypeError`. `--dry-run` passes, then the run exits 2 at model construction.
The requested keys are now merged over the model's own `rope_parameters` (so `rope_theta` is kept and can still be
overridden), the legacy key `type` is read as `rope_type`, the settings go to the text sub-config of a multimodal
checkpoint, and the finished config object is what is loaded. With a config object, `from_pretrained` hands unknown
keyword arguments to the model constructor, so the sliding-window override is applied to the config as well.
Reproduced both failures and checked linear, dynamic and yarn on transformers 5.13.0 with a tiny local Llama config on
CPU, building the model and running a forward pass. The tests do the same, and use no network. The 5.3-5.9 range the
declared floor still admits is not exercised here.
Refs #103
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TRL's trainers call huggingface_hub's `send_telemetry` when they are built, and huggingface_hub sends a small anonymous usage ping to huggingface.co unless `HF_HUB_DISABLE_TELEMETRY` is true or offline mode is on. The privacy page says ForgeLM "does not collect telemetry, does not send usage data anywhere", and the air-gap page said only that ForgeLM has no telemetry service of its own. A default online training run therefore contradicted the page. `main()` now sets `HF_HUB_DISABLE_TELEMETRY=1` before anything imports huggingface_hub, which reads it once at import; importing `forgelm.cli` does not import it (a test pins that, so a later top-level import cannot silently defeat the default). `setdefault` leaves an operator's own value alone, so `HF_HUB_DISABLE_TELEMETRY=0` opts back in. Using ForgeLM as a library does not go through `main()` and is not covered; the air-gap page (English and Turkish) says so. Checked against the installed TRL 1.7.1 and huggingface_hub: both read the variable. Whether huggingface_hub makes other background requests beyond the telemetry ping was not examined. Refs #95 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, and document the real memorisation check The purge reference and the user manual both said that the implicit `--output-dir` fallback "emits a WARNING naming the resolved dir". It did not: nothing was logged. The fallback is the corpus's directory, and the memorisation check (`data.erasure_warning_memorisation`) looks for `<output_dir>/final_model/` there, a place a training run rarely writes its model. A default `forgelm purge --row-id X --corpus data/train.jsonl` therefore ended with a completed-erasure record and no memorisation event, so a reader of the audit chain alone would conclude that no model had consumed the row. `--row-id` without `--output-dir` now logs one warning naming the resolved directory and the `final_model/` it will look for, which makes the documented behaviour true. The audit-event catalogue, the purge reference, the GDPR erasure guide, the user manual and the risk-treatment plan (English and Turkish) described the condition as "a final_model/ exists for any run that consumed the corpus"; the code checks one literal directory and cannot tell which runs consumed the corpus. They now say that. Reading `training.output_dir` from `--config` and scanning `final_model*` / `checkpoint-*` is the larger fix and stays open on the issue; it changes the content of an erasure audit event (additively) and wants its own review. Refs #102 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…n.enforce warning to the CLI The first version added a validator to RetentionConfig, which pushed forgelm/config.py to 1915 lines against its deferred-split budget of 1903 (tests/test_check_module_size.py caught it). The docstring and field description are now line-neutral and the warning is emitted where the config is loaded by the CLI, which is also where an operator sees it. Refs #92, #98 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ightly floor job transformers>=5.3.0 was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue config.json overrides trust_remote_code, fixed 5.5.0) and CVE-2026-9856 (a chat_template name from tokenizer_config.json is used unsanitised as a file name by save_pretrained, fixed 5.10.0). Both reach ForgeLM from a Hub repo: it loads a tokenizer and saves it again to final_model/, on export and on merge. Reproduced on 5.3.0: a traversal name wrote a file outside the output directory; on 5.10.1 the save raises. 5.10.0 was yanked, so the floor is 5.10.1. transformers 5.10 requires huggingface_hub>=1.5.0, so that floor moves too. The nightly minimum-versions job pins both, and a test now compares every nightly pin with pyproject, checks the transformers floor against the advisories, and saves a crafted tokenizer (fails on 5.3.0, passes on 5.10.1). The transformers-facing test modules pass on a 5.10.1 / huggingface_hub 1.5.0 install. The top of the range (5.15+ warmup_ratio, the string fsdp form removed in 5.20) is not touched here. Refs #109 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cannot run, and warn about two ignored knobs distributed.fsdp_strategy accepted no_shard and hybrid_shard (the latter described as the multi-node option). transformers 5.10 made FSDP2 the default, and FSDP2 raises "fsdp=hybrid_shard is only available with FSDP1" when TrainingArguments is built, so such a config passed --dry-run and then exited 2 after the model was loaded. The strategy is now a Literal of full_shard and shard_grad_op, so the config is refused at load time with exit 1. fsdp_auto_wrap: false and fsdp_backward_prefetch: backward_post are still accepted but change nothing under FSDP2 (the plugin always applies the transformer-layer wrap policy and takes no prefetch policy; checked against TrainingArguments on 5.10.1 and 5.13.0). The CLI now logs a warning when either is set away from its default, and the field descriptions, reference pages, user manuals and config_template.yaml say so. The warning lives in cli/_config_load.py because config.py and trainer.py are at their deferred-split line budgets. A new test builds TrainingArguments for all 32 accepted variants through ForgeTrainer._apply_distributed_config, so a config the schema accepts can no longer be one TrainingArguments refuses. Moving to the FSDP2 form (fsdp=True plus an explicit fsdp_config) is not part of this change. Depends on the transformers floor moving to 5.10.1 (#109); the docs say "transformers 5.10+, which ForgeLM requires". Refs #104 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tput-dir warning The warning that `forgelm purge --row-id` now logs when --output-dir is omitted adds eight lines to a module on the deferred-split list, and the budget has to equal the measured size. It is a raise rather than a split for the reason the previous raise gave (C-18): the message has to sit where the default is resolved. The history note says why. Refs #102 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…des lack Second-review finding: the wizard labelled log_only as 'the only mode with an effect to promise', but no retention mode has an effect. It now says log_only is the default and behaves the same as the other two for now. Refs #92 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The changelog entry and the nightly comment pointed at tests/test_dependency_floor.py, which does not exist; the comparison lives in tests/test_packaging.py. Refs #109 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md # tests/test_cli.py
# Conflicts: # CHANGELOG.md # forgelm/cli/_dispatch.py
# Conflicts: # CHANGELOG.md
|
Sorry @cemililik, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 23 hours and 3 minutes by commenting |
Contributor
|
Warning Review limit reachedEnable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Next included review available in 9 minutes. View limit details
Comment |
Reviewer's GuideThis PR consolidates six fixes across model loading, CLI privacy and configuration diagnostics, dependency security, FSDP validation, and purge behavior, with corresponding English/Turkish documentation and targeted regression tests. Reviewers should focus on startup ordering for telemetry, config mutation semantics for Transformers 5, dependency-floor consistency, early-vs-late FSDP failures, and the deliberately limited purge memorisation scope. Sequence diagram for Transformers 5 model configurationsequenceDiagram
participant CLI as ForgeLM model loader
participant AC as transformers.AutoConfig
participant AP as transformers.AutoModel
CLI->>AC: AutoConfig.from_pretrained()
AC-->>CLI: model_config
CLI->>CLI: merge rope_scaling into rope_parameters
CLI->>CLI: set sliding_window on model_config
CLI->>AP: from_pretrained(config=model_config)
AP-->>CLI: configured model
Sequence diagram for CLI telemetry suppressionsequenceDiagram
participant CLI as ForgeLM CLI
participant Hub as huggingface_hub
participant TRL as TRL trainer
CLI->>CLI: _disable_hub_telemetry()
CLI->>CLI: setdefault HF_HUB_DISABLE_TELEMETRY=1
CLI->>Hub: import huggingface_hub
CLI->>TRL: build trainer
TRL->>Hub: send_telemetry()
Hub-->>TRL: telemetry disabled
Flow diagram for early FSDP configuration validationflowchart LR
Config["Load configuration"] --> Strategy{"fsdp_strategy"}
Strategy -->|full_shard or shard_grad_op| Warnings["Warn if inert FSDP2 options are set"]
Strategy -->|no_shard or hybrid_shard| Reject["Reject config with exit 1"]
Warnings --> Trainer["Build FSDP2 trainer"]
Flow diagram for purge memorisation scopeflowchart LR
Purge["forgelm purge --row-id"] --> Output["Resolve --output-dir"]
Output -->|omitted| Warn["Warn with resolved path and final_model path"]
Output --> Check["Check output_dir/final_model"]
Check -->|exists| Event["Emit data.erasure_warning_memorisation"]
Check -->|absent| Complete["Complete purge without memorisation event"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This was referenced Oct 18, 2026
# Conflicts: # CHANGELOG.md
cemililik
added a commit
that referenced
this pull request
Oct 22, 2026
…the code; harden the user-manual schema guard (#1050) ## Summary Finishes #101 (theme D0-6, *Personal data: audit and masking coverage*, 30 findings) and the neighbouring pages that carried the same root cause: documentation, help text, field descriptions, comments and site copy that presented PII masking, the erasure commands and the retention horizons as stronger than the code makes them, plus the guard that should have caught a `chat:` / `output:` block Pydantic rejects. After the first version of this pull request was reviewed, a second pass re-checked every statement against the running commands and the official texts, and a third pass (fact-check, legal and language review, mutation testing) re-checked that delta; this description covers the pull request as it now stands. **This is compliance wording. Please read the before/after table and the legal sign-off list.** ## Related issues Fixes #101. Refs #428, Refs #429, Refs #444, Refs #241, Refs #243, Refs #589, Refs #172, Refs #295, Refs #275, Refs #393, Refs #351, Refs #630, Refs #757, Refs #368, Refs #743, Refs #586, Refs #595, Refs #768, Refs #724, Refs #798, Refs #601, Refs #574, Refs #719, Refs #179, Refs #509, Refs #554, Refs #555, Refs #680, Refs #567, Refs #610, Refs #789. Each of these is a per-page or per-module issue of the October review whose documentation findings this change resolves in whole or in part; a maintainer ticks the findings and closes the ones that are complete (the GPU-cost, encryption-at-rest and notebook issues are, for example). Not worked on, mentioned for context: the `purge --kind artefacts` resolver rewrite stays with the artefacts issue (see the note posted on it) and the chat and `gpu_profiles` blocks the guard lists as known-open belong to other tracked issues; this change also corrects the content of the chat page those issues describe. New issues for what this change found but does not do: #1051 (phone numbers masked only partly), #1052 (wizards omit the retention block at defaults), #1053 (`forgelm ingest` rows have no `id`), #1054 (`--check-policy --config` needs a complete config), #1055 (native review of the DE/FR/ES/ZH strings), #1056 (site localization polish), #1057 (legal sign-off of the wording). ##⚠️ Localization exception `docs/standards/localization.md` keeps German, French, Spanish and Chinese as deferred tiers. As in #1000 (owner decision), **I wrote the DE / FR / ES / ZH site strings myself with no native review**; English and Turkish are authoritative. #1055 tracks the review. Touched keys: `home.cap.gdpr.body`, `home.use.domain.body`, `features.data.pii.body`, `compliance.gdpr15.body`, `compliance.gdpr17.body`, `compliance.art10.body`, `features.gov.{access,purge}.{title,body}`, `privacy.body`, `wizard.retention.*` (title and five hints). The legal pages' "last updated" date moved to 2026-10-22 because the privacy text changed; it is one key shared with the terms page (#1056). ## What this changes for a deployer | Topic | Said | Now says | |---|---|---| | **Article 17(3)(b)** | it "preserves audit / accounting records as a legal-obligation defence"; the audit log is "the legal-obligation record"; AI Act log duties as the example | it disapplies erasure only to the extent a specific Union or Member State obligation (or a public-interest task) requires the processing; keeping entries needs a documented basis; ForgeLM never edits or deletes an entry | | **Article 15 / 17 framing** | "Article 17 erasure with purge and Article 15 access with reverse-pii"; "zero matches is a valid answer" | `reverse-pii` is an input to an Article 15 response, `purge` one step of an Article 17 erasure (trained weights unchanged); a complete response also needs an identity check, one month (12(3)), a copy of the data, the 15(1)(a)-(h) information and the 15(2) safeguards, Article 19 notices, and refusal reasons within one month (12(4)); zero matches does not answer an access request | | **Digest correlation** | query with `--salt-source env_var` to match purge digests | byte-identical only for the same value, `--output-dir` and `FORGELM_AUDIT_SECRET` state, in the default plaintext mode; `--salt-source` is a hash-mask scan for externally masked corpora: not needed for the digests to match, and on a plaintext corpus it reports zero matches even when the person is there (verified end to end; the earlier advice was wrong) | | **Salted digests** | no personal data leaves the log | personal data (recital 26), and pseudonymisation (Art. 4(5)) only if the salt is kept separately; by default the salt file sits next to the log | | **Retention horizons** | "honour" GDPR deadlines, staging TTL is a scheduled cleanup, wizard "matches EU AI Act guidance" | operator-set limits that `--check-policy` reports and never enforces; defaults are ForgeLM choices; no `retention:` block checks nothing, `retention: {}` enables defaults, `0` stops a report; age source is the run's first audit event, else the log's first event, else mtime (so a shared output directory can report a fresh bundle as overdue); ForgeLM creates neither `raw_documents/` nor `ingestion_output/` | | **CI gate / exit codes** | `jq '.violations \| length'`; usage errors exit 1 | `jq -e '.success == true and has("count") and .count == 0'` under `set -o pipefail`; argument-parser usage errors exit 2; the purge JSON envelope section is regenerated from real output | | **`purge --kind artefacts`** | deletes the run's compliance bundle | matches files under `compliance/` whose names contain the run id, so it finds nothing in the standard bundle, emits `data.erasure_failed` (`NoMatchingArtefacts`), exits 1 and deletes nothing; the help text and the error say why | | **PII detectors** | "what GDPR / AI Act Article 10 cares about"; nine categories; German Steuer-ID and French NIR "checksum / key validation"; audit report lists row indices; audit "catches what masking missed" | common structured identifiers, personal data under Art. 4(1); eight detectors, the German ID-card serial and the NIR are shape-only; the report carries counts; `forgelm audit` re-applies the same regexes; `--pii-ml` counts names and locations and reports no `organization` for English | | **GPU-cost page** | a `run_complete` audit event, a model-card cost block, `--gpus`, `nvidia-smi`, a price table that did not match the code | what `_collect_resource_usage` records and where, the real price table and `cost_source` values, the CPU/MPS placeholder rate, a calibration recipe that runs | | **Chat page** | GGUF loading, a YAML `seed:`, `max_length` history, an invented transcript | no GGUF, no seed or greedy mode, last 50 pairs, the real REPL output and every flag | | **Site (six languages)** | masking as automatic; "every row of every split"; "forensically linked" | masking is opt-in (`--pii-mask`); lines containing an identifier you supply, literal and case-sensitive; `0` stops a retention report | | **QMS SOPs / SOC** | "minimum 5 years"; salted SHA-256 "needs no encryption" | a policy-set period (Art. 5(1)(e); Art. 18 is a separate duty); digests are personal data and the log and salt are encrypted alike | ## Changes - Documentation: GDPR guide, purge / reverse-pii / configuration / compliance-summary / event-catalog references, GDPR, GDPR-erasure, human-oversight, PII-masking, PII-ML, audit, ingestion, GPU-cost, chat and LoRA manual pages (EN and TR together), design note, QMS SOPs, deployer guide, localization glossary, `sync-bilingual-docs` skill scope. - Code text only (behaviour, flags and exit codes unchanged): `--help` and epilog strings, field descriptions, docstrings and comments, wizard prompts, the `--kind artefacts` error and the `ingest --pii-mask` note. `config.py`, `_parser.py` and `_purge.py` keep their module-size budgets. - Guard `tools/check_usermanual_schema_drift.py`: top-level blocks are checked everywhere (including full-config summary blocks); two exemption tables with reasons; one block per row per language; unreadable blocks fail; YAML 1.2 booleans; missing PyYAML exits 1; success line counts what was walked. 104 tests; two real snippets it flagged are fixed (`lora.md`, `model-card.md`). - Tests: Article 17(3) invariant per sentence over every public text file; broader retired-phrase lists with line numbers; positive pins for the CI gate, the correlation rule and the exit-2 wording; pages checked against the code (price table, chat flags, detector count); real-exporter `--kind artefacts` tripwire (checked by mutation); site tests per locale and for static fallbacks. ## Finding by finding (#101) | Finding | Resolution | |---|---| | CR-A11-001 (primary) | page rewrite (#1000), runnable example and real output line, `--pii-ml` pointer, the correlation rule and regression tests (this PR) | | CR-A17a-007 | guard and chat / GPU-cost examples (this PR); the gdpr.md block was removed by #1000 | | TECH-T11-002 | Art. 17(3)(b), 5(1)(e), 87 / 32 / 4(1) citations, "compliant out of the box" (this PR) | | DOC-D3B02-001, -007, -008, -009, -010, -024; DOC-BYAML-004 | fixed by the page rewrite in #1000; pinned by retired-phrase tests (EN and TR) | | DOC-D3B02-047 | example passes `--output-dir`, shows the real output line (pinned against the real command) | | DOC-BLEGAL-004 (+ design lines), -005 | Art. 17(3)(b) rewording | | DOC-BLEGAL-041, -047 | site cards in six languages; -050 hedged by #1000, pinned | | DOC-BLEGAL-064, -097, -146, DOC-BFIELD-019 | retention wording and field descriptions | | DOC-BLEGAL-068, -179, -180, -126, DOC-D3B07-044 | PII tier / detector citations | | DOC-D1B05-017, DOC-D2B02-025 | legacy staging marker documented | | DOC-D1B05-005 | **documented, not fixed**: warning in the guide, references and manual, in `--help` and in the error, pinned by a tripwire that fails on purpose when the resolver is fixed. The code half is noted on the artefacts issue | | DOC-D2B02-024, -051 | fixed by #999 / #1000; now pinned | ## Trade-offs worth your review - **`purge --kind artefacts`** is documented rather than fixed (owner decision). - **Known-open guard rows** keep the two fabricated blocks visible; fixing a page forces its row to be deleted. - **Scope.** The second pass widened the diff beyond #101's 30 findings because the same false statements sat on the pages next to them (GPU cost, chat, PII masking, QMS, human oversight). Larger neighbours that need code are filed, not fixed. - **Legal wording.** The statements listed in #1057 were checked against the official texts but not by a lawyer. - **Wording I could not verify here:** the DE / FR / ES / ZH strings (#1055) and the claim about Presidio versions, which I left out of the documentation. ## Review process Each step (erasure / retention, PII citations, GDPR page and site, guard) got two review rounds with independent agents (legal and factual, bilingual and language quality, mutation testing, operator walkthroughs against the running commands). The open pull request then got two independent whole-diff reviews; their findings were checked against the code before applying (one of them showed that the correlation advice added in an earlier round was wrong, and it is corrected on eight surfaces plus an end-to-end test). The tests and the guard were then mutation-tested (237 mutants; the real gaps are closed), the documentation was fact-checked against the running commands, and the legal wording and the Turkish and site strings were reviewed against the official texts. ## Type - [x] Bug fix (documentation, help text and comments that stated more than the code does) - [x] Documentation - [x] Test coverage - [x] CI / infrastructure (the user-manual schema-drift guard) ## Testing - [x] `pytest tests/` passes - [x] `ruff check .` passes - [x] `ruff format --check .` passes - [x] New tests added for changed code - [x] `python -m forgelm --config config_template.yaml --dry-run` works - [x] The full CLAUDE.md gauntlet exits 0 on the final commit (guards, mypy strict, `node --test`, bandit warnings only) ## Checklist - [x] Ruff formatted; module-size budgets unchanged - [x] English docs and their Turkish mirrors updated together (`check_bilingual_parity` / `check_bilingual_code_blocks` pass) - [x] CHANGELOG entry under `[Unreleased]` - [x] The issue I link (#101) has been triaged (no `needs-triage`) and is not blocked; the issues referenced above carry no `blocked` label 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Documentation** * Clarified the scope and limitations of `purge` and `reverse-pii`, including matching requirements, audit digests, retention reporting, and steps needed to complete access or erasure responses. * Updated guidance for PII detection and masking, chat behavior, GPU cost reporting, retention settings, and supported configuration. Clarified that retention checks report overdue items without deleting them and corpus erasure does not change trained weights. * Revised compliance, configuration, and operational guidance in English and Turkish, with related updates to other translated site content. * **Validation** * Expanded strict documentation checks to flag unreadable YAML, undeclared top-level blocks, and stale exceptions.** <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Six code fixes combined into one pull request, replacing #984, #985, #987, #989, #990 and #992 (to be closed as superseded). Each fixes a case where the code failed, or a documented option did nothing, without saying so. Each was mutation-checked on its own; together they pass the full gauntlet.
Related issues
Refs #103, #95, #92, #98, #109, #104, #102. Not closed by this change: the top-of-range problems named in #109 (
warmup_ratioon transformers 5.15+, the stringfsdpform removed in 5.20), the FSDP2 migration (#104), and readingtraining.output_dirfor the purge memorisation check (#102).Changes
training.rope_scalingcrashed every model load on transformers 5 (KeyError: 'rope_type', orrope_thetalost). The requested keys are merged over the model'srope_parametersand the finished config is loaded;sliding_window_attentiontravels in the same config object.forgelm/model.pyHF_HUB_DISABLE_TELEMETRY=1(viasetdefault) beforehuggingface_hubis imported, so TRL's trainer no longer sends Hugging Face's anonymous usage ping the privacy page says does not exist. Library use is not covered; the air-gap page says so.cli/_config_load.py,cli/_dispatch.pyretention.enforcewas documented as a gate (warn_on_excess,block_on_excess→EXIT_EVAL_FAILURE) but no command reads it. Docs, field description, both wizards say it is reserved; the CLI warns when a config sets either of the last two.config.py(in place),cli/_config_load.py,wizard/_collectors.pytransformers>=5.3.0was called the CVE floor but admitted CVE-2026-5241 and CVE-2026-9856 (a craftedchat_templatename writes outside the save directory). Floor is now>=5.10.1(5.10.0 is yanked),huggingface_hub>=1.5.0, the nightly floor job pins both, and tests keep it from drifting.pyproject.toml,nightly.yml,tests/fsdp_strategy: no_shard/hybrid_shardpassed--dry-runand exited 2 after the model loaded (FSDP2 rejects them). They are now rejected when the config loads (exit 1), and two knobs FSDP2 ignores (fsdp_auto_wrap: false,backward_post) log a warning.config.py(in place),cli/_config_load.pyforgelm purge --row-idwithout--output-dirsilently pointed the memorisation check at the corpus directory. It now warns and names the path. The purge module's line budget is raised by 8 with abudget_historynote (the project's documented option).cli/subcommands/_purge.py,tools/check_module_size.pyconfig.py(0 lines of headroom on the module-size ratchet) andtrainer.pychange only by in-place string edits; new warnings live incli/_config_load.py.Merge-conflict handling
The six branches touched
CHANGELOG.md,tests/test_cli.py(two appended test classes) and the import line incli/_dispatch.py; all three were resolved by keeping both sides (the import line now imports_disable_hub_telemetryand_warn_inert_fsdp_options). The affected test modules pass together (621 tests).Type
Testing
CLAUDE.mdgauntlet from a clean worktree of this branch: exit 0.rope_scaling(linear, dynamic, yarn); the traversal reproduction (fails on transformers 5.3.0, passes on 5.10.1); all 32 accepted FSDP variants building realTrainingArguments.huggingface_hub1.5.0 overlay (443 tests). The nightly minimum-versions job runs on its schedule after merge; I did not dispatch it.Checklist
[Unreleased]🤖 Generated with Claude Code
Summary by Sourcery
Harden CLI behavior and dependency compatibility across model loading, privacy, retention, distributed training, and purge workflows.
Bug Fixes:
Enhancements:
Build:
Tests:
Chores: