From 22d01699b17d57ef967bfdea4c97fd61335ad570 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 18 Oct 2026 10:25:39 +0300 Subject: [PATCH 1/2] fix(deps): raise the transformers floor to 5.10.1 and pin it in the nightly floor job transformers>=5.3.0 was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue config.json overrides trust_remote_code, fixed 5.5.0) and CVE-2026-9856 (a chat_template name from tokenizer_config.json is used unsanitised as a file name by save_pretrained, fixed 5.10.0). Both reach ForgeLM from a Hub repo: it loads a tokenizer and saves it again to final_model/, on export and on merge. Reproduced on 5.3.0: a traversal name wrote a file outside the output directory; on 5.10.1 the save raises. 5.10.0 was yanked, so the floor is 5.10.1. transformers 5.10 requires huggingface_hub>=1.5.0, so that floor moves too. The nightly minimum-versions job pins both, and a test now compares every nightly pin with pyproject, checks the transformers floor against the advisories, and saves a crafted tokenizer (fails on 5.3.0, passes on 5.10.1). The transformers-facing test modules pass on a 5.10.1 / huggingface_hub 1.5.0 install. The top of the range (5.15+ warmup_ratio, the string fsdp form removed in 5.20) is not touched here. Refs #109 Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/nightly.yml | 15 ++-- CHANGELOG.md | 8 ++ docs/reference/supply_chain_security-tr.md | 2 +- docs/reference/supply_chain_security.md | 2 +- .../en/getting-started/installation.md | 2 +- .../tr/getting-started/installation.md | 2 +- pyproject.toml | 8 +- tests/test_model.py | 41 ++++++++++ tests/test_packaging.py | 79 +++++++++++++++++++ tools/pip_audit_ignores.yaml | 3 + 10 files changed, 150 insertions(+), 12 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index ae7d4821..0cba6dc6 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -237,14 +237,17 @@ jobs: # EXPLICITLY so this job tests the true documented minimum — trl puts # no upper bound on transformers, so an unpinned resolve would silently # pull the newest 5.x and stop exercising the floor. transformers - # 5.3.0 is the CVE-2026-4372 fix floor and requires torch>=2.4 / - # huggingface_hub>=1.3 / tokenizers>=0.22; trl 1.0.0 pulls the - # accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the pyproject - # [project].dependencies minimums. + # 5.10.1 is the first installable release past CVE-2026-4372, + # CVE-2026-5241 and CVE-2026-9856 (5.10.0, which carries the last of + # them, was yanked); it requires huggingface_hub>=1.5 / + # tokenizers>=0.22, and ForgeLM itself needs torch>=2.4; trl 1.0.0 + # pulls the accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the + # pyproject [project].dependencies minimums (tests/test_dependency_floor.py + # compares them). pip install \ "torch==2.4.0" \ - "transformers==5.3.0" \ - "huggingface_hub==1.3.0" \ + "transformers==5.10.1" \ + "huggingface_hub==1.5.0" \ "peft==0.19.0" \ "accelerate==1.4.0" \ "datasets==4.7.0" \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d44dd02..0647d97c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,14 @@ All notable changes to ForgeLM are documented here. ### Changed +- Raise the `transformers` floor from `>=5.3.0` to `>=5.10.1` (and `huggingface_hub` from `>=1.3.0` to `>=1.5.0`, which + transformers 5.10 requires). `5.3.0` was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue + `config.json` overrides `trust_remote_code: false`, fixed in 5.5.0) and CVE-2026-9856 (a `chat_template` name in + `tokenizer_config.json` is used unsanitised as a file name by `save_pretrained`, so `final_model/` saves, GGUF export + and merges could write outside the output directory; fixed in 5.10.0, which PyPI yanked, hence 5.10.1). The nightly + minimum-versions job pins the new floor, and `tests/test_dependency_floor.py` keeps it from drifting from + `pyproject.toml` and checks that a traversal `chat_template` name cannot escape on save. Environments pinned to + transformers 5.3 to 5.9 must upgrade (#109). - Record monitoring default provenance in `defaulted_fields` and mark literal URL projection with `endpoint_redacted` (also true when the URL already contains only scheme/host/port). Literal endpoints retain scheme/host/port in JSON evidence, including IPv6 brackets; userinfo, path, diff --git a/docs/reference/supply_chain_security-tr.md b/docs/reference/supply_chain_security-tr.md index 0b74c5a2..a81303ae 100644 --- a/docs/reference/supply_chain_security-tr.md +++ b/docs/reference/supply_chain_security-tr.md @@ -151,7 +151,7 @@ bir [`tools/pip_audit_ignores.yaml`](../../tools/pip_audit_ignores.yaml) taşır (şu an dokuz no-fix torch advisory'si ve bir OSV-yanlış-sınıflandırılmış markdown kaydı; daha önce taşıdığı iki transformers advisory'si `transformers>=5.3.0` floor'u onları pip-audit -tarafından raporlanamaz hâle getirince kaldırıldı). Bu dosya yalnız projenin kendi workflow'u tarafından +tarafından raporlanamaz hâle getirince kaldırıldı; floor artık `>=5.10.1`). Bu dosya yalnız projenin kendi workflow'u tarafından `--ignores` ile tüketilir; deployer'lar miras almaz. Her release cycle'da gözden geçirilir. diff --git a/docs/reference/supply_chain_security.md b/docs/reference/supply_chain_security.md index 13cac6c0..e070e770 100644 --- a/docs/reference/supply_chain_security.md +++ b/docs/reference/supply_chain_security.md @@ -149,7 +149,7 @@ ForgeLM's own nightly does carry a checked-in project-internal triage (currently nine no-fix torch advisories and one OSV-misclassified markdown record; the two transformers advisories it once carried were dropped once the `transformers>=5.3.0` floor made them -unreportable by pip-audit). That file is consumed only by the +unreportable by pip-audit; the floor is now `>=5.10.1`). That file is consumed only by the project's own workflow via `--ignores`; deployers do not inherit it. Review at every release cycle. diff --git a/docs/usermanuals/en/getting-started/installation.md b/docs/usermanuals/en/getting-started/installation.md index 591e25f0..e25d3894 100644 --- a/docs/usermanuals/en/getting-started/installation.md +++ b/docs/usermanuals/en/getting-started/installation.md @@ -22,7 +22,7 @@ ForgeLM ships as a single PyPI package with optional dependency groups (called * ::: :::note -**Intel Macs (x86_64) are not supported.** ForgeLM requires `torch>=2.4.0` (pulled in by `transformers>=5.3.0`), and PyPI publishes no `torch>=2.4` wheel for x86_64 macOS. Use Apple Silicon, Linux, or Windows. +**Intel Macs (x86_64) are not supported.** ForgeLM requires `torch>=2.4.0` (the minimum that `transformers` 5 supports), and PyPI publishes no `torch>=2.4` wheel for x86_64 macOS. Use Apple Silicon, Linux, or Windows. ::: ## Base install diff --git a/docs/usermanuals/tr/getting-started/installation.md b/docs/usermanuals/tr/getting-started/installation.md index 5c5517a5..b6bec76e 100644 --- a/docs/usermanuals/tr/getting-started/installation.md +++ b/docs/usermanuals/tr/getting-started/installation.md @@ -22,7 +22,7 @@ ForgeLM, PyPI'da tek bir paket olarak gelir; ağır özellikler için opsiyonel ::: :::note -**Intel Mac'ler (x86_64) desteklenmiyor.** ForgeLM `torch>=2.4.0` gerektirir (`transformers>=5.3.0` üzerinden gelir) ve PyPI'da x86_64 macOS için `torch>=2.4` wheel'i yok. Apple Silicon, Linux veya Windows kullanın. +**Intel Mac'ler (x86_64) desteklenmiyor.** ForgeLM `torch>=2.4.0` gerektirir (`transformers` 5'in desteklediği asgari sürüm) ve PyPI'da x86_64 macOS için `torch>=2.4` wheel'i yok. Apple Silicon, Linux veya Windows kullanın. ::: ## Temel kurulum diff --git a/pyproject.toml b/pyproject.toml index 313b9bb6..bc845a15 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -50,7 +50,11 @@ dependencies = [ # the transformers-5 migration (v0.9.0). Apple Silicon, Linux and Windows # are unaffected. The previous `numpy<2; darwin x86_64` ABI-guard marker is # gone with the platform it protected (torch 2.4+ ships against NumPy 2.x). - "transformers>=5.3.0,<6.0.0", + # 5.10.1 is the first installable release past three advisories that reach ForgeLM's own load and save path from + # a Hub repo: CVE-2026-4372 (fixed 5.3.0), CVE-2026-5241 (a LightGlue config.json overrides trust_remote_code; + # fixed 5.5.0) and CVE-2026-9856 (chat_template names used unsanitised as file names by save_pretrained; fixed + # 5.10.0, which PyPI yanked, hence 5.10.1). It also needs huggingface_hub>=1.5.0. + "transformers>=5.10.1,<6.0.0", "peft>=0.19.0,<1.0.0", "datasets>=4.7.0,<6.0.0", "accelerate>=1.4.0,<2.0.0", @@ -58,7 +62,7 @@ dependencies = [ "pydantic>=2.0.0,<3.0.0", "trl>=1.0.0,<2.0.0", "tensorboard>=2.15.0,<3.0.0", - "huggingface_hub>=1.3.0,<2.0.0", + "huggingface_hub>=1.5.0,<2.0.0", "requests>=2.32.2,<3.0.0", # requests-toolbelt is required by forgelm._http for the HostHeaderSSLAdapter # used to close the DNS-rebinding TOCTOU window in safe_post / safe_get diff --git a/tests/test_model.py b/tests/test_model.py index 771a671e..24698891 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -152,6 +152,47 @@ def test_existing_pad_token_preserved(self): assert fake_tok.pad_token == "[PAD]" +class TestChatTemplateNameCannotEscapeTheSaveDirectory: + """CVE-2026-9856: a ``chat_template`` name from a Hub repo was used as a file name by ``save_pretrained``. + + ForgeLM loads a tokenizer from a Hub repo and saves it again to ``final_model/`` (trainer, GGUF export, merge), so + a crafted ``tokenizer_config.json`` could write a file outside the output directory. Fixed in transformers + 5.10.0 (yanked) and 5.10.1, which is why the declared floor is ``>=5.10.1``. This uses the installed transformers + and a tiny local tokenizer, so it also fails on a floor install that predates the fix. + """ + + def _crafted_tokenizer_dir(self, root): + import json + + from tokenizers import Tokenizer, models, pre_tokenizers + from transformers import PreTrainedTokenizerFast + + tok = Tokenizer(models.WordLevel({"[UNK]": 0, "hello": 1, "": 2}, unk_token="[UNK]")) + tok.pre_tokenizer = pre_tokenizers.Whitespace() + source = root / "hub_repo" + PreTrainedTokenizerFast(tokenizer_object=tok, unk_token="[UNK]", eos_token="").save_pretrained(source) + config_file = source / "tokenizer_config.json" + config = json.loads(config_file.read_text(encoding="utf-8")) + config["chat_template"] = [ + {"name": "../../../ESCAPED_BY_CHAT_TEMPLATE", "template": "{{ messages }}"}, + {"name": "default", "template": "{{ messages }}"}, + ] + config_file.write_text(json.dumps(config), encoding="utf-8") + return source + + def test_a_traversal_name_is_refused_and_nothing_is_written_outside(self, tmp_path): + source = self._crafted_tokenizer_dir(tmp_path) + out = tmp_path / "run" / "final_model" + cfg = _cfg(model={"name_or_path": str(source)}) + + with pytest.raises(ValueError, match="Invalid chat template name"): + tokenizer = model_mod._load_tokenizer(cfg, trust_remote_code=False) + tokenizer.save_pretrained(out) + + escaped = [p for p in tmp_path.rglob("*") if p.is_file() and "ESCAPED" in p.name] + assert escaped == [], f"a chat_template name wrote outside the save directory: {escaped}" + + class TestBuildLoraConfig: """``_build_lora_config`` resolves the PEFT method and target modules.""" diff --git a/tests/test_packaging.py b/tests/test_packaging.py index ab8c562e..3018791c 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -244,6 +244,85 @@ def test_setuptools_security_floor_meets_pysec_2026_3447() -> None: ) +def _load_runtime_dependencies() -> list: + try: + import tomllib # type: ignore[import-not-found] + except ModuleNotFoundError: # pragma: no cover — Python 3.10 fallback + try: + import tomli as tomllib # type: ignore[import-not-found, no-redef] + except ModuleNotFoundError: + pytest.skip("Neither tomllib (3.11+) nor tomli is available; runtime dependency assertion skipped.") + pyproject_path = Path(__file__).resolve().parent.parent / "pyproject.toml" + with pyproject_path.open("rb") as fh: + pyproject = tomllib.load(fh) + return pyproject.get("project", {}).get("dependencies", []) + + +# transformers 5.10.1 is the first installable release past three advisories that reach ForgeLM's own load and +# save path from a Hub repo: CVE-2026-4372 (5.3.0), CVE-2026-5241 (a LightGlue config.json overrides +# trust_remote_code; 5.5.0) and CVE-2026-9856 (a chat_template name used as a file name by save_pretrained; +# 5.10.0, which PyPI yanked). A floor of ">=5.3.0" was documented as "the CVE fix floor" and was not. +_TRANSFORMERS_SECURITY_FLOOR = Version("5.10.1") + + +def test_transformers_floor_is_past_cve_2026_5241_and_cve_2026_9856() -> None: + """``transformers`` must not admit a release that predates the chat_template / trust_remote_code fixes.""" + found = [ + Requirement(spec) + for spec in _load_runtime_dependencies() + if canonicalize_name(Requirement(spec).name) == "transformers" + ] + assert len(found) == 1, f"expected exactly one transformers requirement in [project].dependencies, found {found!r}" + specifier_set = found[0].specifier + lower_bounds = [ + Version(spec.version.rstrip(".*")) for spec in specifier_set if spec.operator in _LOWER_BOUND_OPERATORS + ] + assert lower_bounds, f"transformers entry {str(found[0])!r} has no lower bound" + assert max(lower_bounds) >= _TRANSFORMERS_SECURITY_FLOOR, ( + f"transformers entry {str(found[0])!r} floors at {max(lower_bounds)}, below {_TRANSFORMERS_SECURITY_FLOOR}, " + "the first installable release past CVE-2026-5241 and CVE-2026-9856. Do not lower it without re-verifying " + "both advisories against the release you lower it to." + ) + for vulnerable in ("5.3.0", "5.4.0", "5.9.0", "5.10.0"): + assert not specifier_set.contains(Version(vulnerable), prereleases=True), ( + f"transformers entry {str(found[0])!r} still admits {vulnerable}" + ) + + +def test_nightly_floor_job_pins_the_runtime_dependency_minimums() -> None: + """Every ``name==X`` the nightly minimum-versions job installs is the ``>=X`` floor in pyproject. + + The job installs with ``--no-deps`` afterwards, so it can only test the documented minimum if its list is the + same list. It is a second hand-kept copy of every floor, and it kept saying ``transformers==5.3.0`` after the + floor was meant to be a security floor. + """ + import re + + nightly = (Path(__file__).resolve().parents[1] / ".github" / "workflows" / "nightly.yml").read_text( + encoding="utf-8" + ) + step = nightly.split("Install minimum dependency versions", 1)[1].split("- name:", 1)[0] + pins = { + canonicalize_name(name): version for name, version in re.findall(r'"([A-Za-z0-9_.-]+)==([0-9][^"]*)"', step) + } + assert pins, "the nightly minimum-versions step no longer installs `name==version` pins — update this test" + + floors = {} + for spec in _load_runtime_dependencies(): + req = Requirement(spec) + lowers = [s.version for s in req.specifier if s.operator == ">="] + if lowers: + floors[canonicalize_name(req.name)] = lowers[0] + + for name, version in pins.items(): + assert name in floors, f"the nightly floor job pins {name}=={version}, which is not a runtime dependency" + assert version == floors[name], ( + f"the nightly floor job pins {name}=={version} but pyproject.toml declares {name}>={floors[name]}; " + "the job would test a minimum nobody documents" + ) + assert {"transformers", "huggingface-hub"} <= set(pins), "the floor job must pin transformers and huggingface_hub" + + def _lint_job_pins() -> dict[str, Requirement]: """Requirements the ci.yml lint job installs directly. diff --git a/tools/pip_audit_ignores.yaml b/tools/pip_audit_ignores.yaml index cd16a831..e70e7619 100644 --- a/tools/pip_audit_ignores.yaml +++ b/tools/pip_audit_ignores.yaml @@ -40,6 +40,9 @@ ignores: # so a suppression would be dead weight: # - CVE-2026-1839 — fixed in transformers 5.0.0rc3. # - PYSEC-2025-217 / CVE-2025-14929 (X-CLIP RCE) — OSV last_affected 5.0.0rc0. + # The floor has since moved to transformers>=5.10.1, the first installable + # release past CVE-2026-5241 (fixed 5.5.0) and CVE-2026-9856 (fixed 5.10.0, + # yanked), so neither needs a suppression either. # ----- torch (added 2026-05-21, triage in issue #58) ----- # From e8d9683605ac3eaff584843a1d7f007f621eeb56 Mon Sep 17 00:00:00 2001 From: Cemil ILIK Date: Sun, 18 Oct 2026 11:17:57 +0300 Subject: [PATCH 2/2] docs(deps): name the test file that really compares the nightly pins The changelog entry and the nightly comment pointed at tests/test_dependency_floor.py, which does not exist; the comparison lives in tests/test_packaging.py. Refs #109 Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/nightly.yml | 2 +- CHANGELOG.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 0cba6dc6..32b9e699 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -242,7 +242,7 @@ jobs: # them, was yanked); it requires huggingface_hub>=1.5 / # tokenizers>=0.22, and ForgeLM itself needs torch>=2.4; trl 1.0.0 # pulls the accelerate>=1.4 and datasets>=4.7 floor. Keep 1:1 with the - # pyproject [project].dependencies minimums (tests/test_dependency_floor.py + # pyproject [project].dependencies minimums (tests/test_packaging.py # compares them). pip install \ "torch==2.4.0" \ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0647d97c..c5f41020 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,7 @@ All notable changes to ForgeLM are documented here. `config.json` overrides `trust_remote_code: false`, fixed in 5.5.0) and CVE-2026-9856 (a `chat_template` name in `tokenizer_config.json` is used unsanitised as a file name by `save_pretrained`, so `final_model/` saves, GGUF export and merges could write outside the output directory; fixed in 5.10.0, which PyPI yanked, hence 5.10.1). The nightly - minimum-versions job pins the new floor, and `tests/test_dependency_floor.py` keeps it from drifting from + minimum-versions job pins the new floor, and `tests/test_packaging.py` keeps it from drifting from `pyproject.toml` and checks that a traversal `chat_template` name cannot escape on save. Environments pinned to transformers 5.3 to 5.9 must upgrade (#109). - Record monitoring default provenance in `defaulted_fields` and mark literal URL projection with