Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ All notable changes to ForgeLM are documented here.

### Changed

- Admit newer majors of four optional or dev dependencies: `pypdf` 6 and `xxhash` 4 (the `ingestion` extra), `datasketch` 2 (the
`ingestion-scale` extra) and `pytest-randomly` 4 and 5 (the `dev` extra). The ingestion, data-audit and doctor test modules pass on pypdf
6.21.1, xxhash 4.0.1 and datasketch 2.0.0 with the same results as on the previous majors; CI installs datasketch 2.0.0 and
pytest-randomly 5.0.0 for the full suite. `huggingface_hub` stays below 2: ForgeLM has not been tested with it.
- Raise the `transformers` floor from `>=5.3.0` to `>=5.10.1` (and `huggingface_hub` from `>=1.3.0` to `>=1.5.0`, which
transformers 5.10 requires). `5.3.0` was described as the CVE fix floor but still admitted CVE-2026-5241 (a LightGlue
`config.json` overrides `trust_remote_code: false`, fixed in 5.5.0) and CVE-2026-9856 (a `chat_template` name in
Expand Down
8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -145,12 +145,12 @@ export = [
# so a bare install still works.
# OCR is intentionally out of scope (handle externally with Tesseract).
ingestion = [
"pypdf>=4.0.0,<6.0.0",
"pypdf>=4.0.0,<7.0.0",
"python-docx>=1.0.0,<2.0.0",
"ebooklib>=0.18,<1.0.0",
"beautifulsoup4>=4.12.0,<5.0.0",
"langdetect>=1.0.9,<2.0.0",
"xxhash>=3.0.0,<4.0.0",
"xxhash>=3.0.0,<5.0.0",
]

# Optional MinHash LSH near-duplicate detector for >50K-row corpora.
Expand All @@ -159,7 +159,7 @@ ingestion = [
# operators auditing larger datasets where MinHash's recall/precision
# trade-off is the industry standard (NeMo Curator, Dolma, RedPajama).
ingestion-scale = [
"datasketch>=1.6.0,<2.0.0",
"datasketch>=1.6.0,<3.0.0",
]

# Optional Presidio ML-NER PII detector. Adds the unstructured-identifier
Expand Down Expand Up @@ -187,7 +187,7 @@ dev = [
# surface in CI instead of waiting for a Python micro-bump or runner
# change to flip collection order. Run locally via
# ``pytest --randomly-seed=<n>`` to reproduce a failure.
"pytest-randomly>=3.15.0,<4.0.0",
"pytest-randomly>=3.15.0,<6.0.0",
# Bounded on purpose. ruff 0.16.0 began formatting Python blocks *inside
# Markdown*, taking the format check from 277 files to 562 — on an unchanged
# tree that is 32 newly "unformatted" documents and a red lint job nobody
Expand Down
Loading