[dependabot]

Bumps tokenizers from 0.23.1 to 0.23.2.

Release notes

Sourced from tokenizers's releases.

v0.23.2

This is the last v0 release, we are moving to v1!!

More details coming soon 👀

What's Changed

• Update daachorse from 1.0.1 to 3.0.0 by @​musicinmybrain in huggingface/tokenizers#2050
• Ci test fixes by @​sebpop in huggingface/tokenizers#2079
• Avoid full vocab clone in get_vocab_size() by @​eunseo9311 in huggingface/tokenizers#2074
• ci: pin benchmark jobs to a larger dedicated runner by @​ArthurZucker in huggingface/tokenizers#2084
• ci: target benchmark runners by group, not nonexistent label by @​ArthurZucker in huggingface/tokenizers#2090
• Bindings: acquire model read-lock once per call instead of per pre-token by @​sebpop in huggingface/tokenizers#2072
• ci: split audit in a separate job by @​SBrandeis in huggingface/tokenizers#2103
• ci: install libcairo on benchmark runners by @​SBrandeis in huggingface/tokenizers#2100
• feat(ci): enable npm's trusted publisher for releases by @​McPatate in huggingface/tokenizers#2111
• chore: bump pyo3 to version 0.29 by @​SBrandeis in huggingface/tokenizers#2115
• perf: add benchmarks for the decode API by @​SBrandeis in huggingface/tokenizers#2099
• ci: cache HF test data and authenticate Hub downloads to avoid rate limits by @​SBrandeis in huggingface/tokenizers#2102
• Implementing Parity-aware BPE by @​cimeister in huggingface/tokenizers#1974
• Make doc builds faster by @​mishig25 in huggingface/tokenizers#2158
• chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www by @​dependabot[bot] in huggingface/tokenizers#2208
• chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www by @​dependabot[bot] in huggingface/tokenizers#2244
• chore: remove stale examples (kills 50% of dependabot traffic) by @​ArthurZucker in huggingface/tokenizers#2285
• chore(node): drop 8 unused devDependencies by @​ArthurZucker in huggingface/tokenizers#2286
• ci: pin every action to one SHA, drop stale audit suppressions by @​ArthurZucker in huggingface/tokenizers#2291
• chore(node): take node security alerts from 33 to 0 by @​ArthurZucker in huggingface/tokenizers#2287
• fix(python): pin the ruff rule set so a ruff release can't redden main by @​ArthurZucker in huggingface/tokenizers#2292
• Add a ParityBpeTrainer example and list it in the trainers API docs by @​cimeister in huggingface/tokenizers#2217

New Contributors

• @​eunseo9311 made their first contribution in huggingface/tokenizers#2074
• @​cimeister made their first contribution in huggingface/tokenizers#1974

Full Changelog: huggingface/tokenizers@v0.23.1...v0.23.2

Commits

• 88a4498 add lock
• e4ea65f real release
• 7c5964b add lock
• bc405ad push rc0
• d582781 Add a ParityBpeTrainer example and list it in the trainers API docs (#2217)
• 447890f fix(python): pin the ruff rule set so a ruff release can't redden main (#2292)
• 68b3ab7 chore(node): take node security alerts from 33 to 0 (#2287)
• 74ef81f ci: pin every action to one SHA, drop stale audit suppressions (#2291)
• b26727f chore(node): drop 8 unused devDependencies (#2286)
• 7c7cfd8 chore: remove stale examples (kills 50% of dependabot traffic) (#2285)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Labels

The following labels could not be found: dependencies. Please create it before Dependabot can add it to a pull request.

Please fix the above issues or remove invalid values from dependabot.yml.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#25', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '25', 'title': 'chore: bump tokenizers from 0.23.1 to 0.23.2', 'author': 'dependabot[bot]', 'additions': 5, 'deletions': 5, 'changed_files': 1, 'review': '## Summary\nDependency bump of tokenizers from 0.23.1 to 0.23.2 (and daachorse to 3.0.3) with only Cargo.lock changes. No code impact detected.\n\n## Issues\nNo significant issues found.\n\n## Verdict\n✅ APPROVE\nThe update is limited to lock‑file metadata, checksums are correct, and there are no breaking API changes affecting the current codebase. Safe to merge.'}

---

Auto-generated by GitMind — AI-powered code analysis.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#25', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '25', 'title': 'chore: bump tokenizers from 0.23.1 to 0.23.2', 'author': 'dependabot[bot]', 'additions': 5, 'deletions': 5, 'changed_files': 1, 'review': '## Summary\nDependency version bump in Cargo.lock without corresponding Cargo.toml update.\n\n## Issues\nCargo.toml — [HIGH] Cargo.toml version mismatch for tokenizers — The PR updates the tokenizers crate version in Cargo.lock from 0.23.1 to 0.23.2, but there is no change to Cargo.toml. If Cargo.toml still pins the old version, the repository will have an inconsistent dependency specification, potentially causing build failures or unexpected behavior. Align the version constraint in Cargo.toml with the lockfile change.\n\n## Verdict\n⚠️ REQUEST CHANGES\nThe lockfile and manifest must be consistent for a reliable build. Update Cargo.toml to reflect the new tokenizers version before merging.'}

---

Auto-generated by GitMind — AI-powered code analysis.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#25', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '25', 'title': 'chore: bump tokenizers from 0.23.1 to 0.23.2', 'author': 'dependabot[bot]', 'additions': 5, 'deletions': 5, 'changed_files': 1, 'review': '## Summary\nDependency bump of tokenizers from 0.23.1 to 0.23.2 (and daachorse to 3.0.3) with correct checksum updates. No code changes or functional impact detected.\n\n## Issues\nNo significant issues found.\n\n## Verdict\n✅ APPROVE\nThe PR only updates Cargo.lock to reflect a safe, non‑breaking version bump. All checksum values are correctly updated, and no source code modifications are required. The change should not affect functionality, performance, or security.'}

---

Auto-generated by GitMind — AI-powered code analysis.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#25', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '25', 'title': 'chore: bump tokenizers from 0.23.1 to 0.23.2', 'author': 'dependabot[bot]', 'additions': 5, 'deletions': 5, 'changed_files': 1, 'review': '## Summary\nDependency bump updates Cargo.lock but does not modify Cargo.toml to reflect the new tokenizers version, risking a version mismatch.\n\n## Issues\nCargo.lock:1825 — [MEDIUM] Missing version bump in Cargo.toml — The PR updates tokenizers from 0.23.1 to 0.23.2 in Cargo.lock but does not change the version constraint in Cargo.toml. If the manifest still pins the old version, the build may fail or use the older crate, leading to inconsistency between the lockfile and the manifest.\n\n## Verdict\n⚠️ REQUEST CHANGES\nEnsure the manifest (Cargo.toml) is updated to require the new tokenizers version before merging.'}

---

Auto-generated by GitMind — AI-powered code analysis.

[git-mind-pr-guardian]

GitMind PR Review

{'pr_url': '#25', 'repo': 'Ayyankhan101/Transformer-In-Rust', 'pr_num': '25', 'title': 'chore: bump tokenizers from 0.23.1 to 0.23.2', 'author': 'dependabot[bot]', 'additions': 5, 'deletions': 5, 'changed_files': 1, 'review': '## Summary\nDependency bump of tokenizers from 0.23.1 to 0.23.2 (and daachorse to 3.0.3) with correct lockfile updates. No code changes required and no breaking changes introduced.\n\n## Issues\nNo significant issues found.\n\n## Verdict\n✅ APPROVE\nThe change is limited to Cargo.lock, updates checksums correctly, and does not affect the codebase or introduce regressions.'}

---

Auto-generated by GitMind — AI-powered code analysis.