Add opt-in Hugging Face Rust JSON tokenizer integration - #22548
Draft
JacobSzwejbka wants to merge 5 commits into
Draft
JacobSzwejbka wants to merge 5 commits into
JacobSzwejbka wants to merge 5 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22548
Note: Links to docs will display an error until the docs builds have been completed. ❌ 64 New Failures, 10 Unclassified FailuresAs of commit 1e71054 with merge base 83c4dc1 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
JacobSzwejbka
force-pushed
the
hf-rust-tokenizer-tok
branch
from
September 24, 2026 18:40
f0de270 to
9629e50
Compare
JacobSzwejbka
force-pushed
the
hf-rust-tokenizer-tok
branch
3 times, most recently
from
September 24, 2026 19:06
582d583 to
9108d0d
Compare
JacobSzwejbka
changed the title
Point at the rebased tokenizer dependency with corrected BOS/EOS semantics, safe decoder gating, and regression coverage. Apply the required CMake formatting. AI-assisted-by: Codex
JacobSzwejbka
force-pushed
the
hf-rust-tokenizer-tok
branch
from
September 24, 2026 19:13
9108d0d to
f38fa5c
Compare
JacobSzwejbka
changed the title
JacobSzwejbka
force-pushed
the
hf-rust-tokenizer-tok
branch
2 times, most recently
from
September 24, 2026 20:30
32a0d78 to
5764a02
Compare
AI-assisted-by: Codex
JacobSzwejbka
force-pushed
the
hf-rust-tokenizer-tok
branch
from
September 24, 2026 20:31
5764a02 to
5e3887a
Compare
Generated with assistance from Codex.
Generated with assistance from Codex.
JacobSzwejbka
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Depends on meta-pytorch/tokenizers#212.
This advances the tokenizer submodule and wires its opt-in Hugging Face Rust JSON backend into the LLM runner.
EXECUTORCH_BUILD_HF_RUST_TOKENIZERremains off by default. When enabled, the backend accepts ordinary Hubtokenizer.jsonfiles through published Hugging Face v1.0.0-rc.2 crates and uses the upstream Rust encode/decode pipeline. Failed Rust loads retain the existing tokenizer fallback chain.There is no
.tokcode or format selector in this PR. Experimental.tokwork has moved to draft pytorch/executorch#23139, stacked on meta-pytorch/tokenizers#218, and is blocked on upstream stabilization.Review and merge order
Land meta-pytorch/tokenizers#212 first. Then update this PR's submodule pointer to the canonical merge commit, rebase onto ExecuTorch main, and mark this PR ready.
Hugging Face's v1 RC post lists inference-only C/C++ bindings for ExecuTorch and llama.cpp on the v1 roadmap. Their experimental bindings PR #2375 is still open. The local bridge is deliberately narrow and can be replaced after that upstream ABI exposes the vocabulary and special-token metadata ExecuTorch requires.
Validation
Formatting and focused C++ adapter tests pass. The dependency PR validates Rust lint/tests, a clean CMake build, installed-package linkage, and a real GPT-2 JSON round trip. With the feature disabled, ExecuTorch does not invoke Cargo or link the optional backend.
AI-assisted by Codex.