Skip to content

Add optional Hugging Face Rust JSON tokenizer backend - #212

Open
JacobSzwejbka wants to merge 5 commits into
meta-pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok
Open

Add optional Hugging Face Rust JSON tokenizer backend#212
JacobSzwejbka wants to merge 5 commits into
meta-pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok

Split experimental tok support from JSON integration

e5c2469
Select commit
Loading
Failed to load commit list.
Meta CLA / Meta CLA Check succeeded Sep 24, 2026 in 0s

Contributor License Agreement is valid!

View more details on Meta CLA