Add opt-in Hugging Face Rust JSON tokenizer integration - #22548

Add opt-in Hugging Face Rust JSON tokenizer integration#22548

JacobSzwejbka wants to merge 5 commits into

pytorch:mainpytorch/executorch:mainfrom

JacobSzwejbka:hf-rust-tokenizer-tokJacobSzwejbka/executorch-hf-tokenizer:hf-rust-tokenizer-tokCopy head branch name to clipboard