Skip to content

Add experimental Hugging Face .tok tokenizer path - #23139

Draft
JacobSzwejbka wants to merge 4 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok-followup
Draft

JacobSzwejbka wants to merge 4 commits into
pytorch:mainfrom
JacobSzwejbka:hf-rust-tokenizer-tok-followup

Update tokenizer size measurements

25dddf4
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

2 warnings

The logs for this run have expired and are no longer available.