Add experimental Hugging Face .tok tokenizer path

[JacobSzwejbka]

Summary

This is the experimental .tok wiring follow-up to pytorch/executorch#22548 and meta-pytorch/tokenizers#218. Do not merge it before either dependency.

It adds the ExecuTorch format selector needed to build the Rust tokenizer as .tok-only or with both JSON and .tok support. The default remains JSON.

.tok is not part of Hugging Face tokenizers v1.0.0-rc.2. The tokenizer dependency draft pins an experimental Hugging Face branch commit and is explicitly blocked on that functionality landing in a protected upstream branch or release.

Because this PR is stacked, GitHub currently shows its JSON base changes too. After #22548 and the tokenizer dependency land, rebase this branch and update the tokenizer submodule pointer; the remaining diff will be only the experimental format selection.

AI-assisted by Codex.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23139

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 48 New Failures, 6 Unclassified Failures

As of commit 25dddf4 with merge base 83c4dc1 ():

NEW FAILURES - The following jobs have failed:

• Build Presets / apple (ios) / build (gh)
  ##[error]fatal: Fetched in submodule path 'kernels/optimized/third-party/eigen', but it did not contain 729443409942a1816ddf74b95224003b83f4925c. Direct fetching of that commit failed.
• Build Presets / apple (profiling) / build (gh)
  ##[error]fatal: Fetched in submodule path 'kernels/optimized/third-party/eigen', but it did not contain 729443409942a1816ddf74b95224003b83f4925c. Direct fetching of that commit failed.
• Build Presets / linux (linux, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  Could not load credentials from any providers
• Build Presets / linux (llm, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  Could not load credentials from any providers
• Cadence Build & Test / Resolve CI docker image / resolve (gh)
  ##[error]Refusing to check out fork pull request code from a 'pull_request_target' workflow. This workflow runs with the base repository's GITHUB_TOKEN, secrets, default-branch cache scope, and runner access. Fetching and executing a fork's code in that trusted context commonly leads to "pwn request" vulnerabilities. To opt in, review the risks at https://gh.io/securely-using-pull_request_target and set 'allow-unsafe-pr-checkout: true' on the actions/checkout step.
• pull / android / build-android (gh)
  remote: Token has expired
• pull / test-arm-backend-no-driver (test_pytest_ops_tosa) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-arm-backend-no-driver (test_run_tosa) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-arm-cortex-m-size-test (zephyr-preset) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-build-wasm-linux / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama_runner_eager-linux / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-linux (bf16, custom, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubunt... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-g... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (add_mul, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (ic3, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (linear, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (linear, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (llama3_2_vision_encoder, portable, mt-l-x86iavx512-16-128) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (resnet50, portable, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, mt-l-x86iavx512-8-64) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux-basic (mv3, portable, buck2, mt-l-x86iavx512-8-64, executorch-ubuntu-22.04-clan... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux-basic (mv3, portable, cmake, mt-l-arm64g4-16-62, executorch-ubuntu-22.04-gcc11-... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, mt-l-x86iavx512-8-64, execu... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, mt-l-arm64g4-16-62, executo... / linux-job (gh)
  Could not load credentials from any providers
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  Could not load credentials from any providers
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  Could not load credentials from any providers
• pull / test-qnn-models-linux (mv3) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  Could not load credentials from any providers
• pull / test-vulkan-models-linux / linux-job (gh)
  Could not load credentials from any providers
• pull / unittest / linux / linux-job (gh)
  Could not load credentials from any providers
• pull / unittest-editable / macos / macos-job (gh)
  ##[error]fatal: Fetched in submodule path 'kernels/optimized/third-party/eigen', but it did not contain 729443409942a1816ddf74b95224003b83f4925c. Direct fetching of that commit failed.
• pull / unittest-nxp-neutron / linux-job (gh)
  Could not load credentials from any providers
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  Could not load credentials from any providers
• pull / unittest-wasm-bindings / linux-job (gh)
  Could not load credentials from any providers
• Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, operators) / linux-job (gh)
  Could not load credentials from any providers
• Test ARM Backend / test-arm-vgf / test-backend-linux (arm_vgf_fp, models) / linux-job (gh)
  Could not load credentials from any providers
• Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, operators) / linux-job (gh)
  Could not load credentials from any providers
• Test WebGPU Backend / test-webgpu / test-backend-linux (webgpu, models) / linux-job (gh)
  Could not load credentials from any providers

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Aarch64 Linux Wheels / pytorch/executorch / build-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]fatal: Fetched in submodule path 'kernels/optimized/third-party/eigen', but it did not contain 729443409942a1816ddf74b95224003b83f4925c. Direct fetching of that commit failed.
• Build Aarch64 Linux Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu-aarch64 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build Linux CUDA Wheels / pytorch/executorch / build-manywheel-py3_10-cuda13_0 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]fatal: Fetched in submodule path 'kernels/optimized/third-party/eigen', but it did not contain 729443409942a1816ddf74b95224003b83f4925c. Direct fetching of that commit failed.
• Build Linux CUDA Wheels / pytorch/executorch / upload / upload-manywheel-py3_10-cuda13_0 (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• periodic / test-models-linux (cmake, vit, portable, mt-l-x86iavx512-8-64, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Could not load credentials from any providers
• pull / llm-server / linux / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Could not load credentials from any providers

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.