fastokens: Add version 0.3.2

.github/workflows/build-fastokens.yml

@@ -17,11 +17,13 @@ on:
     paths:
       - '.github/workflows/build-fastokens.yml'
       - 'docs/packages/fastokens.yaml'
+      - 'patches/fastokens/**'
   push:
     branches: [main]
     paths:
       - '.github/workflows/build-fastokens.yml'
       - 'docs/packages/fastokens.yaml'
+      - 'patches/fastokens/**'
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
@@ -62,6 +64,15 @@ jobs:
           ref: v${{ env.FASTOKENS_VERSION }}
           persist-credentials: false
 
+      - name: Checkout python-wheels
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          path: python-wheels
+          persist-credentials: false
+
+      - name: Patch fastokens source
+        run: git apply python-wheels/patches/fastokens/${{ env.FASTOKENS_VERSION }}/*.patch
+
       - name: Build wheels
         uses: pypa/cibuildwheel@1828c10ab37f080699c7b81cea34097c684a7074  # v4.2.0
         with:

docs/packages/fastokens.yaml

@@ -9,3 +9,4 @@ versions:
   - filename: fastokens-0.3.1-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl
     sha256: b67b3667ce0764bf1feee5397175f4b2dfe473bcd88421b5977fd5ef9885ceef
     requires-python: '>=3.9'
+- version: 0.3.2

patches/fastokens/0.3.2/0001-test-give-the-decode-sanitize-fixture-vocab-a-token-.patch

@@ -0,0 +1,42 @@
+From 1f0f5de21aa07524474644cc6466bbf4c2c0a031 Mon Sep 17 00:00:00 2001
+From: Ludovic Henry <git@ludovic.dev>
+Date: Thu, 24 Sep 2026 07:16:36 +0000
+Subject: [PATCH] test: give the decode-sanitize fixture vocab a token 0
+
+_tokenizer()'s vocab was {"h": 1, "e": 2, "l": 3, "o": 4} -- no token
+id 0. Bpe::new() builds id_to_token over 0..=max_token and requires every
+id in that range to be present ("non-contiguous tokens - token N is
+missing"), so Tokenizer.from_json_str() fails for this fixture with a
+model-construction error.
+
+Because ModelConfig is #[serde(tag = "type")] with an Other(Value)
+#[serde(untagged)] fallback variant, serde swallows that inner
+TryFrom<RawBpe> error and silently deserializes into ModelConfig::Other
+instead of propagating it, so every test in this file fails with the
+confusing ValueError: model error: unsupported model type: Other rather
+than the real cause.
+
+Add an unused id-0 entry so the vocab is contiguous from 0; every id the
+tests exercise (1-4) is unchanged.
+
+Upstream-Status: To upstream [no upstream issue/PR access from the riseproject-dev python-wheels session that found this; see https://github.com/riseproject-dev/python-wheels/pull/2234]
+---
+ python/tests/test_decode_sanitize.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/python/tests/test_decode_sanitize.py b/python/tests/test_decode_sanitize.py
+index 76e7d57..e3d5df7 100644
+--- a/python/tests/test_decode_sanitize.py
++++ b/python/tests/test_decode_sanitize.py
+@@ -29,7 +29,7 @@ def _tokenizer():
+             "fuse_unk": False,
+             "byte_fallback": False,
+             "ignore_merges": True,
+-            "vocab": {"h": 1, "e": 2, "l": 3, "o": 4},
++            "vocab": {"<pad>": 0, "h": 1, "e": 2, "l": 3, "o": 4},
+             "merges": [],
+         },
+     }
+-- 
+2.43.0
+