From a76b38cbd2e1fed152d1dbbb1652bab8022ecb8c Mon Sep 17 00:00:00 2001 From: "riseproject-dev[bot]" <330740410+riseproject-dev[bot]@users.noreply.github.com> Date: Wed, 23 Sep 2026 07:27:16 +0000 Subject: [PATCH 1/2] fastokens: Add version 0.3.2 Signed-off-by: riseproject-dev[bot] <330740410+riseproject-dev[bot]@users.noreply.github.com> --- docs/packages/fastokens.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/packages/fastokens.yaml b/docs/packages/fastokens.yaml index a5ca5ece11e..c5e07f5f660 100644 --- a/docs/packages/fastokens.yaml +++ b/docs/packages/fastokens.yaml @@ -9,3 +9,4 @@ versions: - filename: fastokens-0.3.1-cp39-abi3-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl sha256: b67b3667ce0764bf1feee5397175f4b2dfe473bcd88421b5977fd5ef9885ceef requires-python: '>=3.9' +- version: 0.3.2 From c0ff60ee248055fa5917ba5065cd26442fef9f13 Mon Sep 17 00:00:00 2001 From: Ludovic Henry Date: Thu, 24 Sep 2026 07:18:10 +0000 Subject: [PATCH 2/2] fastokens: patch a broken 0.3.2 test fixture (riscv64 CI) The riscv64 build wasn't failing on the version bump itself: the wheel built fine, and the failure was 4 pytest failures in the new (v0.3.2) python/tests/test_decode_sanitize.py, all 'ValueError: model error: unsupported model type: Other'. Root cause upstream: the fixture's vocab is {"h": 1, "e": 2, "l": 3, "o": 4} (no token id 0). Bpe::new() requires vocab ids to be a contiguous range from 0, so building it errors with 'non-contiguous tokens - token 0 is missing'. Because ModelConfig is #[serde(tag = "type")] with an Other(Value) #[serde(untagged)] fallback variant, serde silently swallows that TryFrom error and lands on ModelConfig::Other instead of propagating it -- hence the confusing 'unsupported model type: Other' rather than the real error. This is a plain logic bug, not riscv64-specific; upstream's own CI never catches it because ci.yml runs 'cargo test' (not the Python suite) and never runs pytest against python/tests. Add an unused id-0 vocab entry so the vocab is contiguous; every id the tests actually exercise (1-4) is unchanged. Wire the patch into build-fastokens.yml the same way other patched packages do: a second checkout of python-wheels and a 'git apply' before the build, plus patches/fastokens/** in the pull_request/push path triggers. --- .github/workflows/build-fastokens.yml | 11 +++++ ...code-sanitize-fixture-vocab-a-token-.patch | 42 +++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 patches/fastokens/0.3.2/0001-test-give-the-decode-sanitize-fixture-vocab-a-token-.patch diff --git a/.github/workflows/build-fastokens.yml b/.github/workflows/build-fastokens.yml index f42b6e78695..6ebaefb5759 100644 --- a/.github/workflows/build-fastokens.yml +++ b/.github/workflows/build-fastokens.yml @@ -17,11 +17,13 @@ on: paths: - '.github/workflows/build-fastokens.yml' - 'docs/packages/fastokens.yaml' + - 'patches/fastokens/**' push: branches: [main] paths: - '.github/workflows/build-fastokens.yml' - 'docs/packages/fastokens.yaml' + - 'patches/fastokens/**' concurrency: group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} @@ -62,6 +64,15 @@ jobs: ref: v${{ env.FASTOKENS_VERSION }} persist-credentials: false + - name: Checkout python-wheels + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: python-wheels + persist-credentials: false + + - name: Patch fastokens source + run: git apply python-wheels/patches/fastokens/${{ env.FASTOKENS_VERSION }}/*.patch + - name: Build wheels uses: pypa/cibuildwheel@1828c10ab37f080699c7b81cea34097c684a7074 # v4.2.0 with: diff --git a/patches/fastokens/0.3.2/0001-test-give-the-decode-sanitize-fixture-vocab-a-token-.patch b/patches/fastokens/0.3.2/0001-test-give-the-decode-sanitize-fixture-vocab-a-token-.patch new file mode 100644 index 00000000000..f8cc1e9ce98 --- /dev/null +++ b/patches/fastokens/0.3.2/0001-test-give-the-decode-sanitize-fixture-vocab-a-token-.patch @@ -0,0 +1,42 @@ +From 1f0f5de21aa07524474644cc6466bbf4c2c0a031 Mon Sep 17 00:00:00 2001 +From: Ludovic Henry +Date: Thu, 24 Sep 2026 07:16:36 +0000 +Subject: [PATCH] test: give the decode-sanitize fixture vocab a token 0 + +_tokenizer()'s vocab was {"h": 1, "e": 2, "l": 3, "o": 4} -- no token +id 0. Bpe::new() builds id_to_token over 0..=max_token and requires every +id in that range to be present ("non-contiguous tokens - token N is +missing"), so Tokenizer.from_json_str() fails for this fixture with a +model-construction error. + +Because ModelConfig is #[serde(tag = "type")] with an Other(Value) +#[serde(untagged)] fallback variant, serde swallows that inner +TryFrom error and silently deserializes into ModelConfig::Other +instead of propagating it, so every test in this file fails with the +confusing ValueError: model error: unsupported model type: Other rather +than the real cause. + +Add an unused id-0 entry so the vocab is contiguous from 0; every id the +tests exercise (1-4) is unchanged. + +Upstream-Status: To upstream [no upstream issue/PR access from the riseproject-dev python-wheels session that found this; see https://github.com/riseproject-dev/python-wheels/pull/2234] +--- + python/tests/test_decode_sanitize.py | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/python/tests/test_decode_sanitize.py b/python/tests/test_decode_sanitize.py +index 76e7d57..e3d5df7 100644 +--- a/python/tests/test_decode_sanitize.py ++++ b/python/tests/test_decode_sanitize.py +@@ -29,7 +29,7 @@ def _tokenizer(): + "fuse_unk": False, + "byte_fallback": False, + "ignore_merges": True, +- "vocab": {"h": 1, "e": 2, "l": 3, "o": 4}, ++ "vocab": {"": 0, "h": 1, "e": 2, "l": 3, "o": 4}, + "merges": [], + }, + } +-- +2.43.0 +