Search results

decode raises Tiktoken::UnicodeError when tokens are truncated mid multi-byte character (diverges from Python tiktoken)

Status: Open.

#122 In IAPark/tiktoken_ruby;

· yvonnenieuwerth opened on Jul 13, 2026

[Proposal] Claude Tokenizer

help wantedExtra attention is neededExtra attention is needed

Status: Open.

#15 In IAPark/tiktoken_ruby;

· BarberAlec opened on Sep 16, 2023