A faster than tiktoken tokenizer with first-class support for Vercel's AI SDK.
- Supports the AI SDK tool and message schema
- No WASM; highly portable; 5-7x faster than tiktoken WASM
- >=97% accuracy for top models (GPT-5, Claude Sonnet, Gemini):
| Model | ~500 tokens | ~5k tokens | ~50k tokens |
|---|---|---|---|
| openai/gpt-5 | 99.16% | 99.43% | 99.47% |
| anthropic/claude-opus-4.5 | 98.48% | 98.56% | 99.79% |
| google/gemini-3-pro-preview | 98.70% | 98.77% | 98.91% |
Install:
bun install ai-tokenizerEstimate tokens with the AI SDK:
import Tokenizer, { models } from "ai-tokenizer";
import { count } from "ai-tokenizer/sdk";
import * as encoding from "ai-tokenizer/encoding";
// Find the respective model.
const model = models["openai/gpt-5"];
const tokenizer = new Tokenizer(encoding[model.encoding]);
const result = count({
tokenizer,
model,
// Every message is counted separately
messages,
// Tool descriptions and input schemas are counted
tools,
})
// {
// total: 150,
// messages: [
// {
// total: 120,
// content: [
// { type: "text", total: 100 },
// { type: "tool-call", total: 20, input: 15 }
// ]
// }
// ],
// tools: {
// total: 30,
// definitions: {
// myTool: {
// name: 5,
// description: 10,
// inputSchema: 15
// }
// }
// }
// }
// No API calls are performed; no keys.
const costUSD = result.total * model.pricing.input;Estimate tokens using an encoding:
import { Tokenizer } from "ai-tokenizer"
import * as o200k_base from "ai-tokenizer/encoding/o200k_base"
const tokenizer = new Tokenizer(o200k_base);
const total = tokenizer.count("some text input");Warning
Import encodings selectively based on what you need. Each encoding is 2-8MB uncompressed.
Anthropic changed its tokenizer at Claude Opus 4.7. Four encodings count Claude tokens offline. Each one is named after the first model that counts that way. They match the count_tokens API exactly on all 93 texts in src/fixtures/claude-count-tokens.json, measured on Claude Opus 4.6, Opus 4.7, Opus 4.8 and Opus 5.5. The older claude encoding stays available unchanged.
| Encoding | Models | Frame |
|---|---|---|
claude_v3 |
Claude 3 through Claude Opus 4.6, Sonnet 4.6 and Haiku 4.5 | 7 |
claude_v4_7 |
Claude Opus 4.7 | 11 |
claude_v4_8 |
Claude Opus 4.8, Sonnet 5 and Fable 5 | 6 |
claude_v5 |
Claude Opus 5, Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 | 6 on Opus 5, 8 on the 5.5 and 5.1 models |
claude_v4_7, claude_v4_8 and claude_v5 share one vocabulary. They differ in how the message frame treats the content.
import { Tokenizer } from "ai-tokenizer";
import * as claude_v5 from "ai-tokenizer/encoding/claude_v5";
const tokenizer = new Tokenizer(claude_v5);
tokenizer.count("hello, world"); // 4count()counts the content of one user message.count_tokensalso charges a fixed message frame, shown in the table above. The frame comes from the model config, not from the encoding.- The content count includes what the frame does to the content. On
claude_v3andclaude_v4_7the frame absorbs one leading space. On every encoding, newlines at the end of the content join the frame's own newline token, so a short run of them is free. claude_v5drops trailing ASCII whitespace (space, tab, CR, LF, VT, FF) from the text it counts. These models drop it at the end of each user message and at the end of the system prompt. They do not drop it in assistant messages or intool_resultcontent. The API joins the text blocks of a message with no separator and trims only the last block, so trailing whitespace in earlier blocks still counts. A last block that holds only whitespace is not trimmed:["a", " "]costs what"a "costs onclaude_v4_8. To count a user message with several text blocks, count the joined text withclaude_v5, or withclaude_v4_8when the last block is only whitespace.count("")is 0. The API rejects empty content.- Token ids are synthetic, because Anthropic does not publish its ids. They are stable for one version of this package only.
decode()returns the normalized text, so it is lossy. Plain ASCII prose comes back unchanged. Unicode normalization, folded quotes and spaces, removed control characters, the absorbed leading space, some case changes and trailing whitespace do not.- Counts use Unicode 15.1 data, like the reference. A JavaScript engine with Unicode data older than 15.1 can count text with newer characters differently.
- Some Latin letters added in Unicode 16 (for example U+A7CC, U+A7CD and U+A7DA) count a few tokens off. ctok has the same gap.
- A single space at the end of the text, right after one of the ideographic tone marks U+302A to U+302F, counts 1 token too many on
claude_v3,claude_v4_7andclaude_v4_8("x \u302e "is 5 on the API and 6 here). ctok has the same gap.claude_v5is right, because it trims that space.
The encodings are a TypeScript port of ctok by Sander Land (MIT License), at commit 8c43ffd. scripts/generate-claude-encodings.ts regenerates the data from that commit, and scripts/compare-claude-ctok.ts compares the port with ctok.
Validated against actual API responses with pseudo-random messages:
| Model | ~500 tokens | ~5k tokens | ~50k tokens |
|---|---|---|---|
| alibaba/qwen-3-14b | 96.45% | 96.45% | N/A |
| alibaba/qwen-3-235b | 98.26% | 96.38% | N/A |
| alibaba/qwen-3-30b | 96.45% | 96.27% | N/A |
| alibaba/qwen-3-32b | 95.64% | 95.70% | N/A |
| alibaba/qwen3-235b-a22b-thinking | 96.06% | 96.34% | 95.98% |
| alibaba/qwen3-coder | 94.55% | 96.70% | 94.57% |
| alibaba/qwen3-coder-plus | 94.84% | 96.76% | 94.54% |
| alibaba/qwen3-max | 98.51% | 95.59% | 94.33% |
| alibaba/qwen3-next-80b-a3b-instruct | 95.66% | 96.71% | 95.81% |
| alibaba/qwen3-next-80b-a3b-thinking | 96.02% | 95.70% | 95.09% |
| alibaba/qwen3-vl-instruct | 95.08% | 96.73% | 96.04% |
| amazon/nova-lite | 98.10% | 92.67% | 90.17% |
| amazon/nova-micro | 98.10% | 92.10% | 89.20% |
| amazon/nova-pro | 98.10% | 92.82% | 90.16% |
| anthropic/claude-3-haiku | 97.61% | 99.32% | 99.77% |
| anthropic/claude-3-opus | 98.66% | 98.68% | 99.70% |
| anthropic/claude-3.5-haiku | 97.61% | 99.80% | 99.86% |
| anthropic/claude-3.5-sonnet | 97.93% | 99.26% | 99.67% |
| anthropic/claude-3.5-sonnet-20240620 | 98.01% | 99.98% | 99.86% |
| anthropic/claude-3.7-sonnet | 98.07% | 99.59% | 99.74% |
| anthropic/claude-haiku-4.5 | 98.48% | 98.67% | 99.62% |
| anthropic/claude-opus-4 | 98.07% | 99.39% | 99.74% |
| anthropic/claude-opus-4.1 | 97.78% | 99.83% | 99.80% |
| anthropic/claude-opus-4.5 | 98.48% | 98.56% | 99.79% |
| anthropic/claude-sonnet-4 | 98.22% | 99.89% | 99.83% |
| anthropic/claude-sonnet-4.5 | 98.48% | 98.91% | 99.70% |
| deepseek/deepseek-r1 | 96.24% | 98.67% | 98.10% |
| deepseek/deepseek-r1-distill-llama-70b | 98.20% | 99.08% | N/A |
| deepseek/deepseek-v3 | 95.30% | 95.37% | 95.19% |
| deepseek/deepseek-v3.1 | 97.40% | 95.85% | 96.46% |
| deepseek/deepseek-v3.1-terminus | 96.20% | 97.57% | 97.55% |
| deepseek/deepseek-v3.2-exp | 95.77% | 97.62% | 97.45% |
| deepseek/deepseek-v3.2-exp-thinking | 95.98% | 97.55% | 97.56% |
| google/gemini-2.0-flash | 97.97% | 99.69% | 99.95% |
| google/gemini-2.0-flash-lite | 98.18% | 99.65% | 99.77% |
| google/gemini-2.5-flash | 98.78% | 99.60% | 99.85% |
| google/gemini-2.5-flash-lite | 99.19% | 99.33% | 99.84% |
| google/gemini-2.5-flash-lite-preview-09-2025 | 92.74% | 99.63% | 99.74% |
| google/gemini-2.5-flash-preview-09-2025 | 92.53% | 99.69% | 99.74% |
| google/gemini-2.5-pro | 98.37% | 99.42% | 99.82% |
| google/gemini-3-pro-preview | 98.70% | 98.77% | 98.91% |
| meituan/longcat-flash-chat | 70.65% | 99.15% | 96.13% |
| meta/llama-3.1-8b | 98.69% | 90.67% | 89.69% |
| meta/llama-3.3-70b | 100.00% | 70.03% | 67.23% |
| meta/llama-4-scout | 0.00% | 56.65% | 66.03% |
| minimax/minimax-m2 | 94.13% | 74.53% | 72.83% |
| mistral/codestral | 93.78% | 91.86% | 91.89% |
| mistral/devstral-small | 98.77% | 94.09% | 93.72% |
| mistral/magistral-medium | 98.24% | 94.23% | 93.71% |
| mistral/magistral-small | 97.89% | 94.32% | 93.65% |
| mistral/mistral-large | 98.23% | 94.09% | N/A |
| mistral/mistral-medium | 98.06% | 93.96% | 93.79% |
| mistral/mistral-small | 96.73% | 92.78% | N/A |
| mistral/pixtral-large | 98.94% | 94.25% | 94.27% |
| moonshotai/kimi-k2 | 89.51% | NaN% | 82.80% |
| moonshotai/kimi-k2-0905 | 85.58% | 81.01% | 80.56% |
| moonshotai/kimi-k2-thinking | 95.02% | 92.80% | 91.96% |
| moonshotai/kimi-k2-thinking-turbo | 95.03% | 92.38% | 91.91% |
| moonshotai/kimi-k2-turbo | 94.51% | 92.36% | 91.83% |
| openai/gpt-4-turbo | 99.79% | 99.89% | 99.86% |
| openai/gpt-4.1 | 100.00% | 99.93% | 99.92% |
| openai/gpt-4.1-mini | 100.00% | 99.93% | 99.92% |
| openai/gpt-4o | 100.00% | 99.93% | 99.92% |
| openai/gpt-4o-mini | 100.00% | 99.93% | 99.92% |
| openai/gpt-5 | 99.16% | 99.43% | 99.47% |
| openai/gpt-5-chat | 100.00% | 99.93% | 99.92% |
| openai/gpt-5-codex | 99.17% | 99.43% | 99.47% |
| openai/gpt-5-mini | 99.16% | 99.43% | 99.47% |
| openai/gpt-5-nano | 99.16% | 99.43% | 99.46% |
| openai/gpt-5-pro | 99.17% | 99.43% | 99.47% |
| openai/gpt-5.1-codex | 79.84% | 98.72% | 99.65% |
| openai/gpt-5.1-codex-mini | 99.17% | 99.43% | 99.47% |
| openai/gpt-5.1-instant | 79.84% | 98.72% | 99.65% |
| openai/gpt-5.1-thinking | 79.84% | 98.72% | 99.64% |
| openai/gpt-oss-120b | 80.77% | 69.64% | 69.62% |
| openai/gpt-oss-20b | 68.55% | 60.79% | 60.98% |
| openai/gpt-oss-safeguard-20b | 75.60% | 64.08% | 63.67% |
| openai/o1 | 94.77% | 99.30% | 99.86% |
| openai/o3 | 99.16% | 99.43% | 99.46% |
| openai/o3-mini | 100.00% | NaN% | 99.92% |
| openai/o4-mini | 99.16% | 99.43% | 99.47% |
| prime-intellect/intellect-3 | 93.08% | 95.64% | 97.12% |
| stealth/sonoma-dusk-alpha | 99.47% | 71.65% | 66.69% |
| stealth/sonoma-sky-alpha | 99.46% | 69.52% | 66.49% |
| vercel/v0-1.0-md | 98.99% | 53.45% | 92.42% |
| vercel/v0-1.5-md | 98.96% | 53.55% | 92.39% |
| xai/grok-2 | 95.43% | 96.09% | 95.28% |
| xai/grok-2-vision | 94.44% | 95.68% | N/A |
| xai/grok-3 | 99.57% | 98.67% | 99.23% |
| xai/grok-3-fast | 99.57% | 98.59% | 99.13% |
| xai/grok-3-mini | 99.59% | 98.76% | 99.18% |
| xai/grok-3-mini-fast | 99.59% | 98.59% | 99.13% |
| xai/grok-4 | 99.72% | 73.95% | 67.43% |
| xai/grok-4-fast-non-reasoning | 90.02% | -22.27% | -52.27% |
| xai/grok-4-fast-reasoning | 99.29% | 71.17% | 66.72% |
| xai/grok-4.1-fast-non-reasoning | 99.65% | 71.70% | 66.76% |
| xai/grok-4.1-fast-reasoning | 99.28% | 71.50% | 66.62% |
| xai/grok-code-fast-1 | 80.14% | -13.63% | -51.13% |
| zai/glm-4.5 | 94.55% | 96.26% | 96.16% |
| zai/glm-4.5-air | 96.73% | 96.45% | 96.70% |
| zai/glm-4.5v | 95.17% | 95.56% | 95.36% |
Accuracy shows percentage within actual token count.
Warning
Not every tool/token scenario is tested for accuracy. There will be edge-cases where this is more/less accurate. If you find this inaccurate for your scenario, please open an issue.
Run bun run scripts/generate-accuracy.ts to update this table. Refer to accuracy.json for greater detail.
ai-tokenizer is 5-7x faster than tiktoken for counting tokens. It is on par with gpt-tokenizer.
$ bun bench/versus
clk: ~3.96 GHz
cpu: AMD Ryzen AI 9 HX 370 w/ Radeon 890M
runtime: bun 1.2.19 (x64-linux)
benchmark avg (min … max) p75 / p99 (min … top 1%)
------------------------------------------- -------------------------------
• initialization
------------------------------------------- -------------------------------
ai-tokenizer 13.99 µs/iter 14.14 µs █
(13.44 µs … 15.50 µs) 14.85 µs █
( 0.00 b … 1.24 kb) 308.13 b ██▁█▁█▁▁▁▁█▁▁▁▁▁▁▁▁▁█
tiktoken 85.05 ms/iter 85.68 ms █ █
(81.61 ms … 89.19 ms) 86.55 ms ▅ █ ▅ ▅▅█▅▅ ▅
( 0.00 b … 1.39 mb) 385.33 kb █▁▁▁▁▁▁▁█▁▁█▁█████▁▁█
• encode: small text (~13 chars)
------------------------------------------- -------------------------------
ai-tokenizer 1.33 µs/iter 1.34 µs ▇█
(1.11 µs … 1.90 µs) 1.60 µs ██
( 0.00 b … 2.06 kb) 1.60 kb ▆▃▁▁▁▁▁▂███▃▄▂▃▂▃▂▁▁▂
gpt-tokenizer 1.05 µs/iter 1.13 µs █
(926.82 ns … 1.65 µs) 1.53 µs ██ ▆
( 0.00 b … 864.00 b) 352.86 b ██▂▁▁▁██▆▃▂▂▁▁▁▁▁▁▁▁▁
tiktoken 32.23 µs/iter 30.19 µs █
(27.83 µs … 468.68 µs) 66.56 µs █
( 0.00 b … 1.13 mb) 5.27 kb ██▃▂▂▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁
• encode: medium text (~4.5KB)
------------------------------------------- -------------------------------
ai-tokenizer 89.81 µs/iter 92.53 µs █
(66.07 µs … 372.65 µs) 208.17 µs █ █▇
( 0.00 b … 1.13 mb) 98.84 kb ▄█▃██▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁
gpt-tokenizer 76.97 µs/iter 80.01 µs █ ▆
(66.79 µs … 279.16 µs) 115.17 µs █ █▅
( 0.00 b … 384.00 kb) 5.44 kb ▆█▅█▆██▃▂▁▂▂▁▂▂▁▁▁▁▁▁
tiktoken 651.22 µs/iter 649.48 µs ▄█
(632.83 µs … 874.11 µs) 782.25 µs ██
( 0.00 b … 192.00 kb) 4.31 kb ▄██▆▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁
• encode: large text (~500KB)
------------------------------------------- -------------------------------
ai-tokenizer 19.07 ms/iter 20.23 ms ▅█
(15.19 ms … 20.67 ms) 20.66 ms ▃ ▃██▃▃
( 0.00 b … 26.44 mb) 19.22 mb ▄▁▄▄█▁█▁▁▁▄▁▁▁▄▄█████
gpt-tokenizer 18.19 ms/iter 18.62 ms █
(15.35 ms … 23.68 ms) 22.91 ms █ ▃
( 0.00 b … 31.50 mb) 6.44 mb ▅▃▅▁█▁█▃▃▃▁▁█▁▁▁▁▅▁▁▅
tiktoken 135.38 ms/iter 135.35 ms █ █ █
(134.51 ms … 138.22 ms) 136.61 ms █▅ ▅ ▅█ █▅ ▅
( 0.00 b … 768.00 kb) 648.00 kb ██▁█▁██▁██▁▁▁▁▁▁▁▁▁▁█
• encode: unicode text
------------------------------------------- -------------------------------
ai-tokenizer 118.76 µs/iter 118.95 µs █
(84.61 µs … 500.48 µs) 234.66 µs ██
( 0.00 b … 768.00 kb) 129.37 kb ▃▄▂██▇▄▄▂▂▂▂▁▁▁▁▁▁▁▁▁
gpt-tokenizer 231.35 µs/iter 256.95 µs █▅▇█
(139.24 µs … 2.08 ms) 489.02 µs ████ █
( 0.00 b … 2.41 mb) 260.61 kb ▃▅████▇▆█▆▃▂▂▂▁▁▁▂▁▁▁
tiktoken 836.84 µs/iter 819.36 µs █
(799.44 µs … 1.97 ms) 1.39 ms █▅
( 0.00 b … 192.00 kb) 5.87 kb ██▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
• encode: code
------------------------------------------- -------------------------------
ai-tokenizer 357.64 µs/iter 372.83 µs █▄
(278.58 µs … 1.04 ms) 591.56 µs ▃ ██
( 0.00 b … 1.31 mb) 364.10 kb ██▂▁▂██▅▃▂▂▁▂▂▁▁▁▁▁▁▁
gpt-tokenizer 316.39 µs/iter 316.25 µs █▅
(279.98 µs … 919.92 µs) 622.31 µs ██
( 0.00 b … 192.00 kb) 54.12 kb ▄██▅▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
tiktoken 3.12 ms/iter 3.12 ms ▆█
(3.05 ms … 3.72 ms) 3.52 ms ██▃
( 0.00 b … 192.00 kb) 20.06 kb ▆███▅▃▁▁▁▁▂▂▂▂▁▁▂▁▁▁▂
• encode: mixed content
------------------------------------------- -------------------------------
ai-tokenizer 2.44 ms/iter 2.48 ms ▇█▄
(2.06 ms … 2.89 ms) 2.81 ms ▂████▃
( 0.00 b … 3.56 mb) 2.36 mb ▁▂▂▁▁▁▁██████▄▃▆▄▂▂▁▂
gpt-tokenizer 2.62 ms/iter 2.60 ms █
(2.38 ms … 4.34 ms) 3.96 ms ▅█
(576.00 kb … 4.88 mb) 1.34 mb ▄███▄▂▂▁▁▁▁▂▁▁▂▁▁▁▁▁▁
tiktoken 15.35 ms/iter 15.37 ms ▂█ ▂
(15.17 ms … 16.35 ms) 15.80 ms ██▇█▂
( 0.00 b … 192.00 kb) 93.54 kb █████▇▄▁▄▄▄▁▄▄▄▁▁▇▄▁▄
• decode: large token array
------------------------------------------- -------------------------------
ai-tokenizer 1.57 ms/iter 1.93 ms █
(1.14 ms … 4.02 ms) 3.53 ms ██
( 0.00 b … 6.94 mb) 1.44 mb ██▄▃▁▁▁▁▁▂▃▄▄▃▂▁▁▁▁▁▁
gpt-tokenizer 876.67 µs/iter 857.62 µs █
(628.32 µs … 3.95 ms) 2.34 ms █
( 0.00 b … 6.00 mb) 27.15 kb ▁██▃▄▃▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁
tiktoken 1.64 ms/iter 1.67 ms ▄ █
(1.47 ms … 2.49 ms) 2.33 ms ▆█ ██▃
( 0.00 b … 1.31 mb) 1.14 mb ██████▅▆▂▃▁▁▂▁▂▂▁▁▂▁▁
• count: large text (~500KB)
------------------------------------------- -------------------------------
ai-tokenizer 18.33 ms/iter 19.52 ms █
(14.94 ms … 20.09 ms) 20.05 ms ▃▆ ▆██▃
( 0.00 b … 25.33 mb) 18.64 mb ██▁▁▁▄▁▁▁▁▁▁▄▄▄█████▄
gpt-tokenizer 16.65 ms/iter 18.26 ms █ █
(14.63 ms … 19.14 ms) 19.01 ms ███ ▅ █
( 0.00 b … 7.50 mb) 1.15 mb ▅████▁▅▅▁▁█▅▁▅██▁██▁█
tiktoken 131.95 ms/iter 131.85 ms █
(130.24 ms … 138.35 ms) 133.42 ms █ █
(576.00 kb … 768.00 kb) 720.00 kb █▁██▁█▁█▁▁█▁█▁▁▁▁▁▁▁█Run this yourself with bun bench/versus.