Skip to content

Follow-ups from the 2026-11 token-count update #19

Open

Description

@ThomasK33

This issue tracks follow-ups from the token-count update (PRs #10, #14, #15, #16 and their successors). I add an item when the code it concerns is on main. Each item names the PR that fixes it or the trigger that starts it.

From #10 (Claude encodings, merged)

  • Map Claude models to claude_v3, claude_v4_7, claude_v4_8 and claude_v5 in models.json, with per-model frames and tool overhead. The Claude models PR (branch feat/claude-models) does this.
  • Apply the claude_v5 trailing-whitespace rule only at the end of a user message and the system prompt in count(). The encoding trims every text it counts. The Claude models PR does this.
  • A final single space after U+302A to U+302F counts 1 too many on claude_v3, claude_v4_7 and claude_v4_8. Unicode 16 letters U+A7CC, U+A7CD and U+A7DA count about 1 off. Both gaps come from ctok's reconstructed vocabulary and are documented in the README. Trigger: a ctok upstream fix, then port it.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

Activity

  1. self-assigned this
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions