Skip to content

Mirror https://github.com/ggml-org/llama.cpp/pull/22121 #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ngxson wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Mirror https://github.com/ggml-org/llama.cpp/pull/22121 #103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
74fbcb2
use /// instead of // for llama_token_to_piece docstring
julmb Apr 19, 2026
72d9180
clarify llama_token_to_piece docstring in analogy to llama_detokenize
julmb Apr 19, 2026
b6387d2
perform bounds checking on token in token_to_piece
julmb Apr 19, 2026
492a14f
remove always-true conditional
julmb Apr 19, 2026
3828da4
handle INT32_MIN result of llama_token_to_piece in downstream uses
julmb Apr 20, 2026
af7af8a
add limits includes in files that did not use it before
julmb Apr 26, 2026
6b190ab
add invalid token guard to token_to_piece_for_cache
julmb Apr 27, 2026
0e9b56d
propagate token_to_piece failure
julmb Apr 27, 2026
36fe0b0
consistent punctuation
julmb Apr 28, 2026
2b00549
exit on llama_token_to_piece error
julmb Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions common/common.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1662,6 +1662,9 @@ std::string common_token_to_piece(const struct llama_vocab * vocab, llama_token
std::string piece;
piece.resize(piece.capacity()); // using string internal cache, 15 bytes + '\n'
const int n_chars = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (n_chars < 0) {
piece.resize(-n_chars);
int check = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
Expand All @@ -1684,6 +1687,9 @@ std::string common_detokenize(const struct llama_vocab * vocab, const std::vecto
std::string text;
text.resize(std::max(text.capacity(), tokens.size()));
int32_t n_chars = llama_detokenize(vocab, tokens.data(), (int32_t)tokens.size(), &text[0], (int32_t)text.size(), false, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Detokenization failed: some supplied token is invalid");
}
if (n_chars < 0) {
text.resize(-n_chars);
n_chars = llama_detokenize(vocab, tokens.data(), (int32_t)tokens.size(), &text[0], (int32_t)text.size(), false, special);
Expand Down
1 change: 1 addition & 0 deletions common/speculative.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include <cassert>
#include <cstring>
#include <iomanip>
#include <limits>
#include <map>
#include <cinttypes>

Expand Down
4 changes: 4 additions & 0 deletions examples/batched.swift/Sources/main.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,10 @@ private func tokenize(text: String, add_bos: Bool) -> [llama_token] {
private func token_to_piece(token: llama_token, buffer: inout [CChar]) -> String? {
var result = [CChar](repeating: 0, count: 8)
let nTokens = llama_token_to_piece(vocab, token, &result, Int32(result.count), 0, false)
if nTokens == Int32.min {
print("llama_token_to_piece() failed")
exit(1)
}
Comment on lines +227 to +230

@coderabbitai coderabbitai Bot Apr 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clear the pending UTF-8 buffer on the invalid-token path.

This early return preserves buffer. If the stream was holding the first bytes of a multi-byte scalar, the next valid piece gets decoded against stale bytes and the printed output becomes corrupted.

🩹 Proposed fix 
     if nTokens == Int32.min {
+        buffer.removeAll(keepingCapacity: true)
         return nil
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if nTokens == Int32.min {
return nil
}
if nTokens == Int32.min {
buffer.removeAll(keepingCapacity: true)
return nil
}
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@examples/batched.swift/Sources/main.swift` around lines 227 - 229, The early
return on the invalid-token path leaves the UTF-8 `buffer` intact, causing
subsequent decoding to use stale bytes; modify the branch that checks `if
nTokens == Int32.min { return nil }` to clear or reset the `buffer` (the pending
UTF-8 byte storage) before returning so no partial multi-byte scalar remains.
Locate the check that uses `nTokens` and the `buffer` variable in the
decoding/streaming routine (where `nTokens` is compared to `Int32.min`) and
ensure you explicitly clear/reset `buffer` prior to returning nil.

if nTokens < 0 {
let actualTokensCount = -Int(nTokens)
result = .init(repeating: 0, count: actualTokensCount)
Expand Down
4 changes: 3 additions & 1 deletion examples/llama.swiftui/llama.cpp.swift/LibLlama.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,9 @@ actor LlamaContext {
result.deallocate()
}
let nTokens = llama_token_to_piece(vocab, token, result, 8, 0, false)

if nTokens == Int32.min {
return []
}
Comment on lines +322 to +324

@coderabbitai coderabbitai Bot Apr 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t collapse Int32.min into the normal empty-piece case.

llama_token_to_piece(..., special: false) can legitimately produce 0 bytes, and this helper already models that as []. Returning the same [] for Int32.min hides the invalid-token condition from completion_loop(), so it can’t clear temporary_invalid_cchars or react differently. Please return a distinct failure signal here, e.g. [CChar]? or throws.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@examples/llama.swiftui/llama.cpp.swift/LibLlama.swift` around lines 322 -
324, The current helper collapses the Int32.min error case into a normal
empty-piece result (returning []), which hides invalid-token semantics from
completion_loop(); change the helper that wraps llama_token_to_piece (the
function containing the `if nTokens == Int32.min { return [] }` check) to signal
failure distinctly—either by changing its return type to an optional ([CChar]? )
and return nil on Int32.min, or by making it throw and throw a specific
InvalidToken error; update all callers (notably completion_loop() and any use of
temporary_invalid_cchars) to handle the new nil/throwing case and clear/react to
temporary_invalid_cchars when the helper signals failure.

if nTokens < 0 {
let newResult = UnsafeMutablePointer<Int8>.allocate(capacity: Int(-nTokens))
newResult.initialize(repeating: Int8(0), count: Int(-nTokens))
Expand Down
14 changes: 9 additions & 5 deletions include/llama.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1130,11 +1130,14 @@ extern "C" {
bool add_special,
bool parse_special);

// Token Id -> Piece.
// Uses the vocabulary in the provided context.
// Does not write null terminator to the buffer.
// User can skip up to 'lstrip' leading spaces before copying (useful when encoding/decoding multiple tokens with 'add_space_prefix')
// @param special If true, special tokens are rendered in the output.
/// Token Id -> Piece.
/// Uses the vocabulary in the provided context.
/// Does not write null terminator to the buffer.
/// @return Returns the number of chars/bytes on success, no more than length.
/// @return Returns a negative number on failure - the number of chars/bytes that would have been returned.
/// @return Returns INT32_MIN if the token is not in the vocabulary.
/// @param lstrip User can skip up to 'lstrip' leading spaces before copying (useful when encoding/decoding multiple tokens with 'add_space_prefix')
/// @param special If true, special tokens are rendered in the output.
Comment on lines +1133 to +1140

@coderabbitai coderabbitai Bot Apr 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify that INT32_MIN is a sentinel, not a resize hint.

These return-value bullets still read like “any negative value means required size”, so callers using the usual if (n < 0) resize(-n) pattern can overflow on INT32_MIN. Please spell out that this sentinel must be checked before negating. Also, Line 1142 says “provided context”, but this API takes a llama_vocab *.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@include/llama.h` around lines 1141 - 1148, Update the docblock for the
token-id-to-piece function in include/llama.h to explicitly state that INT32_MIN
is a sentinel meaning "token not in the vocabulary" and must be checked before
any negation (do not use the common pattern resize(-n) without testing for
INT32_MIN), clarify that other negative returns indicate the number of
chars/bytes that would have been returned (resize hint), and correct the phrase
"provided context" to reference the actual parameter type llama_vocab *; keep
the existing notes about lstrip and special unchanged but ensure the
return-value bullets order and wording make the sentinel check unambiguous.

LLAMA_API int32_t llama_token_to_piece(
const struct llama_vocab * vocab,
llama_token token,
Expand All @@ -1147,6 +1150,7 @@ extern "C" {
/// @param text The char pointer must be large enough to hold the resulting text.
/// @return Returns the number of chars/bytes on success, no more than text_len_max.
/// @return Returns a negative number on failure - the number of chars/bytes that would have been returned.
/// @return Returns INT32_MIN if any of the tokens is not in the vocabulary.
/// @param remove_special Allow to remove BOS and EOS tokens if model is configured to do so.
/// @param unparse_special If true, special tokens are rendered in the output.
LLAMA_API int32_t llama_detokenize(
Expand Down
6 changes: 6 additions & 0 deletions src/llama-sampler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3676,13 +3676,19 @@ static void llama_sampler_infill_apply(struct llama_sampler * smpl, llama_token_
}

int len0 = ctx->vocab->token_to_piece(cur_p->data[i0].id, ctx->buf0.data(), ctx->buf0.size(), 0, false);
if (len0 == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (len0 < 0) {
ctx->buf0.resize(len0);
len0 = ctx->vocab->token_to_piece(cur_p->data[i0].id, ctx->buf0.data(), ctx->buf0.size(), 0, false);
assert(len0 > 0);
}

int len1 = ctx->vocab->token_to_piece(cur_p->data[i1].id, ctx->buf1.data(), ctx->buf1.size(), 0, false);
if (len1 == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (len1 < 0) {
ctx->buf1.resize(len1);
len1 = ctx->vocab->token_to_piece(cur_p->data[i1].id, ctx->buf1.data(), ctx->buf1.size(), 0, false);
Expand Down
140 changes: 76 additions & 64 deletions src/llama-vocab.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3137,6 +3137,9 @@ std::string llama_vocab::impl::token_to_piece_for_cache(llama_token token, bool
std::string piece;
piece.resize(piece.capacity()); // using string internal cache
const int n_chars = vocab.token_to_piece(token, &piece[0], piece.size(), 0, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (n_chars < 0) {
piece.resize(-n_chars);
int check = vocab.token_to_piece(token, &piece[0], piece.size(), 0, special);
Expand Down Expand Up @@ -3382,6 +3385,11 @@ std::vector<llama_token> llama_vocab::impl::tokenize(
}

int32_t llama_vocab::impl::token_to_piece(llama_token token, char * buf, int32_t length, int32_t lstrip, bool special) const {
if (token < 0 || token >= (int32_t) id_to_token.size()) {
LLAMA_LOG_ERROR("%s: invalid token %d\n", __func__, token);
return std::numeric_limits<int32_t>::min();
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}

// ref: https://github.com/ggml-org/llama.cpp/pull/7587#discussion_r1620983843
static const int attr_special = LLAMA_TOKEN_ATTR_UNKNOWN | LLAMA_TOKEN_ATTR_CONTROL;
const llama_token_attr attr = token_get_attr(token);
Expand Down Expand Up @@ -3417,82 +3425,80 @@ int32_t llama_vocab::impl::token_to_piece(llama_token token, char * buf, int32_t
}
}

if (0 <= token && token < (int32_t) id_to_token.size()) {
const std::string & token_text = id_to_token[token].text;
switch (get_type()) {
case LLAMA_VOCAB_TYPE_WPM:
case LLAMA_VOCAB_TYPE_SPM:
case LLAMA_VOCAB_TYPE_UGM: {
// NOTE: we accept all unsupported token types,
// suppressing them like CONTROL tokens.
if (attr & (attr_special | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
return _try_copy(token_text.data(), token_text.size());
}
if (attr & LLAMA_TOKEN_ATTR_NORMAL) {
const std::string & token_text = id_to_token[token].text;
switch (get_type()) {
case LLAMA_VOCAB_TYPE_WPM:
case LLAMA_VOCAB_TYPE_SPM:
case LLAMA_VOCAB_TYPE_UGM: {
// NOTE: we accept all unsupported token types,
// suppressing them like CONTROL tokens.
if (attr & (attr_special | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
return _try_copy(token_text.data(), token_text.size());
}
if (attr & LLAMA_TOKEN_ATTR_NORMAL) {
std::string result = token_text;
llama_unescape_whitespace(result);
return _try_copy(result.data(), result.size());
}
if (attr & LLAMA_TOKEN_ATTR_BYTE) {
char byte = (char) token_to_byte(token);
return _try_copy((char*) &byte, 1);
}
break;
}
case LLAMA_VOCAB_TYPE_BPE: {
// NOTE: we accept all unsupported token types,
// suppressing them like CONTROL tokens.
if (attr & (attr_special | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
return _try_copy(token_text.data(), token_text.size());
}
if (attr & LLAMA_TOKEN_ATTR_NORMAL) {
if (escape_whitespaces) {
// SPM-style BPE: tokens contain ▁ for spaces
std::string result = token_text;
llama_unescape_whitespace(result);
return _try_copy(result.data(), result.size());
}
if (attr & LLAMA_TOKEN_ATTR_BYTE) {
char byte = (char) token_to_byte(token);
return _try_copy((char*) &byte, 1);
}
break;
std::string result = llama_decode_text(token_text);
return _try_copy(result.data(), result.size());
}
case LLAMA_VOCAB_TYPE_BPE: {
// NOTE: we accept all unsupported token types,
// suppressing them like CONTROL tokens.
if (attr & (attr_special | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
return _try_copy(token_text.data(), token_text.size());
}
if (attr & LLAMA_TOKEN_ATTR_NORMAL) {
if (escape_whitespaces) {
// SPM-style BPE: tokens contain ▁ for spaces
std::string result = token_text;
llama_unescape_whitespace(result);
return _try_copy(result.data(), result.size());
}
std::string result = llama_decode_text(token_text);
return _try_copy(result.data(), result.size());
}
if (attr & LLAMA_TOKEN_ATTR_BYTE) {
char byte = (char) token_to_byte(token);
return _try_copy((char*) &byte, 1);
}
break;
if (attr & LLAMA_TOKEN_ATTR_BYTE) {
char byte = (char) token_to_byte(token);
return _try_copy((char*) &byte, 1);
}
case LLAMA_VOCAB_TYPE_RWKV: {
std::vector<uint8_t> result = llama_unescape_rwkv_token(token_text);

// If we don't have enough space, return an error
if (result.size() > (size_t)length) {
return -(int)result.size();
}
break;
}
case LLAMA_VOCAB_TYPE_RWKV: {
std::vector<uint8_t> result = llama_unescape_rwkv_token(token_text);

memcpy(buf, result.data(), result.size());
return (int)result.size();
// If we don't have enough space, return an error
if (result.size() > (size_t)length) {
return -(int)result.size();
}
case LLAMA_VOCAB_TYPE_PLAMO2: {
// PLaMo-2 uses similar token handling as BPE/SPM
if (vocab.is_byte(token)) {
// Handle byte tokens like <0xXX>
if (token_text.length() == 6 && token_text.substr(0, 3) == "<0x" && token_text.back() == '>') {
int hex_val = std::stoi(token_text.substr(3, 2), nullptr, 16);
if (length < 1) {
return -1;
}
buf[0] = static_cast<char>(hex_val);
return 1;

memcpy(buf, result.data(), result.size());
return (int)result.size();
}
case LLAMA_VOCAB_TYPE_PLAMO2: {
// PLaMo-2 uses similar token handling as BPE/SPM
if (vocab.is_byte(token)) {
// Handle byte tokens like <0xXX>
if (token_text.length() == 6 && token_text.substr(0, 3) == "<0x" && token_text.back() == '>') {
int hex_val = std::stoi(token_text.substr(3, 2), nullptr, 16);
if (length < 1) {
return -1;
}
buf[0] = static_cast<char>(hex_val);
return 1;
}

// Normal token - just copy the text
std::string result = token_text;
return _try_copy(result.data(), result.size());
}
default:
GGML_ABORT("fatal error");

// Normal token - just copy the text
std::string result = token_text;
return _try_copy(result.data(), result.size());
}
default:
GGML_ABORT("fatal error");
}

return 0;
Expand Down Expand Up @@ -3539,6 +3545,9 @@ int32_t llama_vocab::impl::detokenize(
GGML_ASSERT(avail >= 0);
int32_t n_chars = token_to_piece(tokens[i], text, avail, remove_space, unparse_special);
remove_space = false;
if (n_chars == std::numeric_limits<int32_t>::min()) {
return std::numeric_limits<int32_t>::min();
}
if (n_chars < 0) {
avail = 0;
total -= n_chars;
Expand Down Expand Up @@ -3964,6 +3973,9 @@ std::string llama_vocab::detokenize(const std::vector<llama_token> & tokens, boo
std::string text;
text.resize(std::max(text.capacity(), tokens.size()));
int32_t n_chars = detokenize(tokens.data(), (int32_t)tokens.size(), &text[0], (int32_t)text.size(), false, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Detokenization failed: some supplied token is invalid");
}
if (n_chars < 0) {
text.resize(-n_chars);
n_chars = detokenize(tokens.data(), (int32_t)tokens.size(), &text[0], (int32_t)text.size(), false, special);
Expand Down
4 changes: 4 additions & 0 deletions tests/test-backend-sampler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#undef NDEBUG
#endif

#include <limits>
#include <algorithm>
#include <cstdlib>
#include <cstring>
Expand Down Expand Up @@ -250,6 +251,9 @@ struct test_context {
std::string piece;
piece.resize(piece.capacity()); // using string internal cache, 15 bytes + '\n'
const int n_chars = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (n_chars < 0) {
piece.resize(-n_chars);
int check = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
Expand Down
2 changes: 2 additions & 0 deletions tests/test-tokenizer-random.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,8 @@ def detokenize(self, ids: list[int], remove_special: bool = False, unparse_speci
for i, id in enumerate(ids):
self.token_ids[i] = id
num = self.lib.llama_detokenize(self.model, self.token_ids, len(ids), self.text_buff, len(self.text_buff), remove_special, unparse_special)
if num == - (1 << 31):
raise RuntimeError("error: detokenization failed: some supplied token is invalid")
while num < 0 and len(self.text_buff) < (16 << 20):
self.text_buff = self.ffi.new("uint8_t[]", -2 * num)
num = self.lib.llama_detokenize(self.model, self.token_ids, len(ids), self.text_buff, len(self.text_buff), remove_special, unparse_special)
Expand Down
4 changes: 4 additions & 0 deletions tools/mtmd/mtmd.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#include <windows.h>
#endif

#include <limits>
#include <algorithm>
#include <cerrno>
#include <cstdio>
Expand Down Expand Up @@ -619,6 +620,9 @@ struct mtmd_context {
std::string piece;
piece.resize(piece.capacity()); // using string internal cache, 15 bytes + '\n'
const int n_chars = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
if (n_chars == std::numeric_limits<int32_t>::min()) {
throw std::runtime_error("Token to piece failed: supplied token is invalid");
}
if (n_chars < 0) {
piece.resize(-n_chars);
int check = llama_token_to_piece(vocab, token, &piece[0], piece.size(), 0, special);
Expand Down