detokenizer duplicates text when one call holds several messages for the same uid (latent; blocks multi-token emission) #69

Latent. Production can't hit this today. It is a prerequisite fix for any multi-token emission, such as speculative decoding.

DetokenizeManager.detokenize_with_meta (python/freetoken/tokenizer/detokenize.py:116-190) runs in two passes over msgs:

1. The first pass appends each message's token and builds read_ids / surr_ids from the current surr_offset / read_offset.
2. The second pass advances those offsets.

When one call holds two or more messages for the same uid, every message's slice comes from the same pre-update offsets, so text is emitted twice.

Repro (CPU only):

from transformers import AutoTokenizer
from freetoken.message import DetokenizeMsg
from freetoken.tokenizer.detokenize import DetokenizeManager
tok = AutoTokenizer.from_pretrained("/home/gdevenyi/models/Qwen3.8-Flash-Next-NVFP4")
ids = tok.encode("Hello world, this is a speculative decoding test.")
def run(groups):
    m, out = DetokenizeManager(tok), []
    for g in groups:
        out.extend(m.detokenize([DetokenizeMsg(uid=1, next_token=t, finished=False) for t in g]))
    return "".join(out)
print(repr(run([[t] for t in ids])))                              # 'Hello world, this is a speculative decoding test.'
print(repr(run([ids[i:i+2] for i in range(0, len(ids), 2)])))     # 'HelloHello world,, this is is a speculative speculative decoding test test.'

Why it is latent:

• The tokenizer worker only merges queued messages while len(pending_msg) < local_bs (tokenizer/server.py:178), and local_bs is hard-coded to 1 (server/launch.py:185,205).
• The scheduler emits one DetokenizeMsg per request per step (scheduler/scheduler.py:398).
• So a single call never carries the same uid twice today.

Anything that emits several tokens per request per step would trigger it: speculative decoding, local_bs > 1, or batched replies.

Related:

• The think-end check (detokenize.py:135-140) only looks at next_token.
• completion_tokens_delta is hard-coded to 1 per message (tokenizer/server.py:236).
• Status tok/s counts len(batch.reqs) per step (scheduler/status.py:106).

Fix direction: process same-uid messages sequentially (or group by uid first), and carry token_ids per message. Upstream FlashML-org#71 has a DetokenizeMsg.token_ids approach. FlashML-org#258's version compares against endswith("") (the U+FFFD was stripped), so its offsets never advance; don't copy that one.

Found during the speculative-decoding investigation (2026-09-26).

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt