[AmitMY]

Each word is stored as BOS, bytes, EOS. EOS is the bytes decoder's target (where the word ends), but the bytes encoder is bidirectional and already knows each word's length from the mask, so reading EOS tells it nothing. It now reads BOS and the bytes only:

• WeLTModel.encode_words (training) masks each word's last position (EOS) out of the encoder's input
• WeLTGenerator._encode_words (vLLM inference) drops it likewise
• benchmarks/flops.py counts one position fewer per encoded word

Step FLOPs saved (measured on the ClimbMix / FineWeb-2 mixes): 8.8% for the 25M latent transformer with byte models matched in parameters (25.6 → 23.4 TFLOPs), 5.9–6.7% matched in FLOPs, under 1% with decoder-heavy byte models (where the encoder is 1–2 layers).

Compatibility: checkpoints trained before this change read EOS; retrain them, or evaluate them with the earlier code.

Tests: test_bytes_encoder_ignores_eos (a word's embedding does not depend on its EOS; fails before this change), and test_vllm_matches_megatron keeps training and inference in agreement. Full suite: 137 passed, 1 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

---

Note

Medium Risk
Changes word-embedding computation and breaks weight compatibility with checkpoints trained before this change; training and inference paths were updated together.

Overview
The bytes encoder no longer consumes each word’s EOS token—only BOS and the byte sequence—because the bidirectional encoder already knows word length from the attention mask, so EOS added no information.

Training (WeLTModel.encode_words) masks out the final EOS position before the bytes encoder runs. vLLM inference (WeLTGenerator._encode_words) drops EOS from prompt token ids the same way. FLOP benchmarks count one fewer token per encoded word (n - 1).

Docs in the README describe this behavior. A new test asserts that changing EOS does not change word embeddings. Older checkpoints that were trained with EOS in the encoder input are incompatible unless you retrain or run the previous code.

^{Reviewed by Cursor Bugbot for commit 5b2e7be. Bugbot is set up for automated code reviews on this repo. Configure here.}