[rushilbhat]

Summary

• prevent exact-token synthesis from looping when adjacent tokenizer tokens share the same terminal UTF-8 offset
• resample a fresh candidate when offset-based trimming cannot strictly shorten the text
• add a deterministic ByteLevel BPE regression test

Validation

• cargo fmt --check
• cargo check --all-targets
• cargo test (44 passed)
• cargo clippy with the two unchanged main-branch Rust 1.93 lints allowed; no warnings introduced by this change
• full offline GLM 5.2 tokenizer dry-run of the 1,320-trajectory replay manifest: 18,153 requests, all prompt/output/reset fields matched, exit 0

No benchmark or external model traffic was launched.

[Copilot]

Pull request overview

This PR hardens the Rust agent synthetic-text generator against ByteLevel/BPE edge cases where token offsets can’t be used to safely (and strictly) trim text down to an exact token target, preventing repeated attempts from getting “stuck” on an unshrinkable string.

Changes:

• Ensure offset-based trimming only truncates when it will strictly shorten the UTF-8 string (guard end_offset < text.len()).
• When trimming can’t make progress, resample a fresh synthetic token candidate instead of retrying with an equivalent decoded prefix.
• Add a deterministic regression test using a minimal ByteLevel BPE tokenizer to validate exact-token synthesis succeeds for the problematic offset behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.