Skip to content

fix(ui): truncate previews by rune, not byte, so CJK output stays readable - #40

Merged
AlleyBo55 merged 1 commit into
AlleyBo55:masterfrom
ohomygood:fix/multibyte-truncation
Nov 5, 2026
Merged

AlleyBo55 merged 1 commit into
AlleyBo55:masterfrom
ohomygood:fix/multibyte-truncation

Conversation

@ohomygood

Copy link
Copy Markdown
Contributor

Summary

  • Every preview the product prints — the tool-output block in the REPL, the /skills list, and the "last 5 messages" section of an MCP session_load — cut its text with a byte slice (s[:max]), so non-ASCII text was sliced inside a character. The fragment carried a half-finished UTF-8 sequence and the terminal rendered �.
  • Because CJK characters are three bytes and emoji four, those previews also showed about a third of the text their limit promised: an 80-byte cut kept 26 Chinese characters.
  • Truncation now counts runes: internal/textutil.Truncate keeps the output valid UTF-8, keeps the full rune budget, and marks a cut with .... The three call sites use it, and the three duplicate byte-slicing helpers are gone.

Reproduction

A tool result of 160 Chinese characters (480 bytes), rendered by RenderToolOutput("Bash", out, false).

Before (this branch's parent):

  ✓ Bash
  中文输出中文输出中文输出中文输出中文输出中文输出中\xe6\x96...

\xe6\x96 is the first two bytes of the next 中: the line ends in a replacement character and holds 26 characters where the 80-rune budget promised 80.

After:

  ✓ Bash
  中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文输出中文中...

80 runes on the line: 77 characters plus the ellipsis.

The same byte cut appeared in the skill list (truncateSkillDesc, 80 bytes of a description) and in the MCP session summary (truncate(msg.Content, 200), which also overshot its own limit by the three ellipsis bytes).

Tests

go build ./...                                             ok
go test ./...                                              ok (every package)
go test ./internal/textutil ./internal/repl ./internal/mcp ok
  • internal/textutil/textutil_test.go — ASCII, Chinese, Japanese and emoji cases; the result is always valid UTF-8, never exceeds the rune budget, and a limit at or below the ellipsis length cannot overflow either.
  • internal/repl/render_test.go — the operator-visible boundary: a 160-character Chinese tool result and a Chinese error message both render as valid UTF-8 with no U+FFFD, and the preview keeps all 80 runes.

The two boundary tests fail on master for the intended reason (RenderToolOutput returned invalid UTF-8: "...\xe6\x96...") and pass on this branch.

Notes

  • The orchestrator's truncate is deliberately left alone: its message says [N more bytes] and it caps a payload budget, so it counts bytes on purpose. It is not display text.
  • truncateLog had no call sites and was removed with the other duplicates.

@AlleyBo55

Copy link
Copy Markdown
Owner

The fix is correct, the tests check real behaviour, and every change is an improvement on master. Nothing blocks it.
LGTM

@AlleyBo55
AlleyBo55 merged commit 8a31758 into AlleyBo55:master Nov 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants