+ Printable ASCII is computed.
+ Complete non-ASCII grapheme clusters are measured once, cached for the terminal session, and never guessed.
+ Layout remains pure because all terminal I/O happens before layout begins.
+
+
+ design of record
+ 2026-08-18
+ supersedes the predictive width-model proposal
+
+
+
+
+
+
+
+
+ 01
+
The contract is intentionally small
+
+
+ Width is not a Unicode property owned by mshell.
+ It is an observed property of the terminal currently in front of mshell.
+
+
+
+ The previous design attempted to classify terminals with three parameters and then predict every cluster from that classification.
+ That is attractive but not correct: partial clusterers, Unicode-version drift, configuration, and individual terminal quirks do not form eight clean equivalence classes.
+ This design removes the classification problem.
+
+
+
+
+
Assumption A
+
A complete, self-contained printable grapheme cluster occupies one or two cells on supported terminals.
+
+
+
Assumption B
+
Printable ASCII occupies one cell per byte. Newline and renderer controls are handled explicitly.
+
+
+
Assumption C
+
The width of a complete printable cluster is stable during one terminal-width cache epoch.
+
+
+
Assumption D
+
The terminal answers a standard cursor-position request when exact non-ASCII display is enabled.
+
+
+
+
+ These are runtime contracts, not universal claims about Unicode or every historical terminal.
+ Violations are detected and have a defined fallback.
+
+
+
Terminology
+
+ CSI 6 n, commonly written ESC[6n, is a Device Status Report request.
+ The terminal answers with a Cursor Position Report, CSI row ; column R.
+ This document calls the exchange DSR/CPR.
+ SGR is reserved for styling such as color and bold; it is unrelated to measurement.
+
+
+
Explicit non-goals
+
+
No terminal identity database.
+
No cluster-versus-codepoint model selection.
+
No Mode 2027 inference ladder.
+
No use of the width returned by uniseg.FirstGraphemeCluster….
+
No persistent width cache whose correctness depends on identifying a terminal across sessions.
+
No query during ordinary ASCII typing, cursor motion, or repaint.
+
+
+
+
+ 02
+
Resolve first, then lay out
+
+
+
source bufferUTF-8 + byte cursor
+
→
+
classifyall printable ASCII?
+
→
+
two pathsyes: direct layout · no: segment + resolve + atom layout
+
→
+
layout rowspaint-ready geometry
+
+
+
+ The first boundary is a classification of the complete command buffer.
+ A command containing only printable ASCII bypasses atomization and width resolution and is laid out directly from its source bytes.
+ Every other command takes the general path, where width resolution is impure and may talk to the terminal before layout accepts resolved display atoms.
+ Both layout paths perform no I/O.
+ Rendering accepts only a completed layout.
+
+
+
+
+
Domain
Meaning
May contain
+
+
+
SourceText
The exact command bytes that will be submitted.
UTF-8 text, literal newlines, and source control bytes that must never be emitted raw.
+
DisplayAtom
One indivisible thing the renderer may paint.
A printable ASCII byte, a safe control representation, a complete measured grapheme, a hard break, or a known-width placeholder.
+
Cells
Terminal screen columns.
Integers used only for layout and cursor placement.
+
ScreenBytes
The final output transaction.
Text, SGR styling, erasure, and cursor movement.
+
+
+
+
type ByteOffset int
+type Cells int
+
+type AtomKind int
+
+const (
+ AtomAscii AtomKind = iota
+ AtomControl
+ AtomGrapheme
+ AtomPlaceholder
+ AtomHardBreak
+)
+
+type DisplayAtom struct {
+ SourceStart ByteOffset
+ SourceEnd ByteOffset
+ Text string // source slice or generated safe display text
+ Width Cells // resolved before Layout
+ Kind AtomKind
+}
+
+type WidthCache struct {
+ Epoch uint64
+ Entries map[string]Cells
+}
+
+
+ Defined offset and cell types prevent the original bug class: adding a byte or rune index directly to a terminal column.
+ The cursor is always a byte offset at a grapheme boundary.
+
+
+ AtomAscii and AtomGrapheme use a source slice as Text.
+ AtomControl uses generated safe display text, and a placeholder uses one fixed printable ASCII character.
+ Both retain the original source range; neither emits those source bytes raw.
+
+
+
Reuse is part of the interface
+
+ Resolution and layout accept destination slices owned by the editor state and reuse their backing arrays across generations.
+ Cache keys are cloned only when inserted into the session cache, so a short measured cluster cannot retain an entire pasted command buffer.
+ The first implementation uses straightforward scalar loops; representation or scan optimizations require benchmark evidence.
+
+
+
The prompt is opaque
+
+ The prompt may contain SGR, OSC, arbitrary user code, or cursor movement.
+ Do not parse it into the command layout.
+ Print it, issue one fenced CPR, and treat the reported position as the editing region's origin.
+ The layout input is therefore either (printable command source, cursor, origin column, terminal width) or (resolved command atoms, cursor, origin column, terminal width), never prompt + command.
+
+
+ If the prompt anchor query fails, emit a real newline and start the editing region at column one.
+ That fallback changes presentation but preserves geometry.
+
+
+
+
+ 03
+
The width oracle has four answers
+
+
+
+
Input
Answer
Terminal work
+
+
+
Printable ASCII
One cell per byte.
None.
+
C0 control or DEL other than tab
Two-cell ASCII caret notation.
None.
+
Cached non-ASCII cluster
The previously observed one- or two-cell width.
None.
+
Unknown non-ASCII cluster
Measure it, validate it, then cache the exact byte string.
One DSR/CPR observation.
+
+
+
+
Printable-ASCII layout bypass
+
+ If every byte in the complete command buffer is in the printable ASCII range 0x20 through 0x7e, bypass atomization and width resolution entirely.
+ The ASCII layout function consumes SourceText directly: every byte occupies one cell, every byte boundary is a valid wrapping and cursor boundary, and rows can refer directly to slices of the original command.
+ The normal interactive command therefore needs one validation scan and no per-character atom construction.
+
+
+ A tab, newline, other control byte, DEL, or non-ASCII byte fails this classification and sends the complete command through the general atomizer.
+ That path maps tab to ▸ (U+25B8 BLACK RIGHT-POINTING SMALL TRIANGLE), represents other controls safely, and segments Unicode graphemes.
+ Its display atoms are indivisible; there is no exceptional atom type that layout may sometimes split.
+ Do not retain an independently scanned ASCII prefix when classification fails: a combining codepoint may join its final ASCII base, and the common path should not build general-path state speculatively.
+
+
+
Uniseg has one job
+
+ Use uniseg to identify extended grapheme-cluster boundaries and valid cursor gaps.
+ Ignore every width it reports.
+ After every edit, derive boundaries again; edits can change segmentation, particularly for regional indicators and joiner sequences.
+
+
+
Exact cache keys
+
+ Cache by the exact UTF-8 bytes of the complete cluster.
+ A measured family emoji says nothing about a different family emoji.
+ A measured VS16 sequence says nothing about every sequence containing VS16.
+ This refusal to generalize is what makes the cache an observation rather than another predictive model.
+
+
+
Why the two-cell bound matters
+
+ The one-or-two-cell contract removes the old oversized-cluster splitting design completely.
+ A complete non-ASCII grapheme atom always fits on an empty row when the terminal is at least two columns wide.
+ It also turns measurement validation into a closed check: a result outside one or two is unsupported, not another layout case.
+
+ The cache lives in memory and normally spans prompts.
+ It is tied to a width epoch: the period during which mshell owns and reasserts its terminal modes and assumes the terminal's width behavior is stable.
+ Start a new epoch after terminal reattachment, resume when state cannot be trusted, or any explicit recalibration command.
+ Resize alone does not change cluster width and does not clear the cache.
+
+
+ Do not write the cache to disk in the first implementation.
+ Correct invalidation across terminal upgrades, tmux attachments, font changes, and configuration is harder than remeasuring the small set of clusters a user actually types.
+
+
+
+
+ 04
+
Measure a batch in one transaction
+
+
+ Resolve every distinct cache miss in the newest editor state before layout.
+ A paste containing repeated clusters measures each distinct byte string once.
+ All probes are emitted in one buffered write and all replies are then collected by the normal terminal-input lexer.
+
+
+
+ Resolution is resource-bounded.
+ Measure at most a fixed number of distinct misses in one editor generation and cap both cache entries and candidate byte length.
+ Overflow candidates use the known-width placeholder; they are not allowed to create an unbounded query burst or cache from an adversarial paste.
+ A first implementation can use conservative constants such as 256 probes per generation and 4096 cache entries.
+
+
+
+
server / msh
+
+ probe A + CPRprobe B + CPRprobe C + CPRwait
+
+
terminal
+
+ A replyB replyC reply
+
+
+
+
+ There is one CPR response per distinct unknown cluster, but the requests are not serialized.
+ On SSH or ConPTY the batch can travel as roughly one additional transport round trip, independent of the number of misses in that batch.
+ Ordinary cached or printable-ASCII edits pay no round trip.
+
+
+
Probe from a known cell
+
+ The renderer provides one owned scratch row outside the visible edit contents.
+ Each probe begins at column one, clears pending wrap with CR, writes a known one-cell ASCII sentinel, writes exactly one complete candidate cluster, and requests CPR.
+
+
+
for cluster in misses:
+ write(CR + EL + "A" + cluster + CPR)
+ write(CR + EL)
+
+// Column numbers are one-based.
+measuredWidth = replyColumn - 2
+require replyRow == scratchRow
+require measuredWidth == 1 || measuredWidth == 2
+
+
+ The sentinel forces a clean grapheme boundary for ordinary candidates and makes the arithmetic independent of the cursor position before the scratch transaction.
+ Raw newlines, tabs, C0/C1 controls, and ESC never enter this protocol.
+ The generated tab glyph ▸ may enter as an ordinary measurement candidate.
+ A degenerate cluster that measures zero or modifies the sentinel is not self-contained and is rendered with the failure placeholder instead.
+
+
+ This sentinel protocol requires at least four terminal columns: three cells for the sentinel plus a two-cell candidate, and one further column so the probe never ends in pending wrap.
+ At widths two or three, already cached atoms may still be laid out, but new non-ASCII atoms use placeholders until the terminal is wider.
+
+
+
Scratch-row ownership
+
+ Do not use terminal save/restore-cursor state.
+ Move relative to the renderer's known region, create or reuse a row immediately below the previous editing region, and account for a scroll if creating that row reaches the bottom of the screen.
+ Clear the scratch row before leaving it.
+ After replies are validated, repaint the editing region from its tracked origin.
+
+
+ The initial prompt can perform capability validation before showing editable text.
+ Later cache misses temporarily leave the previous correct frame visible; mshell does not optimistically paint a layout with guessed widths.
+
+
+
Replies share the keyboard stream
+
+ While a batch is outstanding, the lexer routes exactly the expected number of valid CPR tokens to the resolver and queues ordinary input tokens.
+ A generation number associates a requested layout with the editor state that requested it.
+ If more input arrives before resolution completes, keep the valid cache entries, discard the stale layout, apply queued input, and resolve the newest state.
+
One observation, batched with all other misses in the state.
+
Repaint or cursor motion
None.
+
Resize with possible terminal reflow
One re-anchor query; cached widths remain valid.
+
+
+
+
+
+ 05
+
Two pure layout paths
+
+
+ Printable ASCII layout is direct byte and cell arithmetic.
+ General layout begins only after every display atom has a known width, then uses integer arithmetic over indivisible atoms.
+ Both functions are deterministic, allocation-conscious, and incapable of querying the terminal.
+
The printable-ASCII path may wrap at any byte boundary because each source byte is exactly one cell.
+
Every general-path text atom is a complete grapheme cluster, printable ASCII byte, generated control representation, or placeholder with a resolved width.
+
No general-path atom is split across rows. With the supported-terminal contract and terminal width at least two, every first-implementation atom fits on an empty row.
+
The first row begins at the measured prompt column; subsequent soft-wrapped rows begin at column one.
+
Hard newlines are source atoms with explicit row semantics and zero printable width.
+
The cursor may land at any byte boundary on the printable-ASCII path and only at source grapheme boundaries on the general path; either maps to exactly one row and cell gap.
+
Every row width is at most the available terminal width.
+
For printable-ASCII input, both paths produce identical rows and cursor placement.
+
+
+
Soft wrap remains terminal-owned
+
+ The renderer does not insert CRLF at calculated soft-wrap boundaries.
+ It streams the row contents and lets terminal autowrap create soft breaks, preserving selection rejoin and resize reflow.
+ Hard newlines in the source remain hard breaks.
+
+
+ Filling the final column arms the terminal's hidden pending-wrap state.
+ The layout records that state explicitly.
+ When another row follows, printing its first real atom forces the wrap; CR cancels pending wrap when returning to the region origin.
+
+
+
Full-region repaint first
+
+ Start with a whole-region repaint assembled into one output buffer.
+ Home to the known origin with relative movement, paint source slices and fixed placeholders with natural autowrap, erase renderer-owned leftovers, and park the cursor from the layout result.
+ A later desired-versus-actual diff renderer can consume the same layout without changing width resolution.
+
+
+
Styling is orthogonal
+
+ Syntax highlighting produces style spans over source byte offsets.
+ The painter intersects those spans with layout atoms and emits SGR only while painting.
+ SGR bytes have no width and never appear in SourceText, segmentation, the width cache, or layout arithmetic.
+
+
+
+ The more detailed paint-domain vocabulary and region-rendering figures remain in
+ render-pipeline.html.
+ Where that document refers to predictive cluster/codepoint width parameters, this document now supersedes it.
+
+
+
+
+ 06
+
Correct failure beats plausible drift
+
+
+ If mshell cannot establish a width, it must not silently substitute a Unicode width prediction and continue as though the geometry were exact.
+
+
+
+
+
Failure
Required behavior
+
+
+
No CPR support
Continue exact ASCII editing; display unresolved non-ASCII atoms as a fixed one-cell ASCII placeholder.
+
Query timeout
Cancel the batch, restore the owned terminal region, and use placeholders for unresolved atoms.
+
Width outside 1–2
Treat the terminal contract as violated for that atom; use a placeholder and log the observed reply.
+
Wrong row or malformed reply
Reject the complete batch. Never populate the cache from ambiguous measurements.
+
Probe or cache resource limit
Use placeholders for overflow candidates; never issue an unbounded batch.
+
Terminal only two or three cells wide
Do not run the sentinel protocol; cached atoms remain usable and unknown atoms use placeholders.
+
Terminal narrower than two cells
Do not paint interactive contents; continue accepting editing keys or report that the terminal is too narrow.
+
Untrusted state after resume/reattach
Start a new cache epoch, re-anchor, and measure future misses again.
+
+
+
+
+ The placeholder changes display text but not source text.
+ Submission still uses the original UTF-8 bytes.
+ A placeholder atom retains the source start/end offsets of the cluster it represents, so cursor movement and editing remain structurally correct.
+
+
+
Raw controls are never painted
+
+ Raw terminal controls cannot be allowed into the paint stream.
+ They remain in SourceText because editing and submission operate on the exact command.
+ Resolution turns C0 controls and DEL into AtomControl values using two-cell ASCII caret notation: NUL is ^@, Ctrl-Z is ^Z, ESC is ^[, and DEL is ^?.
+ Tab is the exception: it paints as ▸, whose exact one-or-two-cell width is measured and cached for the terminal epoch.
+ C1 controls use a fixed printable ASCII fallback in the first implementation; their source bytes are still preserved.
+ Literal newlines are explicit hard-break atoms.
+ The painter must never copy an AtomControl's source slice to the terminal device.
+
+
+
What correctness means here
+
+
mshell never calls a Unicode table's width estimate an observed terminal width.
+
Every painted non-ASCII cluster has either an exact cache entry from this terminal epoch or a known-width ASCII placeholder.
+
Every layout is computed from the same widths the painter will emit.
+
A failed or stale measurement can delay or degrade one frame, but cannot corrupt future row bookkeeping.
+
Terminal-specific behavior is contained in one resolver instead of leaking through layout and rendering.
+
+
+
+
+ 07
+
Be ruthless with the current branch
+
+
+ The existing Phase 1 work proved useful layout invariants, but its width abstraction is now the wrong abstraction.
+ Preserve tests and row semantics where they still express terminal geometry; remove the predictive model rather than adapting it.
+
+
+
+
+
Current element
Decision
Replacement
+
+
+
WidthParams
Delete
WidthCache plus resolved DisplayAtom.Width.
+
widthByCluster
Delete
Exact DSR/CPR measurement.
+
widthByCodepoints
Delete
No codepoint-width model.
+
Vs16Wide / ambiguous-width parameters
Delete
The exact cluster cache needs no category parameters.
+
uniseg width calls
Delete
Use uniseg boundaries only.
+
LayoutRow and row-end types
Keep and adapt
Consume atoms and the measured prompt-origin column.
+
Property-test structure
Keep
Inject deterministic width maps instead of parameter combinations.
+
Oversize codepoint-model splitting
Delete
The 1–2 cell grapheme contract and minimum two-column terminal remove the case.
+
Opaque prompt measurement
Keep
One fenced anchor CPR per prompt.
+
Region-relative renderer
Keep
Full repaint driven by resolved layout.
+
+
+
+
Implementation order
+
+
Replace width parameters with an injected exact width lookup in layout tests.
+
Add the scalar printable-ASCII classifier and direct ASCII layout path.
+
Introduce reusable DisplayAtom slices and whole-buffer general atomization for every input that fails printable-ASCII classification.
+
Convert general layout to accept indivisible atoms and both layout paths to accept a measured prompt-origin column.
+
Build the batch measurement protocol against a fake terminal and PTY tests.
+
Add the session cache, generation handling, timeouts, and placeholder fallback.
+
Swap the interactive renderer to region-relative repainting.
+
Add terminal-mode lifecycle and platform backends without changing the width/layout interfaces.
+
Run the real-terminal matrix and only then remove the old renderer.
+
+
+
+ ai/interactive-wrapping-plan.md and ai/interactive-wrapping-progress.md describe the superseded predictive-width implementation sequence.
+ They must be rewritten before implementation resumes; they are historical notes until then.
+
+
+
+
+ 08
+
Test the boundaries, not a model
+
+
Pure unit and property tests
+
+
Printable-ASCII classification accepts exactly bytes 0x20 through 0x7e; tab, newline, other controls, DEL, and every non-ASCII byte select the general path.
+
Direct printable-ASCII layout creates no display atoms and produces the same rows and cursor placement as the general atom path for printable-ASCII inputs.
+
Every C0 control and DEL retains its byte range, resolves to the correct safe representation, and can never place the raw control byte in ScreenBytes.
+
Tab paints as ▸; cache hit, exact measurement, timeout fallback, and one- versus two-cell observations all preserve its source offsets.
+
At a mixed ASCII/non-ASCII boundary, classify without constructing partial atoms and segment the complete buffer so combining marks can join the final ASCII base.
+
Segment every stress string and verify atoms tile source bytes exactly.
+
Inject arbitrary width maps containing only one and two; never call a terminal from layout tests.
+
For every grapheme-boundary cursor position and terminal width at least two, rows stay within bounds and the cursor lands on a valid cell gap.
+
Hard breaks, exact soft fills, early soft wraps, trailing newlines, and first-row prompt offsets retain explicit invariants.
+
Cache deduplication measures each exact cluster once per epoch.
+
Probe, cache-entry, and candidate-length limits always produce placeholders rather than unbounded work.
+
Editing that changes segmentation re-derives boundaries and does not reuse a cache entry for a different byte string.
+
Resolution and layout reuse caller-provided backing arrays without retaining stale atoms or rows.
+
+
+
Performance benchmarks
+
+
Benchmark short all-ASCII commands representative of interactive typing, not only large throughput inputs.
+
Benchmark mixed input whose first non-ASCII byte occurs near the beginning and near the end.
+
Benchmark large ASCII and Unicode pastes with allocation counts and bytes per operation reported.
+
Use the scalar printable-ASCII classifier as the baseline; compare word-at-a-time or SIMD validation against it before increasing complexity.
+
Keep the direct ASCII path allocation-free apart from caller-owned row capacity; optimize the general path independently when profiles justify it.
+
+
+
Protocol tests with a fake terminal
+
+
Several probes are written before any reply is required.
+
Replies populate the correct cache keys even when keypress tokens are interleaved.
+
Timeout, malformed reply, wrong row, width zero, and width greater than two leave the cache unchanged.
+
A stale generation retains valid measurements but never paints its stale layout.
+
The scratch row is cleared and terminal mode is restored on every exit path.
+
+
+
Real-terminal matrix
+
+ Run the existing tests/terminal_tests/width_report.py and the new DSR/CPR latency harness in each target environment.
+ Then exercise the actual editor through a PTY with ASCII, combining text, CJK, VS15/VS16, flags, skin tones, keycaps, ZWJ sequences, paste, resize, tmux, SSH, ConPTY, and a terminal that ignores CPR.
+
+
+
Rejected designs
+
+
+
Approach
Reason
+
+
+
Infer a terminal-wide width profile from representative probes.
Fewer queries, but correctness depends on unproved equivalence classes.
+
Use uniseg's cluster width after segmentation.
It describes uniseg's tables, not the current terminal.
+
Measure every occurrence.
Correct but needlessly repeats I/O; exact session caching is simpler.
+
Serialize one probe and wait before sending the next.
Turns a paste into many network round trips.
+
Optimistically render predicted widths, then repair.
Allows a wrong wrap to corrupt the frame and complicates recovery.
+
Persist measurements by terminal identity.
Invalidation is harder than remeasurement and silently risks stale geometry.
+
Split an oversized grapheme into codepoints.
Violates the complete-cluster rendering invariant.
wraptest — pending-wrap behavior across terminals.
+
+
+
+
+
+
+
diff --git a/ai/interactive-wrapping-plan.md b/ai/interactive-wrapping-plan.md
new file mode 100644
index 0000000..7a49058
--- /dev/null
+++ b/ai/interactive-wrapping-plan.md
@@ -0,0 +1,35 @@
+# Interactive wrapping — implementation plan
+Design doc of record: `ai/interactive-wrapping-design.html` (all §09 questions resolved 2026-08-15).
+
+## Decisions locked (2026-08-15)
+
+- Q1: calibration probe ships in v1 (daily driver is konsole, the cluster-but-silent class), built as the *last* phase, after the editor is trusted.
+- Q2: the `⏎` partial-line mark is derived from the column field of the per-prompt CPR; skipped on timeout; last-column-with-pending treated as mid-line.
+- Q3: ISIG stays off; `^C` remains an in-band byte.
+- Q4: control bytes never enter the buffer (paste maps TAB → one space, keeps `\n`, strips other C0); display stand-ins TAB → `▸` U+25B8 (1 cell), other controls `^X` (2 cells); `clusterWidth` stays total.
+- Q5: multiline editing in v1; Alt-Enter inserts a literal newline; pasted newlines are kept; Enter submits.
+- Q6: tmux gets no special-casing — same ladder as any terminal.
+- Dependency `github.com/rivo/uniseg` approved (added at v0.4.7).
+- Windows: full support in v1 — one renderer, two mode backends (termios and console-mode), per design §08.
+- Lone-ESC disambiguation: out of scope for this work.
+
+## Phases
+
+1. **Layout + width (pure functions, no behavior change).**
+ New files `mshell/Width.go`, `mshell/Layout.go` and tests.
+ `WidthParams`/`clusterWidth` per design §05; `Layout(prompt, command, cursor, width, widthFunc)` → rows + cursor(row, col) + pending-wrap flag.
+ Property tests: row widths never exceed terminal width, concatenated rows reconstruct the input, cursor always lands in bounds.
+2. **Renderer swap.**
+ `Render()` repaints the region from the layout model; `ScrollDown`/`ClearScreen`/`ensurePromptNewline` become bookkeeping off the per-prompt CPR; render-path CPRs deleted (with them, the `getCurrentPos` re-entrancy).
+ Delete the dead `StdinReader`/pause-channel code.
+3. **Mode overhaul.**
+ Replace `MakeRaw` with the hand-picked edit mode (§07 table, ONLCR on); delete the cooked/raw prompt dance; donate/steal lifecycle around command execution; bracketed paste with Q4/Q5 sanitization; Alt-Enter binding.
+ Windows console-mode backend implemented here (same interface).
+4. **Fenced query.**
+ One batched write per prompt: CPR (+ DECRQM 2027 / XTVERSION when needed), DA1 fence, timeout; width-model selection ladder (env vars → 2027 → legacy fallback).
+5. **Calibration.**
+ Probe per design §05, on-disk cache keyed by terminal identity, `msh recalibrate`.
+
+## Status
+
+- Phase 1: in progress (2026-08-15).
diff --git a/ai/interactive-wrapping-progress.md b/ai/interactive-wrapping-progress.md
new file mode 100644
index 0000000..0fcdce9
--- /dev/null
+++ b/ai/interactive-wrapping-progress.md
@@ -0,0 +1,104 @@
+# Interactive wrapping — chunk-by-chunk progress
+
+Workflow: Claude proposes each small chunk in chat; Mitchell implements it by hand.
+Claude does NOT write code to the repo. Phases per `ai/interactive-wrapping-plan.md`;
+width semantics per design §05 in `ai/interactive-wrapping-design.html`.
+
+## Phase 1: Width + Layout (pure functions)
+
+- [x] Chunk 1 — `WidthParams`, `clusterWidth` skeleton: ASCII fast path, TAB
+ stand-in (1), control stand-in (2); school branches stubbed.
+ DONE 2026-08-15: implemented in `Main.go` (Mitchell prefers fewer files),
+ tests deferred until schools are in.
+- [ ] Chunk 2 — legacy wcwidth school → `widthByCodepoints`: feed uniseg one
+ rune at a time ([4]byte+EncodeRune, alloc-free) and sum — probe-verified
+ this reproduces the per-codepoint table (family [2 0 2 0 2]=6, ❤️=1,
+ ZWJ/VS/combining=0). Sync `uniseg.EastAsianAmbiguousWidth` global from
+ params in-function. Adds `rivo/uniseg@v0.4.7`. PROPOSED.
+- [ ] Chunk 3 — cluster school → `widthByCluster`: uniseg's whole-cluster width
+ directly (probe-verified: family=2, VS16=2, VS15=1, flag pair=2, amb via
+ global); adjustment when Vs16Wide=false → strip U+FE0F, re-measure.
+ (Chunk 2 DONE 2026-08-15: in Main.go + Main_test.go, test passing;
+ leftover nits: uniseg misfiled as indirect in go.mod, "ambigous" typo,
+ stale "chunks 2 and 3" stub comment.)
+- [ ] Chunk 4 — segmentation: `clusters(s, p) iter.Seq2[string,int]` (Go 1.25
+ range-over-func, uniseg state threaded across boundaries) + `stringWidth`;
+ tests incl. flag-pair segmentation.
+ DONE 2026-08-15 (raw loop, no iterator; ZwjJoins→WidthOnClusters rename
+ agreed but NOT yet applied in code).
+ (Chunk 3 DONE 2026-08-15: widthByCluster in Main.go, tests pass; per-call
+ EastAsianAmbiguousWidth store kept — measured ~25ns, accepted.)
+- [x] Chunk 5 — layout: `layoutInto(dst, text, cursor, width, p) LayoutResult`;
+ text = prompt+command pre-concatenated, cursor = byte offset, rows are
+ no-copy slices with RowEnd enum (Final/SoftExact/SoftEarly/Hard);
+ dst reuse for zero-alloc repaints; CursorCol==width only when PendingWrap.
+ DONE 2026-08-15: in Main.go with TestLayout. Open flag for phase 2:
+ prompts with ANSI colors need SGR-aware width. Chunk 2/3 nits (go.mod
+ indirect, typo, stale comment) confirmed cleaned up.
+- [x] Chunk 6 — RowEndHard: `\n`/`\r\n` clusters end the row (excluded from text
+ and width; cursor-on-newline = end of row; trailing `\n` → empty final
+ row). DONE 2026-08-15: in layoutInto with TestLayoutHardNewline.
+- [ ] Chunk 7 — property tests: seeded-rand generator over a stress alphabet
+ (ASCII/controls/newlines/CJK/ZWJ/VS16/flags/combining/ambiguous), all
+ param combos, cursor at every cluster boundary. Properties: rows tile the
+ input (Hard rows skip exactly "\n"/"\r\n"), Width==stringWidth and
+ ≤ terminal width, end types consistent (SoftEarly ⇒ next cluster wouldn't
+ fit), cursor in bounds, PendingWrap ⇔ final Width==width. Full code
+ provided 2026-08-16; implemented by Mitchell, caught the codepoint-model
+ family-emoji overflow (4 people = 8 cells) on first run. Invariant
+ corrections found: (a) CursorCol==width happens at end of ANY exactly-full
+ row (e.g. cursor on \n after a full row), not only PendingWrap — renderer
+ must key deferred-move on "end of full row"; (b) oversize-cluster rows —
+ handled by chunk 7.5 below, after which row.Width ≤ termWidth is
+ unconditional again.
+- [ ] Chunk 7.5 — oversize clusters + width-2 floor (decided 2026-08-16,
+ supersedes the earlier stand-in-filler idea): terminal width ≥ 2 is a hard
+ assumption (renderer bails below it: paint nothing, input still works;
+ layoutInto documents the precondition with a defensive clamp). A cluster
+ wider than the total width (codepoint model only) SPLITS at codepoint-
+ segment boundaries — segment = nonzero-width codepoint + zero-width
+ followers, so segments are 1–2 cells and always fit. Whole-cluster
+ placement stays the default; splitting engages only when cluster > total
+ width. Codepoint widths are additive, so the row-honesty property survives
+ rows that begin mid-cluster. Tests: drop checkWidth's single-cluster
+ exemption (bound unconditional), unit cases: family at width 2 → four
+ 2-cell rows; family fits whole at line end when next row can hold it.
+ LayoutRow.Text stays a no-copy subslice ALWAYS — stand-ins (▸, ^X) and
+ SGR are paint-time emissions, never string rewrites.
+ Em-dash override (found 2026-08-16): uniseg deliberately widths U+2E3A=3,
+ U+2E3B=4, but both are EAW Neutral and glibc wcwidth=1 — terminals
+ advance 1 cell. clusterWidth overrides both to 1 before model dispatch
+ (else the unit-width ≤ 2 bound and the width-2 floor break, and any
+ command containing ⸻ gets cursor drift at every width). Add ⸺/⸻ to the
+ property-test pieces. Calibration (Phase 5) can verify the assumption
+ per-terminal via probe.
+
+## Phase 2: Renderer swap — not started
+## Phase 3: Mode overhaul — not started
+## Phase 4: Fenced query — not started
+## Phase 5: Calibration — not started
+
+## Terminology
+
+Mitchell's rule: do NOT say "school" (design doc's term). Say "width model":
+cluster model (`widthByCluster`) vs codepoint model (`widthByCodepoints`).
+`WidthParams.ZwjJoins` renamed → `WidthOnClusters` (selects the model; the ZWJ
+probe is just how it gets measured). Chunk 4 = raw loop, no iterator.
+
+## Log
+
+- 2026-08-16: Phase 2 design pinned in `ai/render-pipeline.html` (figures):
+ opaque prompt measured by per-prompt CPR → startCol (no marker convention,
+ no escape parsing; stronger than bash/zsh/fish, which all assume net-zero
+ cursor movement); highlighting = style spans × layout rows intersected by
+ byte offset in the painter (reset at row end, re-apply at row start; ghost
+ joins layout text as a gray span); type vocabulary SourceText/PromptBytes/
+ ScreenBytes/Cells/ByteOffset (defined types, not aliases — typed ints kill
+ the promptLength+index bug class); cursor domain = grapheme-cluster gaps,
+ encoded as ByteOffset always at a boundary; boundaries are DERIVED per
+ keystroke, never stored (clusters unstable under edits: RI fusion) —
+ future chunk: currentCommand []rune → SourceText + ByteOffset.
+- 2026-08-15: workflow started; chunk 1 proposed.
+- 2026-08-15: chunk 1 reworked for a branch-minimal ASCII hot path (len-1 byte
+ dispatch, `isControlCluster` folded in); school fns named `widthByCluster` /
+ `widthByCodepoints` (Mitchell's naming).
diff --git a/ai/render-pipeline.html b/ai/render-pipeline.html
new file mode 100644
index 0000000..447ec45
--- /dev/null
+++ b/ai/render-pipeline.html
@@ -0,0 +1,724 @@
+
+
+
+
+
+mshell Render Pipeline
+
+
+
+
mshell Render Pipeline
+
From keystroke to painted prompt — today's column-math renderer, and where layoutInto takes over in Phase 2.
+ Interactive mode is one loop (InteractiveMode, Main.go:2820): read a token, apply it to state, repaint, repeat.
+ Raw bytes from the terminal are buffered by StdinReaderState.ReadByte and parsed by InteractiveLexer,
+ which assembles escape sequences (arrows, Alt-chords, bracketed paste) into a single TerminalToken.
+ HandleToken is the only mutator: it edits currentCommand []rune and the cursor
+ index (a rune index), plus history and tab-completion state. Nothing in this stage touches the screen.
+
+
+
+
+ Fig 1.
+ The event loop. Editing and painting are already cleanly separated — Phase 2 replaces only the rightmost box's internals.
+
+
+
+
+
+
+
Stage 2 · Render today
+
Column math, one row
+
+ Render (Main.go:1680) rebuilds the whole visible command every keystroke into renderBuffer:
+ jump to the column after the prompt (\033[{promptLength+1}G), clear to end of line, re-lex
+ currentCommand and emit each token wrapped in SGR color codes, append the gray history ghost,
+ draw tab-completion rows below, then park the cursor.
+
+
+ The geometry lives in two expressions, and both encode the same pair of assumptions:
+
+ index is a rune count used as a column offset — so one rune = one cell — and
+ \033[G is an absolute column move on the current row — so the prompt is one row.
+ Wide CJK, emoji, and any command longer than the terminal width break both at once: the terminal auto-wraps
+ where it pleases, and the renderer's idea of the cursor drifts from the screen's.
+
+
+
+
+
+
Stage 3 · Render in Phase 2
+
layoutInto becomes the geometry source
+
+ The swap replaces arithmetic with a model. Render builds the layout text —
+ command + history ghost, rune index converted to a byte offset — and hands it to
+ layoutInto(dst, text, cursor, numCols, params). What comes back is the complete screen geometry:
+ rows that tile the input, an end type per row, and the cursor as (CursorRow, CursorCol).
+ Painting is then a walk over rows — no guessing, no CPR round-trips mid-render.
+
+
+ The prompt itself is opaque: it may contain arbitrary ANSI escapes, so mshell never measures its bytes.
+ It is printed verbatim, and the per-prompt CPR that the design already sends (Q2, batched in the Phase 4
+ fenced query) reports where the cursor actually landed. That measured column seeds the first layout row
+ (a small startCol parameter on layoutInto). This is stronger than what bash
+ (\[ \] markers), zsh (%{ %}), or fish (escape-stripping parser) do — all three
+ assume net-zero cursor movement; the CPR measures it, so even a cursor-moving prompt yields a correct
+ anchor. Content a prompt paints elsewhere on screen (the corner-clock hack) goes stale in every shell alike.
+
+
+
+
+ Fig 2.
+ Only the geometry source changes. The rust box is the pair of assumptions being deleted; the solid green box
+ exists today (Phase 1, tested); the dashed green box is the Phase 2 work. First and last steps are shared.
+
+
+
+
+ Syntax highlighting survives unchanged in spirit but moves to emit time: layoutInto measures the
+ plain text, and the painter injects SGR escapes between clusters as it copies each row out
+ (Stage 5). Escape codes never enter width math — command-side by construction, prompt-side by never
+ measuring the prompt at all. The "SGR-aware width" open flag closes on both fronts.
+
+
+
+
+
+
Stage 4 · The representation rule
+
Representations, splitting, and the width-2 floor
+
+ The screen never shows raw buffer bytes — it shows a representation of each cluster, and the width layer
+ already measures that representation rather than the raw text:
+
+
+
+
+
cluster in buffer
painted as
cells
status
+
+
+
TAB
▸ (U+25B8)
1
in clusterWidth today
+
C0 control / DEL
^X
2
in clusterWidth today
+
+
+
+
+ A cluster wider than the whole terminal (codepoint model only: the family emoji sums to 8; adversarial ZWJ
+ chains are unbounded) is not replaced by a filler — it splits at codepoint-segment boundaries
+ and flows across rows, which is exactly what a codepoint-model terminal does with the raw bytes anyway.
+ A segment is a nonzero-width codepoint plus its zero-width followers (combining marks, ZWJ, selectors), so
+ every segment is 1 or 2 cells and no row ever starts with a zero-width mark. Whole-cluster placement stays
+ the default — a family at an ordinary line end wraps whole; splitting engages only when the cluster exceeds
+ the total width. Codepoint-model widths are additive across any byte range, so rows that begin
+ mid-cluster still satisfy the honesty property unchanged.
+
+
+ This works because of a hard floor: terminal width ≥ 2. Every placement unit — cluster
+ (≤ 2 in the cluster model) or segment (≤ 2 in the codepoint model) — fits by construction. The ≤ 2 bound
+ holds after one override in clusterWidth: uniseg deliberately widths U+2E3A/U+2E3B
+ (two-/three-em dash) as 3/4, but both are East Asian Neutral with wcwidth = 1, and terminals do
+ cell accounting with wcwidth-family tables — so our wrapper returns 1 for both, or any command containing
+ ⸻ would misdraw at every width. A calibration-probe candidate for Phase 5. Below width 2,
+ the renderer bails: paints nothing, input still works, a resize repaints. A one-column terminal is not a
+ supported surface; refusing it honestly beats rendering garbage into it. The guard lives in
+ Render; layoutInto documents width ≥ 2 as a precondition (defensive clamp, not a
+ panic, so a racing resize degrades). The buffer keeps the original bytes throughout — only painting changes,
+ so editing, history, and submission see the real text.
+
+
+
+
+ Fig 3.
+ The rule is a pure function of what both callers already know, so the layout needs no extra flags for the
+ painter — a cell counted is a cell drawn, by construction. If the two ever consult different rules, cursor
+ drift returns; keep them one function.
+
+
+
+
+ Concretely, the oversize branch at the placement site in layoutInto becomes: if
+ clusterWidth(cluster, p) > width, iterate the cluster's codepoint segments and place each
+ through the normal wrap logic (a small segment iterator next to clusterWidth); the painter walks
+ the same segments when emitting. With the width-2 floor, every placement unit fits, the property
+ test's overflow exemption disappears, and row.Width ≤ termWidth becomes unconditional.
+
+
+
+
+
+
Stage 5 · Highlighting
+
Rows × spans: two interval sets, one byte string
+
+ Today highlighting is the render loop: Render iterates tokens and prints each lexeme
+ wrapped in SGR codes. In Phase 2 the emission inverts: tokens become a style map — ordered
+ (startByte, endByte, style) spans that tile the text (the if/else chain at Main.go:1710 becomes a
+ pure tokenStyle(t)) — and the painter walks each layout row's clusters, looking the style up
+ by byte offset. Rows and spans are two independent interval sets over the same string; the painter is
+ their intersection.
+
+
+
+
+ Fig 4.
+ Rows and style spans are independent interval sets over one byte string. A token wrapping across rows needs no
+ special case — row 1's first clusters still land in the string span, so the painter re-applies its style.
+ Full \033[0m at every row end keeps SGR state from bleeding across row-advance bytes (and stops
+ background colors flood-filling on wrap); reset-and-reapply beats attribute diffing — a few bytes, always correct.
+
+
+
+
+
History ghost stops being an append-after hack: it joins the layout text with one gray
+ span, so it wraps correctly — which it can't today. The cursor clamp (cursor ≤ len(command))
+ already keeps the cursor out of it.
+
Underlines (command literal 4;34m, known builtins 4m) are just
+ spans; the style map doesn't care they came from BuiltInList rather than the token type.
+
Lexer-error path (unterminated input) degrades to one unstyled span — the painter doesn't
+ branch.
+
Stand-ins inherit the token's style automatically — lookup is by the cluster's original
+ byte offset — or take a deliberate override (dim/reverse) to read as "not the real glyph".
+
Testable alone:text → spans is pure, with the same tiling property as rows
+ (property 1's checker shape).
+
+
+
+
+
+
Stage 6 · The total picture
+
Four domains, typed at the boundaries
+
+ Everything above reduces to four kinds of data, and the render bugs this project exists to kill are all
+ domain-crossing bugs — the old cursor math added screen cells to a rune count
+ (promptLength + 1 + index, Main.go:1858). Go defined types (not aliases:
+ type SourceText = string checks nothing; type SourceText string makes cross-domain
+ assignment a compile error) let the signatures enforce the picture. Slicing preserves a defined string type —
+ t[a:b] is still SourceText — so LayoutRow.Text stays a typed, no-copy
+ subslice for free; only library boundaries (uniseg) need an explicit string(t), contained inside
+ clusterWidth and the cluster walk.
+
+
type SourceText string // mshell source: command + ghost. Rows and spans slice it.
+type PromptBytes []byte // opaque escapes; printed verbatim, measured only by CPR
+type ScreenBytes []byte // renderBuffer: write-only output, never measured or re-read
+type Cells int // screen geometry: widths, columns, startCol, CursorCol
+type ByteOffset int // offset into SourceText: cursor, span and row bounds
+
+ The typed integers are the higher-value half: Cells and ByteOffset can never be added
+ to each other by accident, which retires the 1858-class bug at compile time. The typed strings then make the
+ flow itself readable: layoutInto(text SourceText, cursor ByteOffset, width Cells, p) → LayoutResult
+ says in its signature which domain every argument lives in.
+
+
+
+
+ Fig 5.
+ The functional flowchart, typed. Plain boxes carry SourceText/ByteOffset; green carries
+ Cells; dashed boxes are opaque byte streams that are never measured (prompt in, screen out).
+ SourceText flows top to bottom unchanged — stand-ins never rewrite it — while the geometry
+ (Cells) is derived beside it and joins it only inside the painter.
+
+
+
+
+ Reading the figure gives the answer to "when do the bytes change": never. SourceText
+ is immutable through the whole pipeline — rows and spans are views into it, and the only transformation
+ (stand-ins, SGR) happens in the painter's one-way emission into ScreenBytes, which nothing ever
+ reads back. The two dashed boxes are mirror images: prompt bytes enter unmeasured (the CPR measures their
+ effect), screen bytes leave unmeasured (the layout already predicted their effect).
+
+
+
+
+
+
Stage 7 · Consequences
+
What the swap deletes, what it must respect
+
+
+
Deleted with the swap
+
+
Both column-math expressions (promptLength+1, promptLength+1+index)
+
Render-path CPR queries and the getCurrentPos re-entrancy they cause
+
ScrollDown / prompt-row guesswork as measurement — becomes bookkeeping off the per-prompt CPR
+
The dead StdinReader / pause-channel code (per the Phase 2 plan)
+
+
+
+
+
Row-end semantics the painter must honor:SoftExact — the terminal's auto-wrap
+ already advanced; do not advance again. SoftEarly and Hard — reposition explicitly.
+ Final — stop, then place the cursor from (CursorRow, CursorCol); a cursor col equal to
+ the terminal width is the deferred-move state, and it can occur at the end of any exactly-full row,
+ not just under PendingWrap.
+
+
+
+ Resolved by design since first draft: ANSI-colored prompts (opaque prompt + per-prompt CPR → startCol,
+ Stage 3) and the history ghost (joins the layout text as a gray span, Stage 5). Still open for
+ Phase 2: the tab-completion block below the prompt (today's row accounting stays, anchored to the layout's
+ last row) and resize handling (reprint prompt, re-query CPR, re-layout). If right-aligned prompt content is ever
+ wanted, follow zsh/fish: offer it as a structured feature mshell draws itself, never via raw cursor movement.
+
+
+
+
+
diff --git a/mshell/Main.go b/mshell/Main.go
index 9a0bb1d..d5c0b65 100644
--- a/mshell/Main.go
+++ b/mshell/Main.go
@@ -25,6 +25,7 @@ import (
"path/filepath"
"time"
"unicode/utf8"
+ "github.com/rivo/uniseg"
)
type CliCommand int
@@ -947,6 +948,122 @@ type TermState struct {
// pathBinManager IPathBinManager
}
+type SourceText string
+type ByteOffset int
+type Cells int
+const UnresolvedWidth Cells = 0
+
+type AtomKind int
+
+const (
+ AtomAscii AtomKind = iota
+ AtomControl
+ AtomGrapheme
+ AtomPlaceholder
+ AtomHardBreak
+)
+
+type DisplayAtom struct {
+ SourceStart ByteOffset
+ SourceEnd ByteOffset
+ Text string
+ Width Cells
+ Kind AtomKind
+}
+
+type WidthCache struct {
+ Epoch uint64
+ Entries map[string]Cells
+}
+
+func controlCaretText(b byte) string {
+ if b == 0x7f {
+ return "^?"
+ }
+ return string([]byte{'^', b + 0x40})
+}
+
+type RowEnd int
+
+const (
+ RowEndFinal RowEnd = iota // Final row
+ RowEndSoftExact // Filled to exactly the perfect width
+ RowEndSoftEarly // We had a wide character at the end, at it didn't fit, so had to end early.
+ RowEndHard // A literal new line has caused us to move
+)
+
+type LayoutRow struct {
+ Text string // slice of the original input, no copy
+ Width int // total cells occupied
+ EndType RowEnd
+}
+
+type LayoutResult struct {
+ Rows []LayoutRow
+ CursorRow int
+ CursorCol int // [0 .. width] Equals width only when PendingWrap and cursor is at the end, a deferred move.
+ PendingWrap bool
+}
+
+func layoutInto(dst []LayoutRow, text string, cursor int, width int, widthOf func(string) int) LayoutResult {
+ if width < 1 {
+ width = 1
+ }
+ res := LayoutResult{}
+ rows := dst[:0] // Reuse the slice
+ rowStart := 0 // byte offset of the current row's first cluster
+ col := 0
+ offset := 0 // byte offset of the current cluster
+ state := -1
+ s := text
+ var cluster string
+ for len(s) > 0 {
+ cluster, s, _, state = uniseg.FirstGraphemeClusterInString(s, state)
+
+ if cluster == "\n" || cluster == "\r\n" { // Hard line break
+ if offset == cursor {
+ res.CursorRow = len(rows)
+ res.CursorCol = col
+ }
+
+ rows = append(rows, LayoutRow{Text: text[rowStart:offset], Width: col, EndType: RowEndHard })
+ offset += len(cluster)
+ rowStart = offset
+ col = 0
+ continue
+ }
+
+ w := widthOf(cluster)
+ if col + w > width && col > 0 { // wrap before this cluster
+ var end RowEnd
+ if col == width {
+ end = RowEndSoftExact
+ } else {
+ end = RowEndSoftEarly
+ }
+
+ rows = append(rows, LayoutRow{Text: text[rowStart:offset], Width: col, EndType: end})
+ rowStart = offset
+ col = 0
+ }
+ if offset == cursor {
+ res.CursorRow = len(rows)
+ res.CursorCol = col
+ }
+ col += w
+ offset += len(cluster)
+ }
+ rows = append(rows, LayoutRow{Text: text[rowStart:], Width: col, EndType: RowEndFinal})
+ if cursor >= len(text) {
+ res.CursorRow = len(rows) - 1
+ res.CursorCol = col
+ }
+
+ res.PendingWrap = col == width
+ res.Rows = rows
+ return res
+}
+
func newLogInstanceID() string {
return strconv.FormatInt(time.Now().UnixNano(), 10)
}
@@ -1283,6 +1400,7 @@ func (state *TermState) clearTabCompletionsDisplay() {
fmt.Fprintf(os.Stdout, "\033[%dG", state.promptLength+1+state.index)
}
+// TODO: Why is this necessary.
func (state *TermState) isTabToken(token TerminalToken) bool {
if t, ok := token.(AsciiToken); ok && t.Char == 9 {
return true
diff --git a/mshell/Main_test.go b/mshell/Main_test.go
index edfad35..cf1cd2a 100644
--- a/mshell/Main_test.go
+++ b/mshell/Main_test.go
@@ -1,10 +1,14 @@
package main
import (
+ "github.com/rivo/uniseg"
"os"
"path/filepath"
"reflect"
"testing"
+ "math/rand"
+ "strings"
+ "slices"
)
func TestHistory(t *testing.T) {
@@ -72,9 +76,9 @@ func TestBuildSharedCompletionInsertUsesBacktickForFilePrefixes(t *testing.T) {
func TestDefaultAppCommandFallsBackToPlatformDefault(t *testing.T) {
tests := []struct {
- goos string
- wantName string
- wantArgs []string
+ goos string
+ wantName string
+ wantArgs []string
}{
{goos: "linux", wantName: "xdg-open", wantArgs: []string{"/tmp/init.msh"}},
{goos: "darwin", wantName: "open", wantArgs: []string{"/tmp/init.msh"}},
@@ -190,3 +194,333 @@ func TestRunEditCommandUsesMSHINITOverride(t *testing.T) {
t.Fatalf("args = %v, want %v", gotArgs, wantArgs)
}
}
+
+func TestClusterSegmentation(t *testing.T) {
+ // Two flags must segment as two 2-rune clusters, not one 4-rune blob.
+ var got []string
+ s := "🇺🇸🇺🇸"
+ state := -1
+ var cl string
+ for len(s) > 0 {
+ cl, s, _, state = uniseg.FirstGraphemeClusterInString(s, state)
+ got = append(got, cl)
+ }
+ if len(got) != 2 || got[0] != "🇺🇸" || got[1] != "🇺🇸" {
+ t.Errorf("flag clusters = %q, want two flags", got)
+ }
+}
+
+func TestAsciiAtomsInto(t *testing.T) {
+ command := SourceText("a\t\x1a\n")
+ got, allAscii := asciiAtomsInto(nil, command)
+ if !allAscii {
+ t.Fatal("seven-bit source should use the Ascii atomizer")
+ }
+
+ want := []DisplayAtom{
+ {SourceStart: 0, SourceEnd: 1, Text: "a", Width: 1, Kind: AtomAscii},
+ {SourceStart: 1, SourceEnd: 2, Text: "\u25B8", Width: UnresolvedWidth, Kind: AtomControl},
+ {SourceStart: 2, SourceEnd: 3, Text: "^Z", Width: 2, Kind: AtomControl},
+ {SourceStart: 3, SourceEnd: 4, Text: "\n", Width: 0, Kind: AtomHardBreak},
+ }
+
+ if !slices.Equal(got, want) {
+ t.Errorf("atoms = %+v, want %+v", got, want)
+ }
+
+ got, allAscii = asciiAtomsInto(got, SourceText("a\u00e9"))
+ if allAscii || len(got) != 0 {
+ t.Errorf("non-ASCII input returned atoms=%+v, ok=%v", got, allAscii)
+ }
+}
+
+func testWidthLookup(widths map[string]int) func(string) int {
+ return func(cluster string) int {
+ if len(cluster) == 1 && cluster[0] >= 0x20 && cluster[0] <= 0x7e {
+ return 1
+ }
+
+ width, ok := widths[cluster]
+ if !ok {
+ panic("missing test width for cluster: " + cluster)
+ }
+ return width
+ }
+}
+
+func sumTestWidths(text string, widthOf func(string) int) int {
+ total := 0
+ state := -1
+
+ for len(text) > 0 {
+ cluster, rest, _, nextState := uniseg.FirstGraphemeClusterInString(text, state)
+ total += widthOf(cluster)
+ text = rest
+ state = nextState
+ }
+
+ return total
+}
+
+func TestLayout(t *testing.T) {
+ widthOf := testWidthLookup(map[string]int{"世": 2,})
+
+ // Exact fill: "abcdef" at width 3 → two full rows, pending wrap.
+ r := layoutInto(nil, "abcdef", 6, 3, widthOf)
+ if len(r.Rows) != 2 || r.Rows[0].Text != "abc" || r.Rows[1].Text != "def" {
+ t.Fatalf("rows = %+v", r.Rows)
+ }
+ if r.Rows[0].EndType != RowEndSoftExact || !r.PendingWrap {
+ t.Errorf("want SoftExact + PendingWrap, got %+v", r)
+ }
+ if r.CursorRow != 1 || r.CursorCol != 3 {
+ t.Errorf("cursor at end = (%d,%d), want (1,3)", r.CursorRow, r.CursorCol)
+ }
+
+ // Early wrap: 世 (2 cells) doesn't fit after "ab" at width 3.
+ r = layoutInto(nil, "ab世", 0, 3, widthOf)
+ if len(r.Rows) != 2 || r.Rows[0].Text != "ab" || r.Rows[0].EndType != RowEndSoftEarly {
+ t.Fatalf("early wrap rows = %+v", r.Rows)
+ }
+ if r.Rows[1].Text != "世" || r.Rows[1].Width != 2 || r.PendingWrap {
+ t.Errorf("second row = %+v", r.Rows[1])
+ }
+
+ // Cursor on the wrap boundary belongs to the new row.
+ r = layoutInto(nil, "abcd", 3, 3, widthOf)
+ if r.CursorRow != 1 || r.CursorCol != 0 {
+ t.Errorf("boundary cursor = (%d,%d), want (1,0)", r.CursorRow, r.CursorCol)
+ }
+
+ // Empty text: one empty row, cursor at origin.
+ r = layoutInto(nil, "", 0, 80, widthOf)
+ if len(r.Rows) != 1 || r.CursorRow != 0 || r.CursorCol != 0 || r.PendingWrap {
+ t.Errorf("empty = %+v", r)
+ }
+
+ // Reuse contract: second call must not grow a new backing array.
+ first := layoutInto(nil, "abcdef", 0, 3, widthOf)
+ second := layoutInto(first.Rows, "xyzuvw", 0, 3, widthOf)
+ if &first.Rows[0] != &second.Rows[0] {
+ t.Error("dst backing array was not reused")
+ }
+}
+
+func TestLayoutHardNewline(t *testing.T) {
+ widthOf := testWidthLookup(nil)
+
+ // Basic split; newline is in neither row's text nor width.
+ r := layoutInto(nil, "ab\ncd", 5, 80, widthOf)
+ if len(r.Rows) != 2 || r.Rows[0].Text != "ab" || r.Rows[1].Text != "cd" {
+ t.Fatalf("rows = %+v", r.Rows)
+ }
+ if r.Rows[0].EndType != RowEndHard || r.Rows[0].Width != 2 {
+ t.Errorf("first row = %+v", r.Rows[0])
+ }
+ if r.CursorRow != 1 || r.CursorCol != 2 {
+ t.Errorf("cursor at end = (%d,%d), want (1,2)", r.CursorRow, r.CursorCol)
+ }
+
+ // Cursor on the newline = end of first row; just after it = start of second.
+ if r := layoutInto(nil, "ab\ncd", 2, 80, widthOf); r.CursorRow != 0 || r.CursorCol != 2 {
+ t.Errorf("cursor on \\n = (%d,%d), want (0,2)", r.CursorRow, r.CursorCol)
+ }
+ if r := layoutInto(nil, "ab\ncd", 3, 80, widthOf); r.CursorRow != 1 || r.CursorCol != 0 {
+ t.Errorf("cursor after \\n = (%d,%d), want (1,0)", r.CursorRow, r.CursorCol)
+ }
+
+ // Trailing newline yields an empty final row, cursor lands on it.
+ r = layoutInto(nil, "ab\n", 3, 80, widthOf)
+ if len(r.Rows) != 2 || r.Rows[1].Text != "" || r.CursorRow != 1 || r.CursorCol != 0 {
+ t.Errorf("trailing newline = %+v", r)
+ }
+
+ // Consecutive newlines produce an empty middle row.
+ r = layoutInto(nil, "a\n\nb", 0, 80, widthOf)
+ if len(r.Rows) != 3 || r.Rows[1].Text != "" || r.Rows[1].EndType != RowEndHard {
+ t.Errorf("blank line = %+v", r.Rows)
+ }
+
+ // Hard break composes with soft wrapping.
+ r = layoutInto(nil, "abcd\nef", 0, 3, widthOf)
+ if len(r.Rows) != 3 || r.Rows[0].EndType != RowEndSoftExact ||
+ r.Rows[1].Text != "d" || r.Rows[1].EndType != RowEndHard || r.Rows[2].Text != "ef" {
+ t.Errorf("wrap+hard = %+v", r.Rows)
+ }
+}
+
+func getClusterBoundaries(text string) []int {
+ s := text
+ state := -1
+ offset := 0
+ var cl string
+ boundaries := make([]int, 0, len(text))
+ for len(s) > 0 {
+ boundaries = append(boundaries, offset)
+ cl, s, _, state = uniseg.FirstGraphemeClusterInString(s, state)
+ offset += len(cl)
+ }
+ boundaries = append(boundaries, len(text))
+
+ return boundaries
+}
+
+func TestLayoutProperties(t *testing.T) {
+ pieces := []string{
+ "a", "Z", " ", // printable ASCII
+ "\n", "\r\n", // hard breaks
+ "世", "界", // wide CJK
+ "é", "e\u0301", // precomposed vs combining
+ "👨👩👧👦", "❤️", "☂\uFE0F", "☂\uFE0E", // ZWJ family, VS16, VS15
+ "🇺🇸", "±", // flag pair, East Asian ambiguous
+ }
+ widths := []int{2, 3, 5, 10, 80}
+
+ widthMaps := []map[string]int{
+ {
+ "世": 2, "界": 2,
+ "é": 1, "e\u0301": 1,
+ "👨👩👧👦": 2, "❤️": 2,
+ "☂\uFE0F": 2, "☂\uFE0E": 1,
+ "🇺🇸": 2, "±": 1,
+ },
+ {
+ "世": 1, "界": 1,
+ "é": 2, "e\u0301": 2,
+ "👨👩👧👦": 1, "❤️": 1,
+ "☂\uFE0F": 1, "☂\uFE0E": 2,
+ "🇺🇸": 1, "±": 2,
+ },
+ }
+
+ rng := rand.New(rand.NewSource(1))
+
+ var dst []LayoutRow
+
+ for i := 0; i < 300; i++ {
+ var sb strings.Builder
+ for n := rng.Intn(31); n > 0; n-- {
+ sb.WriteString(pieces[rng.Intn(len(pieces))])
+ }
+ text := sb.String()
+
+ // Every cluster boundary is valid cursor position
+ boundaries := getClusterBoundaries(text)
+
+ for _, width := range widths {
+ for widthMapIndex, widthMap := range widthMaps {
+ widthOf := testWidthLookup(widthMap)
+ for _, cursor := range boundaries {
+ r := layoutInto(dst, text, cursor, width, widthOf)
+ dst = r.Rows
+ checkLayoutProperties(t, text, cursor, width, widthOf, r)
+ if t.Failed() {
+ t.Fatalf("input %q cursor=%d width=%d widthMap=%d", text, cursor, width, widthMapIndex)
+ }
+ }
+ }
+ }
+ }
+}
+
+func checkLayoutProperties(t *testing.T, text string, cursor int, terminalWidth int, widthOf func(string) int, r LayoutResult) {
+ t.Helper()
+
+ rows := r.Rows
+ if len(rows) == 0 {
+ t.Error("no rows")
+ return
+ }
+
+ // 1. Rows tile the input; a RowEndHard row skips exactly one newline cluster
+ offset := 0
+ for i, row := range rows {
+ if offset+len(row.Text) > len(text) || text[offset:offset+len(row.Text)] != row.Text {
+ t.Errorf("row %d text %q does not match input at offset %d", i, row.Text, offset)
+ return
+ }
+ offset += len(row.Text)
+
+ // The '\r\n' or '\n' is not part of the *Rows* text, so we need to handle moving the offset in that case.
+ if row.EndType == RowEndHard {
+ rest := text[offset:]
+ if strings.HasPrefix(rest, "\r\n") {
+ offset += 2
+ } else if strings.HasPrefix(rest, "\n") {
+ offset += 1
+ } else {
+ t.Errorf("row %d is RowEndHard but input at offset %d is not a newline", i, offset)
+ return
+ }
+ }
+ }
+
+ if offset != len(text) {
+ t.Errorf("rows cover %d bytes, but input is %d bytes", offset, len(text))
+ }
+
+ checkWidth := func(i int, row LayoutRow) {
+ if got := sumTestWidths(row.Text, widthOf); row.Width != got {
+ t.Errorf("row %d Width=%d but summed widths=%d", i, row.Width, got)
+ }
+ if row.Width > terminalWidth {
+ cl, rest, _, _ := uniseg.FirstGraphemeClusterInString(row.Text, -1)
+ if cl != row.Text || rest != "" {
+ t.Errorf("row %d Width=%d exceeds terminal width %d on a multi-cluster row %q", i, row.Width, terminalWidth, row.Text)
+ }
+ }
+ }
+
+ // 2 + 3 for every row but the last: never Final, soft rows justified by the next row.
+ for i, row := range rows[:len(rows)-1] {
+ checkWidth(i, row)
+ switch row.EndType {
+ case RowEndFinal:
+ t.Errorf("row %d of %d is RowEndFinal but not last", i, len(rows))
+ case RowEndSoftExact:
+ if row.Width != terminalWidth {
+ t.Errorf("row %d SoftExact but Width=%d, terminal width %d", i, row.Width, terminalWidth)
+ }
+ case RowEndSoftEarly:
+ if row.Width >= terminalWidth {
+ t.Errorf("row %d SoftEarly but Width=%d, terminal width %d", i, row.Width, terminalWidth)
+ }
+ }
+ if row.EndType == RowEndSoftExact || row.EndType == RowEndSoftEarly {
+ if row.Text == "" {
+ t.Errorf("row %d soft-wrapped but empty", i)
+ } else {
+ next, _, _, _ := uniseg.FirstGraphemeClusterInString(rows[i+1].Text, -1)
+ if w := widthOf(next); row.Width+w <= terminalWidth {
+ t.Errorf("row %d ended soft at width %d but next cluster %q (width %d) would have fit in %d",
+ i, row.Width, next, w, terminalWidth)
+ }
+ }
+ }
+ }
+
+ // The last row: same width rules, and it must be the one Final row.
+ lastRow := rows[len(rows)-1]
+ checkWidth(len(rows)-1, lastRow)
+ if lastRow.EndType != RowEndFinal {
+ t.Errorf("last row EndType=%d, want RowEndFinal", lastRow.EndType)
+ }
+
+ // 4. Cursor Checks
+ if r.CursorRow < 0 || r.CursorRow >= len(rows) {
+ t.Errorf("cursor row %d out of bounds [0,%d]", r.CursorRow, len(rows))
+ } else {
+ if r.CursorCol < 0 || r.CursorCol > rows[r.CursorRow].Width {
+ t.Errorf("cursor col %d out of bounds [0,%d] for row %d", r.CursorCol, rows[r.CursorRow].Width, r.CursorRow)
+ }
+
+ if r.CursorCol == terminalWidth && rows[r.CursorRow].Width != terminalWidth {
+ t.Errorf("cursor col %d at terminal width but row width is %d", r.CursorCol, rows[r.CursorRow].Width)
+ }
+ }
+
+ // 5. PendingWrap tracks the final row exactly filling the terminal
+ if r.PendingWrap != (rows[len(rows)-1].Width == terminalWidth) {
+ t.Errorf("PendingWrap = %v, but last row width = %d, terminal width = %d", r.PendingWrap, rows[len(rows)-1].Width, terminalWidth)
+ }
+}
diff --git a/mshell/go.mod b/mshell/go.mod
index 043c51d..423f939 100644
--- a/mshell/go.mod
+++ b/mshell/go.mod
@@ -5,6 +5,7 @@ go 1.25
require (
github.com/cespare/xxhash v1.1.0
github.com/creack/pty v1.1.24
+ github.com/rivo/uniseg v0.4.7
go.lsp.dev/protocol v0.12.0
golang.org/x/net v0.42.0
golang.org/x/sys v0.34.0
diff --git a/mshell/go.sum b/mshell/go.sum
index 66d4587..9602591 100644
--- a/mshell/go.sum
+++ b/mshell/go.sum
@@ -19,6 +19,8 @@ github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
+github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/segmentio/asm v1.1.3 h1:WM03sfUOENvvKexOLp+pCqgb/WDjsi7EK8gIsICtzhc=
github.com/segmentio/asm v1.1.3/go.mod h1:Ld3L4ZXGNcSLRg4JBsZ3//1+f/TjYl0Mzen/DQy1EJg=
github.com/segmentio/encoding v0.3.4 h1:WM4IBnxH8B9TakiM2QD5LyNl9JSndh88QbHqVC+Pauc=
diff --git a/tests/terminal_tests/cursor_query_latency.go b/tests/terminal_tests/cursor_query_latency.go
new file mode 100644
index 0000000..da0f57a
--- /dev/null
+++ b/tests/terminal_tests/cursor_query_latency.go
@@ -0,0 +1,223 @@
+// cursor_query_latency measures terminal DSR/CPR round-trip latency.
+//
+// Run this through cursor_query_latency.sh in the terminal emulator (and any
+// multiplexer or remote connection) that you want to measure. The interval
+// between requests is deliberately configurable because a burst of queries
+// may exercise different buffering behavior than an occasional shell query.
+package main
+
+import (
+ "errors"
+ "flag"
+ "fmt"
+ "math"
+ "os"
+ "sort"
+ "strings"
+ "time"
+
+ "golang.org/x/term"
+)
+
+const cursorPositionQuery = "\x1b[6n"
+
+type byteResult struct {
+ b byte
+ err error
+}
+
+type sample struct {
+ latency time.Duration
+ row int
+ col int
+}
+
+func main() {
+ count := flag.Int("count", 500, "number of measured queries")
+ warmup := flag.Int("warmup", 20, "number of unmeasured warm-up queries")
+ interval := flag.Duration("interval", 0, "delay between query replies and the next query")
+ timeout := flag.Duration("timeout", 500*time.Millisecond, "maximum time to wait for each reply")
+ raw := flag.Bool("raw", false, "print each measured sample as TSV after the summary")
+ flag.Parse()
+
+ if *count < 1 || *warmup < 0 || *interval < 0 || *timeout <= 0 {
+ fmt.Fprintln(os.Stderr, "count must be positive; warmup and interval non-negative; timeout positive")
+ os.Exit(2)
+ }
+ if !term.IsTerminal(int(os.Stdin.Fd())) || !term.IsTerminal(int(os.Stdout.Fd())) {
+ fmt.Fprintln(os.Stderr, "cursor_query_latency: stdin and stdout must both be the terminal under test")
+ os.Exit(2)
+ }
+
+ oldState, err := term.MakeRaw(int(os.Stdin.Fd()))
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "cursor_query_latency: enter raw mode: %v\n", err)
+ os.Exit(1)
+ }
+ restored := false
+ restore := func() {
+ if !restored {
+ _ = term.Restore(int(os.Stdin.Fd()), oldState)
+ restored = true
+ }
+ }
+ defer restore()
+
+ input := make(chan byteResult, 64)
+ go readBytes(os.Stdin, input)
+
+ all := make([]sample, 0, *count)
+ for i := 0; i < *warmup+*count; i++ {
+ if i > 0 && *interval > 0 {
+ time.Sleep(*interval)
+ }
+ start := time.Now()
+ if _, err := os.Stdout.WriteString(cursorPositionQuery); err != nil {
+ restore()
+ fmt.Fprintf(os.Stderr, "cursor_query_latency: write query %d: %v\n", i+1, err)
+ os.Exit(1)
+ }
+ row, col, err := readCursorPosition(input, *timeout)
+ elapsed := time.Since(start)
+ if err != nil {
+ restore()
+ fmt.Fprintf(os.Stderr, "cursor_query_latency: query %d: %v\n", i+1, err)
+ os.Exit(1)
+ }
+ if i >= *warmup {
+ all = append(all, sample{latency: elapsed, row: row, col: col})
+ }
+ }
+
+ restore()
+ printReport(all, *warmup, *interval, *timeout, *raw)
+}
+
+func readBytes(file *os.File, output chan<- byteResult) {
+ buf := make([]byte, 64)
+ for {
+ n, err := file.Read(buf)
+ for _, b := range buf[:n] {
+ output <- byteResult{b: b}
+ }
+ if err != nil {
+ output <- byteResult{err: err}
+ return
+ }
+ }
+}
+
+func readCursorPosition(input <-chan byteResult, timeout time.Duration) (int, int, error) {
+ timer := time.NewTimer(timeout)
+ defer timer.Stop()
+
+ var reply []byte
+ inReply := false
+ for {
+ select {
+ case result := <-input:
+ if result.err != nil {
+ return 0, 0, result.err
+ }
+ if result.b == 3 {
+ return 0, 0, errors.New("interrupted")
+ }
+
+ if !inReply {
+ if result.b == 0x1b {
+ reply = append(reply[:0], result.b)
+ inReply = true
+ }
+ continue
+ }
+
+ reply = append(reply, result.b)
+ if len(reply) == 2 && result.b != '[' {
+ inReply = false
+ continue
+ }
+ if result.b == 'R' {
+ var row, col int
+ if _, err := fmt.Sscanf(string(reply), "\x1b[%d;%dR", &row, &col); err != nil || row < 1 || col < 1 {
+ return 0, 0, fmt.Errorf("invalid CPR reply %q", reply)
+ }
+ return row, col, nil
+ }
+ if len(reply) > 32 {
+ return 0, 0, fmt.Errorf("overlong CPR reply %q", reply)
+ }
+ case <-timer.C:
+ return 0, 0, fmt.Errorf("no CPR reply within %s", timeout)
+ }
+ }
+}
+
+func printReport(samples []sample, warmup int, interval, timeout time.Duration, raw bool) {
+ durations := make([]time.Duration, len(samples))
+ var total time.Duration
+ for i, s := range samples {
+ durations[i] = s.latency
+ total += s.latency
+ }
+ sort.Slice(durations, func(i, j int) bool { return durations[i] < durations[j] })
+
+ mean := time.Duration(int64(total) / int64(len(durations)))
+ var squaredSeconds float64
+ for _, d := range durations {
+ delta := d.Seconds() - mean.Seconds()
+ squaredSeconds += delta * delta
+ }
+ stddev := time.Duration(math.Sqrt(squaredSeconds/float64(len(durations))) * float64(time.Second))
+
+ fmt.Println("terminal cursor-position query latency")
+ fmt.Printf(" query: DSR CPR ESC[6n samples: %d warmup: %d\n", len(samples), warmup)
+ fmt.Printf(" interval: %s timeout: %s\n", interval, timeout)
+ fmt.Printf(" TERM=%s TERM_PROGRAM=%s COLORTERM=%s\n",
+ envOrDash("TERM"), envOrDash("TERM_PROGRAM"), envOrDash("COLORTERM"))
+ fmt.Printf(" shell=%s tmux=%s ssh=%s\n",
+ envOrDash("SHELL"), yesNoEnv("TMUX"), yesNoEnv("SSH_CONNECTION"))
+ fmt.Println()
+ fmt.Printf(" min %9s p50 %9s p95 %9s p99 %9s max %9s\n",
+ compactDuration(durations[0]), compactDuration(percentile(durations, 0.50)),
+ compactDuration(percentile(durations, 0.95)), compactDuration(percentile(durations, 0.99)),
+ compactDuration(durations[len(durations)-1]))
+ fmt.Printf(" mean %8s stddev %s\n", compactDuration(mean), compactDuration(stddev))
+
+ if raw {
+ fmt.Println()
+ fmt.Println("sample\tlatency_ns\trow\tcol")
+ for i, s := range samples {
+ fmt.Printf("%d\t%d\t%d\t%d\n", i+1, s.latency.Nanoseconds(), s.row, s.col)
+ }
+ }
+}
+
+func percentile(sorted []time.Duration, fraction float64) time.Duration {
+ index := int(math.Ceil(fraction*float64(len(sorted)))) - 1
+ if index < 0 {
+ index = 0
+ }
+ return sorted[index]
+}
+
+func compactDuration(d time.Duration) string {
+ if d < time.Millisecond {
+ return fmt.Sprintf("%.1fµs", float64(d)/float64(time.Microsecond))
+ }
+ return fmt.Sprintf("%.3fms", float64(d)/float64(time.Millisecond))
+}
+
+func envOrDash(name string) string {
+ value := strings.TrimSpace(os.Getenv(name))
+ if value == "" {
+ return "-"
+ }
+ return value
+}
+
+func yesNoEnv(name string) string {
+ if os.Getenv(name) == "" {
+ return "no"
+ }
+ return "yes"
+}
diff --git a/tests/terminal_tests/cursor_query_latency.sh b/tests/terminal_tests/cursor_query_latency.sh
new file mode 100755
index 0000000..1c75421
--- /dev/null
+++ b/tests/terminal_tests/cursor_query_latency.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+set -eu
+
+# Build from the repository module so this standalone experiment uses the same
+# Go version and x/term dependency as mshell itself. The timed region is inside
+# the binary, so compilation and process startup are never part of a sample.
+script_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
+repo_dir=$(CDPATH='' cd -- "$script_dir/../.." && pwd)
+build_dir=$(mktemp -d "${TMPDIR:-/tmp}/mshell-cpr-latency.XXXXXX")
+
+cleanup() {
+ rm -rf -- "$build_dir"
+}
+trap cleanup EXIT HUP INT TERM
+
+cd "$repo_dir/mshell"
+go build -o "$build_dir/cursor-query-latency" ../tests/terminal_tests/cursor_query_latency.go
+
+if [ "$#" -gt 0 ]; then
+ "$build_dir/cursor-query-latency" "$@"
+ exit
+fi
+
+echo "Burst profile (maximum query throughput)"
+"$build_dir/cursor-query-latency" -count 1000 -warmup 50 -interval 0
+echo
+echo "Paced profile (5 ms between replies and queries)"
+"$build_dir/cursor-query-latency" -count 500 -warmup 20 -interval 5ms
+echo
+echo "Idle profile (100 ms between replies and queries)"
+"$build_dir/cursor-query-latency" -count 100 -warmup 10 -interval 100ms
diff --git a/tests/terminal_tests/width_report.py b/tests/terminal_tests/width_report.py
new file mode 100755
index 0000000..dc90bd3
--- /dev/null
+++ b/tests/terminal_tests/width_report.py
@@ -0,0 +1,241 @@
+#!/usr/bin/env python3
+"""Terminal width stress report.
+
+Probes the *terminal emulator* (not mshell) with normal and adversarial
+grapheme clusters, measures how many cells the cursor actually advanced for
+each via CPR (ESC[6n), and prints a report:
+
+ - inferred width model (cluster vs codepoint/legacy), VS16 behavior,
+ East Asian ambiguous width
+ - per-probe: how it renders, measured advance, and the cluster-model and
+ codepoint-model predictions
+ - the maximum advance ever observed for a single grapheme cluster, i.e.
+ an empirical answer to "does this terminal ever move more than 2 cells
+ for one cluster?"
+
+Run it directly in the terminal you want to measure:
+
+ ./width_report.py
+
+Probes are erased as they are measured (CR + EL); the report prints after
+the terminal is restored. The report's "renders" column is padded using the
+*measured* cell width, so columns line up on the terminal under test.
+Requires a tty; exits with a message otherwise.
+"""
+
+import os
+import re
+import select
+import sys
+import termios
+import tty
+
+CPR = "\x1b[6n"
+TIMEOUT = 0.30 # seconds to wait for each CPR reply
+RENDER_COL = 12 # cells reserved for the "renders" column
+
+ZWJ = ""
+VS15 = "︎"
+VS16 = "️"
+
+
+def build_probes():
+ """Each probe: (group, label, string, cluster_pred, codepoint_pred).
+
+ Labels are ASCII-only (the report aligns them with plain padding).
+ Predictions are cells for the WHOLE string. None = no firm prediction
+ (that's what we're here to find out).
+ """
+ person = "\U0001f9d1" # neutral person
+ family3 = "\U0001f468\U0001f469\U0001f466" # man+woman+boy
+ family4 = "\U0001f468\U0001f469\U0001f467\U0001f466" # man+woman+girl+boy
+ probes = [
+ ("baseline", "ASCII 'a'", "a", 1, 1),
+ ("baseline", "wide CJK", "世", 2, 2),
+ ("baseline", "precomposed e-acute", "é", 1, 1),
+ ("baseline", "e + combining acute", "e\u0301", 1, 1),
+
+ ("class", "family (3, RGI)", family3, 2, 6),
+ ("class", "family (4, RGI)", family4, 2, 8),
+ ("class", "heart + VS16", "❤" + VS16, 2, 1),
+ ("class", "umbrella + VS15", "☂" + VS15, 1, 2),
+ ("class", "flag pair (US)", "\U0001f1fa\U0001f1f8", 2, 4),
+ ("class", "thumbs up + skin tone", "\U0001f44d\U0001f3fd", 2, 4),
+ ("class", "Hangul, 3 conjoining jamo", "한", 2, 2),
+ ("class", "degree sign (ambiguous)", "°", None, None),
+
+ ("uniseg-quirk", "two-em dash U+2E3A", "⸺", 1, 1),
+ ("uniseg-quirk", "three-em dash U+2E3B", "⸻", 1, 1),
+
+ ("adversarial", "3 regional indicators",
+ "\U0001f1fa\U0001f1f8\U0001f1fa", 4, 6),
+ ("adversarial", "zalgo: e + 12 marks",
+ "e" + "̧̨́̀̂̃̄̆̈̊̋̌", 1, 1),
+ ("adversarial", "lone ZWJ", ZWJ, 0, 0),
+ ("adversarial", "lone VS16", VS16, 0, 0),
+ ("adversarial", "lone combining acute", "́", 0, 0),
+ ]
+ # Synthetic ZWJ chains: one grapheme cluster of N people. Not RGI
+ # sequences past a point, so partial clusterers show themselves here.
+ for n in (2, 4, 8, 16, 32):
+ s = ZWJ.join([person] * n)
+ probes.append(("chain", f"ZWJ chain of {n} people", s, 2, 2 * n))
+ return probes
+
+
+class Term:
+ def __init__(self):
+ self.fd = sys.stdout.fileno()
+ self.in_fd = sys.stdin.fileno()
+ self.saved = termios.tcgetattr(self.in_fd)
+ tty.setraw(self.in_fd)
+
+ def restore(self):
+ termios.tcsetattr(self.in_fd, termios.TCSADRAIN, self.saved)
+
+ def write(self, s):
+ os.write(self.fd, s.encode())
+
+ def cpr(self):
+ """Send CPR, return (row, col) or None on timeout/garbage."""
+ self.write(CPR)
+ buf = b""
+ while True:
+ r, _, _ = select.select([self.in_fd], [], [], TIMEOUT)
+ if not r:
+ return None
+ buf += os.read(self.in_fd, 64)
+ m = re.search(rb"\x1b\[(\d+);(\d+)R", buf)
+ if m:
+ return int(m.group(1)), int(m.group(2))
+
+
+def measure(term, cols, probe):
+ """Print probe at column 1, return cells advanced (may span rows)."""
+ term.write("\r\x1b[K")
+ start = term.cpr()
+ if start is None:
+ return None
+ term.write(probe)
+ end = term.cpr()
+ # Erase everything the probe touched, back to the starting row.
+ term.write("\r\x1b[K")
+ if end is not None:
+ for _ in range(end[0] - start[0]):
+ term.write("\x1b[A\x1b[2K")
+ if end is None:
+ return None
+ return (end[0] - start[0]) * cols + (end[1] - 1)
+
+
+def render_cell(s, cells):
+ """The probe itself, padded to RENDER_COL using its MEASURED width."""
+ if cells is None:
+ return " " * RENDER_COL
+ if cells > RENDER_COL:
+ return "(too wide)".ljust(RENDER_COL)
+ return s + " " * (RENDER_COL - cells)
+
+
+def cps(s, width):
+ """Space-joined hex codepoints, truncated with an ellipsis to fit."""
+ out = " ".join(f"{ord(c):04X}" for c in s)
+ if len(out) > width:
+ out = out[: width - 1].rstrip() + "\u2026"
+ return out
+
+
+def main():
+ if not (sys.stdin.isatty() and sys.stdout.isatty()):
+ print("width_report.py: needs a tty (run it directly in the terminal under test)")
+ return 1
+
+ size = os.get_terminal_size()
+ cols = size.columns
+ probes = build_probes()
+
+ term = Term()
+ results = []
+ try:
+ # Make room so multi-row wraps don't scroll and break row math.
+ term.write("\n" * 4 + "\x1b[4A")
+ if term.cpr() is None:
+ term.restore()
+ print("width_report.py: no CPR reply — this terminal does not answer ESC[6n; cannot measure.")
+ return 1
+ for group, label, s, cl, cp in probes:
+ cells = measure(term, cols, s)
+ results.append((group, label, s, cl, cp, cells))
+ finally:
+ term.write("\r\x1b[K")
+ term.restore()
+
+ # ---- report ----
+ env = os.environ
+ print("terminal width report")
+ print(f" TERM={env.get('TERM', '?')} TERM_PROGRAM={env.get('TERM_PROGRAM', '-')} cols={cols}")
+ print()
+ CP_COL = 28
+ print(f" {'':<12} {'':<26} {'':<{CP_COL}}"
+ f" {'':>4} {'cluster':>7} {'codept':>6} {'':>8}")
+ print(f" {'group':<12} {'probe':<26} {'codepoints':<{CP_COL}}"
+ f" {'#cp':>4} {'width':>7} {'width':>6} {'measured':>8} {'verdict':<10} renders")
+ print(f" {'-'*12} {'-'*26} {'-'*CP_COL} {'-'*4} {'-'*7} {'-'*6} {'-'*8} {'-'*10} {'-'*RENDER_COL}")
+
+ max_cluster_advance = 0
+ max_cluster_label = ""
+ votes = {"cluster": 0, "codepoint": 0}
+ for group, label, s, cl, cp, cells in results:
+ if cells is None:
+ verdict = "NO REPLY"
+ elif cl is None:
+ verdict = "" # informational probe, no prediction
+ elif cl == cp:
+ verdict = "ok" if cells == cl else f"NEITHER ({cl}?)"
+ elif cells == cl:
+ verdict = "cluster"
+ votes["cluster"] += 1
+ elif cells == cp:
+ verdict = "codepoint"
+ votes["codepoint"] += 1
+ else:
+ verdict = "NEITHER"
+ shown = "?" if cells is None else cells
+ print(f" {group:<12} {label:<26} {cps(s, CP_COL):<{CP_COL}}"
+ f" {len(s):>4} {fmt(cl):>7} {fmt(cp):>6} {shown:>8} {verdict:<10} {render_cell(s, cells)}")
+ # Single-cluster probes only (the RI-run probe is 2 clusters, skip it).
+ if cells is not None and group in ("baseline", "class", "uniseg-quirk", "chain") \
+ and cells > max_cluster_advance:
+ max_cluster_advance = cells
+ max_cluster_label = label
+
+ print()
+ if votes["cluster"] or votes["codepoint"]:
+ model = "cluster" if votes["cluster"] >= votes["codepoint"] else "codepoint"
+ print(f" inferred model: {model} (cluster-matching probes: {votes['cluster']},"
+ f" codepoint-matching: {votes['codepoint']})")
+ if votes["cluster"] and votes["codepoint"]:
+ print(" MIXED RESULTS: this terminal is a partial clusterer — no single model fits.")
+ amb = next((c for g, l, s, _, _, c in results if "ambiguous" in l), None)
+ if amb is not None:
+ print(f" East Asian ambiguous width: {amb}")
+ vs16 = next((c for g, l, s, _, _, c in results if l == "heart + VS16"), None)
+ if vs16 is not None:
+ print(f" VS16 promotes to wide: {'yes' if vs16 == 2 else 'no' if vs16 == 1 else f'odd ({vs16})'}")
+ print()
+ print(f" max advance for a single grapheme cluster: {max_cluster_advance}"
+ f" ({max_cluster_label})")
+ if max_cluster_advance <= 2:
+ print(" => on this terminal, no single cluster ever moved more than 2 cells.")
+ else:
+ print(" => this terminal advanced MORE than 2 cells for a single cluster;")
+ print(" it renders cluster internals as separate glyphs (codepoint-style).")
+ return 0
+
+
+def fmt(v):
+ return "-" if v is None else str(v)
+
+
+if __name__ == "__main__":
+ sys.exit(main())