Grapheme cluster processing (DEC private mode 2027)

[raphamorim]

Implements grapheme clustering per contour's terminal-unicode-core proposal: with CSI ? 2027 h, incoming text is segmented into UAX #29 extended grapheme clusters, so a ZWJ emoji or an Indic conjunct occupies one cell slot instead of one per width-bearing codepoint. Ships on the unicode foundation (#1854). Four commits, one per layer.

Manually verified

Built and ran: ruler + printf '\e[?2027h🧑‍🌾|' puts the pipe at column 3 (one two-cell cluster, shaped as a single glyph); the same sequence after \e[?2027l puts it at column 5 (legacy). CSI ? 2027 $p answers CSI ? 2027 ; 1 $y — the DECRQM detection contract apps like notcurses rely on, verified from a live shell.

The design: position-anchored, stateless

The core deviation from ghostty (which filters controls and resets cursor-movement state): no cross-call segmentation state exists. Any no-break sequence accumulates into a single cell, so everything the break rules can look behind at — an emoji ZWJ run (GB11), regional-indicator parity (GB12/13), an Indic conjunct chain (GB9c) — is exactly the previous cell's contents, and try_cluster_append rebuilds the BreakState from them per codepoint (a handful of steps against rio-unicode's conformance-proven machine). Cursor movement, clears, scrolling, and screen switches invalidate clusters automatically; there are no reset hooks to forget. RI parity correctness across cells falls out of the invariant (a complete pair always occupies one cell).

Width follows ghostty's effect rules: valid VS16/VS15 flip the cluster wide/narrow (reusing the existing promotion machinery, extracted as widen_prev_cell with its row-edge wrap dance); an invalid selector is ignored outright (ghostty's .ignore — cell untouched, deviating deliberately from the legacy attach); any width-bearing continuation makes the whole cluster wide; zero-width continuations attach as before. Bulk non-ASCII decoding bails to scalar under the mode; ASCII bulk paths stay fast (ASCII never continues a cluster; the previous-cell anchor self-invalidates across runs).

Layer by layer

1. Mode plumbing: NamedPrivateMode::GraphemeCluster, Mode::GRAPHEME_CLUSTER (bit 24), DECSET/DECRST, DECRQM Set/Reset, RIS reset, shared across main/alt screens (matching ghostty/contour). Default: off.
2. Cluster input path: as above. Mode off is bit-identical — the full legacy suite passes unchanged.
3. Grid-consumer audit: regex search now feeds every codepoint a cell carries to the DFA (a search for a decomposed accent or a ZWJ sequence can finally match, cell-granular positions; bg-only cells stop feeding color bits decoded as chars). Reflow verified atomic (pair + extras id ride one Square through shrink/grow, reclaim-safe, test-pinned); copy/serialize round-trips cluster text into an identical grid; selection needs nothing (clusters are wide pairs); kitty placeholders coexist untouched.
4. Embedder channel: the wasm wire format gains a CELL_HAS_CLUSTER flag (free bit above the wide field, backward compatible) + RioTerm::cluster_text(); the C ABI gets RIO_CELL_HAS_CLUSTER in a reserved style_flags bit (the by-value struct can't grow) + rio_render_state_cell_cluster() with a count-first UTF-32 contract. An end-to-end test drives DECSET 2027 + a ZWJ emoji through inject_output and reads the cluster back through the public surface.

Performance

The cluster path takes the ghostty devlog-006 treatment: a flat grapheme-class table over U+0000..U+20000 (same window as the width table) and a precomputed transition table folding the entire break-rule chain into one indexed byte — (prev_class, next_class, state) -> (break | next_state), 5,832 entries, ~6KB. Both are built at first use by driving rio-unicode's conformance-tested reference implementation, so they can't drift from the rules, and a test asserts every entry against the reference. On the worst-case workload (pure Japanese text + emoji ZWJ clusters, best-of-7): mode-on throughput went from 31.5 to 59.9 Mcps, cutting the overhead vs mode-off from 3.05x to 1.53x. Mode off stays bit-identical and cost-free.

Tests

470+ across the tree, including 12 mode-2027 tests porting ghostty's Terminal.zig corpus concepts (ZWJ scalar/bulk, RI pairing + parity break, GB9c conjuncts with ghostty's wide rule, both selector cases, skin tones, row-edge widening, reflow, text round-trip) and 2 cluster-search tests. Rendering needed zero changes — #1854's mark-shaping already draws cluster cells as single glyphs.

Deliberately deferred

• A config knob for the boot default (default-on decision comes after bake time, the ghostty path — a knob controlling nothing yet is noise).
• rioterm.js renderer-side use of the new flag (separate repo; one conditional).
• tmux passthrough remains ecosystem-limited, as scoped.