Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,30 @@ reads that marker.
a row of `=` drew as a single character. Under insert mode the repeated
characters still overwrite rather than insert, as `docs/LIMITATIONS.md`
says.
- An underline colour in the form crossterm writes it (`CSI 58;5;n m`,
`CSI 58;2;r;g;b m`) no longer reads as attributes (#579). The emulator
took `5` for blink and `n` for an attribute of its own, so a cell lost its
style (`58;5;0` reset it) or turned bold (`58;5;1`); ratatui writes
underline colours by default. The sequence is still listed by
`unsupported()`, now as one parameter (`^[[58:5:1m`).
- Text is measured in the columns the grid gives it (#580). Insert mode
reserved room for a run by its string width, so a ZWJ family, lam-alef or
a skin-toned emoji overwrote the characters after it instead of pushing
them right; `Screen::locate` reported a history column that
`scrollback_cell` did not address; and the `termlens diff` frame sat a
column or two off on such rows. A saved screen holding an emoji the
presentation selector widened (`⚠️`) reads back with the columns it was
written at, and a row an earlier release wrote with one column for it
still reads.
- `CSI ? 7 h`, which every ncurses program sends at startup, is no longer
listed by `unsupported()` (#581). It asks for autowrap, the only
behaviour the emulator has, so no ncurses application could pass
`unsupported().is_empty()`. `CSI ? 7 l` is still listed: autowrap cannot
be turned off, as `docs/LIMITATIONS.md` now says.
- `Screen::row_wrapped` and `docs/LIMITATIONS.md` say that a wide character
which moved to the next row because it did not fit in an empty last
column is not recorded as a wrap, so `logical_text()` splits the line
there (#585).
- `termlens` no longer panics with exit code 101 (#594). A closed or full
stderr (`2>&1 | true`, `2>/dev/full`) made every diagnostic and the
`inspect` trailer panic; the write is now checked, and a trailer that cannot
Expand Down
76 changes: 68 additions & 8 deletions crates/termlens/src/emu/shadow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,11 @@ enum State {
Csi,
}

/// Rewrites colon-form extended colours into the semicolon form understood by
/// `vt100`. Like [`AttrShadow`], it only holds a complete plain SGR and emits
/// every other sequence unchanged.
/// Rewrites the SGR colour forms `vt100` misreads into ones it reads right:
/// colon-form extended colours into the semicolon form it understands, and
/// a semicolon-form underline colour into one colon parameter it skips.
/// Like [`AttrShadow`], it only holds a complete plain SGR and emits every
/// other sequence unchanged.
pub(super) struct ColorNormalizer {
state: State,
pending: Vec<u8>,
Expand All @@ -79,7 +81,7 @@ impl ColorNormalizer {
&mut self.state,
&mut self.pending,
bytes,
normalize_colon_colours,
normalize_for_grid,
)
}
}
Expand Down Expand Up @@ -207,22 +209,56 @@ fn rewrite_stream(
out
}

fn normalize_colon_colours(seq: &[u8]) -> Option<Vec<u8>> {
/// One plain SGR in the form `vt100` reads right, or `None` to pass it
/// through unchanged.
///
/// Two forms need it. Colon-form extended colours (`38:2::r:g:b`) become
/// the semicolon form. And an underline colour (`58`), which vt100 does not
/// model, in the semicolon form (`58;5;n`, `58;2;r;g;b`) becomes one colon
/// parameter (`58:5:n`): vt100 reads parameters one at a time, so it took
/// the arguments for attributes — `58;5;1` was blink and bold, and
/// `58;5;0` a reset. As one parameter it is skipped whole, and the
/// sequence is still recorded as one nobody honoured (#579). crossterm
/// writes the semicolon form for `SetUnderlineColor`.
fn normalize_for_grid(seq: &[u8]) -> Option<Vec<u8>> {
let params = seq.get(2..seq.len().checked_sub(1)?)?;
if params
.iter()
.any(|b| !matches!(b, b'0'..=b'9' | b';' | b':'))
{
return None;
}
if !params.contains(&b':') {
// The common case, checked without allocating: neither form is here.
if !params.contains(&b':') && !params.split(|&b| b == b';').any(|group| group == b"58") {
return None;
}

let groups: Vec<&[u8]> = params.split(|&b| b == b';').collect();
let mut normalized = Vec::with_capacity(groups.len());
let mut changed = false;
for group in groups {
let mut next = 0;
while next < groups.len() {
let group = groups[next];
next += 1;
if group == b"58" {
let arguments = match groups.get(next).copied() {
Some(b"5") => 2,
Some(b"2") => 4,
_ => 0,
};
if arguments > 0 {
let end = (next + arguments).min(groups.len());
let mut joined = group.to_vec();
for argument in &groups[next..end] {
joined.push(b':');
joined.extend_from_slice(argument);
}
normalized.push(joined);
next = end;
changed = true;
continue;
}
}
let parts: Vec<&[u8]> = group.split(|&b| b == b':').collect();
let replacement = match parts.as_slice() {
[first @ (b"38" | b"48"), b"2", b"", red, green, blue]
Expand Down Expand Up @@ -396,7 +432,7 @@ mod tests {
fn normalized(bytes: &[u8]) -> String {
let mut state = State::Ground;
let mut pending = Vec::new();
let mut out = rewrite_stream(&mut state, &mut pending, bytes, normalize_colon_colours);
let mut out = rewrite_stream(&mut state, &mut pending, bytes, normalize_for_grid);
// Anything still held at the end of the stream is not a complete SGR.
out.append(&mut pending);
String::from_utf8_lossy(&out).replace('\x1b', "E")
Expand Down Expand Up @@ -526,6 +562,30 @@ mod tests {
}
}

#[test]
fn a_semicolon_underline_colour_becomes_one_colon_parameter() {
// vt100 reads parameters one at a time and has no SGR 58, so its
// arguments must not reach it as parameters of their own (#579).
assert_eq!(normalized(b"\x1b[58;5;1m"), "E[58:5:1m");
assert_eq!(normalized(b"\x1b[4;58;2;0;1;7;31m"), "E[4;58:2:0:1:7;31m");
// Truncated: what is there is still one parameter.
assert_eq!(normalized(b"\x1b[58;5m"), "E[58:5m");
// Already one parameter, a 58 with no colour form after it, and a
// reset of the underline colour: nothing to group.
for seq in [
&b"\x1b[58:5:1m"[..],
b"\x1b[58m",
b"\x1b[58;3m",
b"\x1b[59m",
b"\x1b[1;5m",
] {
assert_eq!(
normalized(seq),
String::from_utf8_lossy(seq).replace('\x1b', "E")
);
}
}

#[test]
fn colon_colour_normalization_preserves_unrecognized_bytes() {
assert_eq!(normalized(b"\x1b[38:2::10:20:30m"), "E[38;2;10;20;30m");
Expand Down
20 changes: 19 additions & 1 deletion crates/termlens/src/emu/unhandled.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,11 @@ fn params_text(params: &[&[u16]]) -> String {
/// console's handshake in every Windows snapshot's list (#149).
const TRACKED_PRIVATE_MODES: &[u16] = &[9, 1000, 1002, 1003, 1005, 1006, 1015, 1004, 2026, 9001];

/// DECAWM, autowrap. Setting it asks for the only behaviour the emulator
/// has, so `CSI ? 7 h` — which ncurses sends at every start (`smam`) — is
/// honoured; resetting it is not modelled, and stays on record (#581).
const AUTOWRAP: u16 = 7;

/// The SGR parameters [`shadow`](super::shadow) recovers: blink, conceal,
/// strikethrough and their three "off" forms.
///
Expand Down Expand Up @@ -193,7 +198,7 @@ impl Callbacks for Unhandled {
(Some(b'?'), None, 'h' | 'l') => params
.iter()
.flat_map(|p| p.iter())
.all(|m| TRACKED_PRIVATE_MODES.contains(m)),
.all(|m| TRACKED_PRIVATE_MODES.contains(m) || (c == 'h' && *m == AUTOWRAP)),
// An SGR every one of whose parameters the shadow recovers is
// implemented, and naming it here would contradict the cell
// (#320). One parameter outside the set and the whole sequence
Expand Down Expand Up @@ -278,6 +283,19 @@ mod tests {
.collect()
}

#[test]
fn autowrap_on_is_honoured_and_autowrap_off_is_on_record() {
// Every ncurses program sends `CSI ? 7 h` at startup, and it asks
// for the only behaviour there is: naming it made an empty
// `unsupported()` impossible for any of them (#581). Turning wrap
// off is not modelled, so that still is named.
assert_eq!(shapes(b"\x1b[?7hready"), [] as [String; 0]);
assert_eq!(shapes(b"\x1b[?7l"), ["^[[?7l"]);
// In a list, the set is honoured with the others it is set with.
assert_eq!(shapes(b"\x1b[?7;1000h"), [] as [String; 0]);
assert_eq!(shapes(b"\x1b[?7;12h"), ["^[[?7;12h"]);
}

#[test]
fn a_sequence_nobody_honours_is_recorded_once_in_its_written_form() {
assert_eq!(
Expand Down
126 changes: 119 additions & 7 deletions crates/termlens/src/emu/vt100.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ use std::num::NonZeroU16;
use std::sync::Arc;
use std::time::{Duration, Instant};

use unicode_width::UnicodeWidthStr;

use super::seq::{SeqEvent, SeqTracker, TabOp};
use super::shadow::{AttrShadow, ColorNormalizer};
use super::unhandled::Unhandled;
Expand Down Expand Up @@ -381,10 +379,11 @@ impl Emulator for Vt100Emulator {
for (i, &byte) in bytes.iter().enumerate() {
// Insert mode: a printable run pushes the rest of the row right
// instead of overwriting it. vt100 does not model IRM, so the
// room is reserved with the `ICH` it does dispatch, sized in
// columns — a wide character counts twice — and never past the
// right margin (#261). The bytes owed to the grid go first, so
// the cursor column the reservation is made at is current.
// room is reserved with the `ICH` it does dispatch, sized in the
// columns the grid will give the run — its own layout, which no
// `unicode-width` measure is (#580) — and never past the right
// margin (#261). The bytes owed to the grid go first, so the
// cursor column the reservation is made at is current.
if i >= run_until
&& self.tracker.insert_mode()
&& !self.tracker.mid_sequence()
Expand All @@ -395,7 +394,7 @@ impl Emulator for Vt100Emulator {
.position(|&b| !is_printable(b))
.map_or(bytes.len(), |p| i + p);
run_until = end;
let width = String::from_utf8_lossy(&bytes[i..end]).width();
let width = crate::width::columns(&String::from_utf8_lossy(&bytes[i..end]));
self.feed_staged(&bytes[fed..i]);
fed = i;
let screen = self.parser.screen();
Expand Down Expand Up @@ -760,6 +759,12 @@ mod tests {
}
}

fn emu_with_size(rows: u16, cols: u16, bytes: &[u8]) -> Vt100Emulator {
let mut emu = Vt100Emulator::new(rows, cols, 0, crate::graphics::DEFAULT_CAPTURE, false);
feed_all(&mut emu, bytes);
emu
}

fn emu_with(bytes: &[u8]) -> Vt100Emulator {
let mut emu = Vt100Emulator::new(4, 10, 0, crate::graphics::DEFAULT_CAPTURE, false);
feed_all(&mut emu, bytes);
Expand Down Expand Up @@ -1363,6 +1368,113 @@ mod tests {
assert_eq!(emu.snapshot().row_text(0), "\u{6f22}\u{6f22}\u{6f22} ");
}

#[test]
fn the_width_model_is_the_grid_s_own_layout() {
// `width::columns` predicts how far the grid moves the cursor for a
// run of text; insert mode, `locate` and the diff all rely on that
// prediction. Held against vt100 itself, one sample at a time.
for sample in [
"abc",
"\u{6f22}\u{5b57}x",
"e\u{301}x",
"\u{644}\u{627}",
"\u{1f468}\u{200d}\u{1f469}x",
"\u{26a0}\u{fe0f}x",
"\u{2714}\u{fe0f}\u{31}\u{fe0f}\u{20e3}x",
"\u{1f44d}\u{1f3fd}x",
"a\u{fe0f}b",
"\u{1f1fa}\u{1f1f8}x",
] {
let emu = emu_with_size(2, 30, sample.as_bytes());
let (_, col, _) = emu.snapshot().cursor();
assert_eq!(
usize::from(col),
crate::width::columns(sample),
"{sample:?}"
);
}
}

#[test]
fn an_underline_colour_is_one_parameter_not_three_attributes() {
// vt100 does not model SGR 58 and read `58;5;n` a parameter at a
// time: `5` blink, then `n` as an attribute — `0` reset the cell, `1`
// made it bold (#579). crossterm writes exactly this form.
let emu = emu_with(b"\x1b[1;31mA\x1b[58;5;0mB\x1b[0m \x1b[58;5;1mC\x1b[0m");
let s = emu.snapshot();
let style = |col| *s.cell(0, col).unwrap().style();
for col in [0, 1] {
assert_eq!(style(col).fg, Color::Indexed(1), "column {col}");
assert!(style(col).bold, "column {col}");
assert!(!style(col).blink, "column {col}");
}
assert_eq!(style(3), Style::default(), "C asked only for a colour");
// The truecolour form's arguments are 0, 1 and 7: a reset, bold and
// reverse if read as attributes.
let emu = emu_with(b"\x1b[4mx\x1b[58;2;0;1;7my");
let y = *emu.snapshot().cell(0, 1).unwrap().style();
assert!(y.underline && !y.bold && !y.reverse && !y.dim, "{y:?}");
// Still a sequence nobody honoured, so it is still on record.
let shapes: Vec<String> = emu
.snapshot()
.unsupported()
.iter()
.map(String::from)
.collect();
assert_eq!(shapes, ["^[[58:2:0:1:7m"]);
}

#[test]
fn insert_mode_reserves_the_columns_the_grid_gives_the_run() {
// The reservation was the run's string width, and the grid lays a
// ZWJ family out as two wide cells and lam-alef as two letters:
// the difference was overwritten instead of pushed right (#580).
for (run, expected) in [
(
"\u{1f468}\u{200d}\u{1f469}",
"\u{1f468}\u{200d}\u{1f469}ABCDEF",
),
("\u{644}\u{627}", "\u{644}\u{627}ABCDEFGH"),
("\u{26a0}\u{fe0f}", "\u{26a0}\u{fe0f}ABCDEFGH"),
] {
let stream = format!("ABCDEFGHIJ\r\x1b[4h{run}");
let emu = emu_with_size(2, 10, stream.as_bytes());
assert_eq!(emu.snapshot().row_text(0), expected, "{run:?}");
}
}

#[test]
fn locate_reports_the_column_the_grid_gave_a_history_row() {
// History is text; `locate` turns the text before a match into a
// column, and a column `scrollback_cell` can be asked about must be
// the grid's (#580).
let mut emu = Vt100Emulator::new(2, 20, 100, crate::graphics::DEFAULT_CAPTURE, true);
feed_all(
&mut emu,
"\u{1f468}\u{200d}\u{1f469} SECRET\r\nb\r\nc\r\n".as_bytes(),
);
let s = emu.snapshot();
assert_eq!(
s.locate("SECRET"),
Some(crate::screen::Location::History { row: 0, col: 5 })
);
assert_eq!(s.scrollback_cell(0, 5).map(Cell::contents), Some("S"));
}

#[test]
fn a_selector_widened_emoji_survives_the_text_format() {
// The grid gives `⚠️` two columns; the saved text reads back with
// the same cells, so `find` agrees before and after (#580).
let emu = emu_with_size(2, 10, "\u{26a0}\u{fe0f}X \u{2714}\u{fe0f}Y".as_bytes());
let live = emu.snapshot();
let back = Screen::parse(&live.to_string()).unwrap();
for col in 0..10 {
assert_eq!(back.cell(0, col), live.cell(0, col), "column {col}");
}
assert_eq!(live.find("Y"), Some((0, 6)));
assert_eq!(back.find("Y"), Some((0, 6)));
}

#[test]
fn history_rows_keep_the_width_they_were_captured_at() {
// Documented decision (`Terminal::resize`): history is not reflowed
Expand Down
1 change: 1 addition & 0 deletions crates/termlens/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ mod spawn;
mod terminal;
mod utf8;
mod wait;
mod width;

pub use error::{Error, Result};
#[cfg(feature = "decode")]
Expand Down
7 changes: 6 additions & 1 deletion crates/termlens/src/screen.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1194,6 +1194,10 @@ impl Screen {
/// outside the screen.
///
/// This is the fact [`logical_text`](Self::logical_text) joins rows on.
/// The backend records a wrap only where the last column held
/// something, so a wide character that moved to the next row because it
/// did not fit in an empty last column leaves the row `false`
/// (`docs/LIMITATIONS.md`).
#[must_use]
pub fn row_wrapped(&self, row: u16) -> bool {
self.state
Expand Down Expand Up @@ -1427,7 +1431,8 @@ impl Screen {
let Some(byte_off) = folded.find(needle.as_str()) else {
continue;
};
let col = unicode_width::UnicodeWidthStr::width(&folded[..byte_off]);
// The columns the grid gave the text before the match (#580).
let col = crate::width::columns(&folded[..byte_off]);
return Some(Location::History {
row,
col: u16::try_from(col).unwrap_or(u16::MAX),
Expand Down
Loading
Loading