From 5c495299cd34745a7b0e4b19ad971725b9fafe95 Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 18:51:20 +0900 Subject: [PATCH 1/3] feat: decode mojibake (UTF-8 read as windows-1252) --- README.ko.md | 39 +++++++++- README.md | 40 +++++++++- src/encoding.rs | 56 +++++++++----- src/main.rs | 85 ++++++++++++++------- src/mojibake.rs | 199 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/cli.rs | 40 ++++++++++ 6 files changed, 409 insertions(+), 50 deletions(-) create mode 100644 src/mojibake.rs diff --git a/README.ko.md b/README.ko.md index 8d0c003..4d85f4e 100644 --- a/README.ko.md +++ b/README.ko.md @@ -236,7 +236,40 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - `inspect`와 `decode`는 첫 번째 인자일 때만 하위 명령입니다. `hanfix -- inspect`와 `hf`는 지금처럼 글자를 변환합니다. - 유효한 UTF-8이라는 것은 바이트가 글자를 이룬다는 뜻일 뿐, 원래 뜻한 글이라는 보장은 아닙니다. 그래서 hanfix는 스스로 되돌리지 않습니다. 형식은 사용자가 정하고, `inspect`로 후보를 먼저 볼 수 있습니다. -- 모지바케(UTF-8을 다른 인코딩으로 읽은 글자, `안녕` 대신 `안녕`)는 아직 지원하지 않습니다. +- 모지바케는 아래의 별도 형식으로 다룹니다. + +**모지바케**는 UTF-8 글자를 다른 인코딩으로 읽어서 깨진 글자입니다. UTF-8로 저장한 `안녕`을 windows-1252로 열면 +`안녕`이 됩니다. `--type mojibake`는 이 과정을 거꾸로 합니다. 각 글자를 windows-1252 바이트로 되돌리고, 그 +바이트를 UTF-8로 읽습니다. + +```console +$ hanfix inspect '안녕' +Detected: mojibake +Status: valid +Encoding: UTF-8 read as windows-1252 +Reason: every character is in windows-1252, and its 6 bytes are valid UTF-8; valid UTF-8 does not prove this is the original text + +Original: 안녕 +Candidate: 안녕 +$ hanfix decode --type mojibake --source-encoding windows-1252 '안녕' +안녕 +$ hanfix decode --type mojibake --source-encoding windows-1252 '안�' +hanfix: error: lossy: U+FFFD at character 3 means a byte was already replaced; the original cannot be recovered +``` + +- `--source-encoding`(글자를 잘못 읽은 인코딩)는 꼭 지정해야 합니다. 지원하는 것은 `windows-1252`(`cp1252`도 같음) + 하나입니다. 256바이트를 모두 대응시키며, Microsoft가 정의하지 않은 다섯 바이트(`81`, `8D`, `8F`, `90`, `9D`)는 + 브라우저처럼 같은 번호의 보이지 않는 제어 문자로 봅니다. +- 글자 전체를 변환합니다. windows-1252에 없는 글자(`안녕 안녕`의 `안`)가 있거나 바이트가 완전한 UTF-8이 아니면 + 종료 코드 1로 실패하고 표준 출력에는 아무것도 나오지 않습니다. +- `�`(U+FFFD)는 hanfix가 글자를 받기 전에 이미 어떤 바이트가 바뀌었다는 뜻입니다. 어느 바이트였는지는 어떤 도구도 + 알 수 없으므로 `decode`는 실패하고 `inspect`는 lossy(손실)로 알립니다. 복사해 붙여 넣는 과정에서 보이지 않는 제어 + 문자가 빠져도 마찬가지로 되돌릴 수 없습니다. +- `inspect`는 모지바케처럼 보일 때만 후보를 보여 줍니다. UTF-8 여러 바이트 글자를 시작하는 바이트의 글자 뒤에 그 + 글자를 잇는 바이트의 글자가 오는 경우입니다. 한글, ASCII, 일반 악센트 글자(`café`, `naïve`)는 후보가 없습니다. + 되돌린 결과가 다시 모지바케처럼 보이면 이유에 그렇게 적습니다(두 번 잘못 읽은 경우이니 결과를 한 번 더 되돌리세요). +- CP949 / EUC-KR(UTF-8을 한국어 Windows 글자로 연 경우, `?덈뀞` 같은 출력)은 지원하지 않습니다. 약 17,000자의 표가 + 필요하고, 이런 잘못 읽기는 대개 도중에 바이트를 잃습니다. **이스케이프된 이름의 파일 여러 개를 한꺼번에 바꾸기 (PowerShell).** 함수를 정의한 뒤 폴더에서 `Rename-Decoded`를 실행하면 `old -> new` 목록만 보여 줍니다. 실제로 바꾸려면 `-Apply`를 붙입니다. 되돌릴 수 @@ -283,6 +316,7 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --update hanfix inspect [--] [TEXT ...] hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] + hanfix decode --type mojibake --source-encoding windows-1252 [--] [TEXT ...] ``` | 옵션 | 설명 | @@ -294,7 +328,8 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] | `--init SHELL` | `hf` 함수를 정의하는 셸 코드를 출력합니다. | | `--alias NAME` | `--init`이 만드는 함수 이름입니다. 기본값은 `hf`입니다. | | `--` | 옵션의 끝을 표시합니다. 이후는 모두 글자로 봅니다. | -| `--type TYPE` | `decode` 전용: `underscore-hex` 또는 `percent`. 꼭 지정해야 합니다. | +| `--type TYPE` | `decode` 전용: `underscore-hex`, `percent`, `mojibake` 중 하나. 꼭 지정해야 합니다. | +| `--source-encoding ENC` | `decode --type mojibake` 전용: `windows-1252`(또는 `cp1252`). 그때는 꼭 지정해야 합니다. | - 옵션은 맨 앞에서만 읽습니다. 나머지는 전부 변환할 글자로 봅니다. 그래서 `hanfix ㅣㄴ -ㅁ`의 `-ㅁ`은 옵션이 아니라 글자(`-a`)입니다. - `-`로 시작하는 영문 글자를 바꾸려면 앞에 `--`를 붙이세요: `hanfix -- -rk` → `-가`. diff --git a/README.md b/README.md index 6e87d1c..047fb23 100644 --- a/README.md +++ b/README.md @@ -243,7 +243,41 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex the word as typed text. - Valid UTF-8 only means the bytes form characters, not that they are the text someone meant. That is why hanfix never decodes on its own: you name the type, and `inspect` shows the candidate first. -- Mojibake (UTF-8 read as another encoding, `안녕` for `안녕`) is not handled yet. +- Mojibake has its own type, below. + +**Mojibake** is UTF-8 text that something read in another encoding: `안녕` saved as UTF-8 and opened as +windows-1252 shows as `안녕`. `--type mojibake` runs that misreading backwards: every character goes back to +its windows-1252 byte, and the bytes are read as UTF-8. + +```console +$ hanfix inspect '안녕' +Detected: mojibake +Status: valid +Encoding: UTF-8 read as windows-1252 +Reason: every character is in windows-1252, and its 6 bytes are valid UTF-8; valid UTF-8 does not prove this is the original text + +Original: 안녕 +Candidate: 안녕 +$ hanfix decode --type mojibake --source-encoding windows-1252 '안녕' +안녕 +$ hanfix decode --type mojibake --source-encoding windows-1252 '안�' +hanfix: error: lossy: U+FFFD at character 3 means a byte was already replaced; the original cannot be recovered +``` + +- `--source-encoding` is required: the encoding that misread the text. `windows-1252` (also `cp1252`) is the + one supported. All of its 256 bytes are mapped, the five Microsoft leaves undefined (`81`, `8D`, `8F`, `90`, + `9D`) as the invisible control characters of the same number, as browsers do. +- The whole text is converted. A character that windows-1252 does not have (`안` in `안녕 안녕`), or bytes + that are not complete UTF-8, fail with exit 1 and nothing on standard output. +- `�` (U+FFFD) means a byte was already replaced before hanfix saw the text. No tool can say which byte it was, + so `decode` fails and `inspect` reports it as lossy. Copy and paste can also drop those invisible control + characters, and then the text does not decode either. +- `inspect` lists a mojibake candidate only when the text looks like one: a character whose byte starts a + multi-byte UTF-8 character followed by one whose byte continues it. Korean, ASCII and ordinary accented text + (`café`, `naïve`) give none. When the result looks like mojibake again, the reason says so (it was misread + twice; decode the result once more). +- CP949 / EUC-KR (UTF-8 opened as Korean Windows text, `?덈뀞`-like output) is not supported: it would need a + table of about 17,000 characters, and that kind of misreading usually loses bytes on the way. **Rename many files with escaped names (PowerShell).** Define the function, then run `Rename-Decoded` in a folder: it only lists `old -> new`. Add `-Apply` to rename. It skips a name that does not decode, does not @@ -291,6 +325,7 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --update hanfix inspect [--] [TEXT ...] hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] + hanfix decode --type mojibake --source-encoding windows-1252 [--] [TEXT ...] ``` | Option | Description | @@ -302,7 +337,8 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] | `--init SHELL` | Print the shell code that defines `hf`. | | `--alias NAME` | Name of the function `--init` defines. Default `hf`. | | `--` | End of options. Everything after is text. | -| `--type TYPE` | `decode` only: `underscore-hex` or `percent`. Required. | +| `--type TYPE` | `decode` only: `underscore-hex`, `percent` or `mojibake`. Required. | +| `--source-encoding ENC` | `decode --type mojibake` only: `windows-1252` (or `cp1252`). Required there. | - Options are read only at the start; the rest is text to convert. So the `-ㅁ` in `hanfix ㅣㄴ -ㅁ` is text (`-a`), not an option. diff --git a/src/encoding.rs b/src/encoding.rs index d39bd21..1891bdc 100644 --- a/src/encoding.rs +++ b/src/encoding.rs @@ -6,6 +6,8 @@ use std::fmt; +use crate::mojibake; + #[derive(Clone, Copy, Debug, PartialEq)] pub enum Kind { UnderscoreHex, @@ -142,7 +144,7 @@ pub fn decode(kind: Kind, text: &str) -> Result { #[derive(Clone, Copy, Debug, PartialEq)] pub enum Status { Valid, - /// Valid on its own, but the text also has escapes of the other kind, which this one leaves alone. + /// Valid on its own, but the text also fits another type, which this one leaves alone. Ambiguous, Invalid, } @@ -159,31 +161,48 @@ impl Status { #[derive(Debug, PartialEq)] pub struct Candidate { - pub kind: Kind, + /// The `--type` that decodes it. + pub detected: &'static str, + /// How the text came to be: escaped UTF-8, or UTF-8 read as a code page. + pub encoding: String, pub status: Status, - /// `None` when the escapes do not decode. + /// `None` when it does not decode. pub decoded: Option, pub reason: String, } -/// Every kind of escape found in `text`, each decoded on its own. Empty for plain text. +/// Every type the text fits, each decoded on its own: escapes of either kind, and mojibake per code +/// page. Empty for plain text, including plain Korean. pub fn inspect(text: &str) -> Vec { - let found: Vec<_> = KINDS.into_iter().map(|k| (k, decode(k, text))).filter(|(_, r)| r.as_ref().map_or(true, |d| d.bytes > 0)).collect(); + let mut found: Vec<(&'static str, String, mojibake::Outcome)> = Vec::new(); + for kind in KINDS { + match decode(kind, text) { + Ok(d) if d.bytes == 0 => {} + Ok(d) => found.push(( + kind.name(), + "UTF-8".into(), + Ok((d.text, format!("{} escaped bytes are valid UTF-8; other characters are kept", d.bytes))), + )), + Err(e) => found.push((kind.name(), "UTF-8".into(), Err(e.to_string()))), + } + } + for cp in mojibake::CODEPAGES { + if let Some(r) = mojibake::inspect(cp, text) { + found.push(("mojibake", format!("UTF-8 read as {}", cp.name()), r)); + } + } let several = found.len() > 1; found .into_iter() - .map(|(kind, r)| match r { - Ok(d) => Candidate { - kind, + .map(|(detected, encoding, r)| match r { + Ok((decoded, reason)) => Candidate { + detected, + encoding, status: if several { Status::Ambiguous } else { Status::Valid }, - reason: if several { - format!("{} escaped bytes are valid UTF-8, but the text also has other escapes, which this type keeps", d.bytes) - } else { - format!("{} escaped bytes are valid UTF-8; other characters are kept", d.bytes) - }, - decoded: Some(d.text), + reason: if several { reason + "; the text also fits another type, which this one does not decode" } else { reason }, + decoded: Some(decoded), }, - Err(e) => Candidate { kind, status: Status::Invalid, decoded: None, reason: e.to_string() }, + Err(reason) => Candidate { detected, encoding, status: Status::Invalid, decoded: None, reason }, }) .collect() } @@ -207,16 +226,17 @@ pub fn report(text: &str) -> String { }; if found.is_empty() { return format!( - "Detected: none\nReason: no underscore-hex or percent escape found; the text is kept as is\n{note}\nOriginal: {text}\n" + "Detected: none\nReason: no underscore-hex or percent escape and no mojibake found; the text is kept as is\n{note}\nOriginal: {text}\n" ); } let blocks: Vec = found .iter() .map(|c| { let mut s = format!( - "Detected: {}\nStatus: {}\nEncoding: UTF-8\nReason: {}\n{note}\nOriginal: {text}\n", - c.kind.name(), + "Detected: {}\nStatus: {}\nEncoding: {}\nReason: {}\n{note}\nOriginal: {text}\n", + c.detected, c.status.name(), + c.encoding, c.reason ); if let Some(d) = &c.decoded { diff --git a/src/main.rs b/src/main.rs index 0bdfcc3..b18d58b 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1,6 +1,7 @@ //! hanfix: fix commands and text typed in the wrong keyboard layout (Korean 2-set <-> QWERTY). mod encoding; +mod mojibake; use std::io::{self, IsTerminal, Read, Write}; use std::process::ExitCode; @@ -252,6 +253,7 @@ const USAGE: &str = "usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --update hanfix inspect [--] [TEXT ...] hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] + hanfix decode --type mojibake --source-encoding windows-1252 [--] [TEXT ...] "; fn help() -> String { @@ -285,12 +287,14 @@ shell setup (add one line to your shell's start-up file): then, after a mistyped command, type hf (or ㅗㄹ); hf -y runs it without asking. -escaped text (only as the first argument; \"hanfix -- inspect\" converts the word): - hanfix inspect _ED_8A_B8_EB_A6_AC what the escapes are and what they decode to +escaped text and mojibake (only as the first argument; \"hanfix -- inspect\" converts the word): + hanfix inspect _ED_8A_B8_EB_A6_AC what the text fits and what it decodes to hanfix decode --type underscore-hex _ED_8A_B8 트 hanfix decode --type percent %ED%8A%B8%EB%A6%AC 트리 + hanfix decode --type mojibake --source-encoding windows-1252 안녕 안녕 decode prints only the result; text that is not an escape is kept. An incomplete or invalid - UTF-8 sequence prints nothing and fails. + UTF-8 sequence prints nothing and fails. Mojibake takes the whole text back to windows-1252 + bytes; a character not in it, or U+FFFD (already lost), fails. exit status: 0 on success, 1 when decode fails or the input of inspect or decode is not UTF-8, 2 on a usage error. @@ -351,8 +355,14 @@ fn read_stdin() -> Result, ExitCode> { /// `hanfix inspect` and `hanfix decode`. Their text is read strictly as UTF-8: a decoder that /// replaced bad input bytes would hide the very damage it is asked about. +enum Mode { + Inspect, + Escape(encoding::Kind), + Mojibake(mojibake::Codepage), +} + fn escaped(cmd: &str, mut args: Vec) -> ExitCode { - let mut kind = None; + let (mut ty, mut source): (Option, Option) = (None, None); while let Some(a) = args.first().cloned() { if a == "--" { args.remove(0); @@ -363,29 +373,46 @@ fn escaped(cmd: &str, mut args: Vec) -> ExitCode { return ExitCode::SUCCESS; } let (opt, val) = a.split_once('=').map_or((a.as_str(), None), |(o, v)| (o, Some(v.to_string()))); - if cmd != "decode" || opt != "--type" { - if a.starts_with('-') && a.len() > 1 && a.is_ascii() { - return usage_error(&format!("{cmd}: unknown option '{a}' (put -- before text that starts with -)")); + let slot = match opt { + "--type" if cmd == "decode" => &mut ty, + "--source-encoding" if cmd == "decode" => &mut source, + _ => { + if a.starts_with('-') && a.len() > 1 && a.is_ascii() { + return usage_error(&format!("{cmd}: unknown option '{a}' (put -- before text that starts with -)")); + } + break; } - break; - } + }; + let opt = opt.to_string(); args.remove(0); let val = match val { Some(v) => v, None if !args.is_empty() => args.remove(0), - None => return usage_error("--type needs a value"), + None => return usage_error(&format!("{opt} needs a value")), }; - if kind.is_some() { - return usage_error("--type given twice"); + if slot.is_some() { + return usage_error(&format!("{opt} given twice")); } - match encoding::Kind::parse(&val) { - Some(k) => kind = Some(k), - None => return usage_error(&format!("--type: choose underscore-hex or percent, not '{val}'")), - } - } - if cmd == "decode" && kind.is_none() { - return usage_error("decode needs --type underscore-hex or --type percent"); + *slot = Some(val); } + let mode = match (cmd, ty.as_deref(), source.as_deref()) { + ("inspect", ..) => Mode::Inspect, + (_, None, _) => return usage_error("decode needs --type underscore-hex, percent or mojibake"), + (_, Some("mojibake"), None) => { + return usage_error("--type mojibake needs --source-encoding windows-1252") + } + (_, Some("mojibake"), Some(s)) => match mojibake::Codepage::parse(s) { + Some(cp) => Mode::Mojibake(cp), + None => return usage_error(&format!("--source-encoding: windows-1252 is the one supported, not '{s}'")), + }, + (_, Some(t), s) => match (encoding::Kind::parse(t), s) { + (Some(k), None) => Mode::Escape(k), + (Some(_), Some(_)) => return usage_error("--source-encoding goes with --type mojibake only"), + (None, _) => { + return usage_error(&format!("--type: choose underscore-hex, percent or mojibake, not '{t}'")) + } + }, + }; let text = if !args.is_empty() { args.join(" ") } else if io::stdin().is_terminal() { @@ -403,15 +430,17 @@ fn escaped(cmd: &str, mut args: Vec) -> ExitCode { } } }; - let out = match kind { - None => encoding::report(&text), - Some(k) => match encoding::decode(k, &text) { - Ok(d) => d.text + "\n", - Err(e) => { - eprintln!("hanfix: error: {e}"); - return ExitCode::FAILURE; - } - }, + let decoded = match mode { + Mode::Inspect => Ok(encoding::report(&text)), + Mode::Escape(k) => encoding::decode(k, &text).map(|d| d.text + "\n").map_err(|e| e.to_string()), + Mode::Mojibake(cp) => mojibake::decode(cp, &text).map(|s| s + "\n").map_err(|e| e.to_string()), + }; + let out = match decoded { + Ok(out) => out, + Err(e) => { + eprintln!("hanfix: error: {e}"); + return ExitCode::FAILURE; + } }; let _ = io::stdout().write_all(out.as_bytes()); ExitCode::SUCCESS diff --git a/src/mojibake.rs b/src/mojibake.rs new file mode 100644 index 0000000..33db356 --- /dev/null +++ b/src/mojibake.rs @@ -0,0 +1,199 @@ +//! Mojibake: UTF-8 bytes that were read as a single-byte code page (`안녕` for `안녕`). +//! +//! Recovery runs the misreading backwards: every character goes back to its byte in the code page, +//! and the bytes are read as UTF-8. Both steps are strict. A character the code page does not have, +//! or bytes that are not UTF-8, fail the whole text; U+FFFD means a byte was already replaced, and +//! nothing can say which one, so such text is never reported as recovered. + +use std::fmt; + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Codepage { + Windows1252, +} + +pub const CODEPAGES: [Codepage; 1] = [Codepage::Windows1252]; + +// windows-1252 bytes 0x80-0x9F, as the WHATWG Encoding Standard maps them: the five bytes Microsoft +// leaves undefined (81, 8D, 8F, 90, 9D) keep the C1 control of the same number, so all 256 bytes +// round-trip. 0x00-0x7F and 0xA0-0xFF are the code points of the same number. +const WINDOWS_1252_HIGH: [char; 32] = [ + '\u{20AC}', '\u{0081}', '\u{201A}', '\u{0192}', '\u{201E}', '\u{2026}', '\u{2020}', '\u{2021}', + '\u{02C6}', '\u{2030}', '\u{0160}', '\u{2039}', '\u{0152}', '\u{008D}', '\u{017D}', '\u{008F}', + '\u{0090}', '\u{2018}', '\u{2019}', '\u{201C}', '\u{201D}', '\u{2022}', '\u{2013}', '\u{2014}', + '\u{02DC}', '\u{2122}', '\u{0161}', '\u{203A}', '\u{0153}', '\u{009D}', '\u{017E}', '\u{0178}', +]; + +impl Codepage { + pub fn name(self) -> &'static str { + match self { + Codepage::Windows1252 => "windows-1252", + } + } + + pub fn parse(name: &str) -> Option { + match name.to_ascii_lowercase().as_str() { + "windows-1252" | "cp1252" => Some(Codepage::Windows1252), + _ => None, + } + } + + /// The byte that shows as `c`, if the code page has `c`. + fn byte(self, c: char) -> Option { + match c as u32 { + n @ (0..=0x7F | 0xA0..=0xFF) => Some(n as u8), + _ => WINDOWS_1252_HIGH.iter().position(|&h| h == c).map(|i| 0x80 + i as u8), + } + } + + #[cfg(test)] + fn char_of(self, b: u8) -> char { + match b { + 0x80..=0x9F => WINDOWS_1252_HIGH[(b - 0x80) as usize], + _ => b as char, + } + } +} + +#[derive(Debug, PartialEq)] +pub enum Error { + /// U+FFFD at this character. + Lossy(usize), + NotInCodepage { codepage: Codepage, c: char, at: usize }, + /// The bytes are not UTF-8 from this character on (one byte per character, so the byte offset). + NotUtf8 { at: usize, incomplete: bool }, +} + +impl fmt::Display for Error { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match *self { + Error::Lossy(at) => write!( + f, + "lossy: U+FFFD at character {at} means a byte was already replaced; the original cannot be recovered" + ), + Error::NotInCodepage { codepage, c, at } => { + write!(f, "'{c}' (U+{:04X}) at character {at} is not in {}", c as u32, codepage.name()) + } + Error::NotUtf8 { at, incomplete } => { + let what = if incomplete { "an incomplete UTF-8 sequence" } else { "invalid UTF-8" }; + write!(f, "the bytes are {what} at character {at}") + } + } + } +} + +pub fn decode(codepage: Codepage, text: &str) -> Result { + if let Some(at) = text.chars().position(|c| c == '\u{FFFD}') { + return Err(Error::Lossy(at)); + } + let bytes = text + .chars() + .enumerate() + .map(|(at, c)| codepage.byte(c).ok_or(Error::NotInCodepage { codepage, c, at })) + .collect::, Error>>()?; + String::from_utf8(bytes).map_err(|e| Error::NotUtf8 { + at: e.utf8_error().valid_up_to(), + incomplete: e.utf8_error().error_len().is_none(), + }) +} + +/// Whether `text` looks like UTF-8 read as `codepage`: some character whose byte starts a multi-byte +/// UTF-8 character (C2-F4) is followed by one whose byte continues it (80-BF), or the text has U+FFFD +/// next to bytes above 0x7F. Plain Korean, ASCII and ordinary Latin text (`café`, `naïve`) do not. +fn looks_like(codepage: Codepage, text: &str) -> bool { + let bytes: Vec> = text.chars().map(|c| codepage.byte(c)).collect(); + let pair = bytes.windows(2).any(|w| { + matches!(w[0], Some(0xC2..=0xF4)) && matches!(w[1], Some(0x80..=0xBF)) + }); + pair || (text.contains('\u{FFFD}') && bytes.iter().any(|b| b.is_some_and(|b| b >= 0x80))) +} + +/// The recovered text and why, or why it does not recover. +pub type Outcome = Result<(String, String), String>; + +/// What `hanfix inspect` reports for this code page: `None` when the text does not look like +/// mojibake, otherwise its outcome. +pub fn inspect(codepage: Codepage, text: &str) -> Option { + if !looks_like(codepage, text) { + return None; + } + Some(match decode(codepage, text) { + Ok(s) => { + let mut reason = format!( + "every character is in {0}, and its {1} bytes are valid UTF-8; valid UTF-8 does not prove this is the original text", + codepage.name(), + text.chars().count() + ); + if looks_like(codepage, &s) { + reason += "; the result looks like mojibake again (decoded twice?)"; + } + Ok((s, reason)) + } + Err(e) => Err(e.to_string()), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + const W: Codepage = Codepage::Windows1252; + + /// UTF-8 text as a windows-1252 reader shows it. + fn misread(text: &str) -> String { + text.bytes().map(|b| W.char_of(b)).collect() + } + + #[test] + fn table_round_trips_every_byte() { + for b in 0..=255u8 { + assert_eq!(W.byte(W.char_of(b)), Some(b), "{b:02X}"); + } + assert_eq!(W.byte('\u{0080}'), None); + assert_eq!(W.byte('€'), Some(0x80)); + } + + #[test] + fn recovers_real_cases() { + assert_eq!(misread("안녕"), "안녕"); + assert_eq!(decode(W, "안녕"), Ok("안녕".to_string())); + assert_eq!(decode(W, "트리"), Ok("트리".to_string())); + assert_eq!(decode(W, "한글"), Ok("한글".to_string())); + assert_eq!(decode(W, "café"), Ok("café".to_string())); + assert_eq!(decode(W, "report_안녕.pdf"), Ok("report_안녕.pdf".to_string())); + // Bytes 81, 8D, 8F, 90 and 9D show as C1 controls, invisible but present. + for text in ["안녕하세요", "가나다라마바사아자차카타파하", "회의록 2026-10-29.txt", "😀"] { + assert_eq!(decode(W, &misread(text)).as_deref(), Ok(text), "{text}"); + } + } + + #[test] + fn fails_strictly() { + assert_eq!(decode(W, "안�"), Err(Error::Lossy(3))); + assert_eq!(decode(W, "안녕 안녕"), Err(Error::NotInCodepage { codepage: W, c: '안', at: 0 })); + assert_eq!(decode(W, "안ë…"), Err(Error::NotUtf8 { at: 3, incomplete: true })); + assert_eq!(decode(W, "café"), Err(Error::NotUtf8 { at: 3, incomplete: true })); + assert_eq!(decode(W, "naïve"), Err(Error::NotUtf8 { at: 2, incomplete: false })); + // A dropped C1 control (copy and paste often loses them) cannot be guessed back. + // 곁 is EA B3 81. + assert_eq!(misread("곁"), "ê³\u{81}"); + assert_eq!(decode(W, "ê³"), Err(Error::NotUtf8 { at: 0, incomplete: true })); + assert_eq!(decode(W, "plain ascii"), Ok("plain ascii".to_string())); + } + + #[test] + fn inspects() { + for text in ["안녕하세요", "hello_world", "café", "naïve", "©2026", "100% done", ""] { + assert_eq!(inspect(W, text), None, "{text}"); + } + assert!(matches!(inspect(W, "안녕"), Some(Ok((s, _))) if s == "안녕")); + assert!(matches!(inspect(W, "안ë…"), Some(Err(_)))); + let lossy = inspect(W, "안�").unwrap().unwrap_err(); + assert!(lossy.starts_with("lossy: ")); + // Decoded twice: the first pass gives mojibake again, and the reason says so. + let twice = misread(&misread("안녕")); + let (once, reason) = inspect(W, &twice).unwrap().unwrap(); + assert_eq!(once, misread("안녕")); + assert!(reason.contains("looks like mojibake again")); + } +} diff --git a/tests/cli.rs b/tests/cli.rs index 673ed51..f8b0be5 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -97,3 +97,43 @@ fn keyboard_conversion_unchanged() { let out = hanfix(&[], Some("햣 ㄴㅅㅁ션\n".as_bytes())); assert_eq!(out.stdout, b"git status\n"); } + +#[test] +fn mojibake() { + let w = ["decode", "--type", "mojibake", "--source-encoding", "windows-1252"]; + fn with<'a>(w: &[&'a str], text: &'a str) -> Vec<&'a str> { + [w, &[text]].concat() + } + assert_eq!(stdout(&with(&w, "안녕")), "안녕\n"); + assert_eq!(stdout(&with(&w, "report_한글.pdf")), "report_한글.pdf\n"); + assert_eq!(stdout(&["decode", "--type=mojibake", "--source-encoding=cp1252", "café"]), "café\n"); + let out = hanfix(&w, Some("트리\n".as_bytes())); + assert_eq!(out.stdout, "트리\n".as_bytes()); + for (text, error) in [ + ("안\u{FFFD}", "lossy: U+FFFD at character 3"), + ("안녕 안녕", "'안' (U+C548) at character 0 is not in windows-1252"), + ("안ë…", "incomplete UTF-8 sequence at character 3"), + ] { + let out = hanfix(&with(&w, text), None); + assert_eq!(out.status.code(), Some(1), "{text}"); + assert!(out.stdout.is_empty(), "{text}"); + let err = String::from_utf8(out.stderr).unwrap(); + assert!(err.starts_with("hanfix: error: ") && err.contains(error), "{text}: {err}"); + } + let report = stdout(&["inspect", "안녕"]); + assert!(report.starts_with("Detected: mojibake\nStatus: valid\nEncoding: UTF-8 read as windows-1252\n"), "{report}"); + assert!(report.ends_with("Candidate: 안녕\n"), "{report}"); + assert!(stdout(&["inspect", "안\u{FFFD}"]).contains("Status: invalid\nEncoding: UTF-8 read as windows-1252\nReason: lossy: ")); + for text in ["안녕하세요", "hello world", "café"] { + assert!(stdout(&["inspect", text]).starts_with("Detected: none\n"), "{text}"); + } + for args in [ + &["decode", "--type", "mojibake", "안녕"][..], + &["decode", "--type", "mojibake", "--source-encoding", "cp949", "x"], + &["decode", "--type", "percent", "--source-encoding", "windows-1252", "x"], + &["decode", "--type", "mojibake", "--source-encoding", "cp1252", "--source-encoding", "cp1252", "x"], + &["inspect", "--source-encoding", "cp1252", "x"], + ] { + assert_eq!(hanfix(args, None).status.code(), Some(2), "{args:?}"); + } +} From ae3bfaffc728a8c865de6713da92caddd4534812 Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 18:52:24 +0900 Subject: [PATCH 2/3] chore: 0.2.9 --- Cargo.lock | 2 +- Cargo.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 05ae2ee..a77aa07 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,4 +4,4 @@ version = 3 [[package]] name = "hanfix" -version = "0.2.8" +version = "0.2.9" diff --git a/Cargo.toml b/Cargo.toml index 2b0940a..ba39359 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "hanfix" -version = "0.2.8" +version = "0.2.9" edition = "2021" rust-version = "1.80" description = "Fix commands and text typed in the wrong keyboard layout (Korean 2-set <-> QWERTY)" From 55629b91d053c7a64a5ca68a7f0998d6e1721265 Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 19:04:48 +0900 Subject: [PATCH 3/3] fix: fail on U+FFFD in any decoded result; detect mojibake only by complete sequences --- README.ko.md | 18 ++++++--- README.md | 20 ++++++---- src/encoding.rs | 99 +++++++++++++++++++++++++++++++++++-------------- src/main.rs | 8 ++-- src/mojibake.rs | 57 ++++++++++++++++++++++------ tests/cli.rs | 18 ++++++++- 6 files changed, 161 insertions(+), 59 deletions(-) diff --git a/README.ko.md b/README.ko.md index 4d85f4e..d44e26c 100644 --- a/README.ko.md +++ b/README.ko.md @@ -230,8 +230,10 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - 두 형식에 나뉘어 들어간 바이트열(`%ED_8A_B8`)은 어느 쪽으로도 되돌리지 않습니다. - 이어진 이스케이프는 하나의 바이트열로 보고, 완전하고 유효한 UTF-8이어야 합니다. 하나라도 아니면(`_ED_8A`는 3바이트 글자의 앞 2바이트) `decode`는 아무것도 출력하지 않고, 이유를 표준 오류에 쓰고, 종료 코드 1로 끝납니다. - `�`로 바꿔 넣지 않습니다. 표준 입력이 UTF-8이 아니면 `inspect`와 `decode` 모두 종료 코드 1로 끝납니다. -- `decode`는 결과만 출력합니다. 이스케이프가 없는 글자는 그대로 나오고 종료 코드는 0입니다. `inspect`는 UTF-8 + `�`로 바꿔 넣지 않습니다. 되돌린 결과에 `�`가 있어도 같은 방식으로 실패합니다(`%EF%BF%BD`가 바로 `�`). + 이스케이프하기 전에 이미 글자가 손상되었다는 뜻입니다. 표준 입력이 UTF-8이 아니면 `inspect`와 `decode` 모두 + 종료 코드 1로 끝납니다. +- `decode`는 결과만 출력합니다. 이스케이프가 없는(그리고 `�`가 없는) 글자는 그대로 나오고 종료 코드는 0입니다. `inspect`는 UTF-8 글자에서 실패하지 않고 아무것도 바꾸지 않습니다. - `inspect`와 `decode`는 첫 번째 인자일 때만 하위 명령입니다. `hanfix -- inspect`와 `hf`는 지금처럼 글자를 변환합니다. - 유효한 UTF-8이라는 것은 바이트가 글자를 이룬다는 뜻일 뿐, 원래 뜻한 글이라는 보장은 아닙니다. 그래서 hanfix는 @@ -263,10 +265,12 @@ hanfix: error: lossy: U+FFFD at character 3 means a byte was already replaced; t - 글자 전체를 변환합니다. windows-1252에 없는 글자(`안녕 안녕`의 `안`)가 있거나 바이트가 완전한 UTF-8이 아니면 종료 코드 1로 실패하고 표준 출력에는 아무것도 나오지 않습니다. - `�`(U+FFFD)는 hanfix가 글자를 받기 전에 이미 어떤 바이트가 바뀌었다는 뜻입니다. 어느 바이트였는지는 어떤 도구도 - 알 수 없으므로 `decode`는 실패하고 `inspect`는 lossy(손실)로 알립니다. 복사해 붙여 넣는 과정에서 보이지 않는 제어 - 문자가 빠져도 마찬가지로 되돌릴 수 없습니다. -- `inspect`는 모지바케처럼 보일 때만 후보를 보여 줍니다. UTF-8 여러 바이트 글자를 시작하는 바이트의 글자 뒤에 그 - 글자를 잇는 바이트의 글자가 오는 경우입니다. 한글, ASCII, 일반 악센트 글자(`café`, `naïve`)는 후보가 없습니다. + 알 수 없으므로 `decode`는 실패하고 `inspect`는 lossy(손실)로 알립니다. `�`의 세 바이트를 잘못 읽은 `�`도 + 결과에 `�`가 생기므로 마찬가지입니다. 복사해 붙여 넣는 과정에서 보이지 않는 제어 문자가 빠져도 되돌릴 수 + 없습니다. +- `inspect`는 모지바케처럼 보일 때만 후보를 보여 줍니다. 몇 글자의 바이트가 완전한 UTF-8 여러 바이트 글자 하나를 + 이루는 경우입니다(`안`은 `EC 95 88`, `안`). 한글, ASCII, 일반 악센트 글자는 문장 부호가 붙어도(`café’s`, + `résumé…`, `«Café»`) 후보가 없습니다. 되돌린 결과가 다시 모지바케처럼 보이면 이유에 그렇게 적습니다(두 번 잘못 읽은 경우이니 결과를 한 번 더 되돌리세요). - CP949 / EUC-KR(UTF-8을 한국어 Windows 글자로 연 경우, `?덈뀞` 같은 출력)은 지원하지 않습니다. 약 17,000자의 표가 필요하고, 이런 잘못 읽기는 대개 도중에 바이트를 잃습니다. @@ -378,6 +382,8 @@ bind -x '"\eh": READLINE_LINE=$(hanfix -- "$READLINE_LINE"); READLINE_POINT=${#R - **`hf`는 한글을 영문으로만 바꿉니다.** 명령은 영문이므로 이 방향만 의미가 있습니다. - **한 줄에 한 방향만 바꿉니다.** `git ㅔㅕ노`처럼 영문과 한글이 섞여 있으면, 한글이 있으므로 전부 영문 쪽으로 바꿉니다. 의도한 한글 단어가 섞여 있으면 그것까지 영문이 됩니다. - **두벌식만 지원합니다.** 세벌식, 그리고 QWERTY가 아닌 영문 자판(Dvorak 등)은 지원하지 않습니다. +- **분해형(NFD) 글자는 합치지 않습니다.** macOS 파일 이름은 `ì`를 `i`와 결합 악센트로 저장하기도 합니다. 이런 형태의 + 모지바케는 되돌리지 못하고 `inspect`도 찾지 못하므로, 먼저 NFC로 합치세요. ## 개발 diff --git a/README.md b/README.md index 047fb23..2152474 100644 --- a/README.md +++ b/README.md @@ -235,9 +235,10 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - A byte sequence split between the two types (`%ED_8A_B8`) decodes with neither. - Consecutive escapes form one byte sequence that must be complete, valid UTF-8. If any sequence in the text is not (`_ED_8A` is two bytes of a three-byte character), `decode` prints nothing, writes the reason to - standard error and exits with 1. It never substitutes `�`. Input on standard input that is not UTF-8 - makes `inspect` and `decode` exit with 1 too. -- `decode` prints only the result; text with no escapes comes back unchanged with exit 0. `inspect` never + standard error and exits with 1. It never substitutes `�`. A result that contains `�` fails the same way + (`%EF%BF%BD` is `�` itself): the text was already damaged before it was escaped. Input on standard input + that is not UTF-8 makes `inspect` and `decode` exit with 1 too. +- `decode` prints only the result; text with no escapes (and no `�`) comes back unchanged with exit 0. `inspect` never fails on UTF-8 text and changes nothing. - `inspect` and `decode` are subcommands only as the first argument. `hanfix -- inspect` and `hf` still convert the word as typed text. @@ -270,11 +271,12 @@ hanfix: error: lossy: U+FFFD at character 3 means a byte was already replaced; t - The whole text is converted. A character that windows-1252 does not have (`안` in `안녕 안녕`), or bytes that are not complete UTF-8, fail with exit 1 and nothing on standard output. - `�` (U+FFFD) means a byte was already replaced before hanfix saw the text. No tool can say which byte it was, - so `decode` fails and `inspect` reports it as lossy. Copy and paste can also drop those invisible control - characters, and then the text does not decode either. -- `inspect` lists a mojibake candidate only when the text looks like one: a character whose byte starts a - multi-byte UTF-8 character followed by one whose byte continues it. Korean, ASCII and ordinary accented text - (`café`, `naïve`) give none. When the result looks like mojibake again, the reason says so (it was misread + so `decode` fails and `inspect` reports it as lossy. The same goes for `�`, the three bytes of `�` misread: + the result would contain `�`. Copy and paste can also drop those invisible control characters, and then the + text does not decode either. +- `inspect` lists a mojibake candidate only when the text looks like one: some characters whose bytes form + one complete multi-byte UTF-8 character (`안` is `EC 95 88`, `안`). Korean, ASCII and ordinary accented + text give none, also next to typographic punctuation (`café’s`, `résumé…`, `«Café»`). When the result looks like mojibake again, the reason says so (it was misread twice; decode the result once more). - CP949 / EUC-KR (UTF-8 opened as Korean Windows text, `?덈뀞`-like output) is not supported: it would need a table of about 17,000 characters, and that kind of misreading usually loses bytes on the way. @@ -392,6 +394,8 @@ Characters that are not on the layout (digits, spaces, symbols) are left as they - **One direction per line.** In mixed text like `git ㅔㅕ노`, the Hangul means everything goes to English keys, including any Hangul word you meant to keep. - **2-set only.** 3-set (세벌식) and non-QWERTY English layouts (Dvorak etc.) are not supported. +- **Decomposed (NFD) text is not composed.** macOS file names can store `ì` as `i` and a combining accent. + Mojibake in that form does not decode, and `inspect` does not detect it; compose it to NFC first. ## Development diff --git a/src/encoding.rs b/src/encoding.rs index 1891bdc..8acc72b 100644 --- a/src/encoding.rs +++ b/src/encoding.rs @@ -2,7 +2,8 @@ //! //! Only escape runs change; every other character is copied as it is. A run of escapes is one byte //! sequence that must be complete, valid UTF-8, or the whole decode fails: nothing is replaced with -//! U+FFFD and nothing is guessed. +//! U+FFFD and nothing is guessed. A result that contains U+FFFD fails too, since an escaped `EF BF BD` +//! means the text was damaged before it was escaped. use std::fmt; @@ -47,25 +48,38 @@ impl Kind { } #[derive(Debug, PartialEq)] -pub struct DecodeError { - pub kind: Kind, - /// The escape run that failed, as written in the input. - pub run: String, - /// Characters before the run. - pub at: usize, - pub incomplete: bool, +pub enum DecodeError { + Run { + kind: Kind, + /// The escape run that failed, as written in the input. + run: String, + /// Characters before the run. + at: usize, + incomplete: bool, + }, + /// The decoded text has U+FFFD at character `at`: the bytes were damaged before they were escaped, + /// so the result is not the original. `bytes` as in `Decoded`. + Lossy { at: usize, bytes: usize }, } impl fmt::Display for DecodeError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { - let what = if self.incomplete { "incomplete UTF-8 sequence" } else { "invalid UTF-8" }; - // A run can be megabytes long on standard input; the start and the length locate it. - let run = if self.run.len() > 40 { - format!("{}...' ({} characters)", &self.run[..40], self.run.len()) - } else { - format!("{}'", self.run) - }; - write!(f, "{what} in {} escape '{run} at character {}", self.kind.name(), self.at) + match self { + DecodeError::Run { kind, run, at, incomplete } => { + let what = if *incomplete { "incomplete UTF-8 sequence" } else { "invalid UTF-8" }; + // A run can be megabytes long on standard input; the start and the length locate it. + let run = if run.len() > 40 { + format!("{}...' ({} characters)", &run[..40], run.len()) + } else { + format!("{run}'") + }; + write!(f, "{what} in {} escape '{run} at character {at}", kind.name()) + } + DecodeError::Lossy { at, .. } => write!( + f, + "lossy: the decoded text contains U+FFFD at character {at}: it was already damaged before it was escaped" + ), + } } } @@ -127,7 +141,7 @@ pub fn decode(kind: Kind, text: &str) -> Result { match std::str::from_utf8(&run) { Ok(s) => out.push_str(s), Err(e) => { - return Err(DecodeError { + return Err(DecodeError::Run { kind, run: text[start..i].to_string(), at: text[..start].chars().count(), @@ -138,6 +152,9 @@ pub fn decode(kind: Kind, text: &str) -> Result { copied = i; } out.push_str(&text[copied..]); + if let Some(at) = out.chars().position(|c| c == '\u{FFFD}') { + return Err(DecodeError::Lossy { at, bytes }); + } Ok(Decoded { text: out, bytes }) } @@ -177,7 +194,7 @@ pub fn inspect(text: &str) -> Vec { let mut found: Vec<(&'static str, String, mojibake::Outcome)> = Vec::new(); for kind in KINDS { match decode(kind, text) { - Ok(d) if d.bytes == 0 => {} + Ok(Decoded { bytes: 0, .. }) | Err(DecodeError::Lossy { bytes: 0, .. }) => {} Ok(d) => found.push(( kind.name(), "UTF-8".into(), @@ -290,20 +307,23 @@ mod tests { assert_eq!(ok(Kind::Percent, "%GG"), "%GG"); } + /// The failed run, its position and whether it was incomplete. + fn run_error(kind: Kind, text: &str) -> (String, usize, bool) { + match decode(kind, text) { + Err(DecodeError::Run { run, at, incomplete, .. }) => (run, at, incomplete), + r => panic!("{text}: {r:?}"), + } + } + #[test] fn bad_sequences_fail() { - let e = decode(Kind::UnderscoreHex, "_ED_8A").unwrap_err(); - assert!(e.incomplete && e.run == "_ED_8A" && e.at == 0); - let e = decode(Kind::Percent, "x%ED%8A").unwrap_err(); - assert!(e.incomplete && e.run == "%ED%8A" && e.at == 1); - let e = decode(Kind::Percent, "%FF").unwrap_err(); - assert!(!e.incomplete); + assert_eq!(run_error(Kind::UnderscoreHex, "_ED_8A"), ("_ED_8A".into(), 0, true)); + assert_eq!(run_error(Kind::Percent, "x%ED%8A"), ("%ED%8A".into(), 1, true)); + assert_eq!(run_error(Kind::Percent, "%FF"), ("%FF".into(), 0, false)); // A lead byte with what follows it missing still fails, also before plain text. - let e = decode(Kind::UnderscoreHex, "트리_DB_backup.sql").unwrap_err(); - assert!(e.incomplete && e.run == "_DB" && e.at == 2); + assert_eq!(run_error(Kind::UnderscoreHex, "트리_DB_backup.sql"), ("_DB".into(), 2, true)); // Inside a character every escape is taken, so a wrong one is reported, not kept as text. - let e = decode(Kind::UnderscoreHex, "_ED_C3_A9").unwrap_err(); - assert!(!e.incomplete && e.run == "_ED_C3_A9"); + assert_eq!(run_error(Kind::UnderscoreHex, "_ED_C3_A9"), ("_ED_C3_A9".into(), 0, false)); assert!(decode(Kind::UnderscoreHex, "_ED_A0_80").is_err()); // A good run does not save a later bad one. assert!(decode(Kind::UnderscoreHex, "_ED_8A_B8 _ED").is_err()); @@ -319,6 +339,29 @@ mod tests { ); } + #[test] + fn replacement_character_in_the_result_fails() { + // EF BF BD is U+FFFD: the bytes were already damaged when they were escaped. + for (kind, text, at) in [ + (Kind::Percent, "%EF%BF%BD", 0), + (Kind::Percent, "%EC%95%88%EF%BF%BD", 1), + (Kind::UnderscoreHex, "_EF_BF_BD", 0), + (Kind::UnderscoreHex, "x_EC_95_88_EF_BF_BD.txt", 2), + (Kind::Percent, "a\u{FFFD} %41", 1), + ] { + assert!(matches!(decode(kind, text), Err(DecodeError::Lossy { at: a, .. }) if a == at), "{text}"); + } + assert_eq!(decode(Kind::Percent, "\u{FFFD}"), Err(DecodeError::Lossy { at: 0, bytes: 0 })); + assert_eq!( + decode(Kind::Percent, "%EF%BF%BD").unwrap_err().to_string(), + "lossy: the decoded text contains U+FFFD at character 0: it was already damaged before it was escaped" + ); + let c = inspect("%EC%95%88%EF%BF%BD"); + assert!(c.len() == 1 && c[0].status == Status::Invalid && c[0].reason.starts_with("lossy: ")); + // Plain text with U+FFFD and no escapes is not an escape candidate. + assert!(inspect("a\u{FFFD}b").is_empty()); + } + #[test] fn underscore_bytes_that_start_no_character_are_text() { for text in ["IMG_8123.JPG", "DSC_9001.JPG", "test_99.txt", "_8A", "트_8A", "_C0_80", "_C1_BF", "_F5_80_80_80", "_FF"] { diff --git a/src/main.rs b/src/main.rs index b18d58b..bd34320 100644 --- a/src/main.rs +++ b/src/main.rs @@ -293,8 +293,8 @@ escaped text and mojibake (only as the first argument; \"hanfix -- inspect\" con hanfix decode --type percent %ED%8A%B8%EB%A6%AC 트리 hanfix decode --type mojibake --source-encoding windows-1252 안녕 안녕 decode prints only the result; text that is not an escape is kept. An incomplete or invalid - UTF-8 sequence prints nothing and fails. Mojibake takes the whole text back to windows-1252 - bytes; a character not in it, or U+FFFD (already lost), fails. + UTF-8 sequence prints nothing and fails, and so does a result with U+FFFD (already lost). + Mojibake takes the whole text back to windows-1252 bytes; a character not in it fails. exit status: 0 on success, 1 when decode fails or the input of inspect or decode is not UTF-8, 2 on a usage error. @@ -353,14 +353,14 @@ fn read_stdin() -> Result, ExitCode> { Ok(buf) } -/// `hanfix inspect` and `hanfix decode`. Their text is read strictly as UTF-8: a decoder that -/// replaced bad input bytes would hide the very damage it is asked about. enum Mode { Inspect, Escape(encoding::Kind), Mojibake(mojibake::Codepage), } +/// `hanfix inspect` and `hanfix decode`. Their text is read strictly as UTF-8: a decoder that +/// replaced bad input bytes would hide the very damage it is asked about. fn escaped(cmd: &str, mut args: Vec) -> ExitCode { let (mut ty, mut source): (Option, Option) = (None, None); while let Some(a) = args.first().cloned() { diff --git a/src/mojibake.rs b/src/mojibake.rs index 33db356..616cd8d 100644 --- a/src/mojibake.rs +++ b/src/mojibake.rs @@ -2,8 +2,9 @@ //! //! Recovery runs the misreading backwards: every character goes back to its byte in the code page, //! and the bytes are read as UTF-8. Both steps are strict. A character the code page does not have, -//! or bytes that are not UTF-8, fail the whole text; U+FFFD means a byte was already replaced, and -//! nothing can say which one, so such text is never reported as recovered. +//! or bytes that are not UTF-8, fail the whole text; U+FFFD, in the text or in the result, means a +//! byte was already replaced, and nothing can say which one, so such text is never reported as +//! recovered. use std::fmt; @@ -59,6 +60,9 @@ impl Codepage { pub enum Error { /// U+FFFD at this character. Lossy(usize), + /// U+FFFD at this character of the result (`�` is its three bytes): the text was damaged before + /// it was misread. + Damaged(usize), NotInCodepage { codepage: Codepage, c: char, at: usize }, /// The bytes are not UTF-8 from this character on (one byte per character, so the byte offset). NotUtf8 { at: usize, incomplete: bool }, @@ -71,6 +75,10 @@ impl fmt::Display for Error { f, "lossy: U+FFFD at character {at} means a byte was already replaced; the original cannot be recovered" ), + Error::Damaged(at) => write!( + f, + "lossy: the decoded text contains U+FFFD at character {at}: it was already damaged before it was misread" + ), Error::NotInCodepage { codepage, c, at } => { write!(f, "'{c}' (U+{:04X}) at character {at} is not in {}", c as u32, codepage.name()) } @@ -91,21 +99,32 @@ pub fn decode(codepage: Codepage, text: &str) -> Result { .enumerate() .map(|(at, c)| codepage.byte(c).ok_or(Error::NotInCodepage { codepage, c, at })) .collect::, Error>>()?; - String::from_utf8(bytes).map_err(|e| Error::NotUtf8 { + let s = String::from_utf8(bytes).map_err(|e| Error::NotUtf8 { at: e.utf8_error().valid_up_to(), incomplete: e.utf8_error().error_len().is_none(), - }) + })?; + match s.chars().position(|c| c == '\u{FFFD}') { + Some(at) => Err(Error::Damaged(at)), + None => Ok(s), + } } -/// Whether `text` looks like UTF-8 read as `codepage`: some character whose byte starts a multi-byte -/// UTF-8 character (C2-F4) is followed by one whose byte continues it (80-BF), or the text has U+FFFD -/// next to bytes above 0x7F. Plain Korean, ASCII and ordinary Latin text (`café`, `naïve`) do not. +/// Whether `text` looks like UTF-8 read as `codepage`: some characters map to one complete multi-byte +/// UTF-8 character, a lead byte (C2-F4) and exactly the continuation bytes it needs. An accented letter +/// that happens to be a lead byte before punctuation in 80-BF (`café’s`, `résumé…`, `«Café»`) is not +/// enough, and neither is U+FFFD; plain Korean and ASCII never are. fn looks_like(codepage: Codepage, text: &str) -> bool { let bytes: Vec> = text.chars().map(|c| codepage.byte(c)).collect(); - let pair = bytes.windows(2).any(|w| { - matches!(w[0], Some(0xC2..=0xF4)) && matches!(w[1], Some(0x80..=0xBF)) - }); - pair || (text.contains('\u{FFFD}') && bytes.iter().any(|b| b.is_some_and(|b| b >= 0x80))) + bytes.iter().enumerate().any(|(i, b)| { + let len = match b { + Some(0xC2..=0xDF) => 2, + Some(0xE0..=0xEF) => 3, + Some(0xF0..=0xF4) => 4, + _ => return false, + }; + let seq: Option> = bytes.get(i..i + len).and_then(|w| w.iter().copied().collect()); + seq.is_some_and(|s| std::str::from_utf8(&s).is_ok()) + }) } /// The recovered text and why, or why it does not recover. @@ -179,17 +198,31 @@ mod tests { assert_eq!(misread("곁"), "ê³\u{81}"); assert_eq!(decode(W, "ê³"), Err(Error::NotUtf8 { at: 0, incomplete: true })); assert_eq!(decode(W, "plain ascii"), Ok("plain ascii".to_string())); + // `�` is EF BF BD, U+FFFD: the text was decoded lossily, saved, and then misread. + assert_eq!(decode(W, "안녕�"), Err(Error::Damaged(2))); + assert_eq!(decode(W, "café�"), Err(Error::Damaged(4))); } #[test] fn inspects() { - for text in ["안녕하세요", "hello_world", "café", "naïve", "©2026", "100% done", ""] { + for text in [ + "안녕하세요", "hello_world", "café", "naïve", "©2026", "100% done", "", + // A lead byte before punctuation in 80-BF is not a complete sequence. + "café’s", "résumé…", "„Café“", "«Café»", "naïve’s", "×÷", "España°", "naïve �", + ] { assert_eq!(inspect(W, text), None, "{text}"); } assert!(matches!(inspect(W, "안녕"), Some(Ok((s, _))) if s == "안녕")); + assert!(matches!(inspect(W, "·"), Some(Ok((s, _))) if s == "·")); + assert!(matches!(inspect(W, "😀"), Some(Ok((s, _))) if s == "😀")); assert!(matches!(inspect(W, "안ë…"), Some(Err(_)))); let lossy = inspect(W, "안�").unwrap().unwrap_err(); assert!(lossy.starts_with("lossy: ")); + // U+FFFD in the result: lossy, and no "decoded twice" note on what is left. + for text in ["안녕�", "café�"] { + let e = inspect(W, text).unwrap().unwrap_err(); + assert!(e.starts_with("lossy: the decoded text contains U+FFFD") && !e.contains("twice"), "{text}"); + } // Decoded twice: the first pass gives mojibake again, and the reason says so. let twice = misread(&misread("안녕")); let (once, reason) = inspect(W, &twice).unwrap().unwrap(); diff --git a/tests/cli.rs b/tests/cli.rs index f8b0be5..92c5f01 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -54,6 +54,18 @@ fn decode_failure_writes_nothing_to_stdout() { } } +#[test] +fn replacement_character_in_the_result_fails() { + for (kind, text) in [("percent", "%EC%95%88%EF%BF%BD"), ("underscore-hex", "_EF_BF_BD")] { + let out = hanfix(&["decode", "--type", kind, text], None); + assert_eq!(out.status.code(), Some(1), "{text}"); + assert!(out.stdout.is_empty(), "{text}"); + let err = String::from_utf8(out.stderr).unwrap(); + assert!(err.contains("lossy: the decoded text contains U+FFFD"), "{text}: {err}"); + } + assert!(stdout(&["inspect", "%EC%95%88%EF%BF%BD"]).contains("Status: invalid\nEncoding: UTF-8\nReason: lossy: ")); +} + #[test] fn inspect_never_fails() { for text in ["hello_world", "100% done", "_ED_8A", "%GG", "트리"] { @@ -113,6 +125,7 @@ fn mojibake() { ("안\u{FFFD}", "lossy: U+FFFD at character 3"), ("안녕 안녕", "'안' (U+C548) at character 0 is not in windows-1252"), ("안ë…", "incomplete UTF-8 sequence at character 3"), + ("안녕�", "lossy: the decoded text contains U+FFFD at character 2"), ] { let out = hanfix(&with(&w, text), None); assert_eq!(out.status.code(), Some(1), "{text}"); @@ -124,7 +137,10 @@ fn mojibake() { assert!(report.starts_with("Detected: mojibake\nStatus: valid\nEncoding: UTF-8 read as windows-1252\n"), "{report}"); assert!(report.ends_with("Candidate: 안녕\n"), "{report}"); assert!(stdout(&["inspect", "안\u{FFFD}"]).contains("Status: invalid\nEncoding: UTF-8 read as windows-1252\nReason: lossy: ")); - for text in ["안녕하세요", "hello world", "café"] { + let report = stdout(&["inspect", "안녕�"]); + assert!(report.contains("Status: invalid\nEncoding: UTF-8 read as windows-1252\nReason: lossy: "), "{report}"); + assert!(!report.contains("Candidate") && !report.contains("twice"), "{report}"); + for text in ["안녕하세요", "hello world", "café", "café’s", "résumé…", "„Café“", "«Café»"] { assert!(stdout(&["inspect", text]).starts_with("Detected: none\n"), "{text}"); } for args in [