From 1095fde49f315a7bfb15893069b044008fcd250e Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 18:29:10 +0900 Subject: [PATCH 1/3] feat: inspect and decode underscore-hex and percent escaped text --- README.ko.md | 74 +++++++++++++- README.md | 76 +++++++++++++- src/encoding.rs | 255 +++++++++++++++++++++++++++++++++++++++++++++++ src/main.rs | 104 +++++++++++++++++-- tests/cli.rs | 96 ++++++++++++++++++ tests/shells.ps1 | 27 +++++ 6 files changed, 619 insertions(+), 13 deletions(-) create mode 100644 src/encoding.rs create mode 100644 tests/cli.rs diff --git a/README.ko.md b/README.ko.md index c2204f4..c645110 100644 --- a/README.ko.md +++ b/README.ko.md @@ -191,12 +191,79 @@ hanfix --to en 한글 -> gksrmf **방향은 자동으로 정합니다.** 한글이 하나라도 있으면 영문 키로, 없으면 한글로 바꿉니다. 자동 판단이 틀리면 `--to ko` 또는 `--to en`으로 방향을 정하세요. +### 이스케이프된 한글 되돌리기 + +파일 이름이나 URL에 한글이 `트리` 대신 `_ED_8A_B8_EB_A6_AC`나 `%ED%8A%B8%EB%A6%AC`처럼 UTF-8 바이트로 +이스케이프되어 들어 있을 때가 있습니다. `hanfix inspect`는 무엇이 들어 있는지 설명하고, `hanfix decode`는 +지정한 형식으로만 되돌립니다. + +```console +$ hanfix inspect _ED_8A_B8_EB_A6_AC +Detected: underscore-hex +Status: valid +Encoding: UTF-8 +Reason: 6 escaped bytes are valid UTF-8; other characters are kept + +Original: _ED_8A_B8_EB_A6_AC +Candidate: 트리 +$ hanfix decode --type underscore-hex _ED_8A_B8_EB_A6_AC +트리 +$ hanfix decode --type percent %ED%8A%B8%EB%A6%AC +트리 +$ hanfix decode --type underscore-hex report__ED_8A_B8.pdf +report_트.pdf +$ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex +트 +``` + +- **underscore-hex** (`_XX`): `_` 뒤에 대문자 16진수 두 자리, 바이트 `80`-`FF`만 해당합니다. ASCII는 이스케이프할 + 필요가 없으므로 `backup_2026_10_29`, `hello_bed`, `report__ED_8A_B8`의 첫 `_`는 그대로 두고, `__`를 공백으로 + 바꾸지도 않습니다. +- **percent** (`%XX`): URL 인코딩입니다. 모든 바이트, 대소문자 모두 받습니다(`%20`은 공백). 뒤에 16진수 두 + 자리가 없는 `%`는 글자로 둡니다(`100% done`). `+`는 그대로 `+`입니다. 공백을 뜻하는 것은 HTML 폼 데이터뿐입니다. +- 이어진 이스케이프는 하나의 바이트열로 보고, 완전하고 유효한 UTF-8이어야 합니다. 하나라도 아니면(`_ED_8A`는 + 3바이트 글자의 앞 2바이트) `decode`는 아무것도 출력하지 않고, 이유를 표준 오류에 쓰고, 종료 코드 1로 끝납니다. + `�`로 바꿔 넣지 않습니다. +- `decode`는 결과만 출력합니다. 이스케이프가 없는 글자는 그대로 나오고 종료 코드는 0입니다. `inspect`는 일반 + 글자에서 실패하지 않고 아무것도 바꾸지 않습니다. +- `inspect`와 `decode`는 첫 번째 인자일 때만 하위 명령입니다. `hanfix -- inspect`와 `hf`는 지금처럼 글자를 변환합니다. +- 유효한 UTF-8이라는 것은 바이트가 글자를 이룬다는 뜻일 뿐, 원래 뜻한 글이라는 보장은 아닙니다. 그래서 hanfix는 + 스스로 되돌리지 않습니다. 형식은 사용자가 정하고, `inspect`로 후보를 먼저 볼 수 있습니다. +- 모지바케(UTF-8을 다른 인코딩으로 읽은 글자, `안녕` 대신 `안녕`)는 아직 지원하지 않습니다. + +**이스케이프된 이름의 파일 여러 개를 한꺼번에 바꾸기 (PowerShell).** 함수를 정의한 뒤 폴더에서 +`Rename-Decoded`를 실행하면 `old -> new` 목록만 보여 줍니다. 실제로 바꾸려면 `-Apply`를 붙입니다. 되돌릴 수 +없는 이름, 바뀌지 않는 이름, 파일 이름이 될 수 없는 결과, 이미 있는 파일을 덮어쓰게 되는 경우는 건너뜁니다. + +```powershell +function Rename-Decoded { + param([string]$Path = '.', [ValidateSet('underscore-hex', 'percent')][string]$Type = 'underscore-hex', [switch]$Apply) + $enc = [Console]::OutputEncoding + [Console]::OutputEncoding = [Text.Encoding]::UTF8 + try { + foreach ($f in Get-ChildItem -LiteralPath $Path -File) { + $new = (hanfix decode --type $Type -- $f.Name 2>$null) -join "`n" + if ($LASTEXITCODE -ne 0) { Write-Warning "skip $($f.Name): does not decode"; continue } + if ($new -eq $f.Name) { continue } + if ($new -in '.', '..' -or $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { + Write-Warning "skip $($f.Name): '$new' is not a file name"; continue + } + if (Test-Path -LiteralPath (Join-Path $f.DirectoryName $new)) { Write-Warning "skip $($f.Name): $new exists"; continue } + "$($f.Name) -> $new" + if ($Apply) { Rename-Item -LiteralPath $f.FullName -NewName $new } + } + } finally { [Console]::OutputEncoding = $enc } +} +``` + ### 옵션 ``` usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --init {bash,zsh,fish,powershell} [--alias NAME] hanfix --update + hanfix inspect [--] [TEXT ...] + hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] ``` | 옵션 | 설명 | @@ -208,10 +275,11 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] | `--init SHELL` | `hf` 함수를 정의하는 셸 코드를 출력합니다. | | `--alias NAME` | `--init`이 만드는 함수 이름입니다. 기본값은 `hf`입니다. | | `--` | 옵션의 끝을 표시합니다. 이후는 모두 글자로 봅니다. | +| `--type TYPE` | `decode` 전용: `underscore-hex` 또는 `percent`. 꼭 지정해야 합니다. | - 옵션은 맨 앞에서만 읽습니다. 나머지는 전부 변환할 글자로 봅니다. 그래서 `hanfix ㅣㄴ -ㅁ`의 `-ㅁ`은 옵션이 아니라 글자(`-a`)입니다. - `-`로 시작하는 영문 글자를 바꾸려면 앞에 `--`를 붙이세요: `hanfix -- -rk` → `-가`. -- 종료 코드는 성공 0, 잘못된 사용 2입니다. +- 종료 코드는 성공 0, `decode` 실패 1, 잘못된 사용 2입니다. ### 입력 중인 줄 바로 고치기 (선택) @@ -260,9 +328,9 @@ bind -x '"\eh": READLINE_LINE=$(hanfix -- "$READLINE_LINE"); READLINE_POINT=${#R ## 개발 ```sh -cargo test # 조합 사례, 한글 음절 11,172자 전체의 왕복 변환, 셸 코드 +cargo test # 조합 사례, 한글 음절 11,172자 전체의 왕복 변환, 셸 코드, decode 사례, CLI sh tests/shells.sh # bash, zsh, fish에서 hf가 실제로 고친 명령을 실행하는지 (설치된 hanfix 사용) -pwsh tests/shells.ps1 +pwsh tests/shells.ps1 # PowerShell에서 같은 확인과 위의 Rename-Decoded 예시 ``` 빠른 설치의 세팅 단계도 테스트합니다. 코드 블록 앞의 `` 주석이 대상을 표시하고, CI가 공통 diff --git a/README.md b/README.md index 9f29875..d2487f8 100644 --- a/README.md +++ b/README.md @@ -195,12 +195,81 @@ hanfix --to en 한글 -> gksrmf **The direction is automatic.** Text with any Hangul goes back to English keys; anything else is composed into Hangul. If the guess is wrong, use `--to ko` or `--to en`. +### Decode escaped Korean text + +File names and URLs sometimes carry Korean as escaped UTF-8 bytes: `_ED_8A_B8_EB_A6_AC` or +`%ED%8A%B8%EB%A6%AC` instead of `트리`. `hanfix inspect` explains what it finds; `hanfix decode` turns it back, +using only the type you name. + +```console +$ hanfix inspect _ED_8A_B8_EB_A6_AC +Detected: underscore-hex +Status: valid +Encoding: UTF-8 +Reason: 6 escaped bytes are valid UTF-8; other characters are kept + +Original: _ED_8A_B8_EB_A6_AC +Candidate: 트리 +$ hanfix decode --type underscore-hex _ED_8A_B8_EB_A6_AC +트리 +$ hanfix decode --type percent %ED%8A%B8%EB%A6%AC +트리 +$ hanfix decode --type underscore-hex report__ED_8A_B8.pdf +report_트.pdf +$ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex +트 +``` + +- **underscore-hex** (`_XX`): `_` and two upper-case hex digits, for bytes `80`-`FF` only. ASCII never needs + escaping, so `backup_2026_10_29`, `hello_bed` and the first `_` of `report__ED_8A_B8` stay as they are, and + `__` never becomes a space. +- **percent** (`%XX`): URL encoding, any byte, either case (`%20` is a space). A `%` without two hex digits + after it is text (`100% done`). `+` stays `+`; it means a space only in HTML form data. +- Consecutive escapes form one byte sequence that must be complete, valid UTF-8. If any sequence in the + text is not (`_ED_8A` is two bytes of a three-byte character), `decode` prints nothing, writes the reason to + standard error and exits with 1. It never substitutes `�`. +- `decode` prints only the result; text with no escapes comes back unchanged with exit 0. `inspect` never + fails on plain text and changes nothing. +- `inspect` and `decode` are subcommands only as the first argument. `hanfix -- inspect` and `hf` still convert + the word as typed text. +- Valid UTF-8 only means the bytes form characters, not that they are the text someone meant. That is why + hanfix never decodes on its own: you name the type, and `inspect` shows the candidate first. +- Mojibake (UTF-8 read as another encoding, `안녕` for `안녕`) is not handled yet. + +**Rename many files with escaped names (PowerShell).** Define the function, then run `Rename-Decoded` in a +folder: it only lists `old -> new`. Add `-Apply` to rename. It skips a name that does not decode, does not +change, would not be a valid file name, or would overwrite an existing file. + + +```powershell +function Rename-Decoded { + param([string]$Path = '.', [ValidateSet('underscore-hex', 'percent')][string]$Type = 'underscore-hex', [switch]$Apply) + $enc = [Console]::OutputEncoding + [Console]::OutputEncoding = [Text.Encoding]::UTF8 + try { + foreach ($f in Get-ChildItem -LiteralPath $Path -File) { + $new = (hanfix decode --type $Type -- $f.Name 2>$null) -join "`n" + if ($LASTEXITCODE -ne 0) { Write-Warning "skip $($f.Name): does not decode"; continue } + if ($new -eq $f.Name) { continue } + if ($new -in '.', '..' -or $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { + Write-Warning "skip $($f.Name): '$new' is not a file name"; continue + } + if (Test-Path -LiteralPath (Join-Path $f.DirectoryName $new)) { Write-Warning "skip $($f.Name): $new exists"; continue } + "$($f.Name) -> $new" + if ($Apply) { Rename-Item -LiteralPath $f.FullName -NewName $new } + } + } finally { [Console]::OutputEncoding = $enc } +} +``` + ### Options ``` usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --init {bash,zsh,fish,powershell} [--alias NAME] hanfix --update + hanfix inspect [--] [TEXT ...] + hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] ``` | Option | Description | @@ -212,11 +281,12 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] | `--init SHELL` | Print the shell code that defines `hf`. | | `--alias NAME` | Name of the function `--init` defines. Default `hf`. | | `--` | End of options. Everything after is text. | +| `--type TYPE` | `decode` only: `underscore-hex` or `percent`. Required. | - Options are read only at the start; the rest is text to convert. So the `-ㅁ` in `hanfix ㅣㄴ -ㅁ` is text (`-a`), not an option. - To convert English text that starts with `-`, put `--` first: `hanfix -- -rk` → `-가`. -- Exit status: 0 on success, 2 on a usage error. +- Exit status: 0 on success, 1 when `decode` fails, 2 on a usage error. ### Fix the line you are typing (optional) @@ -268,9 +338,9 @@ Characters that are not on the layout (digits, spaces, symbols) are left as they ## Development ```sh -cargo test # composition cases, round trip of all 11,172 Hangul syllables, shell code +cargo test # composition cases, round trip of all 11,172 Hangul syllables, shell code, decode cases, the CLI sh tests/shells.sh # in bash, zsh and fish, hf actually runs the fixed command (uses the installed hanfix) -pwsh tests/shells.ps1 +pwsh tests/shells.ps1 # the same in PowerShell, and the Rename-Decoded example above ``` The setup steps under Quick install are tested too. A `` comment before a code block marks it, and CI runs diff --git a/src/encoding.rs b/src/encoding.rs new file mode 100644 index 0000000..8b13659 --- /dev/null +++ b/src/encoding.rs @@ -0,0 +1,255 @@ +//! Escaped text back to UTF-8: `_XX` (underscore-hex) and `%XX` (percent) byte escapes. +//! +//! Only escape runs change; every other character is copied as it is. A run of escapes is one byte +//! sequence that must be complete, valid UTF-8, or the whole decode fails: nothing is replaced with +//! U+FFFD and nothing is guessed. + +use std::fmt; + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Kind { + UnderscoreHex, + Percent, +} + +pub const KINDS: [Kind; 2] = [Kind::UnderscoreHex, Kind::Percent]; + +impl Kind { + pub fn name(self) -> &'static str { + match self { + Kind::UnderscoreHex => "underscore-hex", + Kind::Percent => "percent", + } + } + + pub fn parse(name: &str) -> Option { + KINDS.into_iter().find(|k| k.name() == name) + } + + /// The byte escaped at `i`, if an escape starts there. + /// + /// Underscore-hex takes upper-case digits and bytes 0x80-0xFF only: ASCII never needs the escape, and + /// ordinary names are full of `_20`, `_10` or `_be` (`backup_2026_10_29`, `hello_bed`) that are not one. + /// Percent follows RFC 3986: any byte, either case. A marker without two hex digits after it is text. + fn escape_at(self, b: &[u8], i: usize) -> Option { + let (marker, hex): (u8, fn(u8) -> bool) = match self { + Kind::UnderscoreHex => (b'_', |c| c.is_ascii_digit() || (b'A'..=b'F').contains(&c)), + Kind::Percent => (b'%', |c| c.is_ascii_hexdigit()), + }; + if b.get(i) != Some(&marker) || !b.get(i + 1).is_some_and(|&c| hex(c)) || !b.get(i + 2).is_some_and(|&c| hex(c)) { + return None; + } + let v = u8::from_str_radix(std::str::from_utf8(&b[i + 1..i + 3]).ok()?, 16).ok()?; + (self == Kind::Percent || v >= 0x80).then_some(v) + } +} + +#[derive(Debug, PartialEq)] +pub struct DecodeError { + pub kind: Kind, + /// The escape run that failed, as written in the input. + pub run: String, + /// Characters before the run. + pub at: usize, + pub incomplete: bool, +} + +impl fmt::Display for DecodeError { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + let what = if self.incomplete { "incomplete UTF-8 sequence" } else { "invalid UTF-8" }; + write!(f, "{what} in {} escape '{}' at character {}", self.kind.name(), self.run, self.at) + } +} + +#[derive(Debug, PartialEq)] +pub struct Decoded { + pub text: String, + /// Escaped bytes found; 0 means the text had nothing to decode and is returned unchanged. + pub bytes: usize, +} + +pub fn decode(kind: Kind, text: &str) -> Result { + let b = text.as_bytes(); + let (mut out, mut bytes, mut i, mut copied) = (String::with_capacity(text.len()), 0, 0, 0); + while i < b.len() { + if kind.escape_at(b, i).is_none() { + i += 1; + continue; + } + // Markers and hex digits are ASCII, so `i` is a char boundary here. + out.push_str(&text[copied..i]); + let start = i; + let mut run = Vec::new(); + while let Some(v) = kind.escape_at(b, i) { + run.push(v); + i += 3; + } + bytes += run.len(); + match std::str::from_utf8(&run) { + Ok(s) => out.push_str(s), + Err(e) => { + return Err(DecodeError { + kind, + run: text[start..i].to_string(), + at: text[..start].chars().count(), + incomplete: e.error_len().is_none(), + }) + } + } + copied = i; + } + out.push_str(&text[copied..]); + Ok(Decoded { text: out, bytes }) +} + +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Status { + Valid, + /// Valid on its own, but the text also has escapes of the other kind, which this one leaves alone. + Ambiguous, + Invalid, +} + +impl Status { + fn name(self) -> &'static str { + match self { + Status::Valid => "valid", + Status::Ambiguous => "ambiguous", + Status::Invalid => "invalid", + } + } +} + +#[derive(Debug, PartialEq)] +pub struct Candidate { + pub kind: Kind, + pub status: Status, + /// `None` when the escapes do not decode. + pub decoded: Option, + pub reason: String, +} + +/// Every kind of escape found in `text`, each decoded on its own. Empty for plain text. +pub fn inspect(text: &str) -> Vec { + let found: Vec<_> = KINDS.into_iter().map(|k| (k, decode(k, text))).filter(|(_, r)| r.as_ref().map_or(true, |d| d.bytes > 0)).collect(); + let several = found.len() > 1; + found + .into_iter() + .map(|(kind, r)| match r { + Ok(d) => Candidate { + kind, + status: if several { Status::Ambiguous } else { Status::Valid }, + reason: if several { + format!("{} escaped bytes are valid UTF-8, but the text also has other escapes, which this type keeps", d.bytes) + } else { + format!("{} escaped bytes are valid UTF-8; other characters are kept", d.bytes) + }, + decoded: Some(d.text), + }, + Err(e) => Candidate { kind, status: Status::Invalid, decoded: None, reason: e.to_string() }, + }) + .collect() +} + +/// The `hanfix inspect` report. +pub fn report(text: &str) -> String { + let found = inspect(text); + if found.is_empty() { + return format!( + "Detected: none\nReason: no underscore-hex or percent escape found; the text is kept as is\n\nOriginal: {text}\n" + ); + } + let blocks: Vec = found + .iter() + .map(|c| { + let mut s = format!( + "Detected: {}\nStatus: {}\nEncoding: UTF-8\nReason: {}\n\nOriginal: {text}\n", + c.kind.name(), + c.status.name(), + c.reason + ); + if let Some(d) = &c.decoded { + s += &format!("Candidate: {d}\n"); + } + s + }) + .collect(); + blocks.join("\n") +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ok(kind: Kind, text: &str) -> String { + decode(kind, text).unwrap().text + } + + #[test] + fn decodes() { + assert_eq!(ok(Kind::UnderscoreHex, "_ED_8A_B8"), "트"); + assert_eq!(ok(Kind::UnderscoreHex, "_ED_8A_B8_EB_A6_AC"), "트리"); + assert_eq!(ok(Kind::Percent, "%ED%8A%B8"), "트"); + assert_eq!(ok(Kind::Percent, "%ED%8A%B8%EB%A6%AC"), "트리"); + assert_eq!(ok(Kind::Percent, "%ed%8a%b8"), "트"); + assert_eq!(ok(Kind::Percent, "a%20b"), "a b"); + } + + #[test] + fn keeps_the_text_around_escapes() { + assert_eq!(ok(Kind::UnderscoreHex, "file__ED_8A_B8.txt"), "file_트.txt"); + assert_eq!(ok(Kind::UnderscoreHex, "report__ED_8A_B8.pdf"), "report_트.pdf"); + assert_eq!(ok(Kind::UnderscoreHex, "_ED_8A_B8x_EB_A6_AC"), "트x리"); + assert_eq!(ok(Kind::UnderscoreHex, "a__b___ED_8A_B8__"), "a__b__트__"); + assert_eq!(ok(Kind::Percent, "%ED%8A%B8%EB%A6%AC_ED_8A_B8"), "트리_ED_8A_B8"); + assert_eq!(ok(Kind::UnderscoreHex, "%ED%8A%B8_EB_A6_AC"), "%ED%8A%B8리"); + assert_eq!(ok(Kind::Percent, "a+b%2Bc"), "a+b+c"); + } + + #[test] + fn plain_text_is_unchanged() { + for text in ["hello_world", "backup_2026_10_29", "100% done", "normal.txt", "hello_bed", "_", "%", "%G", "%GG", "_ed_8a", "트리", ""] { + for kind in KINDS { + if kind == Kind::Percent && text == "%GG" { + continue; + } + assert_eq!(decode(kind, text), Ok(Decoded { text: text.to_string(), bytes: 0 }), "{text}"); + } + } + assert_eq!(ok(Kind::Percent, "%GG"), "%GG"); + } + + #[test] + fn bad_sequences_fail() { + let e = decode(Kind::UnderscoreHex, "_ED_8A").unwrap_err(); + assert!(e.incomplete && e.run == "_ED_8A" && e.at == 0); + let e = decode(Kind::Percent, "x%ED%8A").unwrap_err(); + assert!(e.incomplete && e.run == "%ED%8A" && e.at == 1); + let e = decode(Kind::Percent, "%FF").unwrap_err(); + assert!(!e.incomplete); + let e = decode(Kind::UnderscoreHex, "트_8A").unwrap_err(); + assert_eq!((e.at, e.incomplete), (1, false)); + // A good run does not save a later bad one. + assert!(decode(Kind::UnderscoreHex, "_ED_8A_B8 _ED").is_err()); + assert_eq!( + decode(Kind::UnderscoreHex, "_ED_8A").unwrap_err().to_string(), + "incomplete UTF-8 sequence in underscore-hex escape '_ED_8A' at character 0" + ); + } + + #[test] + fn inspects() { + assert!(inspect("hello_world").is_empty()); + assert_eq!( + report("_ED_8A_B8_EB_A6_AC"), + "Detected: underscore-hex\nStatus: valid\nEncoding: UTF-8\n\ + Reason: 6 escaped bytes are valid UTF-8; other characters are kept\n\n\ + Original: _ED_8A_B8_EB_A6_AC\nCandidate: 트리\n" + ); + let c = inspect("_ED_8A"); + assert_eq!((c.len(), c[0].status, c[0].decoded.as_deref()), (1, Status::Invalid, None)); + let c = inspect("%ED%8A%B8_EB_A6_AC"); + assert!(c.len() == 2 && c.iter().all(|c| c.status == Status::Ambiguous)); + assert!(report("100% done").starts_with("Detected: none\n")); + } +} diff --git a/src/main.rs b/src/main.rs index f1ccdd6..59d9fb3 100644 --- a/src/main.rs +++ b/src/main.rs @@ -1,5 +1,7 @@ //! hanfix: fix commands and text typed in the wrong keyboard layout (Korean 2-set <-> QWERTY). +mod encoding; + use std::io::{self, IsTerminal, Read, Write}; use std::process::ExitCode; @@ -248,6 +250,8 @@ fn init_script(shell: &str, name: &str) -> String { const USAGE: &str = "usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] hanfix --init {bash,zsh,fish,powershell} [--alias NAME] hanfix --update + hanfix inspect [--] [TEXT ...] + hanfix decode --type {underscore-hex,percent} [--] [TEXT ...] "; fn help() -> String { @@ -281,14 +285,21 @@ shell setup (add one line to your shell's start-up file): then, after a mistyped command, type hf (or ㅗㄹ); hf -y runs it without asking. -exit status: 0 on success, 2 on a usage error. +escaped text (only as the first argument; \"hanfix -- inspect\" converts the word): + hanfix inspect _ED_8A_B8_EB_A6_AC what the escapes are and what they decode to + hanfix decode --type underscore-hex _ED_8A_B8 트 + hanfix decode --type percent %ED%8A%B8%EB%A6%AC 트리 + decode prints only the result; text that is not an escape is kept. An incomplete or invalid + UTF-8 sequence prints nothing and fails. + +exit status: 0 on success, 1 when decode fails, 2 on a usage error. ", SHELLS.join(", ") ) } fn usage_error(msg: &str) -> ExitCode { - eprint!("{USAGE}hanfix: error: {msg}\n"); + eprintln!("{USAGE}hanfix: error: {msg}"); ExitCode::from(2) } @@ -328,7 +339,87 @@ fn update() -> ExitCode { } } +fn read_stdin() -> Result, ExitCode> { + let mut buf = Vec::new(); + io::stdin().read_to_end(&mut buf).map_err(|e| { + eprintln!("hanfix: error: cannot read standard input: {e}"); + ExitCode::FAILURE + })?; + Ok(buf) +} + +/// `hanfix inspect` and `hanfix decode`. Their text is read strictly as UTF-8: a decoder that +/// replaced bad input bytes would hide the very damage it is asked about. +fn escaped(cmd: &str, mut args: Vec) -> ExitCode { + let mut kind = None; + while let Some(a) = args.first().cloned() { + if a == "--" { + args.remove(0); + break; + } + if a == "-h" || a == "--help" { + print!("{USAGE}"); + return ExitCode::SUCCESS; + } + let (opt, val) = a.split_once('=').map_or((a.as_str(), None), |(o, v)| (o, Some(v.to_string()))); + if cmd != "decode" || opt != "--type" { + if a.starts_with('-') && a.len() > 1 && a.is_ascii() { + return usage_error(&format!("{cmd}: unknown option '{a}' (put -- before text that starts with -)")); + } + break; + } + args.remove(0); + let val = match val { + Some(v) => v, + None if !args.is_empty() => args.remove(0), + None => return usage_error("--type needs a value"), + }; + match encoding::Kind::parse(&val) { + Some(k) => kind = Some(k), + None => return usage_error(&format!("--type: choose underscore-hex or percent, not '{val}'")), + } + } + if cmd == "decode" && kind.is_none() { + return usage_error("decode needs --type underscore-hex or --type percent"); + } + let text = if !args.is_empty() { + args.join(" ") + } else if io::stdin().is_terminal() { + return usage_error(&format!("{cmd}: no text given (pass it as arguments or on standard input)")); + } else { + let buf = match read_stdin() { + Ok(b) => b, + Err(code) => return code, + }; + match String::from_utf8(buf) { + Ok(s) => s.trim_end_matches(['\n', '\r']).to_string(), + Err(e) => { + eprintln!("hanfix: error: standard input is not UTF-8 (at byte {})", e.utf8_error().valid_up_to()); + return ExitCode::FAILURE; + } + } + }; + let out = match kind { + None => encoding::report(&text), + Some(k) => match encoding::decode(k, &text) { + Ok(d) => d.text + "\n", + Err(e) => { + eprintln!("hanfix: error: {e}"); + return ExitCode::FAILURE; + } + }, + }; + let _ = io::stdout().write_all(out.as_bytes()); + ExitCode::SUCCESS +} + fn run(mut args: Vec) -> ExitCode { + // Subcommands only in the first place, so "hanfix -- inspect" and the shell functions' + // "hanfix --to en -- $last" still convert the word. + if let Some(cmd @ ("inspect" | "decode")) = args.first().map(String::as_str) { + let cmd = cmd.to_string(); + return escaped(&cmd, args.split_off(1)); + } let (mut to, mut shell, mut name) = (None, None, "hf".to_string()); // Leading options only: the text itself may start with "-" ("-ㅁ" is a mistyped "-a"). while let Some(a) = args.first().cloned() { @@ -394,11 +485,10 @@ fn run(mut args: Vec) -> ExitCode { } else if io::stdin().is_terminal() { return usage_error("no text given (pass it as arguments or on standard input)"); } else { - let mut buf = Vec::new(); - if let Err(e) = io::stdin().read_to_end(&mut buf) { - eprintln!("hanfix: error: cannot read standard input: {e}"); - return ExitCode::FAILURE; - } + let buf = match read_stdin() { + Ok(b) => b, + Err(code) => return code, + }; let text = String::from_utf8_lossy(&buf); fix(text.trim_end_matches(['\n', '\r']), to) + "\n" }; diff --git a/tests/cli.rs b/tests/cli.rs new file mode 100644 index 0000000..5f2969d --- /dev/null +++ b/tests/cli.rs @@ -0,0 +1,96 @@ +//! The binary as a shell sees it: arguments, standard input, standard output and error, exit status. + +use std::io::Write; +use std::process::{Command, Output, Stdio}; + +fn hanfix(args: &[&str], stdin: Option<&[u8]>) -> Output { + let mut child = Command::new(env!("CARGO_BIN_EXE_hanfix")) + .args(args) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .unwrap(); + let mut pipe = child.stdin.take().unwrap(); + pipe.write_all(stdin.unwrap_or_default()).unwrap(); + drop(pipe); + child.wait_with_output().unwrap() +} + +fn stdout(args: &[&str]) -> String { + let out = hanfix(args, None); + assert!(out.status.success(), "{args:?}: {}", String::from_utf8_lossy(&out.stderr)); + String::from_utf8(out.stdout).unwrap() +} + +#[test] +fn decode_prints_only_the_result() { + assert_eq!(stdout(&["decode", "--type", "underscore-hex", "_ED_8A_B8_EB_A6_AC"]), "트리\n"); + assert_eq!(stdout(&["decode", "--type=percent", "%ED%8A%B8%EB%A6%AC"]), "트리\n"); + assert_eq!(stdout(&["decode", "--type", "underscore-hex", "--", "file__ED_8A_B8.txt"]), "file_트.txt\n"); + assert_eq!(stdout(&["decode", "--type", "underscore-hex", "backup_2026_10_29"]), "backup_2026_10_29\n"); + assert_eq!(stdout(&["decode", "--type", "percent", "100%", "done"]), "100% done\n"); +} + +#[test] +fn decode_reads_stdin() { + let out = hanfix(&["decode", "--type", "underscore-hex"], Some(b"_ED_8A_B8\n")); + assert!(out.status.success()); + assert_eq!(out.stdout, "트\n".as_bytes()); + let out = hanfix(&["inspect"], Some(b"\xff_ED")); + assert_eq!(out.status.code(), Some(1)); + assert!(out.stdout.is_empty() && String::from_utf8_lossy(&out.stderr).contains("not UTF-8")); +} + +#[test] +fn decode_failure_writes_nothing_to_stdout() { + for (kind, text) in [("underscore-hex", "_ED_8A"), ("percent", "%ED%8A"), ("percent", "ok %ED%8A%B8 %FF")] { + let out = hanfix(&["decode", "--type", kind, text], None); + assert_eq!(out.status.code(), Some(1), "{text}"); + assert!(out.stdout.is_empty(), "{text}"); + assert!(String::from_utf8_lossy(&out.stderr).starts_with("hanfix: error: "), "{text}"); + } +} + +#[test] +fn inspect_never_fails() { + for text in ["hello_world", "100% done", "_ED_8A", "%GG", "트리"] { + let out = stdout(&["inspect", text]); + assert!(out.starts_with("Detected: "), "{text}: {out}"); + } + assert!(stdout(&["inspect", "_ED_8A_B8"]).ends_with("Candidate: 트\n")); +} + +#[test] +fn usage_errors() { + for args in [ + &["decode", "_ED_8A_B8"][..], + &["decode", "--type", "base64", "x"], + &["decode", "--type"], + &["inspect", "--type", "percent", "x"], + &["inspect", "-x"], + ] { + assert_eq!(hanfix(args, None).status.code(), Some(2), "{args:?}"); + } +} + +// The words still convert when they are not the first argument: after "--", after an option, and the +// way the shell functions call hanfix ("hanfix --to en -- $last"). +#[test] +fn subcommand_words_as_text() { + let inspect_ko = stdout(&["--to", "ko", "--", "inspect"]); + assert_eq!(stdout(&["--", "inspect"]), inspect_ko); + assert_eq!(stdout(&["--", "decode"]), stdout(&["--to", "ko", "--", "decode"])); + assert!(!inspect_ko.starts_with("Detected")); + assert_eq!(stdout(&["--to", "en", "--", inspect_ko.trim_end()]), "inspect\n"); + assert_eq!(stdout(&["--to", "en", "--", "decode --type percent x"]), "decode --type percent x\n"); + assert_eq!(stdout(&["ㅑㅜ넫ㅊㅅ"]), "inspect\n"); +} + +#[test] +fn keyboard_conversion_unchanged() { + assert_eq!(stdout(&["ㅣㄴ", "-ㅁ"]), "ls -a\n"); + assert_eq!(stdout(&["dkssudgktpdy"]), "안녕하세요\n"); + let out = hanfix(&[], Some("햣 ㄴㅅㅁ션\n".as_bytes())); + assert_eq!(out.stdout, b"git status\n"); +} diff --git a/tests/shells.ps1 b/tests/shells.ps1 index ef35067..ad29fd5 100644 --- a/tests/shells.ps1 +++ b/tests/shells.ps1 @@ -31,3 +31,30 @@ Add-History -InputObject ([pscustomobject]@{ $out = (hf -y) -join '|' if ($out -ne 'cdg_run:echo hanfix-ok|hanfix-ok') { throw "FAIL powershell __cdg_run: got '$out'" } Write-Host 'ok powershell with codigest' + +# The README's Rename-Decoded example, run as written: a preview renames nothing, -Apply renames only +# what decodes to a new, free name. The Korean README carries the same code. +$readme = Get-Content -Raw -Encoding utf8 (Join-Path $PSScriptRoot '..' 'README.md') +if ($readme -notmatch '(?s)\s*```powershell\r?\n(.*?)```') { throw 'FAIL rename example: block not found' } +$example = $Matches[1] +if (-not (Get-Content -Raw -Encoding utf8 (Join-Path $PSScriptRoot '..' 'README.ko.md')).Contains($example)) { + throw 'FAIL rename example: README.ko.md differs' +} +Invoke-Expression $example +$dir = Join-Path ([IO.Path]::GetTempPath()) "hanfix-rename-$PID" +New-Item -ItemType Directory $dir | Out-Null +try { + $tree = [string][char]0xD2B8 + [char]0xB9AC + $ri = [string][char]0xB9AC + '_ED_8A_B8_EB_A6_AC.txt', 'backup_2026_10_29.txt', 'bad_ED_8A.txt', '_EB_A6_AC.md', "$ri.md" | + ForEach-Object { New-Item -ItemType File (Join-Path $dir $_) | Out-Null } + $before = (Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|' + $preview = @(Rename-Decoded -Path $dir 3>$null) + if (($preview -join '|') -ne "_ED_8A_B8_EB_A6_AC.txt -> $tree.txt") { throw "FAIL rename preview: got '$($preview -join '|')'" } + if (((Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|') -ne $before) { throw 'FAIL rename preview changed files' } + Rename-Decoded -Path $dir -Apply 3>$null | Out-Null + $after = (Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|' + $want = ('_EB_A6_AC.md', 'backup_2026_10_29.txt', 'bad_ED_8A.txt', "$ri.md", "$tree.txt" | Sort-Object) -join '|' + if ($after -ne $want) { throw "FAIL rename apply: got '$after'" } +} finally { Remove-Item -LiteralPath $dir -Recurse -Force } +Write-Host 'ok powershell Rename-Decoded example' From b60cb85252dcfd587df6630016a539c6dd1a12bb Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 18:32:58 +0900 Subject: [PATCH 2/3] fix: stop Rename-Decoded on a hanfix without decode; note lower-case _xx in inspect; 0.2.8 --- Cargo.lock | 2 +- Cargo.toml | 2 +- README.ko.md | 5 ++++- README.md | 6 +++++- src/encoding.rs | 24 ++++++++++++++++++++++-- tests/shells.ps1 | 11 +++++++++++ 6 files changed, 44 insertions(+), 6 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d202c24..05ae2ee 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4,4 +4,4 @@ version = 3 [[package]] name = "hanfix" -version = "0.2.7" +version = "0.2.8" diff --git a/Cargo.toml b/Cargo.toml index 7dc181b..2b0940a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "hanfix" -version = "0.2.7" +version = "0.2.8" edition = "2021" rust-version = "1.80" description = "Fix commands and text typed in the wrong keyboard layout (Korean 2-set <-> QWERTY)" diff --git a/README.ko.md b/README.ko.md index c645110..ae692f7 100644 --- a/README.ko.md +++ b/README.ko.md @@ -218,7 +218,7 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - **underscore-hex** (`_XX`): `_` 뒤에 대문자 16진수 두 자리, 바이트 `80`-`FF`만 해당합니다. ASCII는 이스케이프할 필요가 없으므로 `backup_2026_10_29`, `hello_bed`, `report__ED_8A_B8`의 첫 `_`는 그대로 두고, `__`를 공백으로 - 바꾸지도 않습니다. + 바꾸지도 않습니다. `_ed_8a_b8`처럼 소문자로 된 것은 되돌리지 않으며, `inspect`가 이를 보면 안내 한 줄을 덧붙입니다. - **percent** (`%XX`): URL 인코딩입니다. 모든 바이트, 대소문자 모두 받습니다(`%20`은 공백). 뒤에 16진수 두 자리가 없는 `%`는 글자로 둡니다(`100% done`). `+`는 그대로 `+`입니다. 공백을 뜻하는 것은 HTML 폼 데이터뿐입니다. - 이어진 이스케이프는 하나의 바이트열로 보고, 완전하고 유효한 UTF-8이어야 합니다. 하나라도 아니면(`_ED_8A`는 @@ -238,6 +238,9 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex ```powershell function Rename-Decoded { param([string]$Path = '.', [ValidateSet('underscore-hex', 'percent')][string]$Type = 'underscore-hex', [switch]$Apply) + # hanfix 0.2.7 and older read "decode ..." as text to convert and exit 0, which would rename every file. + $probe = (hanfix decode --type percent -- '%41' 2>$null) -join "`n" + if ($LASTEXITCODE -ne 0 -or $probe -ne 'A') { throw 'hanfix too old: update with hanfix --update' } $enc = [Console]::OutputEncoding [Console]::OutputEncoding = [Text.Encoding]::UTF8 try { diff --git a/README.md b/README.md index d2487f8..546614a 100644 --- a/README.md +++ b/README.md @@ -222,7 +222,8 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - **underscore-hex** (`_XX`): `_` and two upper-case hex digits, for bytes `80`-`FF` only. ASCII never needs escaping, so `backup_2026_10_29`, `hello_bed` and the first `_` of `report__ED_8A_B8` stay as they are, and - `__` never becomes a space. + `__` never becomes a space. Lower-case runs such as `_ed_8a_b8` are not decoded; `inspect` adds a note + when it sees one. - **percent** (`%XX`): URL encoding, any byte, either case (`%20` is a space). A `%` without two hex digits after it is text (`100% done`). `+` stays `+`; it means a space only in HTML form data. - Consecutive escapes form one byte sequence that must be complete, valid UTF-8. If any sequence in the @@ -244,6 +245,9 @@ change, would not be a valid file name, or would overwrite an existing file. ```powershell function Rename-Decoded { param([string]$Path = '.', [ValidateSet('underscore-hex', 'percent')][string]$Type = 'underscore-hex', [switch]$Apply) + # hanfix 0.2.7 and older read "decode ..." as text to convert and exit 0, which would rename every file. + $probe = (hanfix decode --type percent -- '%41' 2>$null) -join "`n" + if ($LASTEXITCODE -ne 0 -or $probe -ne 'A') { throw 'hanfix too old: update with hanfix --update' } $enc = [Console]::OutputEncoding [Console]::OutputEncoding = [Text.Encoding]::UTF8 try { diff --git a/src/encoding.rs b/src/encoding.rs index 8b13659..227e187 100644 --- a/src/encoding.rs +++ b/src/encoding.rs @@ -151,19 +151,33 @@ pub fn inspect(text: &str) -> Vec { .collect() } +/// Two `_xx` in a row, each 0x80 or more, with a lower-case hex letter: what an upper-case escape run +/// would be. One such byte alone is never UTF-8, and `hello_bed` should not get the note. +fn has_lower_case_underscore(text: &str) -> bool { + let byte = |w: &[u8]| { + w[0] == b'_' && w[1] >= b'8' && w[1..3].iter().all(|c| c.is_ascii_digit() || (b'a'..=b'f').contains(c)) + }; + text.as_bytes().windows(6).any(|w| byte(&w[..3]) && byte(&w[3..]) && w[1..].iter().any(|c| c.is_ascii_lowercase())) +} + /// The `hanfix inspect` report. pub fn report(text: &str) -> String { let found = inspect(text); + let note = if has_lower_case_underscore(text) { + "Note: lower-case _xx is not treated as an escape (underscore-hex is upper case only)\n" + } else { + "" + }; if found.is_empty() { return format!( - "Detected: none\nReason: no underscore-hex or percent escape found; the text is kept as is\n\nOriginal: {text}\n" + "Detected: none\nReason: no underscore-hex or percent escape found; the text is kept as is\n{note}\nOriginal: {text}\n" ); } let blocks: Vec = found .iter() .map(|c| { let mut s = format!( - "Detected: {}\nStatus: {}\nEncoding: UTF-8\nReason: {}\n\nOriginal: {text}\n", + "Detected: {}\nStatus: {}\nEncoding: UTF-8\nReason: {}\n{note}\nOriginal: {text}\n", c.kind.name(), c.status.name(), c.reason @@ -251,5 +265,11 @@ mod tests { let c = inspect("%ED%8A%B8_EB_A6_AC"); assert!(c.len() == 2 && c.iter().all(|c| c.status == Status::Ambiguous)); assert!(report("100% done").starts_with("Detected: none\n")); + let note = "Note: lower-case _xx is not treated as an escape"; + assert!(report("_ed_8a_b8").contains(note)); + assert!(report("%ED%8A%B8_eb_a6_ac").contains(note)); + for text in ["hello_bed", "backup_2026_10_29", "_ED_8A_B8", "a_7f"] { + assert!(!report(text).contains(note), "{text}"); + } } } diff --git a/tests/shells.ps1 b/tests/shells.ps1 index ad29fd5..1b88c57 100644 --- a/tests/shells.ps1 +++ b/tests/shells.ps1 @@ -56,5 +56,16 @@ try { $after = (Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|' $want = ('_EB_A6_AC.md', 'backup_2026_10_29.txt', 'bad_ED_8A.txt', "$ri.md", "$tree.txt" | Sort-Object) -join '|' if ($after -ne $want) { throw "FAIL rename apply: got '$after'" } + + # An older hanfix (0.2.7) treats "decode ..." as text: the example must stop before listing or renaming. + $real = (Get-Command hanfix -CommandType Application | Select-Object -First 1).Source + function hanfix { & $real -- @args } + try { + $before = (Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|' + $threw = $null + try { Rename-Decoded -Path $dir -Apply 3>$null | Out-Null } catch { $threw = $_.Exception.Message } + if ($threw -ne 'hanfix too old: update with hanfix --update') { throw "FAIL rename with old hanfix: got '$threw'" } + if (((Get-ChildItem -LiteralPath $dir -Name | Sort-Object) -join '|') -ne $before) { throw 'FAIL rename with old hanfix changed files' } + } finally { Remove-Item Function:hanfix } } finally { Remove-Item -LiteralPath $dir -Recurse -Force } Write-Host 'ok powershell Rename-Decoded example' From 7504d53444b06f76b5a721c5a3420cf078eaff74 Mon Sep 17 00:00:00 2001 From: isaac Date: Thu, 29 Oct 2026 18:43:58 +0900 Subject: [PATCH 3/3] fix: keep underscore-hex bytes that start no character as text; safer Rename-Decoded --- README.ko.md | 34 ++++++++++++++++------- README.md | 36 ++++++++++++++++++------- src/encoding.rs | 70 +++++++++++++++++++++++++++++++++++++++++++++--- src/main.rs | 6 ++++- tests/cli.rs | 3 +++ tests/shells.ps1 | 12 +++++++++ 6 files changed, 138 insertions(+), 23 deletions(-) diff --git a/README.ko.md b/README.ko.md index ae692f7..8d0c003 100644 --- a/README.ko.md +++ b/README.ko.md @@ -219,12 +219,19 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex - **underscore-hex** (`_XX`): `_` 뒤에 대문자 16진수 두 자리, 바이트 `80`-`FF`만 해당합니다. ASCII는 이스케이프할 필요가 없으므로 `backup_2026_10_29`, `hello_bed`, `report__ED_8A_B8`의 첫 `_`는 그대로 두고, `__`를 공백으로 바꾸지도 않습니다. `_ed_8a_b8`처럼 소문자로 된 것은 되돌리지 않으며, `inspect`가 이를 보면 안내 한 줄을 덧붙입니다. +- underscore-hex의 이어진 이스케이프는 UTF-8 글자를 시작할 수 없는 바이트(`80`-`BF`, `C0`, `C1`, `F5`-`FF`)에서도 + 끝나고, 그 바이트는 글자로 둡니다. 그래서 `IMG_8123.JPG`와 `test_99.txt`는 그대로 나오고, + `_ED_8A_B8_99.txt`는 `트_99.txt`가 됩니다. 글자를 시작하는 바이트는 나머지 바이트가 있어야 하므로, + `트리_DB_backup.sql`을 `_ED_8A_B8_EB_A6_AC_DB_backup.sql`로 쓴 이름은 실패합니다. `_DB`는 2바이트 글자의 + 시작인데 뒤가 없습니다. - **percent** (`%XX`): URL 인코딩입니다. 모든 바이트, 대소문자 모두 받습니다(`%20`은 공백). 뒤에 16진수 두 - 자리가 없는 `%`는 글자로 둡니다(`100% done`). `+`는 그대로 `+`입니다. 공백을 뜻하는 것은 HTML 폼 데이터뿐입니다. + 자리가 없는 `%`는 글자로 둡니다(`100% done`). `+`는 그대로 `+`입니다. 공백을 뜻하는 것은 HTML 폼 데이터뿐입니다. 모든 `%XX`를 + 이스케이프로 보므로 `%ED%8A%B8%99`는 실패합니다. +- 두 형식에 나뉘어 들어간 바이트열(`%ED_8A_B8`)은 어느 쪽으로도 되돌리지 않습니다. - 이어진 이스케이프는 하나의 바이트열로 보고, 완전하고 유효한 UTF-8이어야 합니다. 하나라도 아니면(`_ED_8A`는 3바이트 글자의 앞 2바이트) `decode`는 아무것도 출력하지 않고, 이유를 표준 오류에 쓰고, 종료 코드 1로 끝납니다. - `�`로 바꿔 넣지 않습니다. -- `decode`는 결과만 출력합니다. 이스케이프가 없는 글자는 그대로 나오고 종료 코드는 0입니다. `inspect`는 일반 + `�`로 바꿔 넣지 않습니다. 표준 입력이 UTF-8이 아니면 `inspect`와 `decode` 모두 종료 코드 1로 끝납니다. +- `decode`는 결과만 출력합니다. 이스케이프가 없는 글자는 그대로 나오고 종료 코드는 0입니다. `inspect`는 UTF-8 글자에서 실패하지 않고 아무것도 바꾸지 않습니다. - `inspect`와 `decode`는 첫 번째 인자일 때만 하위 명령입니다. `hanfix -- inspect`와 `hf`는 지금처럼 글자를 변환합니다. - 유효한 UTF-8이라는 것은 바이트가 글자를 이룬다는 뜻일 뿐, 원래 뜻한 글이라는 보장은 아닙니다. 그래서 hanfix는 @@ -233,7 +240,8 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex **이스케이프된 이름의 파일 여러 개를 한꺼번에 바꾸기 (PowerShell).** 함수를 정의한 뒤 폴더에서 `Rename-Decoded`를 실행하면 `old -> new` 목록만 보여 줍니다. 실제로 바꾸려면 `-Apply`를 붙입니다. 되돌릴 수 -없는 이름, 바뀌지 않는 이름, 파일 이름이 될 수 없는 결과, 이미 있는 파일을 덮어쓰게 되는 경우는 건너뜁니다. +없는 이름, 바뀌지 않는 이름, 파일 이름이 될 수 없는 결과(제어 문자, 끝의 점이나 공백, `NUL` 같은 Windows 장치 이름), 이미 있는 파일이나 같은 +실행에서 먼저 바꾸기로 한 이름과 겹치는 경우(대소문자 무시)는 건너뜁니다. ```powershell function Rename-Decoded { @@ -241,17 +249,25 @@ function Rename-Decoded { # hanfix 0.2.7 and older read "decode ..." as text to convert and exit 0, which would rename every file. $probe = (hanfix decode --type percent -- '%41' 2>$null) -join "`n" if ($LASTEXITCODE -ne 0 -or $probe -ne 'A') { throw 'hanfix too old: update with hanfix --update' } + # Every name in the folder and every new name planned so far, ignoring case, so the preview lists + # exactly what -Apply does. + $taken = [Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase) + Get-ChildItem -LiteralPath $Path -Force -Name | ForEach-Object { [void]$taken.Add($_) } $enc = [Console]::OutputEncoding [Console]::OutputEncoding = [Text.Encoding]::UTF8 try { foreach ($f in Get-ChildItem -LiteralPath $Path -File) { - $new = (hanfix decode --type $Type -- $f.Name 2>$null) -join "`n" + $out = @(hanfix decode --type $Type -- $f.Name 2>$null) if ($LASTEXITCODE -ne 0) { Write-Warning "skip $($f.Name): does not decode"; continue } - if ($new -eq $f.Name) { continue } - if ($new -in '.', '..' -or $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { + $new = $out[0] + if ($new -ceq $f.Name) { continue } + # One line, no control characters, no trailing dot or space, which Windows drops ("a." -> "a"), + # and no Windows device name. + if ($out.Count -ne 1 -or $new -match '[\x00-\x1F\x7F]|[. ]$|^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(\.|$)' -or + $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { Write-Warning "skip $($f.Name): '$new' is not a file name"; continue } - if (Test-Path -LiteralPath (Join-Path $f.DirectoryName $new)) { Write-Warning "skip $($f.Name): $new exists"; continue } + if (-not $taken.Add($new)) { Write-Warning "skip $($f.Name): $new exists or is planned"; continue } "$($f.Name) -> $new" if ($Apply) { Rename-Item -LiteralPath $f.FullName -NewName $new } } @@ -282,7 +298,7 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] - 옵션은 맨 앞에서만 읽습니다. 나머지는 전부 변환할 글자로 봅니다. 그래서 `hanfix ㅣㄴ -ㅁ`의 `-ㅁ`은 옵션이 아니라 글자(`-a`)입니다. - `-`로 시작하는 영문 글자를 바꾸려면 앞에 `--`를 붙이세요: `hanfix -- -rk` → `-가`. -- 종료 코드는 성공 0, `decode` 실패 1, 잘못된 사용 2입니다. +- 종료 코드는 성공 0, `decode` 실패와 `inspect`·`decode`의 입력이 UTF-8이 아닐 때 1, 잘못된 사용 2입니다. ### 입력 중인 줄 바로 고치기 (선택) diff --git a/README.md b/README.md index 546614a..6e87d1c 100644 --- a/README.md +++ b/README.md @@ -224,13 +224,21 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex escaping, so `backup_2026_10_29`, `hello_bed` and the first `_` of `report__ED_8A_B8` stay as they are, and `__` never becomes a space. Lower-case runs such as `_ed_8a_b8` are not decoded; `inspect` adds a note when it sees one. +- A run of underscore-hex escapes also ends at a byte that cannot start a UTF-8 character (`80`-`BF`, `C0`, + `C1`, `F5`-`FF`), which stays text: `IMG_8123.JPG` and `test_99.txt` come back unchanged, and + `_ED_8A_B8_99.txt` becomes `트_99.txt`. A byte that does start a character still needs the rest of it, so + a name like `트리_DB_backup.sql` written as `_ED_8A_B8_EB_A6_AC_DB_backup.sql` fails: `_DB` starts a + two-byte character and nothing follows. - **percent** (`%XX`): URL encoding, any byte, either case (`%20` is a space). A `%` without two hex digits - after it is text (`100% done`). `+` stays `+`; it means a space only in HTML form data. + after it is text (`100% done`). `+` stays `+`; it means a space only in HTML form data. Every `%XX` is an + escape, so `%ED%8A%B8%99` fails. +- A byte sequence split between the two types (`%ED_8A_B8`) decodes with neither. - Consecutive escapes form one byte sequence that must be complete, valid UTF-8. If any sequence in the text is not (`_ED_8A` is two bytes of a three-byte character), `decode` prints nothing, writes the reason to - standard error and exits with 1. It never substitutes `�`. + standard error and exits with 1. It never substitutes `�`. Input on standard input that is not UTF-8 + makes `inspect` and `decode` exit with 1 too. - `decode` prints only the result; text with no escapes comes back unchanged with exit 0. `inspect` never - fails on plain text and changes nothing. + fails on UTF-8 text and changes nothing. - `inspect` and `decode` are subcommands only as the first argument. `hanfix -- inspect` and `hf` still convert the word as typed text. - Valid UTF-8 only means the bytes form characters, not that they are the text someone meant. That is why @@ -239,7 +247,8 @@ $ printf '%s' '_ED_8A_B8' | hanfix decode --type underscore-hex **Rename many files with escaped names (PowerShell).** Define the function, then run `Rename-Decoded` in a folder: it only lists `old -> new`. Add `-Apply` to rename. It skips a name that does not decode, does not -change, would not be a valid file name, or would overwrite an existing file. +change, would not be a valid file name (control characters, a trailing dot or space, a Windows device name such as `NUL`), or would take the name +of an existing file or of another file renamed in the same run (ignoring case). ```powershell @@ -248,17 +257,25 @@ function Rename-Decoded { # hanfix 0.2.7 and older read "decode ..." as text to convert and exit 0, which would rename every file. $probe = (hanfix decode --type percent -- '%41' 2>$null) -join "`n" if ($LASTEXITCODE -ne 0 -or $probe -ne 'A') { throw 'hanfix too old: update with hanfix --update' } + # Every name in the folder and every new name planned so far, ignoring case, so the preview lists + # exactly what -Apply does. + $taken = [Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase) + Get-ChildItem -LiteralPath $Path -Force -Name | ForEach-Object { [void]$taken.Add($_) } $enc = [Console]::OutputEncoding [Console]::OutputEncoding = [Text.Encoding]::UTF8 try { foreach ($f in Get-ChildItem -LiteralPath $Path -File) { - $new = (hanfix decode --type $Type -- $f.Name 2>$null) -join "`n" + $out = @(hanfix decode --type $Type -- $f.Name 2>$null) if ($LASTEXITCODE -ne 0) { Write-Warning "skip $($f.Name): does not decode"; continue } - if ($new -eq $f.Name) { continue } - if ($new -in '.', '..' -or $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { + $new = $out[0] + if ($new -ceq $f.Name) { continue } + # One line, no control characters, no trailing dot or space, which Windows drops ("a." -> "a"), + # and no Windows device name. + if ($out.Count -ne 1 -or $new -match '[\x00-\x1F\x7F]|[. ]$|^(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])(\.|$)' -or + $new.IndexOfAny([IO.Path]::GetInvalidFileNameChars()) -ge 0) { Write-Warning "skip $($f.Name): '$new' is not a file name"; continue } - if (Test-Path -LiteralPath (Join-Path $f.DirectoryName $new)) { Write-Warning "skip $($f.Name): $new exists"; continue } + if (-not $taken.Add($new)) { Write-Warning "skip $($f.Name): $new exists or is planned"; continue } "$($f.Name) -> $new" if ($Apply) { Rename-Item -LiteralPath $f.FullName -NewName $new } } @@ -290,7 +307,8 @@ usage: hanfix [-h] [-V] [--to {ko,en}] [--] [TEXT ...] - Options are read only at the start; the rest is text to convert. So the `-ㅁ` in `hanfix ㅣㄴ -ㅁ` is text (`-a`), not an option. - To convert English text that starts with `-`, put `--` first: `hanfix -- -rk` → `-가`. -- Exit status: 0 on success, 1 when `decode` fails, 2 on a usage error. +- Exit status: 0 on success, 1 when `decode` fails or the input of `inspect` or `decode` is not UTF-8, 2 on + a usage error. ### Fix the line you are typing (optional) diff --git a/src/encoding.rs b/src/encoding.rs index 227e187..d39bd21 100644 --- a/src/encoding.rs +++ b/src/encoding.rs @@ -57,7 +57,13 @@ pub struct DecodeError { impl fmt::Display for DecodeError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let what = if self.incomplete { "incomplete UTF-8 sequence" } else { "invalid UTF-8" }; - write!(f, "{what} in {} escape '{}' at character {}", self.kind.name(), self.run, self.at) + // A run can be megabytes long on standard input; the start and the length locate it. + let run = if self.run.len() > 40 { + format!("{}...' ({} characters)", &self.run[..40], self.run.len()) + } else { + format!("{}'", self.run) + }; + write!(f, "{what} in {} escape '{run} at character {}", self.kind.name(), self.at) } } @@ -68,11 +74,30 @@ pub struct Decoded { pub bytes: usize, } +/// Bytes that follow a UTF-8 lead byte; 0 for a byte no character starts with (80-BF, C0, C1, F5-FF). +fn continuation_bytes(lead: u8) -> usize { + match lead { + 0x00..=0x7F => 0, + 0xC2..=0xDF => 1, + 0xE0..=0xEF => 2, + 0xF0..=0xF4 => 3, + _ => usize::MAX, + } +} + +impl Kind { + /// An escape that may start a run. In underscore-hex a byte no UTF-8 character starts with is text: + /// `IMG_8123` and `test_99` are names, not escapes. Percent escapes are always escapes. + fn starts_run(self, b: &[u8], i: usize) -> bool { + self.escape_at(b, i).is_some_and(|v| self == Kind::Percent || continuation_bytes(v) != usize::MAX) + } +} + pub fn decode(kind: Kind, text: &str) -> Result { let b = text.as_bytes(); let (mut out, mut bytes, mut i, mut copied) = (String::with_capacity(text.len()), 0, 0, 0); while i < b.len() { - if kind.escape_at(b, i).is_none() { + if !kind.starts_run(b, i) { i += 1; continue; } @@ -80,7 +105,19 @@ pub fn decode(kind: Kind, text: &str) -> Result { out.push_str(&text[copied..i]); let start = i; let mut run = Vec::new(); + // Continuation bytes the current character still needs. At a character start, an underscore-hex + // byte that cannot begin one ends the run (`_ED_8A_B8_99` is 트 then `_99`); inside a character + // every escape is taken, so a bad one is reported rather than left as text. + let mut need = 0; while let Some(v) = kind.escape_at(b, i) { + if need == 0 { + if !kind.starts_run(b, i) { + break; + } + need = continuation_bytes(v).min(3); + } else { + need -= 1; + } run.push(v); i += 3; } @@ -241,14 +278,39 @@ mod tests { assert!(e.incomplete && e.run == "%ED%8A" && e.at == 1); let e = decode(Kind::Percent, "%FF").unwrap_err(); assert!(!e.incomplete); - let e = decode(Kind::UnderscoreHex, "트_8A").unwrap_err(); - assert_eq!((e.at, e.incomplete), (1, false)); + // A lead byte with what follows it missing still fails, also before plain text. + let e = decode(Kind::UnderscoreHex, "트리_DB_backup.sql").unwrap_err(); + assert!(e.incomplete && e.run == "_DB" && e.at == 2); + // Inside a character every escape is taken, so a wrong one is reported, not kept as text. + let e = decode(Kind::UnderscoreHex, "_ED_C3_A9").unwrap_err(); + assert!(!e.incomplete && e.run == "_ED_C3_A9"); + assert!(decode(Kind::UnderscoreHex, "_ED_A0_80").is_err()); // A good run does not save a later bad one. assert!(decode(Kind::UnderscoreHex, "_ED_8A_B8 _ED").is_err()); assert_eq!( decode(Kind::UnderscoreHex, "_ED_8A").unwrap_err().to_string(), "incomplete UTF-8 sequence in underscore-hex escape '_ED_8A' at character 0" ); + let long = "_ED_8A_B8".repeat(1000) + "_ED"; + assert_eq!( + decode(Kind::UnderscoreHex, &long).unwrap_err().to_string(), + "incomplete UTF-8 sequence in underscore-hex escape '_ED_8A_B8_ED_8A_B8_ED_8A_B8_ED_8A_B8_ED_...' \ + (9003 characters) at character 0" + ); + } + + #[test] + fn underscore_bytes_that_start_no_character_are_text() { + for text in ["IMG_8123.JPG", "DSC_9001.JPG", "test_99.txt", "_8A", "트_8A", "_C0_80", "_C1_BF", "_F5_80_80_80", "_FF"] { + assert_eq!(decode(Kind::UnderscoreHex, text), Ok(Decoded { text: text.to_string(), bytes: 0 }), "{text}"); + } + assert_eq!(ok(Kind::UnderscoreHex, "_ED_8A_B8_99.txt"), "트_99.txt"); + assert_eq!(ok(Kind::UnderscoreHex, "IMG_8123__ED_8A_B8.jpg"), "IMG_8123_트.jpg"); + assert_eq!(ok(Kind::UnderscoreHex, "_ED_8A_B8_99_EB_A6_AC"), "트_99리"); + assert_eq!(ok(Kind::UnderscoreHex, "_C3_A9_F0_9F_98_80"), "é😀"); + assert!(inspect("IMG_8123.JPG").is_empty()); + // Percent stays strict: every %XX is an escape. + assert!(decode(Kind::Percent, "%ED%8A%B8%99").is_err()); } #[test] diff --git a/src/main.rs b/src/main.rs index 59d9fb3..0bdfcc3 100644 --- a/src/main.rs +++ b/src/main.rs @@ -292,7 +292,8 @@ escaped text (only as the first argument; \"hanfix -- inspect\" converts the wor decode prints only the result; text that is not an escape is kept. An incomplete or invalid UTF-8 sequence prints nothing and fails. -exit status: 0 on success, 1 when decode fails, 2 on a usage error. +exit status: 0 on success, 1 when decode fails or the input of inspect or decode is not UTF-8, +2 on a usage error. ", SHELLS.join(", ") ) @@ -374,6 +375,9 @@ fn escaped(cmd: &str, mut args: Vec) -> ExitCode { None if !args.is_empty() => args.remove(0), None => return usage_error("--type needs a value"), }; + if kind.is_some() { + return usage_error("--type given twice"); + } match encoding::Kind::parse(&val) { Some(k) => kind = Some(k), None => return usage_error(&format!("--type: choose underscore-hex or percent, not '{val}'")), diff --git a/tests/cli.rs b/tests/cli.rs index 5f2969d..673ed51 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -29,6 +29,8 @@ fn decode_prints_only_the_result() { assert_eq!(stdout(&["decode", "--type=percent", "%ED%8A%B8%EB%A6%AC"]), "트리\n"); assert_eq!(stdout(&["decode", "--type", "underscore-hex", "--", "file__ED_8A_B8.txt"]), "file_트.txt\n"); assert_eq!(stdout(&["decode", "--type", "underscore-hex", "backup_2026_10_29"]), "backup_2026_10_29\n"); + assert_eq!(stdout(&["decode", "--type", "underscore-hex", "IMG_8123.JPG"]), "IMG_8123.JPG\n"); + assert_eq!(stdout(&["decode", "--type", "underscore-hex", "_ED_8A_B8_99.txt"]), "트_99.txt\n"); assert_eq!(stdout(&["decode", "--type", "percent", "100%", "done"]), "100% done\n"); } @@ -67,6 +69,7 @@ fn usage_errors() { &["decode", "_ED_8A_B8"][..], &["decode", "--type", "base64", "x"], &["decode", "--type"], + &["decode", "--type", "percent", "--type", "underscore-hex", "%41_ED_8A_B8"], &["inspect", "--type", "percent", "x"], &["inspect", "-x"], ] { diff --git a/tests/shells.ps1 b/tests/shells.ps1 index 1b88c57..132ef50 100644 --- a/tests/shells.ps1 +++ b/tests/shells.ps1 @@ -57,6 +57,18 @@ try { $want = ('_EB_A6_AC.md', 'backup_2026_10_29.txt', 'bad_ED_8A.txt', "$ri.md", "$tree.txt" | Sort-Object) -join '|' if ($after -ne $want) { throw "FAIL rename apply: got '$after'" } + # Preview and -Apply agree: no trailing dot or space (Windows would drop it), no control characters, no device name, + # and no two files given the same name in one run (ignoring case). + $p = Join-Path $dir 'p' + New-Item -ItemType Directory $p | Out-Null + 'a%0Db.txt', 'a%2E', 'b%20', 'N%55L', 'x%41.txt', 'x%61.txt' | ForEach-Object { New-Item -ItemType File (Join-Path $p $_) | Out-Null } + $preview = @(Rename-Decoded -Path $p -Type percent 3>$null) -join '|' + $applied = @(Rename-Decoded -Path $p -Type percent -Apply 3>$null) -join '|' + if ($preview -ne 'x%41.txt -> xA.txt' -or $applied -ne $preview) { throw "FAIL rename percent: '$preview' / '$applied'" } + $after = (Get-ChildItem -LiteralPath $p -Name | Sort-Object) -join '|' + $want = ('a%0Db.txt', 'a%2E', 'b%20', 'N%55L', 'x%61.txt', 'xA.txt' | Sort-Object) -join '|' + if ($after -ne $want) { throw "FAIL rename percent apply: got '$after'" } + # An older hanfix (0.2.7) treats "decode ..." as text: the example must stop before listing or renaming. $real = (Get-Command hanfix -CommandType Application | Select-Object -First 1).Source function hanfix { & $real -- @args }