Skip to content

feat: decode mojibake (UTF-8 read as windows-1252) - #14

Merged
NaNIsaac merged 3 commits into
mainfrom
feat/mojibake
Oct 29, 2026
Merged

NaNIsaac merged 3 commits into
mainfrom
feat/mojibake

Conversation

@NaNIsaac

@NaNIsaac NaNIsaac commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Adds mojibake recovery: UTF-8 text that was read as windows-1252.

hanfix inspect '안녕'                                                  Detected: mojibake ... Candidate: 안녕
hanfix decode --type mojibake --source-encoding windows-1252 '안녕'     안녕

Scope: which source encodings, and why no crate

  • windows-1252 only (alias cp1252). It is the common case (UTF-8 opened by Windows tools and old
    web pages) and exact as a 32-entry table: 0x00-0x7F and 0xA0-0xFF are the code points of the same number,
    0x80-0x9F come from the table. The five bytes Microsoft leaves undefined (81, 8D, 8F, 90, 9D) map to the
    C1 controls of the same number, as the WHATWG Encoding Standard does, so all 256 bytes round-trip (unit test).
    Korean UTF-8 often contains these bytes (곁 = EA B3 81), so a table without them would fail real text.
  • CP949 / EUC-KR: not in this PR. Standard Rust has no codecs, and a hand table would be about 17,000
    entries; the honest option is encoding_rs (a new dependency). That misreading (UTF-8 shown as CP949,
    ?덈뀞-like) also usually loses bytes on the way (odd byte counts, ? substitution), so most real cases
    would come out as lossy anyway. Worth adding only with real samples that are not lossy.
  • No new crates.

Contract (tested)

  • decode --type mojibake --source-encoding ENC TEXT maps every character of TEXT back to its ENC byte and
    reads the bytes as UTF-8, strictly. A character not in ENC, or bytes that are not complete, valid UTF-8:
    exit 1, empty stdout, one hanfix: error: ... line naming the character and position. ASCII-only text comes
    back unchanged.
  • U+FFFD anywhere in the input, or in the decoded result of any type (�, %EF%BF%BD, _EF_BF_BD):
    exit 1, empty stdout, lossy: ...; inspect reports the candidate as invalid with that reason and no
    "decoded twice" note. Never claimed as recovered. This also applies to percent and underscore-hex, which
    printed 안� with exit 0 before.
  • --source-encoding is required with --type mojibake and refused with the other types and with inspect;
    given twice, or an unknown encoding (cp949), is a usage error (exit 2).
  • inspect lists a mojibake candidate per code page only when the text contains at least one complete,
    well-formed multi-byte UTF-8 sequence (a lead byte C2-F4 and exactly the continuation bytes it needs).
    Plain Korean, ASCII, café, naïve, ©2026, and accented text next to typographic punctuation
    (café’s, résumé…, „Café“, «Café») give no candidate. Status is rule-based:
    valid, ambiguous (the text also fits an escape type), invalid; no confidence numbers. The reason says
    valid UTF-8 does not prove the original, and notes when the result looks like mojibake again (misread
    twice).
  • The escape types are unchanged. Candidate now carries the detected type name and the encoding line
    (UTF-8 or UTF-8 read as windows-1252) instead of the escape kind.

Files

  • src/mojibake.rs: the windows-1252 table, strict decode, the "looks like" rule, unit tests from real
    cases (안녕 -> 안녕, 트리 -> 트리, 한글 -> 한글, café -> café, round trips of Korean text with C1
    bytes and an emoji, double misreading, lossy and truncated input).
  • src/encoding.rs: inspect merges mojibake candidates; report prints the encoding line per candidate.
  • src/main.rs: --source-encoding, usage, help.
  • tests/cli.rs: decode by argument and stdin, error contract, inspect output, plain text, usage errors.
  • README.md, README.ko.md: mojibake section, options, limits (CP949 / EUC-KR, dropped control characters,
    decomposed NFD text).

Rename-Decoded still offers underscore-hex and percent only; mojibake file names can be added if wanted.
Version 0.2.9 (0.2.8 is released).

local-ci: 55629b9 | cargo test --release --locked (15 unit + 9 CLI pass), cargo clippy --all-targets --locked -- -D warnings clean, pwsh tests/shells.ps1 with this build first on PATH (3 ok) on Windows | not run: tests/shells.sh (bash/zsh/fish, Linux only; init scripts untouched), macOS, Termux

@NaNIsaac
NaNIsaac merged commit 8be6661 into main Oct 29, 2026
20 of 21 checks passed
@NaNIsaac
NaNIsaac deleted the feat/mojibake branch October 29, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant