Repository navigation
feat: decode mojibake (UTF-8 read as windows-1252) - #14
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds mojibake recovery: UTF-8 text that was read as windows-1252.
Scope: which source encodings, and why no crate
cp1252). It is the common case (UTF-8 opened by Windows tools and oldweb pages) and exact as a 32-entry table: 0x00-0x7F and 0xA0-0xFF are the code points of the same number,
0x80-0x9F come from the table. The five bytes Microsoft leaves undefined (81, 8D, 8F, 90, 9D) map to the
C1 controls of the same number, as the WHATWG Encoding Standard does, so all 256 bytes round-trip (unit test).
Korean UTF-8 often contains these bytes (곁 = EA B3 81), so a table without them would fail real text.
entries; the honest option is
encoding_rs(a new dependency). That misreading (UTF-8 shown as CP949,?덈뀞-like) also usually loses bytes on the way (odd byte counts,?substitution), so most real caseswould come out as lossy anyway. Worth adding only with real samples that are not lossy.
Contract (tested)
decode --type mojibake --source-encoding ENC TEXTmaps every character of TEXT back to its ENC byte andreads the bytes as UTF-8, strictly. A character not in ENC, or bytes that are not complete, valid UTF-8:
exit 1, empty stdout, one
hanfix: error: ...line naming the character and position. ASCII-only text comesback unchanged.
�,%EF%BF%BD,_EF_BF_BD):exit 1, empty stdout,
lossy: ...;inspectreports the candidate asinvalidwith that reason and no"decoded twice" note. Never claimed as recovered. This also applies to percent and underscore-hex, which
printed
안�with exit 0 before.--source-encodingis required with--type mojibakeand refused with the other types and withinspect;given twice, or an unknown encoding (
cp949), is a usage error (exit 2).inspectlists a mojibake candidate per code page only when the text contains at least one complete,well-formed multi-byte UTF-8 sequence (a lead byte C2-F4 and exactly the continuation bytes it needs).
Plain Korean, ASCII,
café,naïve,©2026, and accented text next to typographic punctuation(
café’s,résumé…,„Café“,«Café») give no candidate. Status is rule-based:valid,ambiguous(the text also fits an escape type),invalid; no confidence numbers. The reason saysvalid UTF-8 does not prove the original, and notes when the result looks like mojibake again (misread
twice).
Candidatenow carries the detected type name and the encoding line(
UTF-8orUTF-8 read as windows-1252) instead of the escape kind.Files
src/mojibake.rs: the windows-1252 table, strict decode, the "looks like" rule, unit tests from realcases (
안녕-> 안녕,트리-> 트리,한글-> 한글,café-> café, round trips of Korean text with C1bytes and an emoji, double misreading, lossy and truncated input).
src/encoding.rs:inspectmerges mojibake candidates; report prints the encoding line per candidate.src/main.rs:--source-encoding, usage, help.tests/cli.rs: decode by argument and stdin, error contract, inspect output, plain text, usage errors.README.md,README.ko.md: mojibake section, options, limits (CP949 / EUC-KR, dropped control characters,decomposed NFD text).
Rename-Decodedstill offersunderscore-hexandpercentonly; mojibake file names can be added if wanted.Version 0.2.9 (0.2.8 is released).
local-ci: 55629b9 |
cargo test --release --locked(15 unit + 9 CLI pass),cargo clippy --all-targets --locked -- -D warningsclean,pwsh tests/shells.ps1with this build first on PATH (3 ok) on Windows | not run:tests/shells.sh(bash/zsh/fish, Linux only; init scripts untouched), macOS, Termux