From bfede034e1e8b3e3f63ba6711296cd30dfb0572d Mon Sep 17 00:00:00 2001 From: Duc-Tam Nguyen Date: Mon, 10 Aug 2026 14:01:19 +0700 Subject: [PATCH] Answer the charset declaration question from fixCharsetMetas set declared from a anywhere in the tree, so a page whose only charset meta sits in , which is the malformed markup case the walk exists to handle, came out of sanitize with no declaration in at all. Readers pre-scan the first 1024 bytes for the encoding, so the declaration was never found and every multibyte character mojibaked, which is the failure issue #16 reports. Rewriting stale values stays whole-document. Only the declared question moves back to , so a stray body meta is rewritten to utf-8 and still gets its own declaration first. rewriteContentTypeCharset also starts at the first field rather than the second, so content="charset=iso-8859-1" with no media type is rewritten instead of being left to contradict the injected declaration. A real media type has no equals sign, so the existing Cut rejects it. Follow-up to #81. --- CHANGELOG.md | 7 +++++++ sanitize/sanitize.go | 42 +++++++++++++++++++++++++++++++++++++-- sanitize/sanitize_test.go | 33 +++++++++++++++++++++++++++++- 3 files changed, 79 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a638e3..798543a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,13 @@ All notable changes to kage are recorded here. The format follows directly on its article and keeps that article's title metadata ([#62](https://github.com/tamnd/kage/issues/62)). - Non-UTF-8 `` and Content-Type charset declarations are rewritten to `utf-8`, matching the encoding kage writes to disk ([#16](https://github.com/tamnd/kage/issues/16)). + Rewriting covers the whole document, but whether the page *declares* an + encoding is answered from `` alone. A charset meta that Chrome left in + `` was otherwise treated as the document's declaration, so nothing was + inserted into `` and the only declaration sat past the 1024 bytes a + reader pre-scans, which is the mojibake #16 is about. + A legacy `content="charset=iso-8859-1"` with no media type is now rewritten + too, instead of being left to contradict the injected UTF-8 declaration. - Relative links on redirected pages resolve against the browser's final URL and the document's first ``, while the page remains saved under the URL that was originally discovered. Consumed `href` attributes are diff --git a/sanitize/sanitize.go b/sanitize/sanitize.go index 3cf81d1..8f5e92f 100644 --- a/sanitize/sanitize.go +++ b/sanitize/sanitize.go @@ -277,8 +277,14 @@ func ensureCharset(root *html.Node) (added, rewritten bool) { if head == nil { return false, false } + // Rewriting stale values is a whole-document job: a declaration Chrome moved + // into still contradicts the UTF-8 bytes kage writes. fix := fixCharsetMetas(root) - if fix.declared { + // Whether the document *declares* an encoding is a question, though. + // Readers pre-scan only the first 1024 bytes, so a meta stranded in + // is never found; treating it as a declaration left the saved page with + // nothing a reader could act on, which is the mojibake of issue #16. + if headDeclaresCharset(head) { return false, fix.rewritten } meta := &html.Node{ @@ -332,10 +338,42 @@ func fixCharsetMetas(n *html.Node) charsetMetaFix { // rewriteContentTypeCharset rewrites a charset parameter while preserving the // media type and other parameters. It accepts optional whitespace around '='. +// headDeclaresCharset reports whether itself carries an encoding +// declaration, in either the or the legacy Content-Type form. +// Content inside