Skip to content

Handle non-UTF8 response header values - #220

Open
dhairyajangir wants to merge 1 commit into
adafruit:mainfrom
dhairyajangir:fix/143-opaque-header-values-20261112
Open

dhairyajangir wants to merge 1 commit into
adafruit:mainfrom
dhairyajangir:fix/143-opaque-header-values-20261112

Conversation

@dhairyajangir

@dhairyajangir dhairyajangir commented Nov 11, 2026 •

Copy link
Copy Markdown

Fixes #143.

Session.get() raises UnicodeError when an ordinary response header contains a Latin-1 byte such as 0xF6, before its valid body can be read.

Keep the existing UTF-8 conversion first. When it fails, map the original header-value octets to their corresponding Unicode code points using a small UTF-8 bytearray helper. This preserves the supplied value without needing a Latin-1 runtime codec. Header-name decoding and body decoding retain their existing paths.

Keep invalid UTF-8 Content-Length and Transfer-Encoding values on their original error path. In particular, converting a raw NBSP must not make an invalid framing value recognizable through Unicode whitespace handling.

HTTP field-value rules permit opaque high octets. The helper uses the UTF-8/buffer APIs supported by CircuitPython.

Validation:

  • Same final 26 header cases against unchanged main: 8 failed, 18 passed.
  • Complete candidate test suite: 72 passed, including the local server and all three live HTTP cases.
  • All configured pre-commit hooks passed, including Ruff 0.15.8 and REUSE 6.2.0.

Tests cover ASCII and existing UTF-8 behavior, Latin-1 and mixed/opaque values, the complete high-octet range, colons, duplicate cookies, chunked bodies, UTF-8 bodies, and invalid names/framing. Host validation used CPython 3.11.17; target API compatibility was checked against current and stable CircuitPython source. Board execution and heap measurements remain untested.

Codex assisted reproduction, implementation and validation.

@justmobilize justmobilize left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ISO-8859-1 encoded header values throws Unicode exception

2 participants