email.message_from_binary_file creates invalid RFC2047 encodings

[msapiro]

Bug report

Bug description:

The following illustrates the issue.

>>> # Here's an example file
>>> with open('badx2.eml') as fp:
...     print(fp.read())
...     
From: "Sweepstakes Alert–Dyson Vacuum" <DysonVacuumSweeps@stellaryx.space>

body

>>> # Note that the `–` above is a unicode En Dash
>>> # Make a message object
>>> from email import message_from_binary_file
>>> with open('badx2.eml', 'rb') as fp:
...     msg = message_from_binary_file(fp)
...     
>>> # and look at it
>>> print(msg.as_string())
From: =?unknown-8bit?q?=22Sweepstakes_Alert=E2=80=93Dyson_Vacuum=22_=3CDysonVacuumSweeps=40stellaryx=2Espace=3E?=

body

>>> # Note that the email address is improperly included in the encoded word

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[fatelei]

UTF‑8 headers (requires server support)

print(msg.as_string(policy=policy.SMTPUTF8))

ASCII‑safe (works everywhere)

print(msg.as_string(policy=policy.SMTP))

try

[medmunds]

I believe this is working as expected, and should be closed as not a bug.

A (utf-8 encoded, 8-bit) en dash is not a valid character in an email header, unless the message is being handled by an SMTP server that specifically advertises support for the smtputf8 extension (see RFC 6531 and 6532). If you are processing a message with RFC 6532 8-bit headers, you must share that knowledge with Python's email package by using a policy with EmailPolicy.utf8 set True. Example:

message_from_binary_file(fp, policy=email.policy.SMTPUTF8)

Without that knowledge, the email parser has no way of knowing if the 8-bit characters in the header are utf-8, or iso-8859-1, or some other encoding. So it identifies them as using an "unknown-8bit" charset.

[msapiro]

I believe this is working as expected, and should be closed as not a bug.

It is not working as expected. The issue is not that the en dash is encoded as unknown-8bit. The issue is that the entire header including the email address is RFC 2047 encoded. Email addresses MUST not be RFC 2047 encoded.

[medmunds]

Email addresses MUST not be RFC 2047 encoded.

That is absolutely correct. Sorry I overlooked that in your report.

But this is essentially garbage in, garbage out (invalid header in → different invalid header out). Given that it's occurring in the legacy compat32 email parser and generator, I wouldn't expect it to be fixed.

When you call message_from_bytes() without specifying a policy, you're getting the legacy compat32 policy. The legacy parser does not support 8-bit headers. And the legacy generator's as_string() dates from the Python 2 days and largely unaware of Unicode strings.

from email import message_from_bytes, policy
raw = 'From: "Sweepstakes Alert–Dyson Vacuum" <DysonVacuumSweeps@stellaryx.space>'.encode()
message_from_bytes(raw).as_string()
# 'From: =?unknown-8bit?q?=22Sweepstakes_Alert=E2=80=93Dyson_Vacuum=22_=3CDysonVacuumSweeps=40stellaryx=2Espace=3E?=\n\n'

Note that you can get the original (invalid) header back from the legacy generator's as_bytes():

message_from_bytes(raw).as_bytes()
# b'From: "Sweepstakes Alert\xe2\x80\x93Dyson Vacuum" <DysonVacuumSweeps@stellaryx.space>\n\n'

Switching to the modern email.policy.default (which is confusingly not the default policy) provides more tolerant handling of the invalid input, plus RFC 2047 compliant output:

message_from_bytes(raw, policy=policy.default).as_string()
# 'From: Sweepstakes =?unknown-8bit?q?Alert=E2=80=93Dyson?= Vacuum\n <DysonVacuumSweeps@stellaryx.space>\n\n'
message_from_bytes(raw, policy=policy.default)["From"].defects
# (UndecodableBytesDefect('Non-ASCII characters found in header token'),)

[msapiro]

When you call message_from_bytes() without specifying a policy, you're getting the legacy compat32 policy. The legacy parser does not support 8-bit headers. And the legacy generator's as_string() dates from the Python 2 days and largely unaware of Unicode strings.

I understand that.

The example I posted was a contrived minimal example. The actual issue arises in Mailman 3.

A much larger message with that From: header is posted to a list (we have also seen other messages with non-ascii display names). It is converted to a mailman.email.message.Message object (a subclass of Python's email.Message class) using compat32 policy. This message object is processed completely okay through Mailman, but it is converted to bytes by the as_bytes() method and passed to the HyperKitty archiver which converts it to an email.EmailMessage object using policy default, and it is there that the issue manifests and causes archiving to fail.

For various historical reasons, we are reluctant to change this process.

Granted, this may not be a serious issue as the original messages are non- compliant, but in any case, the email package should not be creating invalid encodings. At a minimum it should flag this with a defect in the message object.

[medmunds]

Hmm, is Mailman also using compat32 as_string() somewhere in the process?

…it is converted to bytes by the as_bytes() method…

compat32 as_bytes() does not seem to apply rfc2047 to the addr-spec—see example in my previous comment. Which is why I'm speculating some other operations must be involved. (Does Mailman override any header field methods?)

… At a minimum it should flag this with a defect in the message object. …

The modern email parser does flag it as a defect—example in my previous comment. (Field-level defects are not propagated up to the message object, which is bug #106144. And I don't think the legacy Message object has any way to access field-level defects.)

[msapiro]

compat32 as_bytes() does not seem to apply rfc2047 to the addr-spec—see example in my previous comment. Which is why I'm speculating some other operations must be involved. (Does Mailman override any header field methods?)

I know that as_bytes() does not rfc2047 encode the header in my 3 line example, but in the case of a real message it does.