email.Message param parsing problem II

[customdesigned]

BPO	795081
Nosy	@warsaw, @devdanzin, @bitdancer, @zvyn
Files	virus5: Example (deactivated) virus with wierd headers issue795081.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2003-08-26.03:37:10.000>
labels = ['easy', 'type-feature', 'library', 'expert-email']
title = 'email.Message param parsing problem II'
updated_at = <Date 2016-12-12.12:01:53.266>
user = 'https://bugs.python.org/customdesigned'

bugs.python.org fields:

activity = <Date 2016-12-12.12:01:53.266>
actor = 'bpoaugust'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2003-08-26.03:37:10.000>
creator = 'customdesigned'
dependencies = []
files = ['8230', '39701']
hgrepos = []
issue_num = 795081
keywords = ['patch', 'easy']
message_count = 9.0
messages = ['53986', '53987', '53988', '53989', '63201', '82025', '245292', '245317', '282986']
nosy_count = 8.0
nosy_names = ['barry', 'collinwinter', 'customdesigned', 'tony_nelson', 'ajaksu2', 'r.david.murray', 'bpoaugust', 'zvyn']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue795081'
versions = ['Python 3.3']

Linked PRs

• gh-39128: Fix email.utils.unquote() parameter parsing #141702

[customdesigned]

The enclosed real life (inactivated) virus message
causes email.Message to fail to find the multipart
attachments. This is because the headers following
Content-Type are indented, causing email.Message to
properly append them to Content-Type. The trick is
that the boundary is quoted, and Outhouse^H^H^H^H^Hlook
apparently gets a value of 'bound' for boundary,
whereas email.Message gets the value
'"bound"\n\tX-Priority...'. email.Utils.unqoute
apparently gives up and doesn't remove any quotes.

I believe that unqoute should return just what is
between the quotes, so that '"abc" def' would be
unquoted to 'abc'. In fact, my email filtering
software (http://bmsi.com/python/milter.html) works
correctly on all kinds of screwy mail using my version
of unquote using this heuristic. I believe that header
used by the virus is invalid, so a STRICT parser should
reject it, but a tolerant parser (such as a virus
scanner would use) should use the heuristic.

Here is a brief script to show the problem (attached
file in test/virus5):
----------t.py----------

import email

msg = email.message_from_file(open('test/virus5','r'))
print msg.get_params()

---

$ python2 t.py
[('multipart/mixed', ''), ('boundary',
'"bound"\n\tX-Priority: 3\n\tX-MSMail-Priority:
Normal\n\tX-Mailer: Microsoft Outlook Express
5.50.4522.1300\n\tX-MimeOLE: Produced By Microsoft
MimeOLE V5.50.4522.1300')]

[customdesigned]

Logged In: YES
user_id=142072

Here is a proposed fix for email.Util.unquote (except it
should test for a 'strict' mode flag, which is current only
in Parser):

def unquote(str):
    """Remove quotes from a string."""
    if len(str) > 1:
        if str.startswith('"'):
          if str.endswith('"'):
            str = str[1:-1]
          else: # remove garbage after trailing quote
            try: str = str[1:str[1:].index('"')+1]
            except: return str
          return str.replace('\\\\', '\\').replace('\\"', '"')
        if str.startswith('<') and str.endswith('>'):
            return str[1:-1]
    return str

Actually, I replaced only email.Message._unquotevalue for my
application to minimize the impact. That would also be a
good place to check for a STRICT flag stored with the
message object. Perhaps the Parser should set the Message
_strict flag from its own _strict flag.

[warsaw]

Logged In: YES
user_id=12800

Moving this to feature requests for Python 2.4. If
appropriate, the email-sig should address this in the
intended new lax parser for email 3.0 / Python 2.4. We
can't add this to the Python 2.3 (or earlier) maintenance
releases.

[collinwinter]

I'm still seeing this behaviour as of Python 2.6a0.

Barry: I take it email-sig didn't get around to discussing this?

[tonynelson]

If I understand RFC2822 3.2.2. Quoted characters (heh), unquoting must
be done in one pass, so the current replace().replace() is wrong. It
will change '\\"' to '"', but it should become '\"' when unquoted.

This seems to work:

re.sub(r'\\(.)',r'\1',s)

I haven't encountered a problem with this; I just came across it while
looking at the file Utils.py (Python 2.4, but unchanged in trunk). I
will submit a new bug if desired.

[devdanzin]

Good candidate for the email sprint. Fix suggested inline.