UTF-8 encoding not enforced

[jwilk]

BPO	19942
Nosy	@loewis, @vstinner, @benjaminp, @jwilk, @ezio-melotti, @serhiy-storchaka
Files	test1.py test3.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-12-10.11:48:59.865>
labels = ['type-bug', 'expert-unicode']
title = 'UTF-8 encoding not enforced'
updated_at = <Date 2013-12-11.14:17:06.415>
user = 'https://github.com/jwilk'

bugs.python.org fields:

activity = <Date 2013-12-11.14:17:06.415>
actor = 'benjamin.peterson'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2013-12-10.11:48:59.865>
creator = 'jwilk'
dependencies = []
files = ['33078', '33092']
hgrepos = []
issue_num = 19942
keywords = []
message_count = 3.0
messages = ['205795', '205827', '205901']
nosy_count = 6.0
nosy_names = ['loewis', 'vstinner', 'benjamin.peterson', 'jwilk', 'ezio.melotti', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19942'
versions = ['Python 3.4']

[jwilk]

I created a Python file which contained a non-UTF-8 string literal (but no Unicode literals), and added "UTF-8" encoding declaration to it. I expected that Python will raise SyntaxError when importing such module, but it doesn't:

$ python --version
Python 2.7.6

$ python -c 'import test1' && echo ok
ok

Curiously enough, if I change the declaration to "UTF8", then the exception is raised as expected:

$ sed -e 's/UTF-8/UTF8/' < test1.py > test2.py
$ python -c 'import test2'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "test2.py", line 2
SyntaxError: 'utf8' codec can't decode byte 0xa1 in position 5: invalid start byte

[benjaminp]

Yes, this is a silly bug where we "shortcut" decoding of utf-8 files by not checking if its valid UTF-8. However, this behavior has been around for a long time, so I'm not going to change it in 2.7.x.

[jwilk]

With a slightly adapted test case, I see the same behavior in Python 3.3.3. Perhaps it would be worth fixing the bug in Python 3.4?

[terryjreedy]

@vstinner @serhiy-storchaka @pablogsal After downloading from bpo, running test1.py, with executable code, results in SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte (wrong position) with the caret under the last char of the string (!). Running test3.py, with the bad byte in a comment raises no error. Loading into an IDLE editor fails when trying to decode. I did not try changing to 'utf8'.

I presume the justification for not checking utf-8 files when loaded is that compiling will normally catch the error. If this is still intended, should this be closed?

[StanFromIreland]

I can not reproduce with any version $\ge 3.12$, I assume this was fixed by another commit and issue at some point.

[vstinner]

I confirm that I cannot reproduce the issue on Python 3.15:

$ ./python test1.py 
  File "/home/vstinner/python/main/test1.py", line 2
    s = '�Hello world!'
        ^^^^^^^^^^^^^
SyntaxError: (unicode error) 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte

Let's close this old Python 2 issue.