RecursionError when parsing unwieldy expression (regression from 2.7 -> 3.x)

[mjo]

BPO	42649
Nosy	@terryjreedy, @jdevries3133
Files	DPT_defn repro.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-15.17:55:57.375>
labels = ['interpreter-core', 'build', '3.9']
title = 'RecursionError when parsing unwieldy expression (regression from 2.7 -> 3.x)'
updated_at = <Date 2021-08-13.18:53:06.576>
user = 'https://bugs.python.org/mjo'

bugs.python.org fields:

activity = <Date 2021-08-13.18:53:06.576>
actor = 'jack__d'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-12-15.17:55:57.375>
creator = 'mjo'
dependencies = []
files = ['49683', '50215']
hgrepos = []
issue_num = 42649
keywords = []
message_count = 4.0
messages = ['383084', '383351', '399551', '399552']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'mjo', 'jack__d']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue42649'
versions = ['Python 3.9']

[mjo]

The attached file contains a huge, ugly expression that can be parsed in python-2.7 but not python-3.x (I've tested 3.7, 3.8, and 3.9). With the 3.x versions, I get

  $ python3 DPT_defn 
  RecursionError: maximum recursion depth exceeded during compilation

Python-2.7, on the other hand, is able to parse & compile much larger expressions -- this example was whittled down until it was roughly minimal.

[terryjreedy]

One can use sys.setrecursionlimit to increase allowed recursion depth.

But... When I ran code from 3.10.0a3 IDLE editor, something hung. Without touching recursion limit, when I split statement apart into

ex = compile("""<300 lines>""", '', 'eval')
print(eval(ex))

I got 106994.5 in under a second. With exec, ditto except that None is printed. Same last result with 3.9.1.

[jdevries3133]

I spent some time experimenting with making the expression bigger and the recursion limit lower in python2. It seems like in python2, the depth that the compiler will recurse is unrelated to sys.recursionlimit.

Then, I lowered resource limits on stack and heap by ~1000x and increased the size of the expression by 10,000x, which caused a segmentation fault. Not only that, but the interpreter won't even respond to KeyboardInterrupt while it is doing the compiling. I believe that the recursion depth of the 2.x compiler is simply unbound, which seems like a bug to me.

As I tinkered with these things, I build out a reproduction script, which I've attached.

I think that documenting this change is more pragmatic than thinking about "fixing" it (not sure it can truly be fixed, as that would mean re-incorporating a bug). I don't know where such a note would belong, though. This document is the closed I can find, but it's too high-level for a detail like this to belong::

https://docs.python.org/3/howto/pyporting.html

[jdevries3133]

edit; typo:

**This document is the **closest** I can find

[StanFromIreland]

I now get almost instantaneous parsing with the provided reproducer, is this still an issue?

[terryjreedy]

On main running repro.py, with referenced to non-existent resource module removed, in IDLE from editor, still hangs. I am not now sure what I did as compiling and executing separately is the default. This likely is a issue in code and codeop modules rather than python.

EDIT: mjo and jdevries no longer get emails for issues.