Missing validation for allowed constant values in ast.literal_eval()

[skirpichev]

Bug report

Bug description:

Maybe it's a feature, but I see something wrong here:

>>> import ast
>>> value = object()
>>> value
<object object at 0x7f7489580880>
>>> node = ast.Expression(body=ast.Constant(value))
>>> ast.fix_missing_locations(node)
Expression(body=Constant(value=<object object at 0x7f7489580880>, kind=None))
>>> ast.literal_eval(node)
<object object at 0x7f7489580880>
>>> compile(node, "<test>", "eval")
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    compile(node, "<test>", "eval")
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: got an invalid type in Constant: object

Note, that for unary/binary ops values are checked, e.g.:

>>> node = ast.Expression(body=ast.UnaryOp(op=ast.USub(), operand=ast.Constant(value)))
>>> ast.fix_missing_locations(node)
Expression(body=UnaryOp(op=USub(), operand=Constant(value=<object object at 0x7f7489580880>, kind=None)))
>>> ast.literal_eval(node)
Traceback (most recent call last):
  File "<python-input-10>", line 1, in <module>
    ast.literal_eval(node)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.14/ast.py", line 106, in literal_eval
    return _convert(node_or_string)
  File "/usr/local/lib/python3.14/ast.py", line 105, in _convert
    return _convert_signed_num(node)
  File "/usr/local/lib/python3.14/ast.py", line 74, in _convert_signed_num
    operand = _convert_num(node.operand)
  File "/usr/local/lib/python3.14/ast.py", line 70, in _convert_num
    _raise_malformed_node(node)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.14/ast.py", line 67, in _raise_malformed_node
    raise ValueError(msg + f': {node!r}')
ValueError: malformed node or string on line 1: Constant(value=<object object at 0x7f7489580880>, kind=None)
>>> compile(node, "<test>", "eval")
Traceback (most recent call last):
  File "<python-input-11>", line 1, in <module>
    compile(node, "<test>", "eval")
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: got an invalid type in Constant: object

(Tested in the main branch too.)

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

No response

Linked PRs

• gh-141778: Missing validation for allowed constant values in ast.literal_eval() #141933
• gh-141778: add missing validation in ast.literal_eval() for non-string input #142969

[skirpichev]

CC @Eclips4

[sergey-miryanov]

IIUC, Constant can't get object value according documentation:

A constant value. The value attribute of the Constant literal contains the Python object it represents. The values represented can be instances of str, bytes, int, float, complex, and bool, and the constants None and Ellipsis.

[skirpichev]

I'm not sure it's relevant. The docs don't tell what happens when the value is not from the above list. Should it raise an exception? Or blow up a nuke? E.g. ast.Expr docs says:

value holds one of the other nodes in this section, a Constant, a Name, a Lambda, a Yield or YieldFrom node.

But behold:

>>> import ast
>>> class Spam: ...
>>> ast.Expr(Spam())
Expr(value=<__main__.Spam object at 0x7fc238db30e0>)

My argument was that the literal_eval() has such validation, but only partially.

[sergey-miryanov]

Augh, I just noticed a difference in the first example - literal_eval works, but compile doesn't.
Sorry.

[serhiy-storchaka]

I think this is a GIGO case. ast.literal_eval() works for string argument. It works if the string was parsed to AST by ast.parse(). It does not work for arbitrary invalid AST, this is not its purpose. You can make compile() to fail by simply setting invalid lineno -- ast.literal_eval() ignores lineno.

I propose to close this issue as "won't fix".

[skirpichev]

ast.literal_eval() works for string argument.

Not only. But I realize, it's major case.

It works if the string was parsed to AST by ast.parse().

Nop.

>>> import ast
>>> ast.parse('1j+1', mode='eval')
Expression(body=BinOp(left=Constant(value=1j, kind=None), op=Add(), right=Constant(value=1, kind=None)))
>>> ast.literal_eval(_)
Traceback (most recent call last):
  File "<python-input-11>", line 1, in <module>
    ast.literal_eval(_)
    ~~~~~~~~~~~~~~~~^^^
  File "/usr/local/lib/python3.14/ast.py", line 106, in literal_eval
    return _convert(node_or_string)
  File "/usr/local/lib/python3.14/ast.py", line 105, in _convert
    return _convert_signed_num(node)
  File "/usr/local/lib/python3.14/ast.py", line 79, in _convert_signed_num
    return _convert_num(node)
  File "/usr/local/lib/python3.14/ast.py", line 70, in _convert_num
    _raise_malformed_node(node)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.14/ast.py", line 67, in _raise_malformed_node
    raise ValueError(msg + f': {node!r}')
ValueError: malformed node or string on line 1: BinOp(left=Constant(value=1j, kind=None), op=Add(), right=Constant(value=1, kind=None))
>>> ast.literal_eval(ast.Expression(body=ast.BinOp(left=ast.Constant(value=1), op=ast.Add(), right=ast.Constant(value=1j))))
(1+1j)

(Though, literal_eval() also reject string '1j+1', this looks consistent.)

You can make compile() to fail by simply setting invalid lineno -- ast.literal_eval() ignores lineno.

That's why I did fix_missing_locations() in my initial examples.

It does not work for arbitrary invalid AST, this is not its purpose.

Then it should reject invalid input, not silently accept it. We do this for some cases. Why not do it for all?

BTW, the fix is trivial:

diff --git a/Lib/ast.py b/Lib/ast.py
index 45181779c57..646a50f57cd 100644
--- a/Lib/ast.py
+++ b/Lib/ast.py
@@ -20,6 +20,7 @@
 :copyright: Copyright 2008 by Armin Ronacher.
 :license: Python License.
 """
+import types
 from _ast import *
 
 
@@ -68,8 +69,13 @@ def _convert_literal(node):
     """
     Used by `literal_eval` to convert an AST node into a value.
     """
-    if isinstance(node, Constant):
-        return node.value
+    if (
+        isinstance(node, Constant)
+        and type(value := node.value) in (int, float, complex, str,
+                                          bytes, bool, types.NoneType,
+                                          types.EllipsisType)
+    ):
+        return value
     if isinstance(node, Dict) and len(node.keys) == len(node.values):
         return dict(zip(
             map(_convert_literal, node.keys),

[picnixz]

Instead of importing type, I suggest caching type(None) and type(...) as global variables.

[Eclips4]

I'm actually in favor of using the new isinstance checks (or type(x) in (...)). This isn't working as expected, but the real question is: Should we validate the input in the ast.literal_eval? In my opinion, yes.

[skirpichev]

I hope someone will finish patch. Fits to "easy" camp for me.