Merge remote-tracking branch 'upstream/main' into gh-149306-wave-error-messages

Author: htjworld

.github/workflows/posix-deps-apt.sh

@@ -26,9 +26,16 @@ apt-get -yq --no-install-recommends install \
     xvfb \
     zlib1g-dev
 
-# Workaround missing libmpdec-dev on ubuntu 24.04:
-# https://launchpad.net/~ondrej/+archive/ubuntu/php
-# https://deb.sury.org/
-sudo add-apt-repository ppa:ondrej/php
-apt-get update
-apt-get -yq --no-install-recommends install libmpdec-dev
+# Workaround missing libmpdec-dev on ubuntu 24.04 by building mpdecimal
+# from source. ppa:ondrej/php (launchpad.net) are unreliable 
+# (https://status.canonical.com) so fetch the tarball directly
+# from the upstream host.
+# https://www.bytereef.org/mpdecimal/
+MPDECIMAL_VERSION=4.0.1
+curl -fsSL "https://www.bytereef.org/software/mpdecimal/releases/mpdecimal-${MPDECIMAL_VERSION}.tar.gz" \
+    | tar -xz -C /tmp
+(cd "/tmp/mpdecimal-${MPDECIMAL_VERSION}" \
+    && ./configure --prefix=/usr/local \
+    && make -j"$(nproc)" \
+    && make install)
+ldconfig

Doc/c-api/perfmaps.rst

@@ -31,7 +31,7 @@ Note that holding an :term:`attached thread state` is not required for these API
    or ``-2`` on failure to create a lock. Check ``errno`` for more information
    about the cause of a failure.
 
-.. c:function:: int PyUnstable_WritePerfMapEntry(const void *code_addr, unsigned int code_size, const char *entry_name)
+.. c:function:: int PyUnstable_WritePerfMapEntry(const void *code_addr, size_t code_size, const char *entry_name)
 
    Write one single entry to the ``/tmp/perf-$pid.map`` file. This function is
    thread safe. Here is what an example entry looks like::

Doc/deprecations/pending-removal-in-3.20.rst

@@ -38,3 +38,8 @@ Pending removal in Python 3.20
   - :mod:`zlib`
 
   (Contributed by Hugo van Kemenade and Stan Ulbrych in :gh:`76007`.)
+
+* :mod:`ast`:
+
+  * Creating instances of abstract AST nodes (such as :class:`ast.AST`
+    or :class:`!ast.expr`) is deprecated and will raise an error in Python 3.20.

Doc/faq/programming.rst

@@ -1924,7 +1924,7 @@ correctly using identity tests:
 
    .. code-block:: python
 
-      _sentinel = object()
+      _sentinel = sentinel('_sentinel')
 
       def pop(self, key, default=_sentinel):
           if key in self:

Doc/glossary.rst

@@ -39,10 +39,11 @@ Glossary
       ABCs with the :mod:`abc` module.
 
    annotate function
-      A function that can be called to retrieve the :term:`annotations <annotation>`
-      of an object. This function is accessible as the :attr:`~object.__annotate__`
-      attribute of functions, classes, and modules. Annotate functions are a
-      subset of :term:`evaluate functions <evaluate function>`.
+      A callable that can be called to retrieve the :term:`annotations <annotation>` of
+      an object. Annotate functions are usually :term:`functions <function>`,
+      automatically generated as the :attr:`~object.__annotate__` attribute of functions,
+      classes, and modules. Annotate functions are a subset of
+      :term:`evaluate functions <evaluate function>`.
 
    annotation
       A label associated with a variable, a class

Doc/howto/descriptor.rst

@@ -594,7 +594,7 @@ a pure Python equivalent:
 
     def object_getattribute(obj, name):
         "Emulate PyObject_GenericGetAttr() in Objects/object.c"
-        null = object()
+        null = sentinel('null')
         objtype = type(obj)
         cls_var = find_name_in_mro(objtype, name, null)
         descr_get = getattr(type(cls_var), '__get__', null)
@@ -1635,12 +1635,12 @@ by member descriptors:
 
 .. testcode::
 
-    null = object()
+    null = sentinel('null')
 
     class Member:
 
         def __init__(self, name, clsname, offset):
-            'Emulate PyMemberDef in Include/structmember.h'
+            'Emulate PyMemberDef in Include/descrobject.h'
             # Also see descr_new() in Objects/descrobject.c
             self.name = name
             self.clsname = clsname

Doc/howto/perf_profiling.rst

@@ -217,8 +217,9 @@ Example, using the :mod:`sys` APIs in file :file:`example.py`:
 How to obtain the best results
 ------------------------------
 
-For best results, Python should be compiled with
-``CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"`` as this allows
+For best results, keep frame pointers enabled. On supported GCC-compatible
+toolchains, CPython builds itself with ``-fno-omit-frame-pointer`` and, when
+available, ``-mno-omit-leaf-frame-pointer`` by default. These flags allow
 profilers to unwind using only the frame pointer and not on DWARF debug
 information. This is because as the code that is interposed to allow ``perf``
 support is dynamically generated it doesn't have any DWARF debugging information

Doc/library/annotationlib.rst

@@ -510,6 +510,81 @@ annotations from the class and puts them in a separate attribute:
          return typ
 
 
+Creating a custom callable annotate function
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Custom :term:`annotate functions <annotate function>` may be literal functions like those
+automatically generated for functions, classes, and modules. Or, they may wish to utilise
+the encapsulation provided by classes, in which case any :term:`callable` can be used as
+an :term:`annotate function`.
+
+To provide the :attr:`~Format.VALUE`, :attr:`~Format.STRING`, or
+:attr:`~Format.FORWARDREF` formats directly, an :term:`annotate function` must provide
+the following attribute:
+
+* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
+  raise a :exc:`NotImplementedError` when called with a supported format.
+
+To provide the :attr:`~Format.VALUE_WITH_FAKE_GLOBALS` format, which is used to
+automatically generate :attr:`~Format.STRING` or :attr:`~Format.FORWARDREF` if they are
+not supported directly, :term:`annotate functions <annotate function>` must provide the
+following attributes:
+
+* A callable ``__call__`` with signature ``__call__(format, /) -> dict``, that does not
+  raise a :exc:`NotImplementedError` when called with
+  :attr:`~Format.VALUE_WITH_FAKE_GLOBALS`.
+* A :ref:`code object <code-objects>` ``__code__`` containing the compiled code for the
+  annotate function.
+* Optional: A tuple of the function's positional defaults ``__kwdefaults__``, if the
+  function represented by ``__code__`` uses any positional defaults.
+* Optional: A dict of the function's keyword defaults ``__defaults__``, if the function
+  represented by ``__code__`` uses any keyword defaults.
+* Optional: All other :ref:`function attributes <inspect-types>`.
+
+.. code-block:: python
+
+   class Annotate:
+       called_formats = []
+
+       def __call__(self, format=None, /, *, _self=None):
+           # When called with fake globals, `_self` will be the
+           # actual self value, and `self` will be the format.
+           if _self is not None:
+               self, format = _self, self
+
+           self.called_formats.append(format)
+           if format <= 2:  # VALUE or VALUE_WITH_FAKE_GLOBALS
+               return {"x": MyType}
+           raise NotImplementedError
+
+       __code__ = __call__.__code__
+       __defaults__ = (None,)
+       __kwdefaults__ = property(lambda self: dict(_self=self))
+
+       __globals__ = {}
+       __builtins__ = {}
+       __closure__ = None
+
+This can then be called with:
+
+.. code-block:: pycon
+
+   >>> from annotationlib import call_annotate_function, Format
+   >>> call_annotate_function(Annotate(), format=Format.STRING)
+   {'x': 'MyType'}
+
+Or used as the annotate function for an object:
+
+.. code-block:: pycon
+
+   >>> from annotationlib import get_annotations, Format
+   >>> class C:
+   ...   pass
+   >>> C.__annotate__ = Annotate()
+   >>> get_annotations(Annotate(), format=Format.STRING)
+   {'x': 'MyType'}
+
+
 Limitations of the ``STRING`` format
 ------------------------------------
 

Doc/library/ast.rst

@@ -42,7 +42,7 @@ Node classes
 
 .. class:: AST
 
-   This is the base of all AST node classes.  The actual node classes are
+   This is the abstract base of all AST node classes.  The actual node classes are
    derived from the :file:`Parser/Python.asdl` file, which is reproduced
    :ref:`above <abstract-grammar>`.  They are defined in the :mod:`!_ast` C
    module and re-exported in :mod:`!ast`.
@@ -168,6 +168,15 @@ Node classes
    arguments that were set as attributes of the AST node, even if they did not
    match any of the fields of the AST node. These cases now raise a :exc:`TypeError`.
 
+.. deprecated-removed:: next 3.20
+
+    In the :ref:`grammar above <abstract-grammar>`, the AST node classes that
+    correspond to production rules with variants (aka "sums") are abstract
+    classes. Previous versions of Python allowed for the creation of direct
+    instances of these abstract node classes. This behavior is deprecated and
+    will be removed in Python 3.20.
+
+
 .. note::
     The descriptions of the specific node classes displayed here
     were initially adapted from the fantastic `Green Tree
@@ -271,18 +280,25 @@ Root nodes
 Literals
 ^^^^^^^^
 
-.. class:: Constant(value)
+.. class:: Constant(value, kind)
 
    A constant value. The ``value`` attribute of the ``Constant`` literal contains the
    Python object it represents. The values represented can be instances of :class:`str`,
    :class:`bytes`, :class:`int`, :class:`float`, :class:`complex`, and :class:`bool`,
    and the constants :data:`None` and :data:`Ellipsis`.
 
+   The ``kind`` attribute is an optional string. For string literals with a
+   ``u`` prefix, ``kind`` is set to ``'u'``. For all other
+   constants, ``kind`` is ``None``.
+
    .. doctest::
 
         >>> print(ast.dump(ast.parse('123', mode='eval'), indent=4))
         Expression(
             body=Constant(value=123))
+        >>> print(ast.dump(ast.parse("u'hello'", mode='eval'), indent=4))
+        Expression(
+            body=Constant(value='hello', kind='u'))
 
 
 .. class:: FormattedValue(value, conversion, format_spec)
@@ -2536,6 +2552,20 @@ and classes for traversing abstract syntax trees:
       Added the *color* parameter.
 
 
+.. function:: compare(a, b, /, *, compare_attributes=False)
+
+   Recursively compares two ASTs.
+
+   *compare_attributes* affects whether AST attributes are considered
+   in the comparison. If *compare_attributes* is ``False`` (default), then
+   attributes are ignored. Otherwise they must all be equal. This
+   option is useful to check whether the ASTs are structurally equal but
+   differ in whitespace or similar details. Attributes include line numbers
+   and column offsets.
+
+   .. versionadded:: 3.14
+
+
 .. _ast-compiler-flags:
 
 Compiler flags
@@ -2571,20 +2601,6 @@ effects on the compilation of a program:
    .. versionadded:: 3.8
 
 
-.. function:: compare(a, b, /, *, compare_attributes=False)
-
-   Recursively compares two ASTs.
-
-   *compare_attributes* affects whether AST attributes are considered
-   in the comparison. If *compare_attributes* is ``False`` (default), then
-   attributes are ignored. Otherwise they must all be equal. This
-   option is useful to check whether the ASTs are structurally equal but
-   differ in whitespace or similar details. Attributes include line numbers
-   and column offsets.
-
-   .. versionadded:: 3.14
-
-
 .. _ast-cli:
 
 Command-line usage

Doc/library/compression.zstd.rst

@@ -331,10 +331,14 @@ Compressing and decompressing data in memory
 
       If *max_length* is non-negative, the method returns at most *max_length*
       bytes of decompressed data. If this limit is reached and further
-      output can be produced, the :attr:`~.needs_input` attribute will
-      be set to ``False``. In this case, the next call to
+      output can be produced (or EOF is reached), the :attr:`~.needs_input`
+      attribute will be set to ``False``. In this case, the next call to
       :meth:`~.decompress` may provide *data* as ``b''`` to obtain
-      more of the output.
+      more of the output. The full content can thus be read like::
+
+        process_output(d.decompress(data, max_length))
+        while not d.eof and not d.needs_input:
+            process_output(d.decompress(b"", max_length))
 
       If all of the input data was decompressed and returned (either
       because this was less than *max_length* bytes, or because