gh-148641: Implement PEP 829 - startup configuration files (#149109)

Implement PEP 829 - startup configuration files
Also add `pkgutil.resolve_name(..., strict=True)` 

Co-authored-by: Brett Cannon <brett@python.org>

Author: warsaw

Doc/deprecations/pending-removal-in-3.18.rst

@@ -10,3 +10,9 @@ Pending removal in Python 3.18
     specifier ``'N'``, which is only supported in the :mod:`!decimal` module's
     C implementation, has been deprecated since Python 3.13.
     (Contributed by Serhiy Storchaka in :gh:`89902`.)
+
+* Deprecations defined by :pep:`829`:
+
+  * ``import`` lines in :file:`{name}.pth` files are silently ignored.
+
+  (Contributed by Barry Warsaw in :gh:`148641`.)

Doc/deprecations/pending-removal-in-3.20.rst

@@ -39,6 +39,16 @@ Pending removal in Python 3.20
 
   (Contributed by Hugo van Kemenade and Stan Ulbrych in :gh:`76007`.)
 
+* Deprecations defined by :pep:`829`:
+
+  * Warnings are produced for ``import`` lines found in :file:`{name}.pth`
+    files.
+
+  * :file:`{name}.pth` files are no longer decoded in the locale encoding by
+    default.  They **MUST** be encoded in ``utf-8-sig``.
+
+  (Contributed by Barry Warsaw in :gh:`148641`.)
+
 * :mod:`ast`:
 
   * Creating instances of abstract AST nodes (such as :class:`ast.AST`

Doc/library/pkgutil.rst

@@ -194,7 +194,7 @@ support.
       The :mod:`importlib.resources` module provides structured access to
       module resources.
 
-.. function:: resolve_name(name)
+.. function:: resolve_name(name, *, strict=False)
 
    Resolve a name to an object.
 
@@ -208,6 +208,7 @@ support.
 
    * ``W(.W)*``
    * ``W(.W)*:(W(.W)*)?``
+   * ``W(.W)*:(W(.W)*)``
 
    The first form is intended for backward compatibility only. It assumes that
    some part of the dotted name is a package, and the rest is an object
@@ -222,6 +223,11 @@ support.
    hierarchy within that package. Only one import is needed in this form. If
    it ends with the colon, then a module object is returned.
 
+   The first two forms are accepted when ``strict=False`` (the default).
+
+   The third form requires both the module name and callable, separated by
+   a colon. Only this form is accepted when ``strict=True``.
+
    The function will return an object (which might be a module), or raise one
    of the following exceptions:
 
@@ -233,3 +239,7 @@ support.
    hierarchy within the imported package to get to the desired object.
 
    .. versionadded:: 3.9
+
+   .. versionchanged:: 3.15
+
+      The optional keyword-only ``strict`` flag was added.

Doc/library/site.rst

@@ -17,7 +17,7 @@ import can be suppressed using the interpreter's :option:`-S` option.
 
 Importing this module normally appends site-specific paths to the module search path
 and adds :ref:`callables <site-consts>`, including :func:`help` to the built-in
-namespace. However, Python startup option :option:`-S` blocks this and this module
+namespace. However, Python startup option :option:`-S` blocks this, and this module
 can be safely imported with no automatic modifications to the module search path
 or additions to the builtins.  To explicitly trigger the usual site-specific
 additions, call the :func:`main` function.
@@ -71,40 +71,121 @@ the user site prefixes are also implicitly not searched for site-packages.
    single: # (hash); comment
    pair: statement; import
 
-A path configuration file is a file whose name has the form :file:`{name}.pth`
-and exists in one of the four directories mentioned above; its contents are
-additional items (one per line) to be added to ``sys.path``.  Non-existing items
-are never added to ``sys.path``, and no check is made that the item refers to a
-directory rather than a file.  No item is added to ``sys.path`` more than
-once.  Blank lines and lines beginning with ``#`` are skipped.  Lines starting
-with ``import`` (followed by space or tab) are executed.
+The :mod:`!site` module recognizes two startup configuration files of the form
+:file:`{name}.pth` for path configurations, and :file:`{name}.start` for
+pre-first-line code execution.  Both files can exist in one of the four
+directories mentioned above.  Within each directory, these files are sorted
+alphabetically by filename, then parsed in sorted order.
 
-.. note::
+.. _site-pth-files:
 
-   An executable line in a :file:`.pth` file is run at every Python startup,
-   regardless of whether a particular module is actually going to be used.
-   Its impact should thus be kept to a minimum.
-   The primary intended purpose of executable lines is to make the
-   corresponding module(s) importable
-   (load 3rd-party import hooks, adjust :envvar:`PATH` etc).
-   Any other initialization is supposed to be done upon a module's
-   actual import, if and when it happens.
-   Limiting a code chunk to a single line is a deliberate measure
-   to discourage putting anything more complex here.
+Path extensions (:file:`.pth` files)
+------------------------------------
+
+:file:`{name}.pth` contains additional items (one per line) to be appended to
+``sys.path``.  Items that name non-existing directories are never added to
+``sys.path``, and no check is made that the item refers to a directory rather
+than a file.  No item is added to ``sys.path`` more than once.  Blank lines
+and lines beginning with ``#`` are skipped.
+
+For backward compatibility, lines starting with ``import`` (followed by space
+or tab) are executed with :func:`exec`.
 
 .. versionchanged:: 3.13
+
    The :file:`.pth` files are now decoded by UTF-8 at first and then by the
    :term:`locale encoding` if it fails.
 
+.. versionchanged:: next
+
+   :file:`.pth` file lines starting with ``import`` are deprecated.  During
+   the deprecation period, such lines are still executed (except in the case
+   below), but a diagnostic message is emitted only when the :option:`-v` flag
+   is given.
+
+   ``import`` lines in :file:`{name}.pth` are silently ignored when a
+   :ref:`matching <site-start-files>` :file:`{name}.start` file exists.
+
+   Errors on individual lines no longer abort processing of the rest of the
+   file.  Each error is reported and the remaining lines continue to be
+   processed.
+
+.. deprecated-removed:: next 3.20
+
+   Decoding :file:`{name}.pth` files in any encoding other than ``utf-8-sig``
+   is deprecated in Python 3.15, and support for decoding from the locale
+   encoding will be removed in Python 3.20.
+
+   ``import`` lines in :file:`{name}.pth` files are deprecated and will be
+   silently ignored in Python 3.18 and 3.19.  In Python 3.20 a warning will be
+   produced for ``import`` lines in :file:`{name}.pth` files.
+
+
+.. _site-start-files:
+
+Startup entry points (:file:`.start` files)
+-------------------------------------------
+
+.. versionadded:: next
+
+A startup entry point file is a file whose name has the form
+:file:`{name}.start` and exists in one of the site-packages directories
+described above.  Each file specifies entry points to be called during
+interpreter startup, using the ``pkg.mod:callable`` syntax understood by
+:func:`pkgutil.resolve_name`.
+
+Each non-blank line that does not begin with ``#`` must contain an entry
+point reference in the form ``pkg.mod:callable``.  The colon and callable
+portion are mandatory.  Each callable is invoked with no arguments, and
+any return value is discarded.
+
+:file:`.start` files are processed after all :file:`.pth` path extensions
+have been applied to :data:`sys.path`, ensuring that paths are available
+before any startup code runs.
+
+Unlike :data:`sys.path` extensions from :file:`.pth` files, duplicate entry
+points are **not** de-duplicated --- if an entry point appears more than once,
+it will be called more than once.
+
+If an exception occurs during resolution or invocation of an entry point,
+a traceback is printed to :data:`sys.stderr` and processing continues with
+the remaining entry points.
+
+:file:`.start` files must be encoded in UTF-8.
+
+:pep:`829` defined the original specification for these features.
+
+.. note::
+
+   If a :file:`{name}.start` file exists alongside a :file:`{name}.pth` file
+   with the same base name, any ``import`` lines in the :file:`.pth` file are
+   ignored in favor of the entry points in the :file:`.start` file.
+
+.. note::
+
+   Executable lines (``import`` lines in :file:`{name}.pth` files and
+   :file:`{name}.start` file entry points) are always run at Python startup
+   (unless :option:`-S` is given to disable the ``site.py`` module entirely),
+   regardless of whether a particular module is actually going to be used.
+
+.. note::
+
+   :file:`{name}.start` files invoke :func:`pkgutil.resolve_name` with
+   ``strict=True``, which requires the full ``pkg.mod:callable`` form.
+
 .. index::
    single: package
    triple: path; configuration; file
 
+
+Startup file examples
+---------------------
+
 For example, suppose ``sys.prefix`` and ``sys.exec_prefix`` are set to
 :file:`/usr/local`.  The Python X.Y library is then installed in
 :file:`/usr/local/lib/python{X.Y}`.  Suppose this has
 a subdirectory :file:`/usr/local/lib/python{X.Y}/site-packages` with three
-subsubdirectories, :file:`foo`, :file:`bar` and :file:`spam`, and two path
+sub-subdirectories, :file:`foo`, :file:`bar` and :file:`spam`, and two path
 configuration files, :file:`foo.pth` and :file:`bar.pth`.  Assume
 :file:`foo.pth` contains the following::
 
@@ -131,6 +212,45 @@ directory precedes the :file:`foo` directory because :file:`bar.pth` comes
 alphabetically before :file:`foo.pth`; and :file:`spam` is omitted because it is
 not mentioned in either path configuration file.
 
+Let's say that there is also a :file:`foo.start` file containing the
+following::
+
+    # foo package startup code
+
+    foo.submod:initialize
+
+Now, after ``sys.path`` has been extended as above, and before Python turns
+control over to user code, the ``foo.submod`` module is imported and the
+``initialize()`` function from that module is called.
+
+
+.. _site-migration-guide:
+
+Migrating from ``import`` lines in ``.pth`` files to ``.start`` files
+---------------------------------------------------------------------
+
+If your package currently ships a :file:`{name}.pth` file, you can keep all
+``sys.path`` extension lines unchanged.  Only ``import`` lines need to be
+migrated.
+
+To migrate, create a callable (taking zero arguments) within an importable
+module in your package.  Reference it as a ``pkg.mod:callable`` entry point
+in a matching :file:`{name}.start` file.  Move everything on your ``import``
+line after the first semi-colon into the ``callable()`` function.
+
+If your package must straddle older Pythons that do not support :pep:`829`
+and newer Pythons that do, change the ``import`` lines in your
+:file:`{name}.pth` to use the following form:
+
+.. code-block:: python
+
+   import pkg.mod; pkg.mod.callable()
+
+Older Pythons will execute these ``import`` lines, while newer Pythons will
+ignore them in favor of the :file:`{name}.start` file.  After the straddling
+period, remove all ``import`` lines from your :file:`.pth` files.
+
+
 :mod:`!sitecustomize`
 ---------------------
 
@@ -236,10 +356,27 @@ Module contents
       This function used to be called unconditionally.
 
 
-.. function:: addsitedir(sitedir, known_paths=None)
+.. function:: addsitedir(sitedir, known_paths=None, *, defer_processing_start_files=False)
+
+   Add a directory to sys.path and parse the :file:`.pth` and :file:`.start`
+   files found in that directory.  Typically used in :mod:`sitecustomize` or
+   :mod:`usercustomize` (see above).
+
+   The *known_paths* argument is an optional set of case-normalized paths
+   used to prevent duplicate :data:`sys.path` entries.  When ``None`` (the
+   default), the set is built from the current :data:`sys.path`.
+
+   While :file:`.pth` and :file:`.start` files are always parsed, set
+   *defer_processing_start_files* to ``True`` to prevent processing the
+   startup data found in those files, so that you can process them explicitly
+   (this is typically used by the :func:`main` function).
+
+   .. versionchanged:: next
 
-   Add a directory to sys.path and process its :file:`.pth` files.  Typically
-   used in :mod:`sitecustomize` or :mod:`usercustomize` (see above).
+      Also processes :file:`.start` files.  See :ref:`site-start-files`.
+      All :file:`.pth` and :file:`.start` files are now read and
+      accumulated before any path extensions, ``import`` line execution,
+      or entry point invocations take place.
 
 
 .. function:: getsitepackages()
@@ -308,5 +445,6 @@ value greater than 2 if there is an error.
 .. seealso::
 
    * :pep:`370` -- Per user site-packages directory
+   * :pep:`829` -- Startup entry points and the deprecation of import lines in ``.pth`` files
    * :ref:`sys-path-init` -- The initialization of :data:`sys.path`.
 

Doc/whatsnew/3.15.rst

@@ -91,6 +91,7 @@ Summary -- Release highlights
 * :ref:`Improved error messages <whatsnew315-improved-error-messages>`
 * :ref:`The official Windows 64-bit binaries now use the tail-calling interpreter
   <whatsnew315-windows-tail-calling-interpreter>`
+* :pep:`829`: Package Startup Configuration Files
 
 New features
 ============

Lib/pkgutil.py

@@ -9,6 +9,9 @@
 import os.path
 import sys
 
+lazy import re
+
+
 __all__ = [
     'get_importer', 'iter_importers',
     'walk_packages', 'iter_modules', 'get_data',
@@ -398,9 +401,10 @@ def get_data(package, resource):
     return loader.get_data(resource_name)
 
 
-_NAME_PATTERN = None
+_LENIENT_PATTERN = None
+_STRICT_PATTERN = None
 
-def resolve_name(name):
+def resolve_name(name, *, strict=False):
     """
     Resolve a name to an object.
 
@@ -410,6 +414,7 @@ def resolve_name(name):
 
     W(.W)*
     W(.W)*:(W(.W)*)?
+    W(.W)*:(W(.W)*)
 
     The first form is intended for backward compatibility only. It assumes that
     some part of the dotted name is a package, and the rest is an object
@@ -424,6 +429,11 @@ def resolve_name(name):
     hierarchy within that package. Only one import is needed in this form. If
     it ends with the colon, then a module object is returned.
 
+    The first two forms are accepted when `strict=False` (the default).
+
+    The third form requires both the module name and callable, separated by
+    a colon. Only this form is accepted when `strict=True`.
+
     The function will return an object (which might be a module), or raise one
     of the following exceptions:
 
@@ -432,18 +442,26 @@ def resolve_name(name):
     AttributeError - if a failure occurred when traversing the object hierarchy
                      within the imported package to get to the desired object.
     """
-    global _NAME_PATTERN
-    if _NAME_PATTERN is None:
-        # Lazy import to speedup Python startup time
-        import re
-        dotted_words = r'(?!\d)(\w+)(\.(?!\d)(\w+))*'
-        _NAME_PATTERN = re.compile(f'^(?P<pkg>{dotted_words})'
-                                   f'(?P<cln>:(?P<obj>{dotted_words})?)?$',
-                                   re.UNICODE)
-
-    m = _NAME_PATTERN.match(name)
-    if not m:
+    global _LENIENT_PATTERN, _STRICT_PATTERN
+    dotted_words = r'(?!\d)(\w+)(\.(?!\d)(\w+))*'
+    if strict:
+        if _STRICT_PATTERN is None:
+            _STRICT_PATTERN = re.compile(
+                f'^(?P<pkg>{dotted_words})'
+                f'(?P<cln>:(?P<obj>{dotted_words}))$',
+                re.UNICODE)
+        pattern = _STRICT_PATTERN
+    else:
+        if _LENIENT_PATTERN is None:
+            _LENIENT_PATTERN = re.compile(
+                f'^(?P<pkg>{dotted_words})'
+                f'(?P<cln>:(?P<obj>{dotted_words})?)?$',
+                re.UNICODE)
+        pattern = _LENIENT_PATTERN
+
+    if (m := pattern.match(name)) is None:
         raise ValueError(f'invalid format: {name!r}')
+
     gd = m.groupdict()
     if gd.get('cln'):
         # there is a colon - a one-step import is all that's needed