Expand interpreters module doc introduction.

Author: ericsnowcurrently

Doc/library/concurrent.interpreters.rst

@@ -13,17 +13,26 @@
 
 --------------
 
-
-Introduction
-------------
-
 The :mod:`!concurrent.interpreters` module constructs higher-level
 interfaces on top of the lower level :mod:`!_interpreters` module.
 
-.. XXX Add references to the upcoming HOWTO docs in the seealso block.
+The module is primarily meant to provide a basic API for managing
+interpreters (AKA "subinterpreters") and running things in them.
+Running mostly involves switching to an interpreter (in the current
+thread) and calling a function in that execution context.
+
+For concurrency, interpreters themselves (and this module) don't
+provide much more than isolation, which on its own isn't useful.
+Actual concurrency is available separately through
+:mod:`threads <threading>`  See `below <interp-concurrency_>`_
 
 .. seealso::
 
+   :class:`InterpreterPoolExecutor`
+      combines threads with interpreters in a familiar interface.
+
+    .. XXX Add references to the upcoming HOWTO docs in the seealso block.
+
    :ref:`isolating-extensions-howto`
        how to update an extension module to support multiple interpreters
 
@@ -41,18 +50,125 @@ interfaces on top of the lower level :mod:`!_interpreters` module.
 Key details
 -----------
 
-Before we dive into examples, there are a small number of details
+Before we dive in further, there are a small number of details
 to keep in mind about using multiple interpreters:
 
-* isolated, by default
+* `isolated <interp-isolation_>`_, by default
 * no implicit threads
 * not all PyPI packages support use in multiple interpreters yet
 
 .. XXX Are there other relevant details to list?
 
-In the context of multiple interpreters, "isolated" means that
-different interpreters do not share any state.  In practice, there is some
-process-global data they all share, but that is managed by the runtime.
+
+.. _interpreters-intro:
+
+Introduction
+------------
+
+An "interpreter" is effectively the execution context of the Python
+runtime.  It contains all of the state the runtime needs to execute
+a program.  This includes things like the import state and builtins.
+(Each thread, even if there's only the main thread, has some extra
+runtime state, in addition to the current interpreter, related to
+the current exception and the bytecode eval loop.)
+
+The concept and functionality of the interpreter has been a part of
+Python since version 2.2, but the feature was only available through
+the C-API and not well known, and the `isolation <interp-isolation_>`_
+was relatively incomplete until version 3.12.
+
+.. _interp-isolation:
+
+Multiple Interpreters and Isolation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A Python implementation may support using multiple interpreters in the
+same process.  CPython has this support.  Each interpreter is
+effectively isolated from the others (with a limited number of
+carefully managed process-global exceptions to the rule).
+
+That isolation is primarily useful as a strong separation between
+distinct logical components of a program, where you want to have
+careful control of how those components interact.
+
+.. note::
+
+   Interpreters in the same process can technically never be strictly
+   isolated from one another since there are few restrictions on memory
+   access within the same process.  The Python runtime makes a best
+   effort at isolation but extension modules may easily violate that.
+   Therefore, do not use multiple interpreters in security-senstive
+   situations, where they shouldn't have access to each other's data.
+
+Running in an Interpreter
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Running in a different interpreter involves switching to it in the
+current thread and then calling some function.  The runtime will
+execute the function using the current interpreter's state.  The
+:mod:`!concurrent.interpreters` module provides a basic API for
+creating and managing interpreters, as well as the switch-and-call
+operation.
+
+No other threads are automatically started for the operation.
+There is `a helper <interp-call-in-thread_>`_ for that though.
+There is another dedicated helper for calling the builtin
+:func:`exec` in an interpreter.
+
+When :func:`exec` (or :func:`eval`) are called in an interpreter,
+they run using the interpreter's :mod:`!__main__` module as the
+"globals" namespace.  The same is true for functions that aren't
+associated with any module.  This is the same as how scripts invoked
+from the command-line run in the :mod:`!__main__` module.
+
+
+.. _interp-concurrency:
+
+Concurrency and Parallelism
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As noted earlier, interpreters do not provide any concurrency
+on their own.  They strictly represent the isolated execution
+context the runtime will use *in the current thread*.  That isolation
+makes them similar to processes, but they still enjoy in-process
+efficiency, like threads.
+
+All that said, interpreters do naturally support certain flavors of
+concurrency, as a powerful side effect of that isolation.
+There's a powerful side effect of that isolation.  It enables a
+different approach to concurrency than you can take with async or
+threads.  It's a similar concurrency model to CSP or the actor model,
+a model which is relatively easy to reason about.
+
+You can take advantage of that concurrency model in a single thread,
+switching back and forth between interpreters, Stackless-style.
+However, this model is more useful when you combine interpreters
+with multiple threads.  This mostly involves starting a new thread,
+where you switch to another interpreter and run what you want there.
+
+Each actual thread in Python, even if you're only running in the main
+thread, has its own *current* execution context.  Multiple threads can
+use the same interpreter or different ones.
+
+At a high level, you can think of the combination of threads and
+interpreters as threads with opt-in sharing.
+
+As a significant bonus, interpreters are sufficiently isolated that
+they do not share the :term:`GIL`, which means combining threads with
+multiple interpreters enables full multi-core parallelism.
+(This has been the case since Python 3.12.)
+
+Communication Between Interpreters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In practice, multiple interpreters are useful only if we have a way
+to communicate between them.  This usually involves some form of
+message passing, but can even mean sharing data in some carefully
+managed way.
+
+With this in mind, the :mod:`!concurrent.interpreters` module provides
+a :class:`queue.Queue` implementation, available through
+:func:`create_queue`.
 
 
 Reference
@@ -125,6 +241,8 @@ Interpreter objects
       Return the result of calling running the given function in the
       interpreter (in the current thread).
 
+   .. _interp-call-in-thread:
+
    .. method:: call_in_thread(callable, /, *args, **kwargs)
 
       Run the given function in the interpreter (in a new thread).