Ewoks install bootstrap

.github/workflows/test.yml

@@ -79,6 +79,25 @@ jobs:
             enable-coverage: "false"
             extra-pytest-warnings: ""
 
+  # Run the tests that install package managers, on runners without them
+  tests-package-manager-install:
+    needs: build
+    uses: ewoks-kit/.github/.github/workflows/python-tests.yml@main
+    with:
+      os: ${{ matrix.os }}
+      python-version: "3.12"
+      pytest-args: >-
+        -v -ra -W error -m package_manager_install
+        -W ignore::DeprecationWarning:jupyter_client.session
+        -W ignore::DeprecationWarning:orangecanvas.utils.localization
+        -W ignore::DeprecationWarning:orangecanvas.utils.localization.si
+        -W ignore::DeprecationWarning:matplotlib._fontconfig_pattern
+        -W ignore::DeprecationWarning:matplotlib._mathtext
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+
   # Run linter / checks
   checks:
     uses: ewoks-kit/.github/.github/workflows/python-check.yml@main

CHANGELOG.md

@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - add `--in-place`, `--python-version` and `--with-ewoks` arguments.
 - `ewoks convert`: add `--package-manager-name` and `--package-manager-command` arguments.
 - `ewoks execute`: add `--env` argument.
+- Add `ewoks-install.sh` and `ewoks-install.ps1` scripts to install the environment of a workflow without ewoks.
 - Add `ewoks lint` CLI command to check if a workflow is conform to the Ewoks specification.
 
 ### Changed

doc/conf.py

@@ -3,6 +3,13 @@
 
 
 import importlib.metadata
+import os
+from typing import Optional
+
+from ewokscore.tests.examples.graphs import get_graph
+from sphinx.application import Sphinx
+
+from ewoks.bindings import convert_graph
 
 release = importlib.metadata.version("ewoks")
 
@@ -76,3 +83,45 @@
     "footer_start": ["copyright"],
     "footer_end": ["footer_end"],
 }
+
+
+# The ewoks-install scripts
+html_extra_path = ["../src/ewoks/_bootstrap"]
+
+# Root of the documentation version that is built: on Read the Docs, for example
+# .../en/stable, or on GitLab Pages
+_DOCS_URL = (
+    os.environ.get("READTHEDOCS_CANONICAL_URL")
+    or os.environ.get("CI_PAGES_URL")
+    or "https://ewoks.readthedocs.io/en/latest"
+).rstrip("/")
+
+
+def _substitute_docs_url(app, docname, source):
+    # Substitutions are not supported in code blocks
+    source[0] = source[0].replace("|docs_url|", _DOCS_URL)
+
+
+def _save_demo_workflow(app: Sphinx, exception: Optional[Exception]) -> None:
+    """Save the demo workflow next to the ewoks-install scripts, with only ewoks and
+    its engine as requirements instead of all the packages of the documentation
+    build."""
+    if exception is not None or app.builder.format != "html":
+        return
+    graph, _ = get_graph("demo")
+    workflow = convert_graph(graph, None)
+    requirements = workflow["graph"]["requirements"]
+    names = ["ewoks", requirements["ewoks"]["engine"]["name"]]
+    requirements["distributions"] = [
+        {"name": name, "version": importlib.metadata.version(name)} for name in names
+    ]
+    requirements["manager"]["files"] = {}
+    filename = os.path.join(app.outdir, "demo.json")
+    convert_graph(
+        workflow, filename, save_options={"indent": 2}, save_requirements=False
+    )
+
+
+def setup(app):
+    app.connect("source-read", _substitute_docs_url)
+    app.connect("build-finished", _save_demo_workflow)

doc/explanations/task_input_priority.rst

@@ -7,7 +7,7 @@ Details
 A :term:`node <Nodes>` in a :term:`workflow` can get inputs from three different sources:
 
 1. Via the ``data_mapping`` :term:`link <Links>` attribute of an incoming :term:`link <Links>` (see `Link attributes <https://ewokscore.readthedocs.io/en/stable/definitions.html#link-attributes>`_)
-2. Via the ``parameters`` CLI argument (or ``inputs`` for Python) when executing/submitting the :term:`workflow` (see `ewoks execute reference <https://ewoks.readthedocs.io/en/stable/cli.html#ewoks-execute>`_)
+2. Via the ``parameters`` CLI argument (or ``inputs`` for Python) when executing/submitting the :term:`workflow` (see :ref:`ewoks execute reference <cli_execute>`)
 3. Via the ``default_inputs`` :term:`node <Nodes>` attribute of the :term:`node <Nodes>` itself (see `Node attributes <https://ewokscore.readthedocs.io/en/stable/definitions.html#node-attributes>`_)
 
 If the same input is specified by these different sources, :term:`Ewoks` applies the following priorities:

doc/howtoguides/requirements.rst

@@ -13,6 +13,9 @@ What is stored
 
 ``ewoks convert`` and ``ewoks execute -o convert_destination=...`` store
 
+* ``ewoks``: the version of :term:`ewoks` that generated the requirements and the engine
+  that saved the :term:`workflow`. The
+  :ref:`ewoks-install script <install_bootstrap>` installs both to run ``ewoks install``.
 * ``python`` and ``system``: the python interpreter and the operating system.
 * ``distributions``: every installed python package with its version and, when it was not
   installed from the python package index, the git commit or the archive it came from. Any

doc/reference/cli.rst

@@ -63,6 +63,8 @@ ewoks convert
 
     **ewoks convert** can also be used to store ``inputs`` inside the destination :term:`workflow`.
 
+.. _cli_execute:
+
 ewoks execute
 -------------
 

doc/tutorials/install.rst

@@ -40,6 +40,7 @@ Store the requirements
 .. code-block:: json
 
     {
+        "ewoks": {"version": "7.0.0", "engine": {"name": "ewokscore", "version": "5.1.0"}},
         "python": {"version": "3.12.11", "implementation": "CPython", "...": "..."},
         "system": {"system": "Linux", "machine": "x86_64", "...": "..."},
         "distributions": [
@@ -53,6 +54,9 @@ Store the requirements
         }
     }
 
+* ``ewoks`` is the version of :term:`ewoks` that generated the requirements and the engine that
+  saved the :term:`workflow`: ``ewokscore`` for a ``.json`` or ``.yaml`` file and
+  ``ewoksorange`` for an ``.ows`` file.
 * ``distributions`` are the installed python packages. Any :term:`package manager` can
   recreate the environment from this list.
 * ``manager`` is the :term:`package manager` that generated the requirements together with
@@ -63,19 +67,123 @@ Create the environment
 
 ``ewoks install`` creates a python environment for the :term:`workflow`. It prints the python
 interpreter of the environment it created, the command to execute the :term:`workflow` in it
-and the command to remove it again
+and the command to remove it again. These commands work from any terminal and directory
 
 .. code-block:: text
 
     Installed requirements for demo.json
-      Python : ewoks_envs/demo/bin/python
-      Execute: ewoks execute --env ewoks_envs/demo demo.json
-      Remove : rm -rf ewoks_envs/demo
+      Python : /home/user/ewoks_envs/demo/bin/python
+      Execute: /home/user/ewoks_envs/demo/bin/python -m ewoks execute /home/user/demo.json
+      Remove : rm -rf /home/user/ewoks_envs/demo
 
 Without ``--yes`` you are asked to confirm after the packages have been listed. The
 walk-throughs use ``--env-root`` to create the environment in the working directory. Without
 it the environment is created where the :term:`package manager` creates named environments.
 
+.. _install_bootstrap:
+
+Create the environment without ewoks
+++++++++++++++++++++++++++++++++++++
+
+``ewoks install`` needs a python environment with :term:`ewoks`. The ``ewoks-install`` script
+only needs a :term:`package manager`: it installs the version of :term:`ewoks` that generated
+the requirements, and the engine that saved the :term:`workflow`, in a bootstrap environment
+and runs ``ewoks install`` from there. All
+arguments are passed to ``ewoks install``, except for the options of the script itself
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            & ([scriptblock]::Create((irm |docs_url|/ewoks-install.ps1))) demo.json
+
+Without a :term:`workflow` at hand, try it with the demo :term:`workflow` of this
+documentation, which only requires :term:`ewoks`
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            irm |docs_url|/demo.json -OutFile demo.json
+            & ([scriptblock]::Create((irm |docs_url|/ewoks-install.ps1))) demo.json
+
+Alternatively, download the script first, for example to read it before running it
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSf -o ewoks-install.sh |docs_url|/ewoks-install.sh
+            sh ewoks-install.sh demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSf -o ewoks-install.sh |docs_url|/ewoks-install.sh
+            sh ewoks-install.sh demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            irm |docs_url|/ewoks-install.ps1 -OutFile ewoks-install.ps1
+            powershell -ExecutionPolicy Bypass -File ewoks-install.ps1 demo.json
+
+* ``--package-manager-name`` and ``--package-manager-command`` select the
+  :term:`package manager` that creates the bootstrap environment. Without them it is the first
+  one that is available of uv, pixi, conda, poetry and pip-venv.
+* ``--print-command`` prints the ``ewoks install`` command instead of running it.
+* ``--bootstrap-requirement`` installs a package in the bootstrap environment instead of the
+  version of :term:`ewoks` and the engine of the requirements, for example
+  ``--bootstrap-requirement "ewoks>=7"``. Repeat it to install an engine as well, for example
+  ``--bootstrap-requirement ewoks --bootstrap-requirement ewoksorange``. The latest version of
+  :term:`ewoks` is installed when the requirements do not provide one.
+* ``--bootstrap-dir`` is the directory of the bootstrap environments.
+* ``--install-package-manager`` installs the :term:`package manager` of
+  ``--package-manager-name`` in the bootstrap directory when it is not installed. Conda
+  cannot be installed in a directory with spaces.
+
+The script keeps the bootstrap environment in ``~/.ewoks/bootstrap`` and uses it again when
+it installs another :term:`workflow` whose requirements were generated by the same version of
+:term:`ewoks`. The bootstrap environment only runs ``ewoks install``:
+it is not the environment of the :term:`workflow`. Before the environment of the
+:term:`workflow`, the script prints the python interpreter of the bootstrap environment, the
+command to run ``ewoks install`` again and the command to remove the bootstrap environment. When the script fails to create the bootstrap
+environment, it prints how to install without it.
+
 Execute the workflow
 --------------------
 

doc/tutorials/install/conda.rst

@@ -89,6 +89,29 @@ packages of the python package index.
 Re-producer side
 ----------------
 
+Without the producer side, download the demo :term:`workflow` of this documentation, which
+only requires :term:`ewoks`
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            irm |docs_url|/demo.json -OutFile demo.json
+
 Only ``ewoks`` itself is needed to recreate the environment of ``demo.json``
 
 .. tabs::
@@ -124,6 +147,32 @@ The environment of the :term:`workflow` is a conda environment in ``ewoks_envs/d
 ``--env-root`` it is created in the first environment directory of conda, where
 ``conda activate demo`` finds it.
 
+Alternatively, the :ref:`ewoks-install script <install_bootstrap>` recreates the
+environment without an environment with :term:`ewoks`
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            & ([scriptblock]::Create((irm |docs_url|/ewoks-install.ps1))) demo.json
+
+It selects the :term:`package manager` itself and prints the command to execute the
+:term:`workflow` in the environment it creates.
+
 Execute the workflow
 --------------------
 

doc/tutorials/install/pip_venv.rst

@@ -50,6 +50,29 @@ Install any package that provides :term:`tasks <Task>` instead of
 Re-producer side
 ----------------
 
+Without the producer side, download the demo :term:`workflow` of this documentation, which
+only requires :term:`ewoks`
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSfO |docs_url|/demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            irm |docs_url|/demo.json -OutFile demo.json
+
 Only ``ewoks`` itself is needed to recreate the environment of ``demo.json``
 
 .. tabs::
@@ -83,6 +106,32 @@ Only ``ewoks`` itself is needed to recreate the environment of ``demo.json``
 
 The environment of the :term:`workflow` is a virtual environment in ``ewoks_envs/demo``.
 
+Alternatively, the :ref:`ewoks-install script <install_bootstrap>` recreates the
+environment without an environment with :term:`ewoks`
+
+.. tabs::
+
+    .. group-tab:: Linux
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: macOS
+
+        .. code-block:: bash
+
+            curl -LsSf |docs_url|/ewoks-install.sh | sh -s -- demo.json
+
+    .. group-tab:: Windows
+
+        .. code-block:: powershell
+
+            & ([scriptblock]::Create((irm |docs_url|/ewoks-install.ps1))) demo.json
+
+It selects the :term:`package manager` itself and prints the command to execute the
+:term:`workflow` in the environment it creates.
+
 Execute the workflow
 --------------------