Build fprime_py for the Python deployment only, install Python files on every build, and add a CMake hook and component cookiecutter

.github/workflows/check.yml

@@ -50,5 +50,5 @@ jobs:
       - name: Run mypy
         run: python -m mypy
 
-      - name: Run generator tests
+      - name: Run tests
         run: python -m pytest tests

README.md

@@ -45,12 +45,22 @@ autocoder and its dependencies by running `pip install .` in the `fprime-python`
 or newer. The autocoder reads the FPP model directly, so `FPRIME_ENABLE_JSON_MODEL_GENERATION` and
 `fprime-python-model` are no longer needed.
 
-Next, add the path to the download in the `library_locations` list set in settings.ini for a deployment. 
+Next, add the path to the download in the `library_locations` list set in the project's `settings.ini`.
 
 ```ini
 library_locations: ./lib/fprime-python
 ```
 
+Then include `FprimePython` from the project's top-level `CMakeLists.txt`, after `FPrime.cmake` and before
+`fprime_setup_included_code()`. It locates the pip-installed pybind11, compiles the project as position
+independent code, and registers the `fprime_python` build target that generates and builds the bindings.
+
+```cmake
+include("${CMAKE_CURRENT_LIST_DIR}/lib/fprime/cmake/FPrime.cmake")
+include(FprimePython)
+fprime_setup_included_code()
+```
+
 When running the code, the same version of Python must be used as the python used to build it.
 
 ## Setting a Component Up With Python Bindings
@@ -69,6 +79,28 @@ Once finished, the python bindings will be autocoded and included in the next bu
 as shown below). This will also produce a `<component>.template.py` file in the component folder as a basic template for
 implementing components in python.
 
+> [!CAUTION]
+> The `<component>.template.py` is updated on every build unlike F Prime implementation templates.
+
+To include the component in the built python package, add the final Python file to the `SOURCES` list. A component
+implemented in Python has no `.cpp` or `.hpp` implementation files.
+
+```cmake
+register_fprime_module(
+    AUTOCODER_INPUTS
+        "${CMAKE_CURRENT_LIST_DIR}/MyComponent.fpp" # Has annotated component
+    SOURCES
+        "${CMAKE_CURRENT_LIST_DIR}/MyComponent.py" # Has Python implementation
+)
+```
+
+A new component with all of the above in place can be stamped out with the cookiecutter in this repository. Run it
+from the directory that should hold the component:
+
+```bash
+fprime-util new --from-source <path to fprime-python>/cookiecutters/component
+```
+
 Every port, command, event, telemetry channel, parameter and internal port of the component is bound, as is the
 overflow hook of each member with `hook` queue-full behavior (see below). Three component features cannot be bound
 yet, because F Prime declares a handler for each that has no Python equivalent:
@@ -133,29 +165,75 @@ deployment topology MyDeployment {
 An annotated topology must be a `deployment topology`: the binding calls the topology's setup and teardown, and
 F Prime only generates those for a deployment.
 
-> [!CAUTION]
-> The `<component>.template.py` is updated on every build unlike F Prime implementation templates.
+The build produces one Python extension module, `fprime_py`, for the annotated deployment, and installs it with
+the Python files of every module the deployment uses into `build-artifacts/python`. Python files are reinstalled
+by every build of the project or of the deployment, so an edit to one takes effect without rebuilding any C++.
+Building a single component's directory installs nothing.
 
-To include the component in the built python package, add the final Python file to the `SOURCES` list.
+The executable of an annotated deployment is linked against `pybind11::embed` by the build, because the components
+it links call into Python. That executable is not what runs the deployment (see
+[Running a Deployment](#running-a-deployment)), so its `Main.cpp` may be an empty `main`.
 
-```cmake
-register_fprime_module(
-    AUTOCODER_INPUTS
-        "${CMAKE_CURRENT_LIST_DIR}/MyComponent.fpp" # Has annotated component
-    SOURCES
-        "${CMAKE_CURRENT_LIST_DIR}/MyComponent.py" # Has Python implementation
-)
+Deployments whose topology is not annotated run from C++ as usual. Any number of them may be built in the same
+project as a Python deployment, in any order. A build supports one Python deployment: a second annotated deployment
+is skipped with a warning, because both would install to `build-artifacts/python`.
+
+### Custom Bindings
+
+`fprime_py` binds what the FPP model defines. A deployment's hand-written types are not in the model, so the
+project binds them itself by defining `setup_user_deployment`, declared in `FprimePython/FprimePython.hpp`. Every
+Python deployment must define it, and at least bind the `TopologyState` type passed to `setup` and `teardown`:
+
+```c++
+#include "FprimePython/FprimePython.hpp"
+#include "MyDeployment/Top/MyDeploymentTopologyDefs.hpp"
+
+void setup_user_deployment(pybind11::module_& m) {
+    pybind11::class_<MyDeployment::TopologyState>(m, "TopologyState")
+        .def(pybind11::init<>())
+        .def_readwrite("hostname", &MyDeployment::TopologyState::hostname)
+        .def_readwrite("port", &MyDeployment::TopologyState::port);
+}
 ```
 
+Add the file to the `SOURCES` of the module that defines the topology. A member bound with `def_readwrite` must own
+its value: declare a string member as `std::string`. A `const char*` member is left pointing into a Python string
+that is freed once the assignment returns.
+
+### Running a Deployment
+
+The deployment's entry point is a Python file named `fsw_main.py` defining a function `fsw_main()`, listed in the
+`SOURCES` of the deployment so that it is installed with the rest. It constructs the `TopologyState`, calls
+`fprime_py.Os.init()` and the bound `setup`, runs until told to stop, and calls `teardown`. Run it with the Python
+that built the project, either directly or through the runner, which finds `build-artifacts/python` from the
+working directory or any directory above it:
+
+```bash
+python3 build-artifacts/python/fsw_main.py
+fprime-python-runner
+```
+
+Python instances are reached as attributes of the bound topology's `Instances`, e.g.
+`fprime_py.MyDeployment.Instances.myComponent`. Only instances of `@ fprime-python` components are listed.
+`FprimePython.PythonRateGroupDriver` is one such component: instantiate it in place of a C++ timer to drive the
+rate groups from `fsw_main()` with its blocking `driveRateGroup` call.
+
+A deployment may list `FprimePython_PythonConsole` under `CHOOSES_IMPLEMENTATIONS` to send F Prime console output
+through Python's `print`.
+
+A complete walk-through is in the F´ how-to
+[Develop Components in Python](https://fprime.jpl.nasa.gov/latest/docs/how-to/develop/python-development/), and a
+working project in [fprime-python-reference](https://github.com/fprime-community/fprime-python-reference).
+
 ## F Prime Data Types
 
-Modeled types are available to Python. These types are under the namespace module `fprime_python` and then under their
-respective F Prime namespaces. For example, `import fprime_python.Fw.Time.Time` will import the `Fw::Time` type that
-lives under `#include "Fw/Time/Time.hpp`.
+Modeled types are available to Python. These types are under the module `fprime_py` and then under their
+respective F Prime namespaces. For example, `fprime_py.Fw.Time` is the `Fw::Time` type that lives under
+`#include "Fw/Time/Time.hpp"`.
 
 ```python
-from fprime_python.Fw.Time import Time
-fw_time_object = Time()
+import fprime_py
+fw_time_object = fprime_py.Fw.Time()
 ```
 
 Struct members are available as properties and as the autocoded `get_<member>()`/`set_<member>()` functions. Members
@@ -205,7 +283,3 @@ python -m venv .venv && .venv/bin/pip install -e '.[dev]'
 .venv/bin/python -m mypy
 .venv/bin/python -m pytest tests
 ```
-
-## TODO: custom bindings
-
-## TODO: Deployments

cmake/FprimePython.cmake

@@ -0,0 +1,37 @@
+####
+# FprimePython.cmake:
+#
+# Sets a project up to build F´ Python bindings. Include it from the project's top-level CMakeLists.txt, after
+# including FPrime.cmake and before calling `fprime_setup_included_code()`:
+#
+#     include("${CMAKE_CURRENT_LIST_DIR}/lib/fprime/cmake/FPrime.cmake")
+#     include(FprimePython)
+#     fprime_setup_included_code()
+#
+# It must run in the project's top-level directory scope and before any F´ module is added: the bindings of
+# every module are linked into one Python extension, so every module must be compiled as position independent
+# code and must be able to include pybind11.
+####
+include_guard()
+
+# The F´ sub-builds only gather model information, and compile nothing that could need pybind11
+if (DEFINED FPRIME_SUB_BUILD_TARGETS)
+    return()
+endif()
+set(CMAKE_POSITION_INDEPENDENT_CODE ON)
+set(PYBIND11_FINDPYTHON ON)
+
+# pybind11 is installed with the fprime-python package, so the interpreter knows where its CMake package is
+find_program(PYTHON NAMES python3 python)
+execute_process(COMMAND "${PYTHON}" -m pybind11 --cmakedir
+    OUTPUT_VARIABLE FPRIME_PYTHON_PYBIND11_CMAKEDIR
+    OUTPUT_STRIP_TRAILING_WHITESPACE
+    RESULT_VARIABLE FPRIME_PYTHON_PYBIND11_RESULT
+    ERROR_QUIET)
+if (NOT FPRIME_PYTHON_PYBIND11_RESULT EQUAL 0)
+    message(FATAL_ERROR "[fprime_python] '${PYTHON} -m pybind11 --cmakedir' failed. Install fprime-python into the "
+        "Python environment used to build, e.g. 'pip install ./lib/fprime-python'.")
+endif()
+find_package(pybind11 CONFIG REQUIRED PATHS "${FPRIME_PYTHON_PYBIND11_CMAKEDIR}")
+link_libraries("pybind11::module" "pybind11::headers")
+register_fprime_target(target/fprime_python)

cmake/autocoder/fprime_python.cmake

@@ -86,11 +86,15 @@ function(fprime_python_setup_autocode MODULE_NAME AC_INPUT_FILES)
         list(FILTER GENERATED_HPP_FILES INCLUDE REGEX "^.*\\.hpp$")
         list(FILTER GENERATED_JSON_FILES INCLUDE REGEX "^.*\\.json$")
         list(FILTER GENERATED_PY_FILES INCLUDE REGEX "^.*\\.py$")
+        # A bound topology leaves a snippet of its own kind, marking the deployments that run from Python
+        set(GENERATED_TOPOLOGY_FILES "${GENERATED_JSON_FILES}")
+        list(FILTER GENERATED_TOPOLOGY_FILES INCLUDE REGEX "^.*\\.topology\\.json$")
 
         # Append to target properties for tracking
         append_list_property("${GENERATED_HPP_FILES}" TARGET "${MODULE_NAME}" PROPERTY FPRIME_PYTHON_GENERATED_HPP_FILES)
         append_list_property("${GENERATED_JSON_FILES}" TARGET "${MODULE_NAME}" PROPERTY FPRIME_PYTHON_GENERATED_JSON_FILES)
         append_list_property("${GENERATED_PY_FILES}" TARGET "${MODULE_NAME}" PROPERTY FPRIME_PYTHON_GENERATED_PY_FILES)
+        append_list_property("${GENERATED_TOPOLOGY_FILES}" TARGET "${MODULE_NAME}" PROPERTY FPRIME_PYTHON_BOUND_TOPOLOGY_FILES)
 
         # Regenerate when anything in the model changes, not just this module's own translation units: a
         # bound component's ports, commands and types may all be defined elsewhere.

cmake/target/fprime_python.cmake

@@ -53,37 +53,75 @@ endfunction(fprime_python_add_module_target)
 
 # Function `fprime_python_add_deployment_target`:
 #
-# Generates the pybind11 module for the specified deployment target. This will create a shared library that can be
-# imported into Python containing the F´ deployment.
+# Generates the pybind11 module for a Python deployment: one whose topology is annotated `@ fprime-python`. This
+# creates a shared library, importable from Python as `fprime_py`, containing the F´ deployment. Deployments
+# that run from C++ are left alone, so they may be built alongside a Python deployment in any order.
 #
 ####
 function(fprime_python_add_deployment_target MODULE TARGET SOURCES DEPENDENCIES FULL_DEPENDENCIES)
     fprime_python_add_module_target("${MODULE}" "${TARGET}" "${SOURCES}" "${DEPENDENCIES}")
-    # Only a single fprime_py module may exist per build: skip additional
-    # deployments (e.g. other cFS apps within the same cFS mission build)
+    # A deployment without a bound topology runs from C++: it has nothing for Python to import
+    _fprime_python_get_bound_topologies("${MODULE}" "${FULL_DEPENDENCIES}")
+    if (NOT FPRIME_PYTHON_BOUND_TOPOLOGIES)
+        message(STATUS "[fprime_python] Skipping deployment '${MODULE}': no '@ fprime-python' topology")
+        return()
+    endif()
+    # Every Python deployment would install its fprime_py and Python files to the same place
     if (TARGET "fprime_py")
-        message(STATUS "[fprime_python] Skipping deployment '${MODULE}': fprime_py already exists")
+        get_target_property(FIRST_DEPLOYMENT "fprime_py" FPRIME_PYTHON_DEPLOYMENT)
+        message(WARNING "[fprime_python] Skipping Python deployment '${MODULE}': fprime_py is already built for "
+            "'${FIRST_DEPLOYMENT}' and a build supports one Python deployment.")
         return()
     endif()
     message(STATUS "[fprime_python] Deployment '${MODULE}' dependencies: ${FULL_DEPENDENCIES}")
+    # The deployment executable links the bindings of its modules, so it needs the Python library
+    target_link_libraries("${MODULE}" PRIVATE pybind11::embed)
 
-    # Generate the Python module initialization file and add it to a pybind11 module called "fprime_python". This sets
+    # Generate the Python module initialization file and add it to a pybind11 module called "fprime_py". This sets
     # forth the top level module that users will import to access the deployment.
     _fprime_python_generate_init_file("${MODULE}" "${FULL_DEPENDENCIES}")
     pybind11_add_module("fprime_py" "${FPRIME_PYTHON_DEPLOYMENT_INIT_FILE}")
+    set_target_properties("fprime_py" PROPERTIES FPRIME_PYTHON_DEPLOYMENT "${MODULE}")
 
     # Link in all deployment modules and dependencies
     list(REMOVE_ITEM FULL_DEPENDENCIES "${MODULE}")
     target_link_libraries("fprime_py" PUBLIC
         ${FPRIME_PYTHON_DEPLOYMENT_MODULES} ${FULL_DEPENDENCIES} Fw_Types FprimePython)
-    
-    # Install the fprime_python target and any associated python files with a post-build install command
+
+    # Install on every build, not after the link of fprime_py: editing a Python file does not relink it
     _fprime_python_install_helper("${FULL_DEPENDENCIES}")
-    add_custom_command(TARGET "fprime_py" POST_BUILD COMMAND "${CMAKE_COMMAND}"
+    add_custom_target("fprime_py_install" ALL
+        COMMAND "${CMAKE_COMMAND}"
             -DCMAKE_INSTALL_COMPONENT=fprime-python
             -DFPRIME_INSTALL_DEST=${FPRIME_INSTALL_DEST}
             -DFPRIME_BUILD_DIR=${CMAKE_BINARY_DIR}
-            -P ${FPRIME_FRAMEWORK_PATH}/cmake/target/fprime_install.cmake)
+            -P ${FPRIME_FRAMEWORK_PATH}/cmake/target/fprime_install.cmake
+        COMMENT "Installing fprime_py and Python files"
+        VERBATIM)
+    add_dependencies("fprime_py_install" "fprime_py")
+    # Building only the deployment, as `fprime-util build` does from its directory, installs as well
+    add_dependencies("${MODULE}" "fprime_py_install")
+endfunction()
+
+####
+# Function `_fprime_python_get_bound_topologies`:
+#
+# Finds the topologies bound to Python among a deployment and its dependencies. The autocoder records each as a
+# property of the module that defines it.
+#
+# Sets FPRIME_PYTHON_BOUND_TOPOLOGIES in the caller's scope, empty when the deployment runs from C++.
+####
+function(_fprime_python_get_bound_topologies MODULE FULL_DEPENDENCIES)
+    set(BOUND_TOPOLOGIES)
+    foreach(DEPENDENCY IN LISTS FULL_DEPENDENCIES MODULE)
+        if (TARGET "${DEPENDENCY}")
+            get_target_property(TOPOLOGY_FILES "${DEPENDENCY}" FPRIME_PYTHON_BOUND_TOPOLOGY_FILES)
+            if (TOPOLOGY_FILES)
+                list(APPEND BOUND_TOPOLOGIES ${TOPOLOGY_FILES})
+            endif()
+        endif()
+    endforeach()
+    set(FPRIME_PYTHON_BOUND_TOPOLOGIES "${BOUND_TOPOLOGIES}" PARENT_SCOPE)
 endfunction()
 
 ####

cookiecutters/component/README.md

@@ -0,0 +1,11 @@
+# F Prime Python Component Cookiecutter
+
+Stamps out a component implemented in Python: an FPP model annotated `@ fprime-python`, the Python implementation,
+and the `CMakeLists.txt` that registers both. Run it from the directory that should hold the new component, giving
+the path to this directory:
+
+```bash
+fprime-util new --from-source <path to fprime-python>/cookiecutters/component
+```
+
+The new component has one command, one event and one telemetry channel as place-holders.

cookiecutters/component/cookiecutter.json

@@ -0,0 +1,16 @@
+{
+    "component_name": "MyPythonComponent",
+    "component_short_description": "A component implemented in Python",
+    "component_namespace": "Components",
+    "component_kind": [
+        "active",
+        "passive",
+        "queued"
+    ],
+    "__prompts__": {
+        "component_name": "Component name",
+        "component_short_description": "Component short description",
+        "component_namespace": "Component namespace",
+        "component_kind": "Select component kind"
+    }
+}

cookiecutters/component/hooks/pre_gen_project.py

@@ -0,0 +1,7 @@
+from fprime.util.cookiecutter_wrapper import is_valid_name
+
+# Check if the component name is valid
+if is_valid_name("{{ cookiecutter.component_name }}") != "valid":
+    raise ValueError(
+        "Unacceptable component name. Do not use spaces or special characters"
+    )

cookiecutters/component/{{cookiecutter.component_name}}/CMakeLists.txt

@@ -0,0 +1,22 @@
+####
+# F Prime CMakeLists.txt:
+#
+# SOURCES: list of source files (to be compiled)
+# AUTOCODER_INPUTS: list of files to be passed to the autocoders
+# DEPENDS: list of libraries that this module depends on
+#
+# More information in the F´ CMake API documentation:
+# https://fprime.jpl.nasa.gov/latest/docs/reference/api/cmake/API/
+#
+####
+
+# Module names are derived from the path from the nearest project/library/framework
+# root when not specifically overridden by the developer. i.e. The module defined by
+# `Ref/SignalGen/CMakeLists.txt` will be named `Ref_SignalGen`.
+
+register_fprime_library(
+    AUTOCODER_INPUTS
+        "${CMAKE_CURRENT_LIST_DIR}/{{cookiecutter.component_name}}.fpp"
+    SOURCES
+        "${CMAKE_CURRENT_LIST_DIR}/{{cookiecutter.component_name}}.py"
+)