From 6871563a1a885c587f76427b2edef2bc53b9bc0d Mon Sep 17 00:00:00 2001 From: matclim Date: Tue, 20 Oct 2026 17:52:08 +0200 Subject: [PATCH 1/7] docs: document product selectors Add a user-guide page on product_selector: - what each field matches, including that creator matches either the plugin label or the algorithm name, and the empty default suffix - when a layer may be omitted, and that a wrong layer silently prevents delivery instead of failing - ambiguity errors and what they ask for - reading selectors from configuration with config.get - the expected -Wmissing-field-initializers warning Also document the product_selector fields in the header. Assisted by: Claude Haiku 5.5 --- docs/source/guide/product_selectors.rst | 118 ++++++++++++++++++++++++ phlex/core/product_selector.hpp | 12 +++ 2 files changed, 130 insertions(+) create mode 100644 docs/source/guide/product_selectors.rst diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst new file mode 100644 index 000000000..2ba251278 --- /dev/null +++ b/docs/source/guide/product_selectors.rst @@ -0,0 +1,118 @@ +Product selectors +================= + +An algorithm states which data products it receives with one +``phlex::product_selector`` per input, passed to ``input_family(...)``: + +.. code-block:: cpp + + m.transform("fit", fit, concurrency::unlimited) + .input_family(product_selector{.creator = "finder", .layer = "event", .suffix = "seeds"}, + product_selector{.creator = "geometry", .layer = "job"}); + +Every field is optional, and an unset field matches any value. +The product type is not written in the selector: Phlex takes it from the +type of the corresponding algorithm parameter. +Types are compared by their structure rather than by name, so two distinct +C++ types can be treated as the same; set ``suffix`` (or ``creator``) when +products of different types could otherwise match the same selector. + +Fields +------ + +``creator`` + Matches either the *label* of the plugin that created the product (its + key in the ``modules`` table) or the name of the algorithm that created + it. + For products of explicit providers and sources, it matches the creator + name the provider declared. + +``layer`` + The data layer of the product. + See `When the layer is required`_ below. + +``suffix`` + The suffix given to ``output_product_suffixes(...)`` by the creating + algorithm. + An algorithm that does not call ``output_product_suffixes(...)`` creates + its products with the empty suffix, so a suffix is only needed to tell + apart several products of the same type from one creator. + +``stage`` + The stage that created the product; ``"CURRENT"`` means the job's own + stage. + Products from earlier stages can only come from providers. + +An empty string is not accepted for ``creator`` or ``layer``. + +When the layer is required +-------------------------- + +The layer may be omitted only for a single-input transform, observer or +predicate. +The selector then matches the product in whichever layer it is created; +for a product supplied by a provider, the provider's layer is used. +For algorithms with more than one input, and for every fold and unfold, +each selector must name a layer; otherwise registration fails with +``Must specify layers in the product selectors for node ...``. + +Products are delivered only from the layer the selector names. +For products created by algorithms in the job, the layer is checked as +products arrive, not when the graph is built. +A selector that names the wrong layer therefore does not cause an error; +the algorithm is simply never called. +(If no algorithm or provider can create a selected product at all, the job +fails with ``No provider found for the following required products``.) +If an algorithm unexpectedly does not run, check its layers, and run with +``SPDLOG_LEVEL=debug`` to see how its inputs were connected. + +Ambiguous selectors +------------------- + +If a selector matches products that differ in suffix, creator or type, +graph construction fails with ``Ambiguous product selector -- see errors``. +The preceding log messages list the candidates and say which field to add. +Matching products that differ only in their layer are allowed. + +Selectors in configuration files +-------------------------------- + +A selector can be read directly from a plugin's configuration, where it is +an object with any of the keys ``creator``, ``layer``, ``suffix`` and +``stage``: + +.. code-block:: jsonnet + + { + modules: { + fit: { + cpp: 'track_fit', + seeds: { creator: 'finder', layer: 'event', suffix: 'seeds' }, + geometry: { creator: 'geometry', layer: 'job' }, + }, + }, + } + +.. code-block:: cpp + + PHLEX_REGISTER_ALGORITHMS(m, config) + { + m.transform("fit", fit, phlex::concurrency::unlimited) + .input_family(config.get("seeds"), + config.get("geometry")); + } + +A JSON array of selector objects can be read as +``std::array``. +Missing keys leave the corresponding fields unset. +Other keys are ignored without an error, so a misspelled key (``suffx``) +silently widens the selector. + +Designated initializers +----------------------- + +``product_selector`` is meant to be initialized with a subset of its fields. +With ``-Wextra``, GCC and Clang warn about the omitted fields +(``-Wmissing-field-initializers``). +The warning is expected; projects that build with ``-Werror`` may want +``-Wno-missing-field-initializers``. diff --git a/phlex/core/product_selector.hpp b/phlex/core/product_selector.hpp index c0bc2f751..00a6e8cab 100644 --- a/phlex/core/product_selector.hpp +++ b/phlex/core/product_selector.hpp @@ -96,11 +96,23 @@ namespace phlex { }; } + /// @brief Selects the data products an algorithm receives as one input. + /// + /// Every field is optional; an unset field matches any value. Fields are usually set + /// with designated initializers, for example + /// @c product_selector{.creator = "tracker", .layer = "event", .suffix = "tracks"}, or read + /// from a configuration object with @c config.get(key). struct PHLEX_CORE_EXPORT product_selector { + /// Plugin label (configuration key) or algorithm name of the product's creator. detail::creator_name creator; + /// Data layer of the product. Required for every input of multi-input algorithms, + /// folds and unfolds; products from other layers are not delivered. detail::layer_name layer; + /// Product suffix, as given to @c output_product_suffixes(...). std::optional suffix; + /// Stage that created the product; @c "CURRENT" means the job's own stage. std::optional stage; + /// Product type; set by the framework from the algorithm's parameter type. experimental::type_id type; // Check that all products selected by /other/ would satisfy this query From f02cae4765d0fd5707c3dbca34507825c9b3d84c Mon Sep 17 00:00:00 2001 From: matclim Date: Mon, 2 Nov 2026 00:41:13 +0100 Subject: [PATCH 2/7] docs(selectors): address review comments on suffixes, providers, arrays - Say when a suffix is needed without limiting it to one creator: with creator unset, same-typed products of several creators also match. - Separate the ambiguity rules: products from algorithms may differ only in layer, but more than one matching explicit provider is an error even if they differ only in layer, and at most one implicit provider may match. - State that reading std::array uses only the first N elements and throws if there are fewer. Assisted by: Claude Haiku 5.5 --- docs/source/guide/product_selectors.rst | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 2ba251278..1230a0996 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -35,8 +35,10 @@ Fields The suffix given to ``output_product_suffixes(...)`` by the creating algorithm. An algorithm that does not call ``output_product_suffixes(...)`` creates - its products with the empty suffix, so a suffix is only needed to tell - apart several products of the same type from one creator. + its products with the empty suffix. + Set a suffix whenever the other fields would match more than one product + of the right type, whether those products come from one creator or, when + ``creator`` is also unset, from several. ``stage`` The stage that created the product; ``"CURRENT"`` means the job's own @@ -69,10 +71,18 @@ If an algorithm unexpectedly does not run, check its layers, and run with Ambiguous selectors ------------------- -If a selector matches products that differ in suffix, creator or type, -graph construction fails with ``Ambiguous product selector -- see errors``. +If a selector matches products created by algorithms in the job that +differ in suffix, creator or type, graph construction fails with +``Ambiguous product selector -- see errors``. The preceding log messages list the candidates and say which field to add. -Matching products that differ only in their layer are allowed. +Matching products that differ only in their layer are allowed: each is +delivered only for data cells of its own layer. + +Providers are stricter. +If a selector matches more than one explicit provider, graph construction +fails with ``Multiple explicit providers found``, even when the providers +differ only in their layer; set ``layer`` (or another field) to pick one. +Likewise, at most one implicit provider supplied by a source may match. Selectors in configuration files -------------------------------- @@ -104,6 +114,8 @@ an object with any of the keys ``creator``, ``layer``, ``suffix`` and A JSON array of selector objects can be read as ``std::array``. +Only the first ``N`` elements are read: extra elements are ignored without +an error, and an array with fewer than ``N`` elements makes ``get`` throw. Missing keys leave the corresponding fields unset. Other keys are ignored without an error, so a misspelled key (``suffx``) silently widens the selector. From bbe05f9235931e77be1351941f2bbbb712c7e58f Mon Sep 17 00:00:00 2001 From: matclim <43775464+matclim@users.noreply.github.com> Date: Mon, 2 Nov 2026 20:40:04 +0100 Subject: [PATCH 3/7] Update docs/source/guide/product_selectors.rst Co-authored-by: Beojan Stanislaus --- docs/source/guide/product_selectors.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 1230a0996..7ab40b573 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -14,9 +14,7 @@ Every field is optional, and an unset field matches any value. The product type is not written in the selector: Phlex takes it from the type of the corresponding algorithm parameter. Types are compared by their structure rather than by name, so two distinct -C++ types can be treated as the same; set ``suffix`` (or ``creator``) when -products of different types could otherwise match the same selector. - +C++ types can match a selector. Use additional fields if this causes an ambiguity. Fields ------ From c3fb19502adb0918f50346d28902dd55ead350be Mon Sep 17 00:00:00 2001 From: matclim <43775464+matclim@users.noreply.github.com> Date: Mon, 2 Nov 2026 20:40:13 +0100 Subject: [PATCH 4/7] Update docs/source/guide/product_selectors.rst Co-authored-by: Beojan Stanislaus --- docs/source/guide/product_selectors.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 7ab40b573..778d399ad 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -52,6 +52,7 @@ The layer may be omitted only for a single-input transform, observer or predicate. The selector then matches the product in whichever layer it is created; for a product supplied by a provider, the provider's layer is used. +For transforms, outputs are dynamically placed into the same layer as the input product (i.e. the output layer may change from iteration to iteration). For algorithms with more than one input, and for every fold and unfold, each selector must name a layer; otherwise registration fails with ``Must specify layers in the product selectors for node ...``. From 2ebe719e064599ac99c2176ed62868b99f42d85d Mon Sep 17 00:00:00 2001 From: matclim <43775464+matclim@users.noreply.github.com> Date: Mon, 2 Nov 2026 20:40:22 +0100 Subject: [PATCH 5/7] Update docs/source/guide/product_selectors.rst Co-authored-by: Beojan Stanislaus --- docs/source/guide/product_selectors.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 778d399ad..4acb5c21d 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -34,9 +34,7 @@ Fields algorithm. An algorithm that does not call ``output_product_suffixes(...)`` creates its products with the empty suffix. - Set a suffix whenever the other fields would match more than one product - of the right type, whether those products come from one creator or, when - ``creator`` is also unset, from several. + Use ``suffix`` to disambiguate products that could not otherwise be disambiguated with other fields, for example when a ``transform`` creates two products of the same type (or compatible types). ``stage`` The stage that created the product; ``"CURRENT"`` means the job's own From f9053107b86cbb917ed07c95fb135f04c3fb1d1a Mon Sep 17 00:00:00 2001 From: matclim <43775464+matclim@users.noreply.github.com> Date: Mon, 2 Nov 2026 20:40:56 +0100 Subject: [PATCH 6/7] Update docs/source/guide/product_selectors.rst Co-authored-by: Beojan Stanislaus --- docs/source/guide/product_selectors.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 4acb5c21d..76dea4859 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -68,10 +68,10 @@ If an algorithm unexpectedly does not run, check its layers, and run with Ambiguous selectors ------------------- -If a selector matches products created by algorithms in the job that -differ in suffix, creator or type, graph construction fails with -``Ambiguous product selector -- see errors``. -The preceding log messages list the candidates and say which field to add. +If a selector matches products created by algorithms in the job, and specifying a +suffix, creator or type would narrow the matches, the selector is ambiguous. +Graph construction then fails with ``Ambiguous product selector -- see errors``. +The preceding log messages list the candidates and name the field to add. Matching products that differ only in their layer are allowed: each is delivered only for data cells of its own layer. From 565e0299d92fa1e59ea9f79a87ef404c84edc705 Mon Sep 17 00:00:00 2001 From: matclim Date: Tue, 3 Nov 2026 00:11:24 +0100 Subject: [PATCH 7/7] docs(selectors): address second-round review comments - Restore the blank line before the "Fields" heading, which the earlier suggestion commit removed; without it the heading is parsed as part of the preceding paragraph. - Rewrap the two long lines added by suggestion commits. - Replace the tautological "each is delivered only for data cells of its own layer" with what actually happens: the algorithm runs for every data cell holding a matching product, restricted to the selector's layer when it names one. - Mark the silently ignored unknown selector keys as current behaviour and link issue #1035, which tracks rejecting them. Assisted by: Claude Haiku 5.5 --- docs/source/guide/product_selectors.rst | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/source/guide/product_selectors.rst b/docs/source/guide/product_selectors.rst index 76dea4859..8df34a825 100644 --- a/docs/source/guide/product_selectors.rst +++ b/docs/source/guide/product_selectors.rst @@ -14,7 +14,9 @@ Every field is optional, and an unset field matches any value. The product type is not written in the selector: Phlex takes it from the type of the corresponding algorithm parameter. Types are compared by their structure rather than by name, so two distinct -C++ types can match a selector. Use additional fields if this causes an ambiguity. +C++ types can match a selector. +Use additional fields if this causes an ambiguity. + Fields ------ @@ -34,7 +36,9 @@ Fields algorithm. An algorithm that does not call ``output_product_suffixes(...)`` creates its products with the empty suffix. - Use ``suffix`` to disambiguate products that could not otherwise be disambiguated with other fields, for example when a ``transform`` creates two products of the same type (or compatible types). + Use ``suffix`` to disambiguate products that could not otherwise be + disambiguated with other fields, for example when a ``transform`` + creates two products of the same type (or compatible types). ``stage`` The stage that created the product; ``"CURRENT"`` means the job's own @@ -50,7 +54,9 @@ The layer may be omitted only for a single-input transform, observer or predicate. The selector then matches the product in whichever layer it is created; for a product supplied by a provider, the provider's layer is used. -For transforms, outputs are dynamically placed into the same layer as the input product (i.e. the output layer may change from iteration to iteration). +For transforms, outputs are dynamically placed into the same layer as the +input product (i.e. the output layer may change from iteration to +iteration). For algorithms with more than one input, and for every fold and unfold, each selector must name a layer; otherwise registration fails with ``Must specify layers in the product selectors for node ...``. @@ -72,8 +78,10 @@ If a selector matches products created by algorithms in the job, and specifying suffix, creator or type would narrow the matches, the selector is ambiguous. Graph construction then fails with ``Ambiguous product selector -- see errors``. The preceding log messages list the candidates and name the field to add. -Matching products that differ only in their layer are allowed: each is -delivered only for data cells of its own layer. +Matching products that differ only in their layer are allowed. +The algorithm then runs for every data cell that holds a matching product: +only cells of the selector's layer if it names one, otherwise cells of any +of those layers. Providers are stricter. If a selector matches more than one explicit provider, graph construction @@ -114,8 +122,9 @@ A JSON array of selector objects can be read as Only the first ``N`` elements are read: extra elements are ignored without an error, and an array with fewer than ``N`` elements makes ``get`` throw. Missing keys leave the corresponding fields unset. -Other keys are ignored without an error, so a misspelled key (``suffx``) -silently widens the selector. +Other keys are currently ignored without an error, so a misspelled key +(``suffx``) silently widens the selector +(`issue #1035 `_). Designated initializers -----------------------