Interface for framework driver

The current (experimental) interface of the framework driver allows constructions like:

void cells_to_process(framework_driver& d)
{
  unsigned int const num_runs = 2;
  unsigned int const num_subruns = 2;
  unsigned int const num_spills = 3;

  auto job_id = data_cell_index::base_ptr();
  d.yield(job_id);
  for (unsigned int r : std::views::iota(0u, num_runs)) {
    auto run_id = job_id->make_child(r, "run");
    d.yield(run_id);
    for (unsigned int sr : std::views::iota(0u, num_subruns)) {
      auto subrun_id = run_id->make_child(sr, "subrun");
      d.yield(subrun_id);
      for (unsigned int spill : std::views::iota(0u, num_spills)) {
        d.yield(subrun_id->make_child(spill, "spill"));
      }
    }
  }
}

Although this keeps the structure simple for users, it's not clear that it's ideal for the framework:

The framework determines the hierarchy of the data layers based on the d.yield(...) functions encountered.
If the framework knew the hierarchy before processing any of these yield functions, not only could the yield invocations be in vetted against the expected hierarchy, but it would also simplify the formation of the data graph.

We should discuss whether the simplicity of the current interface outweighs the two items discussed above.

Note that allowing the framework to know the hierarchy before executing the driver itself would result in a simple heuristic for the invocation of providers:

If the provider provides a data product from a layer supported by the driver, then the provider reacts to messages generated from the driver
If the provider provides a data product from a layer not support by the driver, then the provider reacts indirectly to messages generated from the driver (i.e. a relation must exist between the driver-supported layers and the provider's layer)

This example driver specifies at least two different ideas:

it is processing a tree that contains three layers (I would call this the shape of the input; it is defining a functor for the application of the framework graph's execution), and
It is doing a depth-first traversal of the tree.

I would like us to consider how we might disentangle the two things.

Agreement:

The hierarchy will be declared to the framework before processing any data-cell indices.
Different policies for supporting hierarchies:
1. A fixed hierarchy may be declared for the entire job.
2. The global hierarchy may be updated based on the hierarchies stored in input files in the middle of the job.
Whenever the global hierarchy is established (or updated), the framework validates it against what the computational graph requires.
1. This may involve creating more index-set nodes, provider nodes, preserver nodes, and translator nodes.
2. The job shall gracefully end when one of the input files contains a hierarchy that does not meet the hierarchy constraints of the computational graph.
Adopt a syntax similar to the above for the driver function.
We will use a registration syntax that looks similar to

void cells_to_process(framework_driver& d) { ...}

PHLEX_REGISTER_DRIVER(m)
{
  auto d = m.fixed_hierarchy("run", "subrun", "spill");
  d.use(cells_to_process);
}

The driver-interface concern is resolved by PR #443, which closed #438, with the resulting fixed-hierarchy and cursor APIs made public by PR #469. Hierarchy declaration is now separate from traversal, available before execution, and used to validate yielded indices and configure index routing.

The broader agreement’s requirements for updating hierarchies between input files, validating established or updated hierarchies against computational-graph constraints, and gracefully rejecting incompatible input-file hierarchies remain follow-up work. Discussion continues in discussion #841, with the remaining requirements summarized in this comment.

Closing this issue as resolved for the driver interface does not imply that those multiple-file hierarchy capabilities are implemented.

Navigation Menu

Repository navigation

Interface for framework driver #395

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Interface for framework driver #395

Description

Activity

knoepfel commented on Mar 5, 2026

marcpaterno commented on Mar 9, 2026

knoepfel commented on Mar 11, 2026

knoepfel commented on Oct 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions