030-release: Prepare clean DPG 0.3.0 core

CHANGELOG.md

@@ -1,5 +1,66 @@
 # Changelog
 
+## 0.3.0
+
+### Added
+
+- Strengthened `context_order="auto"` to validate global graph-path support
+  against observed execution traces, including longer-history recombinations;
+  added a regression test for pairwise-consistent phantom paths.
+- Added context-aware DPG construction through `context_order` in execution
+  trace mode. `context_order=1` preserves legacy predicate identity; integer
+  orders and `"auto"` split nodes by recent execution history.
+- Class outcomes are shared terminal sinks at every context order, and node
+  metadata exposes `predicate`, `context`, and `context_order`.
+- Added enumeration-free local context-order resolution with explicit
+  resolution history and failure reporting.
+- Execution-trace edge construction preserves within-case event order and no
+  longer applies an unstable sort to the constant case identifier.
+- Added exact sklearn `decision_path` routing. Threshold rounding now formats
+  predicate labels without changing the branch selected by the model.
+- Added `decimal_threshold="auto"`, which derives precision from the data and
+  warns when a tree threshold is off the derived grid.
+- Validated the context resolver's optional `max_k` bound so invalid or
+  insufficient caps fail explicitly instead of returning an unresolved order.
+
+### Benchmarking evidence
+
+- The post-fix E2 benchmark completed all 375/375 cells. `context_order="auto"`
+  resolved a mean order of 2.0373 and reported a zero phantom-path rate across
+  every tested classifier family; the corresponding k=1 execution-trace graphs
+  retained phantom paths in the pooled graph for Bagging, Gradient Boosting,
+  and Random Forest.
+- Across that grid, post-fix execution-trace k=1 averaged 7.0742 seconds and
+  886.8 graph nodes per cell, while auto-k averaged 7.2271 seconds and 1005.5
+  nodes: approximately 2.2% more runtime for trace-consistent graph structure.
+- In the E5 alignment benchmark, the shipped unweighted LRC aggregation reached
+  mean Spearman correlation 0.8525 at k=1 and 0.9360 at auto-k, with mean top-10
+  feature overlap increasing from 0.7574 to 0.8759.
+- On the reference ten-tree Random Forests for Iris, Wine, and Breast Cancer,
+  k=1 and auto-k contained exactly the same raw `(feature, operator, threshold)`
+  split predicates. Differences in class-boundary envelopes were caused by
+  contextual predicate/community assignments, not by changed learned splits.
+
+### Compatibility and limitations
+
+- The default remains `context_order=1`; DPG-k is opt-in so existing consumers
+  keep their graph shape. `context_order > 1` requires `execution_trace` mode.
+- `get_trace_consistent_lrc()` remains available for k=1 and is deprecated for
+  contextual graphs; k>1 aggregates ordinary unweighted node LRC by predicate.
+- The routing correction can change graph weights and labels at floating-point
+  boundaries. Residual off-grid behavior is reported by the auto-precision
+  warning rather than hidden.
+- **Regression sink semantics are out of scope for 0.3.0.** `context_order`
+  mechanically builds a graph for regressors (regression leaves are treated
+  as terminal sinks, like class leaves), but there is no "one sink per
+  output" guarantee: a regression sink is only as unique as the 2-decimal
+  rounded leaf value, so two leaves collide into one sink by coincidence of
+  rounding, not by any modeled notion of "output". A principled regression
+  sink policy is deferred to a future release. `class_boundaries` and
+  `communities` remain classifier-only features; calling
+  `DPGExplainer.explain_global(communities=True)` on a regressor now raises a
+  clear `ValueError` instead of an internal `numpy.linalg.LinAlgError`.
+
 ## 0.2.0
 
 ### Added

README.md

@@ -2,12 +2,11 @@
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 [![Python Versions](https://img.shields.io/badge/python-3.10%20|%203.11%20|%203.12-blue.svg)](pyproject.toml)
-[![PyPI](https://img.shields.io/pypi/v/dpg.svg)](https://pypi.org/project/dpg/)
 [![Build Status](https://github.com/Meta-Group/DPG/actions/workflows/ci.yml/badge.svg)](https://github.com/Meta-Group/DPG/actions/workflows/ci.yml)
 [![Documentation Status](https://readthedocs.org/projects/dpg/badge/?version=latest)](https://dpg.readthedocs.io/en/latest/)
 
 <p align="center">
-  <img src="https://github.com/Meta-Group/DPG/blob/main/DPG.png" alt="DPG logo" width="300">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/DPG.png" alt="DPG logo" width="300">
 </p>
 
 
@@ -21,7 +20,9 @@ insightful points. DPG enables graph-based evaluations and the identification of
 towards facilitating comparisons between features and their associated values while offering insights
 into the entire model. DPG provides descriptive metrics that enhance the understanding of the
 decisions inherent in the model, offering valuable insights.
-![DPG overview](https://github.com/Meta-Group/DPG/blob/main/image.png)
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/image.png" width="600" />
+</p>
 
 ---
 
@@ -47,7 +48,9 @@ The concept behind DPG is to convert a generic tree-based ensemble model for cla
 - Nodes represent predicates, i.e., the feature-value associations present in each node of every tree;
 - Edges denote the frequency with which these predicates are satisfied during the model training phase by the samples of the dataset.
 
-![DPG example](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/example.png?raw=true)
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/example.png" width="600" />
+</p>
 
 ## Metrics
 The graph-based nature of DPG provides significant enhancements in the direction of a complete mapping of the ensemble structure.
@@ -61,7 +64,7 @@ The graph-based nature of DPG provides significant enhancements in the direction
 
 |Constraints | Betweenness centrality | Local reaching centrality | Community|
 |------------|------------|--------------|--------------------|
-![](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/example_constraints.png) | ![](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/example_bc.png) | ![](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/example_lrc.png) | ![](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/example_community.png) |
+![](https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/example_constraints.png) | ![](https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/example_bc.png) | ![](https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/example_lrc.png) | ![](https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/example_community.png) |
 |Constraints(Class 1) = val3 < F1 ≤ val1, F2 ≤ val2 | BC(F2 ≤ val2) = 4/24 | LRC(F1 ≤ val1) = 6 / 7 | Community(Class 1) = F1 ≤ val1, F2 ≤ val2 |
 
 ---
@@ -169,6 +172,7 @@ explainer = DPGExplainer(
             },
             "graph_construction": {
                 "mode": "execution_trace",  # or "aggregated_transitions"
+                "context_order": 1,           # 1, an integer > 1, or "auto"
             },
         }
     },
@@ -178,6 +182,31 @@ explainer = DPGExplainer(
 - `"aggregated_transitions"`: current default behavior; filters path variants first, then discovers the DPG.
 - `"execution_trace"`: builds directly from raw traces and filters edges instead of whole-path variants when `perc_var > 0`.
 
+`context_order` controls predicate identity in `execution_trace` mode. `1`
+preserves the legacy graph exactly; an order greater than one uses the last k
+executed predicates as context and keeps one sink per class. `"auto"` resolves
+the smallest order for which every pooled graph path is supported by an
+observed execution trace, including recombinations that only appear after
+multiple hops. Orders greater than one require `mode: "execution_trace"`.
+`context_order="auto"` also requires `execution_trace`; in
+`aggregated_transitions` mode context resolution is not applied.
+
+A phantom path is a multi-hop path assembled by pooling edges from different
+tree executions even though no single execution produced it. DPG-k removes
+these recombinations at the resolved context order while retaining one graph.
+The effective order and each node's context are available through
+`get_context_order()` and `get_node_context(node)`.
+
+**Regression scope (0.3.0):** `context_order` builds without raising for
+regressors (regression leaves, labeled `"Pred <value>"`, are treated as
+terminal sinks just like class leaves), but "one sink per output" is not a
+defined guarantee for regression the way it is for classification: two
+leaves collapse into the same sink only when their rounded values coincide,
+which is an artifact of label rounding, not a modeled invariant. A
+principled regression sink policy is deferred past 0.3.0. `class_boundaries`
+and `communities` remain classifier-only; calling `explain_global`
+with `communities=True` on a regressor raises a clear `ValueError`.
+
 #### Minimal local workflow
 
 ```python
@@ -263,23 +292,26 @@ Important:
 
 ## CLI scripts
 The library contains two different scripts to apply DPG:
-- `run_dpg_standard.py`: with this script it is possible to test DPG on a standard classification dataset provided by `sklearn` such as `iris`, `digits`, `wine`, `breast cancer`, and `diabetes`.
-- `run_dpg_custom.py`: with this script it is possible to apply DPG to your classification dataset, specifying the target class.
+- `examples/run_dpg_standard.py`: with this script it is possible to test DPG on a standard classification dataset provided by `sklearn` such as `iris`, `digits`, `wine`, `breast cancer`, and `diabetes`.
+- `examples/run_dpg_custom.py`: with this script it is possible to apply DPG to your classification dataset, specifying the target class.
 
 ### Implementation notes
 The library also contains two other essential scripts:
 - `core.py` contains all the functions used to calculate and create the DPG and the metrics.
 - `visualizer.py` contains the functions used to manage the visualization of DPG.
 
 ### Output
-The DPG output, through `run_dpg_standard.py` or `run_dpg_custom.py`, produces several files:
+The DPG output, through `examples/run_dpg_standard.py` or `examples/run_dpg_custom.py`, produces several files:
 - the visualization of DPG in a dedicated environment, which can be zoomed and saved;
 - a `.txt` file containing the DPG metrics;
 - a `.csv` file containing the information about all the nodes of the DPG and their associated metrics;
 - a `.txt` file containing the Random Forest statistics (accuracy, confusion matrix, classification report)
 
 ### CLI parameter reference
-Usage: `python run_dpg_standard.py --dataset <dataset_name> --n_learners <integer_number> --pv <threshold_value> --t <integer_number> --model_name <str_model_name> --dir <save_dir_path> --plot --save_plot_dir <save_plot_dir_path> --attribute <attribute> --communities --clusters --threshold_clusters <float> --class_flag --seed <int>`
+Usage: `dpg --dataset <dataset_name> --n_learners <integer_number> --pv <threshold_value> --t <integer_number> --model_name <str_model_name> --dir <save_dir_path> --plot --save_plot_dir <save_plot_dir_path> --attribute <attribute> --communities --clusters --threshold_clusters <float> --class_flag --seed <int>`
+
+After installing DPG, the `dpg` command is the packaged equivalent of
+`python examples/run_dpg_standard.py`.
 Where:
 - `dataset` is the name of the standard classification `sklearn` dataset to be analyzed;
 - `n_learners` is the number of base learners for the ensemble model;
@@ -306,23 +338,29 @@ Where:
 
 Disclaimer: `attribute`, `communities`, and `clusters` are mutually exclusive: DPG supports just one visualization mode at a time.
 
-The usage of `run_dpg_custom.py` is similar, but it requires another parameter:
+The usage of `examples/run_dpg_custom.py` is similar, but it requires another parameter:
 - `target_column`, which is the name of the column to be used as the target variable;
 - while `ds` is the path of the directory where the dataset is.
 
-### Example `run_dpg_standard.py`
+### Example `examples/run_dpg_standard.py`
 Some examples can be appreciated in the `examples` folder: https://github.com/Meta-Group/DPG/tree/main/examples
 
 In particular, the following DPG is obtained by transforming a Random Forest with 5 base learners, trained on Iris dataset.
-The used command is `python run_dpg_standard.py --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples`.
-![Iris DPG](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/iris_bl5_perc0.001_dec2.png)
+The used command is `dpg --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples`.
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/iris_bl5_perc0.001_dec2.png" width="800" />
+</p>
 
 The following visualizations are obtained using the same parameters as the previous example, but they show two different metrics: _Community_ and _Betweenness centrality_.
-The used command for showing communities is `python run_dpg_standard.py --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples --communities`.
-![Iris communities](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/iris_bl5_perc0.001_dec2_communities.png)
+The used command for showing communities is `dpg --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples --communities`.
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/iris_bl5_perc0.001_dec2_communities.png" width="800" />
+</p>
 
-The used command for showing a specific property is `python run_dpg_standard.py --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples --attribute "Betweenness centrality" --class_flag`.
-![Iris betweenness centrality](https://github.com/Meta-Group/DPG/blob/main/dpg_image_examples/iris_bl5_perc0.001_dec2_Betweennesscentrality.png)
+The used command for showing a specific property is `dpg --dataset iris --n_learners 5 --pv 0.001 --t 2 --dir examples --plot --save_plot_dir examples --attribute "Betweenness centrality" --class_flag`.
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Meta-Group/DPG/main/dpg_image_examples/iris_bl5_perc0.001_dec2_Betweennesscentrality.png" width="800" />
+</p>
 
 ***
 ## Citation

config.yaml

@@ -1,9 +1,12 @@
 dpg:
-  default:
-    perc_var: 0.000000001
-    decimal_threshold: 6
-    n_jobs: -1
-  visualization:
+  default:
+    perc_var: 0.000000001
+    decimal_threshold: 6
+    n_jobs: -1
+  graph_construction:
+    mode: "aggregated_transitions"
+    context_order: 1
+  visualization:
     graph_attrs:
       bgcolor: "white"
       rankdir: "R"

docs/conf.py

@@ -14,7 +14,7 @@
 project = "DPG"
 copyright = "2024, Sylvio Barbon Junior, Leonardo Arrighi"
 author = "Sylvio Barbon Junior, Leonardo Arrighi"
-release = "0.2.0"
+release = "0.3.0"
 
 # ---------------------------------------------------------------------------
 # General configuration

docs/index.md

@@ -54,8 +54,9 @@ explainer = DPGExplainer(
     feature_names=X.columns.tolist(),
     target_names=["setosa", "versicolor", "virginica"],
 )
-explanation = explainer.fit(X.values)
-explainer.plot(explanation)
+explainer.fit(X.values)
+explanation = explainer.explain_global()
+explainer.plot("iris_dpg", explanation=explanation)
 ```
 
 ## Contents

docs/quickstart.md

@@ -43,7 +43,7 @@ print(explanation.node_metrics.head())
 print(explanation.edge_metrics.head())
 
 # 5. Visualise
-explainer.plot(explanation, save_dir="results/")
+explainer.plot("iris_dpg", explanation=explanation, save_dir="results/")
 ```
 
 ## What `DPGExplainer` returns
@@ -125,15 +125,16 @@ dpg = DecisionPredicateGraph(
 )
 dpg.fit(X)
 
-dpg.get_trace_consistent_lrc()   # {predicate_label: trace-consistent LRC score}
+dpg.get_trace_consistent_lrc()   # supported for legacy execution-trace k=1
 dpg.get_trace_consistent_trc()   # {predicate_label: set of labels observed downstream}
 dpg.get_trace_signatures()       # list[TraceSignature(signature, predicate_sequence, path_count)]
 ```
 
 - `get_trace_consistent_lrc()` scores each predicate by how much of the label
-  space it was observed to reach *within a single trace*, unlike the
-  pooled-graph NetworkX local reaching centrality, which can credit reach
-  that only exists after aggregating unrelated traces.
+  space it was observed to reach *within a single trace* for legacy k=1,
+  unlike the pooled-graph NetworkX local reaching centrality, which can credit
+  reach that only exists after aggregating unrelated traces. For contextual
+  graphs, use `get_predicate_lrc(graph)` instead.
 - `get_trace_consistent_trc()` returns each predicate's observed downstream
   label sets — every member is guaranteed to have co-occurred later in at
   least one real execution.
@@ -143,6 +144,11 @@ dpg.get_trace_signatures()       # list[TraceSignature(signature, predicate_sequ
 These getters return empty containers until `fit()` is called, and are reset
 on every refit. Outside `execution_trace` mode they remain empty.
 
+For contextual graphs (`context_order > 1`),
+`get_trace_consistent_lrc()` is deprecated; use the graph-based
+`get_predicate_lrc(graph)` aggregation instead. The explainer's node metrics
+already apply the appropriate contextual aggregation automatically.
+
 **`perc_var` does not filter trace artefacts.** In `execution_trace` mode,
 `perc_var` only filters infrequent *edges* out of the pooled visualisation
 graph — it never removes a trace signature or downstream relation. A

dpg/__init__.py

@@ -13,6 +13,7 @@
     classwise_feature_bounds_from_communities,
     plot_class_feature_complexity,
     plot_dpg,
+    plot_dpg_communities,
     plot_dpg_class_bounds_vs_dataset_feature_ranges,
     plot_dpg_constraints_overview,
     plot_dpg_local_paths_aggregate,
@@ -35,6 +36,7 @@
     "DPG_OLIVE_CLASS_PALETTE",
     "resolve_theme_context",
     "plot_dpg",
+    "plot_dpg_communities",
     "plot_dpg_local_paths_aggregate",
     "plot_dpg_reg",
     "plot_dpg_constraints_overview",