feat: show dataplane CPU of the packet-processing cores

AGENTS.md

@@ -10,7 +10,7 @@ The core use case is helping users understand what a firewall is actually doing
 
 Throughput must be calculated from cumulative interface octet counters. The standard Palo Alto and Fortinet dashboards use `ifHCInOctets` / `ifHCOutOctets`. The Palo Alto API dashboard is intentionally API-only and uses the MAC-level `port/rx-bytes` / `port/tx-bytes` counters of physical ports from `show counter interface all` (falling back to `ibytes` / `obytes` only when the `port` block is absent): `ibytes` / `obytes` are dataplane counters that miss hardware-offloaded flows. Do not use dataplane, NPU, session, or feature throughput summaries, which can be incomplete when traffic is offloaded or bypasses their counter path.
 
-For CPU views, the standard SNMP dashboards should show both the global CPU and every per-processor/dataplane CPU exposed by the vendor MIB (`pan_hr_processors` for Palo Alto and `fortinet_processors` for Fortinet). The Palo Alto API dashboard must remain API-only and show management-plane CPU plus every dataplane/core returned by `resource-monitor`. The overview is useful for quick reading; per-CPU lines reveal imbalance and saturated dataplanes.
+For CPU views, the standard SNMP dashboards should show both the global CPU and every per-processor/dataplane CPU exposed by the vendor MIB (`pan_hr_processors` for Palo Alto and `fortinet_processors` for Fortinet). The Palo Alto API dashboard must remain API-only and show management-plane CPU plus every dataplane/core returned by `resource-monitor`. Its all-core dataplane average matches SNMP but includes cores that PAN-OS lists without using them for packet processing (always 0%); `cpu_active_pct` excludes them and is shown next to it. The overview is useful for quick reading; per-CPU lines reveal imbalance and saturated dataplanes.
 
 The expected user journey is:
 

README.md

@@ -807,7 +807,8 @@ Notes:
 `Palo Alto API Performance Monitoring` works for compact and multi-dataplane systems and reads only XML API measurements:
 
 - A current-load strip with 30-minute sparklines: DP CPU average, hottest DP core (including the one-minute peak), MP CPU and RAM, sessions, session table, CPS, and throughput.
-- CPU, RAM, session-utilization, resource-pressure, and per-core panels with dashed 70% and 90% guide lines. The **CPU MP / DP** overview shows the management plane, the average of all dataplanes (multi-dataplane systems only), and one all-core average per dataplane; hottest-core and active-core lines are in each dataplane's CPU summary.
+- CPU, RAM, session-utilization, resource-pressure, and per-core panels with dashed 70% and 90% guide lines. The **CPU MP / DP** overview shows the management plane, the average of all dataplanes (multi-dataplane systems only), the active-core average of all dataplanes, and one all-core average per dataplane; hottest-core and active-core lines are also in each dataplane's CPU summary.
+- **DP CPU (avg)** and **DP CPU (active cores)** tiles. PAN-OS lists every core of a dataplane in `resource-monitor`, but only the pan task cores process packets: on a PA-5580 dataplane, 80 of the 128 listed cores. The others stay at exactly 0% and halve the all-core average, which is also the value SNMP `hrProcessorLoad` reports (45% all-core for 72% on the packet-processing cores during a load test). The collector writes `cpu_active_pct` (mean of the cores with any load in the minute) and `active_cores` on each dataplane's `core=average` point; the active-core figure is the one that reaches 100% at saturation. It is also drawn in the Load Test ramp and scatter plot.
 - An **Ingress Backlog by Dataplane** overview panel, a link-utilization table (interface, In %, Out %, rates, and speed, readable without scrolling), worst-dataplane resource pressure, and the global drop rate stacked by counter category.
 - Collapsible sections for HA (role timeline, sync state, and an **HA Links and Monitoring** table), interfaces and errors/discards, a **VSYS** section and a **Dataplanes** section whose panels repeat side by side for every VSYS or dataplane, zones and logical interfaces, session protocols, DoS/zone-protection drops, filtered global counters, MP load/storage, and environmental sensors split into temperature, fan, power, and alarm panels.
 - A collapsed **Logging and Management Health** section: log rate, logs discarded, content versions, management processes not running, GlobalProtect users, and RAID state.

scripts/build_paloalto_api_dashboard.py

@@ -416,17 +416,24 @@ def kpi_panels(y: int) -> list[dict]:
     return [
         kpi(3001, "DP CPU (avg)", current_value("paloalto_api_dataplane_cpu", "cpu_pct", "DP CPU", ' and r.core == "average"'), 0, y, "percent",
             levels=LOAD_THRESHOLDS, description="Highest per-dataplane average across all cores. This is the value SNMP reports."),
-        kpi(3002, "Hottest DP Core", current_value("paloalto_api_dataplane_cpu", "/^cpu_(max_)?pct$/", "Hottest core", ' and r.core != "average"'), 3, y, "percent",
+        kpi(3009, "DP CPU (active cores)", current_value("paloalto_api_dataplane_cpu", "cpu_active_pct", "DP CPU (active cores)", ' and r.core == "average"'), 3, y, "percent",
+            levels=LOAD_THRESHOLDS, description=(
+                "Highest per-dataplane average over the cores that process packets. PAN-OS lists every core of a dataplane, "
+                "but only the pan task cores carry traffic (for example 80 of 128 on a PA-5580 dataplane); the others stay at 0% "
+                "and pull the all-core average (DP CPU avg, the SNMP value) far below the real load. Cores with no load at all "
+                "during the minute are excluded."
+            )),
+        kpi(3002, "Hottest DP Core", current_value("paloalto_api_dataplane_cpu", "/^cpu_(max_)?pct$/", "Hottest core", ' and r.core != "average"'), 6, y, "percent",
             levels=LOAD_THRESHOLDS, description=(
                 "Busiest individual dataplane core. Uses the per-core peak within each minute (cpu_max_pct) when the collector "
                 "provides it and falls back to the per-core one-minute average (cpu_pct): both fields are merged and the highest "
                 "value per minute wins. A high value with a lower average reveals imbalance or saturated cores hidden by the average."
             )),
-        kpi(3003, "MP CPU", current_value("paloalto_api_management", "mp_cpu_pct", "MP CPU"), 6, y, "percent", levels=LOAD_THRESHOLDS),
-        kpi(3004, "MP RAM", current_value("paloalto_api_management", "memory_used_pct", "MP RAM"), 9, y, "percent",
+        kpi(3003, "MP CPU", current_value("paloalto_api_management", "mp_cpu_pct", "MP CPU"), 9, y, "percent", levels=LOAD_THRESHOLDS, w=2),
+        kpi(3004, "MP RAM", current_value("paloalto_api_management", "memory_used_pct", "MP RAM"), 11, y, "percent", w=2,
             levels=thresholds(("green", None), ("#EAB839", 85), ("red", 95))),
-        kpi(3005, "Active Sessions", current_value("paloalto_api_sessions", "sessions_active", "Sessions"), 12, y, "short"),
-        kpi(3006, "Session Table", current_value("paloalto_api_sessions", "session_utilization_pct", "Session table"), 15, y, "percent",
+        kpi(3005, "Active Sessions", current_value("paloalto_api_sessions", "sessions_active", "Sessions"), 13, y, "short"),
+        kpi(3006, "Session Table", current_value("paloalto_api_sessions", "session_utilization_pct", "Session table"), 16, y, "percent", w=2,
             levels=thresholds(("green", None), ("#EAB839", 80), ("red", 90))),
         kpi(3007, "CPS", current_value("paloalto_api_sessions", "cps", "CPS"), 18, y, "cps"),
         kpi(3008, "Throughput (In + Out)", f'''
@@ -513,9 +520,16 @@ def overview_panels() -> list[dict]:
   |> map(fn: (r) => ({ _time: r._time, _field: "All dataplanes (average)", _value: r.total / float(v: r.count) }))
   |> group(columns: ["_field"])
   |> sort(columns: ["_time"])
-union(tables: [management, overall, dataplanes])
+active = from(bucket: "firewalls")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r._measurement == "paloalto_api_dataplane_cpu" and r.hostname == "${hostname}" and r._field == "cpu_active_pct" and r.core == "average")
+  |> group()
+  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
+  |> map(fn: (r) => ({ _time: r._time, _field: "Dataplanes (active cores)", _value: r._value }))
+  |> group(columns: ["_field"])
+union(tables: [management, overall, active, dataplanes])
   |> keep(columns: ["_time", "_field", "_value"])
-''', 0, 8, 12, 10, "percent", "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one) and the all-core average of each dataplane (equivalent to SNMP). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance."))),
+''', 0, 8, 12, 10, "percent", "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one), the all-core average of each dataplane (equivalent to SNMP) and the average of the packet-processing cores of all dataplanes (active cores: cores at 0% for the whole minute are excluded, which is the real load when PAN-OS lists reserved cores). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance."))),
         guide_lines(percent_range(timeseries(3, "MP RAM Usage", '''
 from(bucket: "firewalls")
   |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
@@ -612,11 +626,14 @@ def physical_rate(field: str, label: str, window: str = "v.windowPeriod") -> str
 '''
 
 
-def dataplane_cpu(label: str, window: str = "v.windowPeriod", *, hottest: bool = False) -> str:
-    """Average of the dataplane all-core averages, or the hottest core of any dataplane."""
+def dataplane_cpu(label: str, window: str = "v.windowPeriod", *, hottest: bool = False, active: bool = False) -> str:
+    """Average of the dataplane all-core (or active-core) averages, or the hottest core of any dataplane."""
     if hottest:
         selector = 'r._field =~ /^cpu_(max_)?pct$/ and r.core != "average"'
         fn = "max"
+    elif active:
+        selector = 'r._field == "cpu_active_pct" and r.core == "average"'
+        fn = "mean"
     else:
         selector = 'r._field == "cpu_pct" and r.core == "average"'
         fn = "mean"
@@ -690,30 +707,34 @@ def load_test_row() -> dict:
     ]
     ramp = timeseries(3611, "Throughput vs Dataplane CPU", (
         named(received, "received") + named(sent, "sent")
-        + named(dataplane_cpu("DP CPU (average)"), "cpu") + named(dataplane_cpu("Hottest DP core", hottest=True), "hottest")
-        + '''union(tables: [received, sent, cpu, hottest])
+        + named(dataplane_cpu("DP CPU (average)"), "cpu") + named(dataplane_cpu("DP CPU (active cores)", active=True), "active")
+        + named(dataplane_cpu("Hottest DP core", hottest=True), "hottest")
+        + '''union(tables: [received, sent, cpu, active, hottest])
   |> keep(columns: ["_time", "_field", "_value"])'''
     ), 0, 4, 24, 10, "bps", (
         "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). "
         "CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. "
+        "DP CPU (active cores) leaves out the cores PAN-OS lists but does not use for packet processing: it is the load that reaches 100% at saturation. "
         "The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute."
     ))
     right_axis(ramp, "/CPU|core/", "percent", percent=True)
     curve = xychart(3612, "CPU vs Throughput", (
         named(physical_rate("in_octets", "throughput_bps", "1m"), "throughput")
-        + named(dataplane_cpu("dp_cpu_pct", "1m"), "cpu") + named(dataplane_cpu("hottest_core_pct", "1m", hottest=True), "hottest")
-        + '''union(tables: [throughput, cpu, hottest])
+        + named(dataplane_cpu("dp_cpu_pct", "1m"), "cpu") + named(dataplane_cpu("active_core_pct", "1m", active=True), "active")
+        + named(dataplane_cpu("hottest_core_pct", "1m", hottest=True), "hottest")
+        + '''union(tables: [throughput, cpu, active, hottest])
   |> group()
   |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
   |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct)
-  |> keep(columns: ["_time", "throughput_bps", "dp_cpu_pct", "hottest_core_pct"])
+  |> keep(columns: ["_time", "throughput_bps", "dp_cpu_pct", "active_core_pct", "hottest_core_pct"])
   |> sort(columns: ["_time"])'''
-    ), 0, 14, 12, 10, "throughput_bps", [("dp_cpu_pct", "DP CPU (average)"), ("hottest_core_pct", "Hottest DP core")],
+    ), 0, 14, 12, 10, "throughput_bps", [("dp_cpu_pct", "DP CPU (average)"), ("active_core_pct", "DP CPU (active cores)"), ("hottest_core_pct", "Hottest DP core")],
         "Dataplane CPU as a function of the received throughput, one point per minute of the selected range: the CPU-versus-throughput curve of a performance test report. Points that pile up at 100% CPU mark the maximum throughput of the platform for this traffic mix.",
         x_label="Throughput received")
     curve["fieldConfig"]["overrides"] = [
         override("throughput_bps", displayName="Throughput received", unit="bps"),
         override("dp_cpu_pct", displayName="DP CPU (average)", unit="percent", min=0, max=100),
+        override("active_core_pct", displayName="DP CPU (active cores)", unit="percent", min=0, max=100),
         override("hottest_core_pct", displayName="Hottest DP core", unit="percent", min=0, max=100),
     ]
     rates = timeseries(3613, "Packet Rate and Connection Rate", (

telegraf/paloalto_api_collector.py

@@ -513,6 +513,16 @@ def walk(node: ET.Element, dataplane: str | None = None):
         summary = {}
         if averages:
             summary["cpu_pct"] = sum(averages) / len(averages)
+            # PAN-OS lists every core of the dataplane, but only the pan task
+            # cores process packets (80 of 128 on a PA-5580 DP); the others stay
+            # at exactly 0% and halve the all-core average that SNMP reports.
+            active = [
+                fields["cpu_pct"]
+                for fields in cores
+                if "cpu_pct" in fields and fields.get("cpu_max_pct", fields["cpu_pct"]) > 0
+            ]
+            summary["cpu_active_pct"] = sum(active) / len(active) if active else 0.0
+            summary["active_cores"] = len(active)
         if maxima:
             summary["cpu_max_pct"] = max(maxima)
         points[(dataplane, "average")] = ({"dataplane": dataplane, "core": "average"}, summary)

tests/test_paloalto_api_collector.py

@@ -167,7 +167,10 @@ def test_dataplane_cpu_ignores_maximum_table(self):
         indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)}
         # The maximum is stored separately and never mixed into cpu_pct.
         self.assertEqual(indexed[("dp0", "0")], {"cpu_pct": 12.0, "cpu_max_pct": 99.0})
-        self.assertEqual(indexed[("dp0", "average")], {"cpu_pct": 12.0, "cpu_max_pct": 99.0})
+        self.assertEqual(
+            indexed[("dp0", "average")],
+            {"cpu_pct": 12.0, "cpu_active_pct": 12.0, "active_cores": 1, "cpu_max_pct": 99.0},
+        )
 
     def test_dataplane_average_point_uses_mean_of_averages_and_max_of_maxima(self):
         result = ET.fromstring(
@@ -180,9 +183,48 @@ def test_dataplane_average_point_uses_mean_of_averages_and_max_of_maxima(self):
         )
         indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)}
         self.assertEqual(indexed[("s1dp0", "1")], {"cpu_pct": 30.0, "cpu_max_pct": 80.0})
-        self.assertEqual(indexed[("s1dp0", "average")], {"cpu_pct": 20.0, "cpu_max_pct": 80.0})
+        self.assertEqual(
+            indexed[("s1dp0", "average")],
+            {"cpu_pct": 20.0, "cpu_active_pct": 20.0, "active_cores": 2, "cpu_max_pct": 80.0},
+        )
         line = line_protocol("m", {}, indexed[("s1dp0", "average")])
-        self.assertEqual(line, "m cpu_max_pct=80.0,cpu_pct=20.0")
+        self.assertEqual(line, "m active_cores=2i,cpu_active_pct=20.0,cpu_max_pct=80.0,cpu_pct=20.0")
+
+    def test_dataplane_active_core_average_skips_cores_that_never_ran(self):
+        # PA-5580 dataplane under load: core 0 and cores 81-127 stay at 0% (not
+        # pan task cores), cores 1-80 process packets at about 72%.
+        averages = "".join(
+            f"<entry><coreid>{core}</coreid><value>{72 if 1 <= core <= 80 else 0}</value></entry>" for core in range(128)
+        )
+        maxima = "".join(
+            f"<entry><coreid>{core}</coreid><value>{85 if 1 <= core <= 80 else 0}</value></entry>" for core in range(128)
+        )
+        result = ET.fromstring(
+            "<result><data-processors><s1dp0><minute>"
+            f"<cpu-load-average>{averages}</cpu-load-average><cpu-load-maximum>{maxima}</cpu-load-maximum>"
+            "</minute></s1dp0></data-processors></result>"
+        )
+        indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)}
+        summary = indexed[("s1dp0", "average")]
+        self.assertEqual(summary["cpu_pct"], 72 * 80 / 128)
+        self.assertEqual(summary["cpu_active_pct"], 72.0)
+        self.assertEqual(summary["active_cores"], 80)
+
+    def test_dataplane_active_core_average_without_maximum_or_load(self):
+        # Without a maximum table the per-core average decides; an idle
+        # dataplane reports 0.0 rather than no value.
+        result = ET.fromstring(
+            "<result><data-processors><dp0><second><cpu-load-average>"
+            "<entry><coreid>0</coreid><value>0</value></entry>"
+            "<entry><coreid>1</coreid><value>40</value></entry>"
+            "</cpu-load-average></second></dp0><dp1><second><cpu-load-average>"
+            "<entry><coreid>0</coreid><value>0</value></entry>"
+            "</cpu-load-average></second></dp1></data-processors></result>"
+        )
+        indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)}
+        self.assertEqual(indexed[("dp0", "average")]["cpu_active_pct"], 40.0)
+        self.assertEqual(indexed[("dp1", "average")]["cpu_active_pct"], 0.0)
+        self.assertEqual(indexed[("dp1", "average")]["active_cores"], 0)
 
     def test_dataplane_command_reads_last_minute(self):
         self.assertIn("<minute><last>1</last></minute>", DATAPLANE_COMMAND)