diff --git a/AGENTS.md b/AGENTS.md index 068bf2d..88b9dda 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ The core use case is helping users understand what a firewall is actually doing Throughput must be calculated from cumulative interface octet counters. The standard Palo Alto and Fortinet dashboards use `ifHCInOctets` / `ifHCOutOctets`. The Palo Alto API dashboard is intentionally API-only and uses the MAC-level `port/rx-bytes` / `port/tx-bytes` counters of physical ports from `show counter interface all` (falling back to `ibytes` / `obytes` only when the `port` block is absent): `ibytes` / `obytes` are dataplane counters that miss hardware-offloaded flows. Do not use dataplane, NPU, session, or feature throughput summaries, which can be incomplete when traffic is offloaded or bypasses their counter path. -For CPU views, the standard SNMP dashboards should show both the global CPU and every per-processor/dataplane CPU exposed by the vendor MIB (`pan_hr_processors` for Palo Alto and `fortinet_processors` for Fortinet). The Palo Alto API dashboard must remain API-only and show management-plane CPU plus every dataplane/core returned by `resource-monitor`. The overview is useful for quick reading; per-CPU lines reveal imbalance and saturated dataplanes. +For CPU views, the standard SNMP dashboards should show both the global CPU and every per-processor/dataplane CPU exposed by the vendor MIB (`pan_hr_processors` for Palo Alto and `fortinet_processors` for Fortinet). The Palo Alto API dashboard must remain API-only and show management-plane CPU plus every dataplane/core returned by `resource-monitor`. Its all-core dataplane average matches SNMP but includes cores that PAN-OS lists without using them for packet processing (always 0%); `cpu_active_pct` excludes them and is shown next to it. The overview is useful for quick reading; per-CPU lines reveal imbalance and saturated dataplanes. The expected user journey is: diff --git a/README.md b/README.md index 7a576ef..0795f7a 100644 --- a/README.md +++ b/README.md @@ -807,7 +807,8 @@ Notes: `Palo Alto API Performance Monitoring` works for compact and multi-dataplane systems and reads only XML API measurements: - A current-load strip with 30-minute sparklines: DP CPU average, hottest DP core (including the one-minute peak), MP CPU and RAM, sessions, session table, CPS, and throughput. -- CPU, RAM, session-utilization, resource-pressure, and per-core panels with dashed 70% and 90% guide lines. The **CPU MP / DP** overview shows the management plane, the average of all dataplanes (multi-dataplane systems only), and one all-core average per dataplane; hottest-core and active-core lines are in each dataplane's CPU summary. +- CPU, RAM, session-utilization, resource-pressure, and per-core panels with dashed 70% and 90% guide lines. The **CPU MP / DP** overview shows the management plane, the average of all dataplanes (multi-dataplane systems only), the active-core average of all dataplanes, and one all-core average per dataplane; hottest-core and active-core lines are also in each dataplane's CPU summary. +- **DP CPU (avg)** and **DP CPU (active cores)** tiles. PAN-OS lists every core of a dataplane in `resource-monitor`, but only the pan task cores process packets: on a PA-5580 dataplane, 80 of the 128 listed cores. The others stay at exactly 0% and halve the all-core average, which is also the value SNMP `hrProcessorLoad` reports (45% all-core for 72% on the packet-processing cores during a load test). The collector writes `cpu_active_pct` (mean of the cores with any load in the minute) and `active_cores` on each dataplane's `core=average` point; the active-core figure is the one that reaches 100% at saturation. It is also drawn in the Load Test ramp and scatter plot. - An **Ingress Backlog by Dataplane** overview panel, a link-utilization table (interface, In %, Out %, rates, and speed, readable without scrolling), worst-dataplane resource pressure, and the global drop rate stacked by counter category. - Collapsible sections for HA (role timeline, sync state, and an **HA Links and Monitoring** table), interfaces and errors/discards, a **VSYS** section and a **Dataplanes** section whose panels repeat side by side for every VSYS or dataplane, zones and logical interfaces, session protocols, DoS/zone-protection drops, filtered global counters, MP load/storage, and environmental sensors split into temperature, fan, power, and alarm panels. - A collapsed **Logging and Management Health** section: log rate, logs discarded, content versions, management processes not running, GlobalProtect users, and RAID state. diff --git a/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json b/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json index 5de54b5..1945a0e 100644 --- a/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json +++ b/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json @@ -224,6 +224,75 @@ "x": 3, "y": 4 }, + "id": 13009, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "value", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "query": "from(bucket: \"firewalls\")\n |> range(start: -30m)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> max()\n |> group()\n |> sort(columns: [\"_time\"])\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (active cores)\", _value: float(v: r._value) }))", + "refId": "A" + } + ], + "title": "DP CPU (active cores)", + "type": "stat", + "description": "Highest per-dataplane average over the cores that process packets. PAN-OS lists every core of a dataplane, but only the pan task cores carry traffic (for example 80 of 128 on a PA-5580 dataplane); the others stay at 0% and pull the all-core average (DP CPU avg, the SNMP value) far below the real load. Cores with no load at all during the minute are excluded.\n\nSource: `show running resource-monitor minute last 1` via the PAN-OS XML API, polled every 1 min by default." + }, + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": {}, + "decimals": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "#EAB839", + "value": 70 + }, + { + "color": "red", + "value": 90 + } + ] + }, + "color": { + "mode": "thresholds" + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 6, + "y": 4 + }, "id": 13002, "options": { "colorMode": "background", @@ -289,8 +358,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 6, + "w": 2, + "x": 9, "y": 4 }, "id": 13003, @@ -358,8 +427,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 9, + "w": 2, + "x": 11, "y": 4 }, "id": 13004, @@ -419,7 +488,7 @@ "gridPos": { "h": 4, "w": 3, - "x": 12, + "x": 13, "y": 4 }, "id": 13005, @@ -487,8 +556,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 15, + "w": 2, + "x": 16, "y": 4 }, "id": 13006, @@ -711,13 +780,13 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "management = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_management\" and r.hostname == \"${hostname}\" and r._field == \"mp_cpu_pct\")\n |> map(fn: (r) => ({ r with _field: \"Management plane\" }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\ndataplanes = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _field: r.dataplane }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\noverall = dataplanes\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, count: 0}, fn: (r, accumulator) => ({ total: accumulator.total + r._value, count: accumulator.count + 1 }))\n |> filter(fn: (r) => r.count > 1)\n |> map(fn: (r) => ({ _time: r._time, _field: \"All dataplanes (average)\", _value: r.total / float(v: r.count) }))\n |> group(columns: [\"_field\"])\n |> sort(columns: [\"_time\"])\nunion(tables: [management, overall, dataplanes])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", + "query": "management = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_management\" and r.hostname == \"${hostname}\" and r._field == \"mp_cpu_pct\")\n |> map(fn: (r) => ({ r with _field: \"Management plane\" }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\ndataplanes = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _field: r.dataplane }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\noverall = dataplanes\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, count: 0}, fn: (r, accumulator) => ({ total: accumulator.total + r._value, count: accumulator.count + 1 }))\n |> filter(fn: (r) => r.count > 1)\n |> map(fn: (r) => ({ _time: r._time, _field: \"All dataplanes (average)\", _value: r.total / float(v: r.count) }))\n |> group(columns: [\"_field\"])\n |> sort(columns: [\"_time\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Dataplanes (active cores)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [management, overall, active, dataplanes])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", "refId": "A" } ], "title": "CPU MP / DP by Slot", "type": "timeseries", - "description": "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one) and the all-core average of each dataplane (equivalent to SNMP). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show system resources` every 1 min\n- `show running resource-monitor minute last 1` every 1 min" + "description": "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one), the all-core average of each dataplane (equivalent to SNMP) and the average of the packet-processing cores of all dataplanes (active cores: cores at 0% for the whole minute are excluded, which is the real load when PAN-OS lists reserved cores). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show system resources` every 1 min\n- `show running resource-monitor minute last 1` every 1 min" }, { "datasource": { @@ -3284,13 +3353,13 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "received = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Received\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\nsent = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"out_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Sent\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (average)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Hottest DP core\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [received, sent, cpu, hottest])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", + "query": "received = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Received\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\nsent = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"out_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Sent\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (average)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (active cores)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Hottest DP core\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [received, sent, cpu, active, hottest])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", "refId": "A" } ], "title": "Throughput vs Dataplane CPU", "type": "timeseries", - "description": "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show counter interface all` every 20 s\n- `show running resource-monitor minute last 1` every 1 min" + "description": "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. DP CPU (active cores) leaves out the cores PAN-OS lists but does not use for packet processing: it is the load that reaches 100% at saturation. The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show counter interface all` every 20 s\n- `show running resource-monitor minute last 1` every 1 min" }, { "datasource": { @@ -3364,6 +3433,30 @@ } ] }, + { + "matcher": { + "id": "byName", + "options": "active_core_pct" + }, + "properties": [ + { + "id": "displayName", + "value": "DP CPU (active cores)" + }, + { + "id": "unit", + "value": "percent" + }, + { + "id": "min", + "value": 0 + }, + { + "id": "max", + "value": 100 + } + ] + }, { "matcher": { "id": "byName", @@ -3423,6 +3516,29 @@ "fixed": "DP CPU (average)" } }, + { + "frame": { + "matcher": { + "id": "byIndex", + "options": 0 + } + }, + "x": { + "matcher": { + "id": "byName", + "options": "Throughput received" + } + }, + "y": { + "matcher": { + "id": "byName", + "options": "DP CPU (active cores)" + } + }, + "name": { + "fixed": "DP CPU (active cores)" + } + }, { "frame": { "matcher": { @@ -3465,7 +3581,7 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "throughput = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"throughput_bps\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"dp_cpu_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"hottest_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [throughput, cpu, hottest])\n |> group()\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct)\n |> keep(columns: [\"_time\", \"throughput_bps\", \"dp_cpu_pct\", \"hottest_core_pct\"])\n |> sort(columns: [\"_time\"])", + "query": "throughput = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"throughput_bps\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"dp_cpu_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"active_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"hottest_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [throughput, cpu, active, hottest])\n |> group()\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct)\n |> keep(columns: [\"_time\", \"throughput_bps\", \"dp_cpu_pct\", \"active_core_pct\", \"hottest_core_pct\"])\n |> sort(columns: [\"_time\"])", "refId": "A" } ], diff --git a/grafana/provisioning/dashboards/Palo_API_Dashboard.json b/grafana/provisioning/dashboards/Palo_API_Dashboard.json index d754cf3..859bb49 100644 --- a/grafana/provisioning/dashboards/Palo_API_Dashboard.json +++ b/grafana/provisioning/dashboards/Palo_API_Dashboard.json @@ -224,6 +224,75 @@ "x": 3, "y": 4 }, + "id": 3009, + "options": { + "colorMode": "background", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "value", + "wideLayout": true + }, + "targets": [ + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "query": "from(bucket: \"firewalls\")\n |> range(start: -30m)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> max()\n |> group()\n |> sort(columns: [\"_time\"])\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (active cores)\", _value: float(v: r._value) }))", + "refId": "A" + } + ], + "title": "DP CPU (active cores)", + "type": "stat", + "description": "Highest per-dataplane average over the cores that process packets. PAN-OS lists every core of a dataplane, but only the pan task cores carry traffic (for example 80 of 128 on a PA-5580 dataplane); the others stay at 0% and pull the all-core average (DP CPU avg, the SNMP value) far below the real load. Cores with no load at all during the minute are excluded.\n\nSource: `show running resource-monitor minute last 1` via the PAN-OS XML API, polled every 1 min by default." + }, + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "fieldConfig": { + "defaults": { + "unit": "percent", + "custom": {}, + "decimals": 1, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "#EAB839", + "value": 70 + }, + { + "color": "red", + "value": 90 + } + ] + }, + "color": { + "mode": "thresholds" + } + }, + "overrides": [] + }, + "gridPos": { + "h": 4, + "w": 3, + "x": 6, + "y": 4 + }, "id": 3002, "options": { "colorMode": "background", @@ -289,8 +358,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 6, + "w": 2, + "x": 9, "y": 4 }, "id": 3003, @@ -358,8 +427,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 9, + "w": 2, + "x": 11, "y": 4 }, "id": 3004, @@ -419,7 +488,7 @@ "gridPos": { "h": 4, "w": 3, - "x": 12, + "x": 13, "y": 4 }, "id": 3005, @@ -487,8 +556,8 @@ }, "gridPos": { "h": 4, - "w": 3, - "x": 15, + "w": 2, + "x": 16, "y": 4 }, "id": 3006, @@ -711,13 +780,13 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "management = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_management\" and r.hostname == \"${hostname}\" and r._field == \"mp_cpu_pct\")\n |> map(fn: (r) => ({ r with _field: \"Management plane\" }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\ndataplanes = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _field: r.dataplane }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\noverall = dataplanes\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, count: 0}, fn: (r, accumulator) => ({ total: accumulator.total + r._value, count: accumulator.count + 1 }))\n |> filter(fn: (r) => r.count > 1)\n |> map(fn: (r) => ({ _time: r._time, _field: \"All dataplanes (average)\", _value: r.total / float(v: r.count) }))\n |> group(columns: [\"_field\"])\n |> sort(columns: [\"_time\"])\nunion(tables: [management, overall, dataplanes])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", + "query": "management = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_management\" and r.hostname == \"${hostname}\" and r._field == \"mp_cpu_pct\")\n |> map(fn: (r) => ({ r with _field: \"Management plane\" }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\ndataplanes = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _field: r.dataplane }))\n |> group(columns: [\"_field\"])\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\noverall = dataplanes\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, count: 0}, fn: (r, accumulator) => ({ total: accumulator.total + r._value, count: accumulator.count + 1 }))\n |> filter(fn: (r) => r.count > 1)\n |> map(fn: (r) => ({ _time: r._time, _field: \"All dataplanes (average)\", _value: r.total / float(v: r.count) }))\n |> group(columns: [\"_field\"])\n |> sort(columns: [\"_time\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Dataplanes (active cores)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [management, overall, active, dataplanes])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", "refId": "A" } ], "title": "CPU MP / DP", "type": "timeseries", - "description": "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one) and the all-core average of each dataplane (equivalent to SNMP). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show system resources` every 1 min\n- `show running resource-monitor minute last 1` every 1 min" + "description": "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one), the all-core average of each dataplane (equivalent to SNMP) and the average of the packet-processing cores of all dataplanes (active cores: cores at 0% for the whole minute are excluded, which is the real load when PAN-OS lists reserved cores). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show system resources` every 1 min\n- `show running resource-monitor minute last 1` every 1 min" }, { "datasource": { @@ -2135,13 +2204,13 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "received = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Received\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\nsent = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"out_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Sent\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (average)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Hottest DP core\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [received, sent, cpu, hottest])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", + "query": "received = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Received\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\nsent = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"out_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"Sent\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (average)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"DP CPU (active cores)\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"Hottest DP core\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [received, sent, cpu, active, hottest])\n |> keep(columns: [\"_time\", \"_field\", \"_value\"])", "refId": "A" } ], "title": "Throughput vs Dataplane CPU", "type": "timeseries", - "description": "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show counter interface all` every 20 s\n- `show running resource-monitor minute last 1` every 1 min" + "description": "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. DP CPU (active cores) leaves out the cores PAN-OS lists but does not use for packet processing: it is the load that reaches 100% at saturation. The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute.\n\nSources via the PAN-OS XML API, polled by default:\n\n- `show counter interface all` every 20 s\n- `show running resource-monitor minute last 1` every 1 min" }, { "datasource": { @@ -2215,6 +2284,30 @@ } ] }, + { + "matcher": { + "id": "byName", + "options": "active_core_pct" + }, + "properties": [ + { + "id": "displayName", + "value": "DP CPU (active cores)" + }, + { + "id": "unit", + "value": "percent" + }, + { + "id": "min", + "value": 0 + }, + { + "id": "max", + "value": 100 + } + ] + }, { "matcher": { "id": "byName", @@ -2274,6 +2367,29 @@ "fixed": "DP CPU (average)" } }, + { + "frame": { + "matcher": { + "id": "byIndex", + "options": 0 + } + }, + "x": { + "matcher": { + "id": "byName", + "options": "Throughput received" + } + }, + "y": { + "matcher": { + "id": "byName", + "options": "DP CPU (active cores)" + } + }, + "name": { + "fixed": "DP CPU (active cores)" + } + }, { "frame": { "matcher": { @@ -2316,7 +2432,7 @@ "type": "influxdb", "uid": "P951FEA4DE68E13C5" }, - "query": "throughput = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"throughput_bps\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"dp_cpu_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"hottest_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [throughput, cpu, hottest])\n |> group()\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct)\n |> keep(columns: [\"_time\", \"throughput_bps\", \"dp_cpu_pct\", \"hottest_core_pct\"])\n |> sort(columns: [\"_time\"])", + "query": "throughput = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and r._field == \"in_octets\")\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> group(columns: [\"_time\"])\n |> sum()\n |> group()\n |> map(fn: (r) => ({ _time: r._time, _field: \"throughput_bps\", _value: r._value * 8.0 }))\n |> group(columns: [\"_field\"])\ncpu = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"dp_cpu_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nactive = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field == \"cpu_active_pct\" and r.core == \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"active_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nhottest = from(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_dataplane_cpu\" and r.hostname == \"${hostname}\" and r._field =~ /^cpu_(max_)?pct$/ and r.core != \"average\")\n |> map(fn: (r) => ({ r with _value: float(v: r._value) }))\n |> group()\n |> aggregateWindow(every: 1m, fn: max, createEmpty: false)\n |> map(fn: (r) => ({ _time: r._time, _field: \"hottest_core_pct\", _value: r._value }))\n |> group(columns: [\"_field\"])\nunion(tables: [throughput, cpu, active, hottest])\n |> group()\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct)\n |> keep(columns: [\"_time\", \"throughput_bps\", \"dp_cpu_pct\", \"active_core_pct\", \"hottest_core_pct\"])\n |> sort(columns: [\"_time\"])", "refId": "A" } ], diff --git a/scripts/build_paloalto_api_dashboard.py b/scripts/build_paloalto_api_dashboard.py index d30f8b0..2e494c2 100644 --- a/scripts/build_paloalto_api_dashboard.py +++ b/scripts/build_paloalto_api_dashboard.py @@ -416,17 +416,24 @@ def kpi_panels(y: int) -> list[dict]: return [ kpi(3001, "DP CPU (avg)", current_value("paloalto_api_dataplane_cpu", "cpu_pct", "DP CPU", ' and r.core == "average"'), 0, y, "percent", levels=LOAD_THRESHOLDS, description="Highest per-dataplane average across all cores. This is the value SNMP reports."), - kpi(3002, "Hottest DP Core", current_value("paloalto_api_dataplane_cpu", "/^cpu_(max_)?pct$/", "Hottest core", ' and r.core != "average"'), 3, y, "percent", + kpi(3009, "DP CPU (active cores)", current_value("paloalto_api_dataplane_cpu", "cpu_active_pct", "DP CPU (active cores)", ' and r.core == "average"'), 3, y, "percent", + levels=LOAD_THRESHOLDS, description=( + "Highest per-dataplane average over the cores that process packets. PAN-OS lists every core of a dataplane, " + "but only the pan task cores carry traffic (for example 80 of 128 on a PA-5580 dataplane); the others stay at 0% " + "and pull the all-core average (DP CPU avg, the SNMP value) far below the real load. Cores with no load at all " + "during the minute are excluded." + )), + kpi(3002, "Hottest DP Core", current_value("paloalto_api_dataplane_cpu", "/^cpu_(max_)?pct$/", "Hottest core", ' and r.core != "average"'), 6, y, "percent", levels=LOAD_THRESHOLDS, description=( "Busiest individual dataplane core. Uses the per-core peak within each minute (cpu_max_pct) when the collector " "provides it and falls back to the per-core one-minute average (cpu_pct): both fields are merged and the highest " "value per minute wins. A high value with a lower average reveals imbalance or saturated cores hidden by the average." )), - kpi(3003, "MP CPU", current_value("paloalto_api_management", "mp_cpu_pct", "MP CPU"), 6, y, "percent", levels=LOAD_THRESHOLDS), - kpi(3004, "MP RAM", current_value("paloalto_api_management", "memory_used_pct", "MP RAM"), 9, y, "percent", + kpi(3003, "MP CPU", current_value("paloalto_api_management", "mp_cpu_pct", "MP CPU"), 9, y, "percent", levels=LOAD_THRESHOLDS, w=2), + kpi(3004, "MP RAM", current_value("paloalto_api_management", "memory_used_pct", "MP RAM"), 11, y, "percent", w=2, levels=thresholds(("green", None), ("#EAB839", 85), ("red", 95))), - kpi(3005, "Active Sessions", current_value("paloalto_api_sessions", "sessions_active", "Sessions"), 12, y, "short"), - kpi(3006, "Session Table", current_value("paloalto_api_sessions", "session_utilization_pct", "Session table"), 15, y, "percent", + kpi(3005, "Active Sessions", current_value("paloalto_api_sessions", "sessions_active", "Sessions"), 13, y, "short"), + kpi(3006, "Session Table", current_value("paloalto_api_sessions", "session_utilization_pct", "Session table"), 16, y, "percent", w=2, levels=thresholds(("green", None), ("#EAB839", 80), ("red", 90))), kpi(3007, "CPS", current_value("paloalto_api_sessions", "cps", "CPS"), 18, y, "cps"), kpi(3008, "Throughput (In + Out)", f''' @@ -513,9 +520,16 @@ def overview_panels() -> list[dict]: |> map(fn: (r) => ({ _time: r._time, _field: "All dataplanes (average)", _value: r.total / float(v: r.count) })) |> group(columns: ["_field"]) |> sort(columns: ["_time"]) -union(tables: [management, overall, dataplanes]) +active = from(bucket: "firewalls") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "paloalto_api_dataplane_cpu" and r.hostname == "${hostname}" and r._field == "cpu_active_pct" and r.core == "average") + |> group() + |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false) + |> map(fn: (r) => ({ _time: r._time, _field: "Dataplanes (active cores)", _value: r._value })) + |> group(columns: ["_field"]) +union(tables: [management, overall, active, dataplanes]) |> keep(columns: ["_time", "_field", "_value"]) -''', 0, 8, 12, 10, "percent", "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one) and the all-core average of each dataplane (equivalent to SNMP). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance."))), +''', 0, 8, 12, 10, "percent", "Management-plane CPU, the average of all dataplanes (only when the firewall has more than one), the all-core average of each dataplane (equivalent to SNMP) and the average of the packet-processing cores of all dataplanes (active cores: cores at 0% for the whole minute are excluded, which is the real load when PAN-OS lists reserved cores). Dashed lines mark 70% and 90%. The Hottest DP Core tile and the Dataplanes row show per-core load and imbalance."))), guide_lines(percent_range(timeseries(3, "MP RAM Usage", ''' from(bucket: "firewalls") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) @@ -612,11 +626,14 @@ def physical_rate(field: str, label: str, window: str = "v.windowPeriod") -> str ''' -def dataplane_cpu(label: str, window: str = "v.windowPeriod", *, hottest: bool = False) -> str: - """Average of the dataplane all-core averages, or the hottest core of any dataplane.""" +def dataplane_cpu(label: str, window: str = "v.windowPeriod", *, hottest: bool = False, active: bool = False) -> str: + """Average of the dataplane all-core (or active-core) averages, or the hottest core of any dataplane.""" if hottest: selector = 'r._field =~ /^cpu_(max_)?pct$/ and r.core != "average"' fn = "max" + elif active: + selector = 'r._field == "cpu_active_pct" and r.core == "average"' + fn = "mean" else: selector = 'r._field == "cpu_pct" and r.core == "average"' fn = "mean" @@ -690,30 +707,34 @@ def load_test_row() -> dict: ] ramp = timeseries(3611, "Throughput vs Dataplane CPU", ( named(received, "received") + named(sent, "sent") - + named(dataplane_cpu("DP CPU (average)"), "cpu") + named(dataplane_cpu("Hottest DP core", hottest=True), "hottest") - + '''union(tables: [received, sent, cpu, hottest]) + + named(dataplane_cpu("DP CPU (average)"), "cpu") + named(dataplane_cpu("DP CPU (active cores)", active=True), "active") + + named(dataplane_cpu("Hottest DP core", hottest=True), "hottest") + + '''union(tables: [received, sent, cpu, active, hottest]) |> keep(columns: ["_time", "_field", "_value"])''' ), 0, 4, 24, 10, "bps", ( "The load ramp: throughput of the physical ports (left axis) against dataplane CPU (right axis). " "CPU that climbs faster than throughput, or throughput that flattens while CPU keeps rising, shows where the platform saturates. " + "DP CPU (active cores) leaves out the cores PAN-OS lists but does not use for packet processing: it is the load that reaches 100% at saturation. " "The dataplane CPU is the one-minute resource-monitor average, so it lags the 20-second throughput by up to a minute." )) right_axis(ramp, "/CPU|core/", "percent", percent=True) curve = xychart(3612, "CPU vs Throughput", ( named(physical_rate("in_octets", "throughput_bps", "1m"), "throughput") - + named(dataplane_cpu("dp_cpu_pct", "1m"), "cpu") + named(dataplane_cpu("hottest_core_pct", "1m", hottest=True), "hottest") - + '''union(tables: [throughput, cpu, hottest]) + + named(dataplane_cpu("dp_cpu_pct", "1m"), "cpu") + named(dataplane_cpu("active_core_pct", "1m", active=True), "active") + + named(dataplane_cpu("hottest_core_pct", "1m", hottest=True), "hottest") + + '''union(tables: [throughput, cpu, active, hottest]) |> group() |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") |> filter(fn: (r) => exists r.throughput_bps and exists r.dp_cpu_pct) - |> keep(columns: ["_time", "throughput_bps", "dp_cpu_pct", "hottest_core_pct"]) + |> keep(columns: ["_time", "throughput_bps", "dp_cpu_pct", "active_core_pct", "hottest_core_pct"]) |> sort(columns: ["_time"])''' - ), 0, 14, 12, 10, "throughput_bps", [("dp_cpu_pct", "DP CPU (average)"), ("hottest_core_pct", "Hottest DP core")], + ), 0, 14, 12, 10, "throughput_bps", [("dp_cpu_pct", "DP CPU (average)"), ("active_core_pct", "DP CPU (active cores)"), ("hottest_core_pct", "Hottest DP core")], "Dataplane CPU as a function of the received throughput, one point per minute of the selected range: the CPU-versus-throughput curve of a performance test report. Points that pile up at 100% CPU mark the maximum throughput of the platform for this traffic mix.", x_label="Throughput received") curve["fieldConfig"]["overrides"] = [ override("throughput_bps", displayName="Throughput received", unit="bps"), override("dp_cpu_pct", displayName="DP CPU (average)", unit="percent", min=0, max=100), + override("active_core_pct", displayName="DP CPU (active cores)", unit="percent", min=0, max=100), override("hottest_core_pct", displayName="Hottest DP core", unit="percent", min=0, max=100), ] rates = timeseries(3613, "Packet Rate and Connection Rate", ( diff --git a/telegraf/paloalto_api_collector.py b/telegraf/paloalto_api_collector.py index ad04189..2450ba3 100644 --- a/telegraf/paloalto_api_collector.py +++ b/telegraf/paloalto_api_collector.py @@ -513,6 +513,16 @@ def walk(node: ET.Element, dataplane: str | None = None): summary = {} if averages: summary["cpu_pct"] = sum(averages) / len(averages) + # PAN-OS lists every core of the dataplane, but only the pan task + # cores process packets (80 of 128 on a PA-5580 DP); the others stay + # at exactly 0% and halve the all-core average that SNMP reports. + active = [ + fields["cpu_pct"] + for fields in cores + if "cpu_pct" in fields and fields.get("cpu_max_pct", fields["cpu_pct"]) > 0 + ] + summary["cpu_active_pct"] = sum(active) / len(active) if active else 0.0 + summary["active_cores"] = len(active) if maxima: summary["cpu_max_pct"] = max(maxima) points[(dataplane, "average")] = ({"dataplane": dataplane, "core": "average"}, summary) diff --git a/tests/test_paloalto_api_collector.py b/tests/test_paloalto_api_collector.py index 8e6feb6..f099ce5 100644 --- a/tests/test_paloalto_api_collector.py +++ b/tests/test_paloalto_api_collector.py @@ -167,7 +167,10 @@ def test_dataplane_cpu_ignores_maximum_table(self): indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)} # The maximum is stored separately and never mixed into cpu_pct. self.assertEqual(indexed[("dp0", "0")], {"cpu_pct": 12.0, "cpu_max_pct": 99.0}) - self.assertEqual(indexed[("dp0", "average")], {"cpu_pct": 12.0, "cpu_max_pct": 99.0}) + self.assertEqual( + indexed[("dp0", "average")], + {"cpu_pct": 12.0, "cpu_active_pct": 12.0, "active_cores": 1, "cpu_max_pct": 99.0}, + ) def test_dataplane_average_point_uses_mean_of_averages_and_max_of_maxima(self): result = ET.fromstring( @@ -180,9 +183,48 @@ def test_dataplane_average_point_uses_mean_of_averages_and_max_of_maxima(self): ) indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)} self.assertEqual(indexed[("s1dp0", "1")], {"cpu_pct": 30.0, "cpu_max_pct": 80.0}) - self.assertEqual(indexed[("s1dp0", "average")], {"cpu_pct": 20.0, "cpu_max_pct": 80.0}) + self.assertEqual( + indexed[("s1dp0", "average")], + {"cpu_pct": 20.0, "cpu_active_pct": 20.0, "active_cores": 2, "cpu_max_pct": 80.0}, + ) line = line_protocol("m", {}, indexed[("s1dp0", "average")]) - self.assertEqual(line, "m cpu_max_pct=80.0,cpu_pct=20.0") + self.assertEqual(line, "m active_cores=2i,cpu_active_pct=20.0,cpu_max_pct=80.0,cpu_pct=20.0") + + def test_dataplane_active_core_average_skips_cores_that_never_ran(self): + # PA-5580 dataplane under load: core 0 and cores 81-127 stay at 0% (not + # pan task cores), cores 1-80 process packets at about 72%. + averages = "".join( + f"{core}{72 if 1 <= core <= 80 else 0}" for core in range(128) + ) + maxima = "".join( + f"{core}{85 if 1 <= core <= 80 else 0}" for core in range(128) + ) + result = ET.fromstring( + "" + f"{averages}{maxima}" + "" + ) + indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)} + summary = indexed[("s1dp0", "average")] + self.assertEqual(summary["cpu_pct"], 72 * 80 / 128) + self.assertEqual(summary["cpu_active_pct"], 72.0) + self.assertEqual(summary["active_cores"], 80) + + def test_dataplane_active_core_average_without_maximum_or_load(self): + # Without a maximum table the per-core average decides; an idle + # dataplane reports 0.0 rather than no value. + result = ET.fromstring( + "" + "00" + "140" + "" + "00" + "" + ) + indexed = {(tags["dataplane"], tags["core"]): fields for tags, fields in parse_dataplane_resources(result)} + self.assertEqual(indexed[("dp0", "average")]["cpu_active_pct"], 40.0) + self.assertEqual(indexed[("dp1", "average")]["cpu_active_pct"], 0.0) + self.assertEqual(indexed[("dp1", "average")]["active_cores"], 0) def test_dataplane_command_reads_last_minute(self): self.assertIn("1", DATAPLANE_COMMAND) diff --git a/tests/test_paloalto_api_dashboard.py b/tests/test_paloalto_api_dashboard.py index 10f3076..376475c 100644 --- a/tests/test_paloalto_api_dashboard.py +++ b/tests/test_paloalto_api_dashboard.py @@ -151,22 +151,39 @@ def test_load_test_row_summarizes_a_capacity_ramp(self): self.assertEqual(axis["matcher"], {"id": "byRegexp", "options": "/CPU|core/"}) properties = {item["id"]: item["value"] for item in axis["properties"]} self.assertEqual((properties["unit"], properties["custom.axisPlacement"], properties["max"]), ("percent", "right", 100)) - for label in ("Received", "Sent", "DP CPU (average)", "Hottest DP core"): + for label in ("Received", "Sent", "DP CPU (average)", "DP CPU (active cores)", "Hottest DP core"): self.assertIn(f'_field: "{label}"', ramp["targets"][0]["query"]) curve = panels["CPU vs Throughput"] self.assertEqual(curve["type"], "xychart") self.assertEqual(curve["options"]["mapping"], "manual") # Matchers name the display names set by the overrides: Grafana # matches fields by display name, so the raw column names find nothing. - self.assertEqual([s["x"]["matcher"]["options"] for s in curve["options"]["series"]], ["Throughput received"] * 2) - self.assertEqual([s["y"]["matcher"]["options"] for s in curve["options"]["series"]], ["DP CPU (average)", "Hottest DP core"]) - self.assertEqual([s["name"]["fixed"] for s in curve["options"]["series"]], ["DP CPU (average)", "Hottest DP core"]) + cpu_series = ["DP CPU (average)", "DP CPU (active cores)", "Hottest DP core"] + self.assertEqual([s["x"]["matcher"]["options"] for s in curve["options"]["series"]], ["Throughput received"] * 3) + self.assertEqual([s["y"]["matcher"]["options"] for s in curve["options"]["series"]], cpu_series) + self.assertEqual([s["name"]["fixed"] for s in curve["options"]["series"]], cpu_series) self.assertIn("aggregateWindow(every: 1m", curve["targets"][0]["query"]) self.assertIn('pivot(rowKey: ["_time"], columnKey: ["_field"]', curve["targets"][0]["query"]) for title in ("Packet Rate and Connection Rate", "Sessions and Session Table", "Drops and Interface Errors"): self.assertEqual(panels[title]["type"], "timeseries", title) self.assertTrue(load_test["collapsed"]) + def test_kpi_strip_shows_all_core_and_active_core_dataplane_cpu(self): + for dashboard in (self.dashboard, self.chassis_dashboard): + strip = sorted( + (panel for panel in dashboard["panels"] if panel["type"] == "stat" and panel["gridPos"]["y"] == 4), + key=lambda panel: panel["gridPos"]["x"], + ) + self.assertEqual([panel["title"] for panel in strip[:3]], ["DP CPU (avg)", "DP CPU (active cores)", "Hottest DP Core"]) + # The tiles fill the 24-column row edge to edge without gaps. + edges = [(panel["gridPos"]["x"], panel["gridPos"]["x"] + panel["gridPos"]["w"]) for panel in strip] + self.assertEqual(edges[0][0], 0) + self.assertEqual(edges[-1][1], 24) + self.assertTrue(all(end == start for (_, end), (start, _) in zip(edges, edges[1:]))) + query = strip[1]["targets"][0]["query"] + self.assertIn('r._field == "cpu_active_pct"', query) + self.assertIn('r.core == "average"', query) + def test_panel_ids_are_unique_and_rows_are_ordered(self): for dashboard in (self.dashboard, self.chassis_dashboard): ids = []