From 84704e317390e44ecad37168f8b7280f258bb321 Mon Sep 17 00:00:00 2001 From: Thomas Date: Sat, 26 Sep 2026 21:41:48 +0200 Subject: [PATCH] feat: estimate offloaded throughput on the API dashboards The API collector now keeps the dataplane ibytes/obytes of each physical port as dp_in_octets/dp_out_octets, next to the MAC-level port counters (only when the port block exists, so a missing port counter never reads as zero offload). The general section of both API dashboards gains a "Throughput Received: Dataplane vs Offloaded" panel: dataplane and offloaded (port minus dataplane) stacked, plus the total line. Small platforms refresh the port counters late and in bursts: on a PA-440 without offload, 20 s differences swung by several Mb/s both ways and 2-minute windows still read up to +27%. A burst reads negative then positive in consecutive windows, while real offload stays positive, so the estimate takes the smaller of two consecutive window differences, over windows of at least 2 minutes, floored at 5% of the total. SNMP exposes no dataplane byte counter, so the SNMP dashboards are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 2 +- README.md | 3 +- docs/adr/0001-paloalto-xml-api-collector.md | 2 +- .../Palo_API_Chassis_Dashboard.json | 289 +++++++++++++----- .../dashboards/Palo_API_Dashboard.json | 271 +++++++++++----- scripts/build_paloalto_api_dashboard.py | 63 +++- telegraf/paloalto_api_collector.py | 5 + tests/test_paloalto_api_collector.py | 13 + tests/test_paloalto_api_dashboard.py | 22 ++ 9 files changed, 505 insertions(+), 165 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 88b9dda..2581035 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ This project should provide an easy-to-install Docker Compose stack for basic Pa The core use case is helping users understand what a firewall is actually doing with minimal setup. Prioritize CPU, RAM, sessions, CPS, disk where useful, interface status, and interface throughput over deep feature-specific monitoring. -Throughput must be calculated from cumulative interface octet counters. The standard Palo Alto and Fortinet dashboards use `ifHCInOctets` / `ifHCOutOctets`. The Palo Alto API dashboard is intentionally API-only and uses the MAC-level `port/rx-bytes` / `port/tx-bytes` counters of physical ports from `show counter interface all` (falling back to `ibytes` / `obytes` only when the `port` block is absent): `ibytes` / `obytes` are dataplane counters that miss hardware-offloaded flows. Do not use dataplane, NPU, session, or feature throughput summaries, which can be incomplete when traffic is offloaded or bypasses their counter path. +Throughput must be calculated from cumulative interface octet counters. The standard Palo Alto and Fortinet dashboards use `ifHCInOctets` / `ifHCOutOctets`. The Palo Alto API dashboard is intentionally API-only and uses the MAC-level `port/rx-bytes` / `port/tx-bytes` counters of physical ports from `show counter interface all` (falling back to `ibytes` / `obytes` only when the `port` block is absent): `ibytes` / `obytes` are dataplane counters that miss hardware-offloaded flows. The API collector also stores `ibytes` / `obytes` as `dp_in_octets` / `dp_out_octets` next to the port counters, only for the offloaded-traffic estimate (port minus dataplane) of the API dashboards. Do not use dataplane, NPU, session, or feature throughput summaries, which can be incomplete when traffic is offloaded or bypasses their counter path. For CPU views, the standard SNMP dashboards should show both the global CPU and every per-processor/dataplane CPU exposed by the vendor MIB (`pan_hr_processors` for Palo Alto and `fortinet_processors` for Fortinet). The Palo Alto API dashboard must remain API-only and show management-plane CPU plus every dataplane/core returned by `resource-monitor`. Its all-core dataplane average matches SNMP but includes cores that PAN-OS lists without using them for packet processing (always 0%); `cpu_active_pct` excludes them and is shown next to it. The overview is useful for quick reading; per-CPU lines reveal imbalance and saturated dataplanes. diff --git a/README.md b/README.md index 0795f7a..c8fdcae 100644 --- a/README.md +++ b/README.md @@ -809,6 +809,7 @@ Notes: - A current-load strip with 30-minute sparklines: DP CPU average, hottest DP core (including the one-minute peak), MP CPU and RAM, sessions, session table, CPS, and throughput. - CPU, RAM, session-utilization, resource-pressure, and per-core panels with dashed 70% and 90% guide lines. The **CPU MP / DP** overview shows the management plane, the average of all dataplanes (multi-dataplane systems only), the active-core average of all dataplanes, and one all-core average per dataplane; hottest-core and active-core lines are also in each dataplane's CPU summary. - **DP CPU (avg)** and **DP CPU (active cores)** tiles. PAN-OS lists every core of a dataplane in `resource-monitor`, but only the pan task cores process packets: on a PA-5580 dataplane, 80 of the 128 listed cores. The others stay at exactly 0% and halve the all-core average, which is also the value SNMP `hrProcessorLoad` reports (45% all-core for 72% on the packet-processing cores during a load test). The collector writes `cpu_active_pct` (mean of the cores with any load in the minute) and `active_cores` on each dataplane's `core=average` point; the active-core figure is the one that reaches 100% at saturation. It is also drawn in the Load Test ramp and scatter plot. +- A **Throughput Received: Dataplane vs Offloaded** panel. The physical ports count every received frame, while the dataplane counter (`ibytes`) misses hardware-offloaded flows, so the difference estimates the offloaded traffic. It is an estimate: it also includes frames dropped before the dataplane and framing overhead, it is averaged over at least 2 minutes, and it reads 0 below 5% of the total because small platforms refresh their port counters in bursts. A firewall without offload, such as a PA-440, shows 0. SNMP has no dataplane byte counter, so the SNMP dashboards have no equivalent. - An **Ingress Backlog by Dataplane** overview panel, a link-utilization table (interface, In %, Out %, rates, and speed, readable without scrolling), worst-dataplane resource pressure, and the global drop rate stacked by counter category. - Collapsible sections for HA (role timeline, sync state, and an **HA Links and Monitoring** table), interfaces and errors/discards, a **VSYS** section and a **Dataplanes** section whose panels repeat side by side for every VSYS or dataplane, zones and logical interfaces, session protocols, DoS/zone-protection drops, filtered global counters, MP load/storage, and environmental sensors split into temperature, fan, power, and alarm panels. - A collapsed **Logging and Management Health** section: log rate, logs discarded, content versions, management processes not running, GlobalProtect users, and RAID state. @@ -827,7 +828,7 @@ Fixed multi-dataplane models such as PA-5580 get the per-DP, interface, MP, sens Every API panel carries an (i) icon next to its title. Hovering it shows what the panel means plus the PAN-OS CLI command behind the data (for example `show session info` or `show running resource-monitor minute last 1`) and its default polling interval, so a value can be checked on the firewall itself. The mapping lives in `MEASUREMENT_SOURCES` in `scripts/build_paloalto_api_dashboard.py`. -Per-VSYS sessions come from `show session meter`. Per-zone and per-VSYS throughput come from the logical interface (`ifnet`) counters already returned by `show counter interface all`, tagged with the zone and VSYS learned from `show interface all`. Per-VSYS CPS and packet rate come from `show session info` scoped to each VSYS (the SNMP `panVsysTotalCps` equivalent); VSYS slots that `show session meter` lists but PAN-OS reports as not configured are skipped and probed again every `system_interval`. The XML API has no per-zone CPS equivalent to the SNMP zone CPS objects, so zone CPS remains SNMP-only. Global throughput uses only the MAC-level port counters (`port/rx-bytes` / `port/tx-bytes`) of physical Ethernet ports; the dataplane `ibytes` / `obytes` counters are a fallback only, because they miss hardware-offloaded flows. Only physical ports are summed, so aggregate `internal`, `vlan`, `loopback`, and `tunnel` counters are not double-counted. The repeated per-interface throughput and errors/discards panels cover the same interfaces as the SNMP dashboard: physical ports from the MAC-level port counters, and subinterfaces, tunnels, VLAN, loopback and aggregate interfaces from the logical (`ifnet`) counters. The `ifnet` counters are maintained by the dataplane, so per-zone, per-VSYS and logical-interface throughput can under-report hardware-offloaded flows; PAN-OS exposes no MAC-level equivalent for logical interfaces. +Per-VSYS sessions come from `show session meter`. Per-zone and per-VSYS throughput come from the logical interface (`ifnet`) counters already returned by `show counter interface all`, tagged with the zone and VSYS learned from `show interface all`. Per-VSYS CPS and packet rate come from `show session info` scoped to each VSYS (the SNMP `panVsysTotalCps` equivalent); VSYS slots that `show session meter` lists but PAN-OS reports as not configured are skipped and probed again every `system_interval`. The XML API has no per-zone CPS equivalent to the SNMP zone CPS objects, so zone CPS remains SNMP-only. Global throughput uses only the MAC-level port counters (`port/rx-bytes` / `port/tx-bytes`) of physical Ethernet ports; the dataplane `ibytes` / `obytes` counters are a fallback only, because they miss hardware-offloaded flows. Only physical ports are summed, so aggregate `internal`, `vlan`, `loopback`, and `tunnel` counters are not double-counted. The repeated per-interface throughput and errors/discards panels cover the same interfaces as the SNMP dashboard: physical ports from the MAC-level port counters, and subinterfaces, tunnels, VLAN, loopback and aggregate interfaces from the logical (`ifnet`) counters. The dataplane `ibytes` / `obytes` of the same physical ports are stored as `dp_in_octets` / `dp_out_octets` (only when the port counters exist) and are used only for the offloaded-traffic estimate. The `ifnet` counters are maintained by the dataplane, so per-zone, per-VSYS and logical-interface throughput can under-report hardware-offloaded flows; PAN-OS exposes no MAC-level equivalent for logical interfaces. ### API Coverage and Deliberate Limits diff --git a/docs/adr/0001-paloalto-xml-api-collector.md b/docs/adr/0001-paloalto-xml-api-collector.md index a7af323..92216d0 100644 --- a/docs/adr/0001-paloalto-xml-api-collector.md +++ b/docs/adr/0001-paloalto-xml-api-collector.md @@ -17,7 +17,7 @@ Prefer exact `${VARIABLE}` references in YAML for new installations. Resolve the Use one general API dashboard for compact and chassis platforms. Per-core CPU carries `dataplane` and `core` tags, and Grafana repeats a collapsible detail row for every dataplane. Collect resource pressure per dataplane, interface operational metadata, HA state, storage, and environmental sensors on slower schedules than session and octet counters. -Keep this dashboard API-only: calculate throughput from deltas of the per-port MAC counters `port/rx-bytes` and `port/tx-bytes` returned by `show counter interface all`, with `ibytes` and `obytes` only as a fallback when the `port` block is missing. Amended 2026-09-24: a load test with half its traffic hardware-offloaded showed `ibytes`/`obytes` (and the session kbps summary) at 47 Gb/s while the port counters and SNMP `ifHCInOctets` read 97 Gb/s; the `ibytes`/`obytes` counters are updated by the dataplane and miss offloaded flows. SNMP `ifHCInOctets` returns the same port counter, cached by the SNMP agent for several seconds. Do not use the session throughput summary. The standard dashboards continue using SNMP `ifHCInOctets` and `ifHCOutOctets`. +Keep this dashboard API-only: calculate throughput from deltas of the per-port MAC counters `port/rx-bytes` and `port/tx-bytes` returned by `show counter interface all`, with `ibytes` and `obytes` only as a fallback when the `port` block is missing. Amended 2026-09-24: a load test with half its traffic hardware-offloaded showed `ibytes`/`obytes` (and the session kbps summary) at 47 Gb/s while the port counters and SNMP `ifHCInOctets` read 97 Gb/s; the `ibytes`/`obytes` counters are updated by the dataplane and miss offloaded flows. SNMP `ifHCInOctets` returns the same port counter, cached by the SNMP agent for several seconds. Amended 2026-09-26: the dataplane `ibytes`/`obytes` of the physical ports are kept as `dp_in_octets`/`dp_out_octets` when the port counters exist, only to estimate offloaded traffic as port minus dataplane; the estimate is averaged over at least 2 minutes and floored at 5% of the total because a PA-440 without offload refreshes its port counters in bursts (20 s differences swung by several Mb/s both ways while 3-minute totals agreed within 3%). Do not use the session throughput summary. The standard dashboards continue using SNMP `ifHCInOctets` and `ifHCOutOctets`. Provide a separate high-end API chassis dashboard for PA-5200, PA-5400, PA-5500, PA-7000, and PA-7500 systems. Fixed multi-DP appliances use the same per-DP views. Detect modular PA-5450, PA-7050, PA-7080, and PA-7500 models from `show system info` before polling `show chassis inventory`, `show chassis status`, and `show chassis power`, avoiding unsupported calls on fixed appliances. diff --git a/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json b/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json index 1945a0e..96cc768 100644 --- a/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json +++ b/grafana/provisioning/dashboards/Palo_API_Chassis_Dashboard.json @@ -1395,6 +1395,127 @@ } ] }, + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "fieldConfig": { + "defaults": { + "unit": "bps", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 25, + "spanNulls": false, + "showPoints": "never", + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "custom.stacking", + "value": { + "mode": "none", + "group": "A" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dataplane" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "blue" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Offloaded (estimate)" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 44 + }, + "id": 13014, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "query": "window = if int(v: v.windowPeriod) > int(v: 2m) then v.windowPeriod else 2m\nfrom(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and (r._field == \"in_octets\" or r._field == \"dp_in_octets\"))\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: window, fn: mean, createEmpty: false)\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.in_octets and exists r.dp_in_octets)\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, dataplane: 0.0}, fn: (r, accumulator) => ({ total: accumulator.total + r.in_octets * 8.0, dataplane: accumulator.dataplane + r.dp_in_octets * 8.0 }))\n |> group()\n |> sort(columns: [\"_time\"])\n |> map(fn: (r) => ({ r with delta: r.total - r.dataplane, _value: r.total - r.dataplane }))\n |> difference(columns: [\"_value\"], keepFirst: true)\n |> map(fn: (r) => {\n offload = if exists r._value then r.delta - (if r._value > 0.0 then r._value else 0.0) else 0.0\n return {\n _time: r._time,\n \"Dataplane\": r.dataplane,\n \"Offloaded (estimate)\": if offload > r.total * 0.05 then offload else 0.0,\n \"Total\": r.total,\n }\n })", + "refId": "A" + } + ], + "title": "Throughput Received: Dataplane vs Offloaded", + "type": "timeseries", + "description": "Traffic received on the physical Ethernet ports, split between what the dataplane counted (ibytes of show counter interface all) and the rest, which is mostly hardware-offloaded flows. Offloaded = port counter (port/rx-bytes) minus dataplane counter. It is an estimate: it also contains frames dropped before the dataplane and framing overhead. Some platforms refresh the port counters late and in bursts, so it is averaged over at least 2 minutes, only counted when the previous window shows it too, and set to 0 below 5% of the total. The Total line is not stacked. No data means the firewall does not return the MAC-level port counters.\n\nSource: `show counter interface all` via the PAN-OS XML API, polled every 20 s by default." + }, { "datasource": { "type": "influxdb", @@ -1440,7 +1561,7 @@ "h": 8, "w": 8, "x": 0, - "y": 44 + "y": 53 }, "id": 13011, "options": { @@ -1518,7 +1639,7 @@ "h": 8, "w": 8, "x": 8, - "y": 44 + "y": 53 }, "id": 13013, "options": { @@ -1578,7 +1699,7 @@ "h": 8, "w": 8, "x": 16, - "y": 44 + "y": 53 }, "id": 13012, "options": { @@ -2091,7 +2212,7 @@ "h": 1, "w": 24, "x": 0, - "y": 52 + "y": 61 }, "id": 9201, "panels": [ @@ -2115,7 +2236,7 @@ "h": 11, "w": 12, "x": 0, - "y": 53 + "y": 62 }, "id": 2010, "options": { @@ -2159,7 +2280,7 @@ "h": 11, "w": 12, "x": 12, - "y": 53 + "y": 62 }, "id": 2024, "options": { @@ -2266,7 +2387,7 @@ "h": 6, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 2040, "options": { @@ -2300,7 +2421,7 @@ "h": 1, "w": 24, "x": 0, - "y": 53 + "y": 62 }, "id": 9202, "panels": [ @@ -2327,7 +2448,7 @@ "h": 10, "w": 14, "x": 0, - "y": 54 + "y": 63 }, "id": 2011, "options": { @@ -2380,7 +2501,7 @@ "h": 10, "w": 10, "x": 14, - "y": 54 + "y": 63 }, "id": 2012, "options": { @@ -2427,7 +2548,7 @@ "h": 8, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 2025, "options": { @@ -2470,7 +2591,7 @@ "h": 1, "w": 24, "x": 0, - "y": 54 + "y": 63 }, "id": 9204, "panels": [ @@ -2497,7 +2618,7 @@ "h": 10, "w": 12, "x": 0, - "y": 55 + "y": 64 }, "id": 2015, "options": { @@ -2553,7 +2674,7 @@ "h": 10, "w": 12, "x": 12, - "y": 55 + "y": 64 }, "id": 2016, "options": { @@ -2609,7 +2730,7 @@ "h": 10, "w": 12, "x": 0, - "y": 65 + "y": 74 }, "id": 2017, "options": { @@ -2662,7 +2783,7 @@ "h": 10, "w": 12, "x": 12, - "y": 65 + "y": 74 }, "id": 2018, "options": { @@ -2696,7 +2817,7 @@ "h": 1, "w": 24, "x": 0, - "y": 55 + "y": 64 }, "id": 9205, "panels": [ @@ -2723,7 +2844,7 @@ "h": 11, "w": 24, "x": 0, - "y": 56 + "y": 65 }, "id": 2019, "options": { @@ -2766,7 +2887,7 @@ "h": 1, "w": 24, "x": 0, - "y": 56 + "y": 65 }, "id": 19013, "panels": [ @@ -2798,7 +2919,7 @@ "h": 4, "w": 3, "x": 0, - "y": 57 + "y": 66 }, "id": 13601, "options": { @@ -2858,7 +2979,7 @@ "h": 4, "w": 3, "x": 3, - "y": 57 + "y": 66 }, "id": 13602, "options": { @@ -2918,7 +3039,7 @@ "h": 4, "w": 3, "x": 6, - "y": 57 + "y": 66 }, "id": 13603, "options": { @@ -2978,7 +3099,7 @@ "h": 4, "w": 3, "x": 9, - "y": 57 + "y": 66 }, "id": 13604, "options": { @@ -3038,7 +3159,7 @@ "h": 4, "w": 3, "x": 12, - "y": 57 + "y": 66 }, "id": 13605, "options": { @@ -3107,7 +3228,7 @@ "h": 4, "w": 3, "x": 15, - "y": 57 + "y": 66 }, "id": 13606, "options": { @@ -3176,7 +3297,7 @@ "h": 4, "w": 3, "x": 18, - "y": 57 + "y": 66 }, "id": 13607, "options": { @@ -3240,7 +3361,7 @@ "h": 4, "w": 3, "x": 21, - "y": 57 + "y": 66 }, "id": 13608, "options": { @@ -3328,7 +3449,7 @@ "h": 10, "w": 24, "x": 0, - "y": 61 + "y": 70 }, "id": 13611, "options": { @@ -3487,7 +3608,7 @@ "h": 10, "w": 12, "x": 0, - "y": 71 + "y": 80 }, "id": 13612, "options": { @@ -3637,7 +3758,7 @@ "h": 10, "w": 12, "x": 12, - "y": 71 + "y": 80 }, "id": 13613, "options": { @@ -3726,7 +3847,7 @@ "h": 9, "w": 12, "x": 0, - "y": 81 + "y": 90 }, "id": 13614, "options": { @@ -3782,7 +3903,7 @@ "h": 9, "w": 12, "x": 12, - "y": 81 + "y": 90 }, "id": 13615, "options": { @@ -3825,7 +3946,7 @@ "h": 1, "w": 24, "x": 0, - "y": 57 + "y": 66 }, "id": 19006, "panels": [ @@ -3892,7 +4013,7 @@ "h": 6, "w": 24, "x": 0, - "y": 58 + "y": 67 }, "id": 11006, "options": { @@ -4023,7 +4144,7 @@ "h": 4, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 11007, "options": { @@ -4176,7 +4297,7 @@ "h": 4, "w": 24, "x": 0, - "y": 68 + "y": 77 }, "id": 11008, "options": { @@ -4210,7 +4331,7 @@ "h": 1, "w": 24, "x": 0, - "y": 58 + "y": 67 }, "id": 19001, "panels": [ @@ -4237,7 +4358,7 @@ "h": 7, "w": 12, "x": 0, - "y": 59 + "y": 68 }, "id": 10008, "options": { @@ -4283,7 +4404,7 @@ "h": 1, "w": 24, "x": 0, - "y": 59 + "y": 68 }, "id": 19008, "panels": [ @@ -4310,7 +4431,7 @@ "h": 10, "w": 12, "x": 0, - "y": 60 + "y": 69 }, "id": 10009, "options": { @@ -4363,7 +4484,7 @@ "h": 10, "w": 12, "x": 12, - "y": 60 + "y": 69 }, "id": 10010, "options": { @@ -4397,7 +4518,7 @@ "h": 1, "w": 24, "x": 0, - "y": 60 + "y": 69 }, "id": 19004, "panels": [ @@ -4424,7 +4545,7 @@ "h": 10, "w": 24, "x": 0, - "y": 61 + "y": 70 }, "id": 10011, "options": { @@ -4470,7 +4591,7 @@ "h": 1, "w": 24, "x": 0, - "y": 61 + "y": 70 }, "id": 19009, "panels": [ @@ -4497,7 +4618,7 @@ "h": 8, "w": 24, "x": 0, - "y": 62 + "y": 71 }, "id": 13101, "options": { @@ -4556,7 +4677,7 @@ "h": 8, "w": 24, "x": 0, - "y": 70 + "y": 79 }, "id": 13103, "options": { @@ -4615,7 +4736,7 @@ "h": 8, "w": 24, "x": 0, - "y": 78 + "y": 87 }, "id": 13102, "options": { @@ -4661,7 +4782,7 @@ "h": 1, "w": 24, "x": 0, - "y": 62 + "y": 71 }, "id": 19010, "panels": [ @@ -4688,7 +4809,7 @@ "h": 10, "w": 12, "x": 0, - "y": 63 + "y": 72 }, "id": 13111, "options": { @@ -4744,7 +4865,7 @@ "h": 10, "w": 12, "x": 12, - "y": 63 + "y": 72 }, "id": 13114, "options": { @@ -4800,7 +4921,7 @@ "h": 10, "w": 24, "x": 0, - "y": 73 + "y": 82 }, "id": 13113, "options": { @@ -4853,7 +4974,7 @@ "h": 9, "w": 24, "x": 0, - "y": 83 + "y": 92 }, "id": 13115, "options": { @@ -4887,7 +5008,7 @@ "h": 1, "w": 24, "x": 0, - "y": 63 + "y": 72 }, "id": 19002, "panels": [ @@ -4936,7 +5057,7 @@ "h": 9, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 13201, "options": { @@ -4997,7 +5118,7 @@ "h": 10, "w": 24, "x": 0, - "y": 73 + "y": 82 }, "id": 13202, "options": { @@ -5073,7 +5194,7 @@ "h": 10, "w": 24, "x": 0, - "y": 83 + "y": 92 }, "id": 10012, "options": { @@ -5134,7 +5255,7 @@ "h": 9, "w": 24, "x": 0, - "y": 93 + "y": 102 }, "id": 10013, "options": { @@ -5180,7 +5301,7 @@ "h": 1, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 19007, "panels": [ @@ -5207,7 +5328,7 @@ "h": 9, "w": 12, "x": 0, - "y": 65 + "y": 74 }, "id": 10014, "options": { @@ -5263,7 +5384,7 @@ "h": 9, "w": 12, "x": 12, - "y": 65 + "y": 74 }, "id": 10015, "options": { @@ -5306,7 +5427,7 @@ "h": 1, "w": 24, "x": 0, - "y": 65 + "y": 74 }, "id": 19003, "panels": [ @@ -5333,7 +5454,7 @@ "h": 9, "w": 12, "x": 0, - "y": 66 + "y": 75 }, "id": 13301, "options": { @@ -5389,7 +5510,7 @@ "h": 9, "w": 12, "x": 12, - "y": 66 + "y": 75 }, "id": 13302, "options": { @@ -5445,7 +5566,7 @@ "h": 9, "w": 12, "x": 0, - "y": 75 + "y": 84 }, "id": 13303, "options": { @@ -5501,7 +5622,7 @@ "h": 9, "w": 12, "x": 12, - "y": 75 + "y": 84 }, "id": 13304, "options": { @@ -5557,7 +5678,7 @@ "h": 9, "w": 24, "x": 0, - "y": 84 + "y": 93 }, "id": 13305, "options": { @@ -5600,7 +5721,7 @@ "h": 1, "w": 24, "x": 0, - "y": 66 + "y": 75 }, "id": 19011, "panels": [ @@ -5627,7 +5748,7 @@ "h": 12, "w": 24, "x": 0, - "y": 67 + "y": 76 }, "id": 10016, "options": { @@ -5680,7 +5801,7 @@ "h": 10, "w": 24, "x": 0, - "y": 79 + "y": 88 }, "id": 13311, "options": { @@ -5714,7 +5835,7 @@ "h": 1, "w": 24, "x": 0, - "y": 67 + "y": 76 }, "id": 19012, "panels": [ @@ -5742,7 +5863,7 @@ "h": 9, "w": 12, "x": 0, - "y": 68 + "y": 77 }, "id": 13501, "options": { @@ -5799,7 +5920,7 @@ "h": 9, "w": 12, "x": 12, - "y": 68 + "y": 77 }, "id": 13502, "options": { @@ -5949,7 +6070,7 @@ "h": 4, "w": 24, "x": 0, - "y": 77 + "y": 86 }, "id": 13503, "options": { @@ -6005,7 +6126,7 @@ "h": 4, "w": 4, "x": 0, - "y": 81 + "y": 90 }, "id": 13504, "options": { @@ -6095,7 +6216,7 @@ "h": 8, "w": 20, "x": 4, - "y": 81 + "y": 90 }, "id": 13505, "options": { @@ -6147,7 +6268,7 @@ "h": 4, "w": 4, "x": 0, - "y": 85 + "y": 94 }, "id": 13506, "options": { @@ -6202,7 +6323,7 @@ "h": 8, "w": 24, "x": 0, - "y": 89 + "y": 98 }, "id": 13507, "options": { @@ -6245,7 +6366,7 @@ "h": 1, "w": 24, "x": 0, - "y": 68 + "y": 77 }, "id": 19005, "panels": [ @@ -6272,7 +6393,7 @@ "h": 10, "w": 12, "x": 0, - "y": 69 + "y": 78 }, "id": 10017, "options": { @@ -6328,7 +6449,7 @@ "h": 10, "w": 12, "x": 12, - "y": 69 + "y": 78 }, "id": 10018, "options": { @@ -6386,7 +6507,7 @@ "h": 10, "w": 12, "x": 0, - "y": 79 + "y": 88 }, "id": 10019, "options": { @@ -6546,7 +6667,7 @@ "h": 10, "w": 12, "x": 12, - "y": 79 + "y": 88 }, "id": 13401, "options": { @@ -6595,7 +6716,7 @@ "h": 9, "w": 12, "x": 0, - "y": 89 + "y": 98 }, "id": 10022, "options": { @@ -6651,7 +6772,7 @@ "h": 9, "w": 12, "x": 12, - "y": 89 + "y": 98 }, "id": 10023, "options": { diff --git a/grafana/provisioning/dashboards/Palo_API_Dashboard.json b/grafana/provisioning/dashboards/Palo_API_Dashboard.json index 859bb49..0de6534 100644 --- a/grafana/provisioning/dashboards/Palo_API_Dashboard.json +++ b/grafana/provisioning/dashboards/Palo_API_Dashboard.json @@ -1395,6 +1395,127 @@ } ] }, + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "fieldConfig": { + "defaults": { + "unit": "bps", + "custom": { + "drawStyle": "line", + "lineInterpolation": "linear", + "lineWidth": 1, + "fillOpacity": 25, + "spanNulls": false, + "showPoints": "never", + "stacking": { + "mode": "normal", + "group": "A" + } + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "Total" + }, + "properties": [ + { + "id": "custom.stacking", + "value": { + "mode": "none", + "group": "A" + } + }, + { + "id": "custom.fillOpacity", + "value": 0 + }, + { + "id": "custom.lineWidth", + "value": 2 + }, + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "text" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Dataplane" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "blue" + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "Offloaded (estimate)" + }, + "properties": [ + { + "id": "color", + "value": { + "mode": "fixed", + "fixedColor": "orange" + } + } + ] + } + ] + }, + "gridPos": { + "h": 9, + "w": 24, + "x": 0, + "y": 35 + }, + "id": 3014, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "influxdb", + "uid": "P951FEA4DE68E13C5" + }, + "query": "window = if int(v: v.windowPeriod) > int(v: 2m) then v.windowPeriod else 2m\nfrom(bucket: \"firewalls\")\n |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n |> filter(fn: (r) => r._measurement == \"paloalto_api_interfaces\" and r.hostname == \"${hostname}\" and exists r.interface and r.interface =~ /(?i)^ethernet/ and r.interface !~ /\\./ and (r._field == \"in_octets\" or r._field == \"dp_in_octets\"))\n |> derivative(unit: 1s, nonNegative: true)\n |> aggregateWindow(every: window, fn: mean, createEmpty: false)\n |> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\n |> filter(fn: (r) => exists r.in_octets and exists r.dp_in_octets)\n |> group(columns: [\"_time\"])\n |> reduce(identity: {total: 0.0, dataplane: 0.0}, fn: (r, accumulator) => ({ total: accumulator.total + r.in_octets * 8.0, dataplane: accumulator.dataplane + r.dp_in_octets * 8.0 }))\n |> group()\n |> sort(columns: [\"_time\"])\n |> map(fn: (r) => ({ r with delta: r.total - r.dataplane, _value: r.total - r.dataplane }))\n |> difference(columns: [\"_value\"], keepFirst: true)\n |> map(fn: (r) => {\n offload = if exists r._value then r.delta - (if r._value > 0.0 then r._value else 0.0) else 0.0\n return {\n _time: r._time,\n \"Dataplane\": r.dataplane,\n \"Offloaded (estimate)\": if offload > r.total * 0.05 then offload else 0.0,\n \"Total\": r.total,\n }\n })", + "refId": "A" + } + ], + "title": "Throughput Received: Dataplane vs Offloaded", + "type": "timeseries", + "description": "Traffic received on the physical Ethernet ports, split between what the dataplane counted (ibytes of show counter interface all) and the rest, which is mostly hardware-offloaded flows. Offloaded = port counter (port/rx-bytes) minus dataplane counter. It is an estimate: it also contains frames dropped before the dataplane and framing overhead. Some platforms refresh the port counters late and in bursts, so it is averaged over at least 2 minutes, only counted when the previous window shows it too, and set to 0 below 5% of the total. The Total line is not stacked. No data means the firewall does not return the MAC-level port counters.\n\nSource: `show counter interface all` via the PAN-OS XML API, polled every 20 s by default." + }, { "datasource": { "type": "influxdb", @@ -1440,7 +1561,7 @@ "h": 8, "w": 8, "x": 0, - "y": 35 + "y": 44 }, "id": 3011, "options": { @@ -1518,7 +1639,7 @@ "h": 8, "w": 8, "x": 8, - "y": 35 + "y": 44 }, "id": 3013, "options": { @@ -1578,7 +1699,7 @@ "h": 8, "w": 8, "x": 16, - "y": 35 + "y": 44 }, "id": 3012, "options": { @@ -1617,7 +1738,7 @@ "h": 1, "w": 24, "x": 0, - "y": 43 + "y": 52 }, "id": 9013, "panels": [ @@ -1649,7 +1770,7 @@ "h": 4, "w": 3, "x": 0, - "y": 44 + "y": 53 }, "id": 3601, "options": { @@ -1709,7 +1830,7 @@ "h": 4, "w": 3, "x": 3, - "y": 44 + "y": 53 }, "id": 3602, "options": { @@ -1769,7 +1890,7 @@ "h": 4, "w": 3, "x": 6, - "y": 44 + "y": 53 }, "id": 3603, "options": { @@ -1829,7 +1950,7 @@ "h": 4, "w": 3, "x": 9, - "y": 44 + "y": 53 }, "id": 3604, "options": { @@ -1889,7 +2010,7 @@ "h": 4, "w": 3, "x": 12, - "y": 44 + "y": 53 }, "id": 3605, "options": { @@ -1958,7 +2079,7 @@ "h": 4, "w": 3, "x": 15, - "y": 44 + "y": 53 }, "id": 3606, "options": { @@ -2027,7 +2148,7 @@ "h": 4, "w": 3, "x": 18, - "y": 44 + "y": 53 }, "id": 3607, "options": { @@ -2091,7 +2212,7 @@ "h": 4, "w": 3, "x": 21, - "y": 44 + "y": 53 }, "id": 3608, "options": { @@ -2179,7 +2300,7 @@ "h": 10, "w": 24, "x": 0, - "y": 48 + "y": 57 }, "id": 3611, "options": { @@ -2338,7 +2459,7 @@ "h": 10, "w": 12, "x": 0, - "y": 58 + "y": 67 }, "id": 3612, "options": { @@ -2488,7 +2609,7 @@ "h": 10, "w": 12, "x": 12, - "y": 58 + "y": 67 }, "id": 3613, "options": { @@ -2577,7 +2698,7 @@ "h": 9, "w": 12, "x": 0, - "y": 68 + "y": 77 }, "id": 3614, "options": { @@ -2633,7 +2754,7 @@ "h": 9, "w": 12, "x": 12, - "y": 68 + "y": 77 }, "id": 3615, "options": { @@ -2676,7 +2797,7 @@ "h": 1, "w": 24, "x": 0, - "y": 44 + "y": 53 }, "id": 9006, "panels": [ @@ -2743,7 +2864,7 @@ "h": 6, "w": 24, "x": 0, - "y": 45 + "y": 54 }, "id": 1006, "options": { @@ -2874,7 +2995,7 @@ "h": 4, "w": 24, "x": 0, - "y": 51 + "y": 60 }, "id": 1007, "options": { @@ -3027,7 +3148,7 @@ "h": 4, "w": 24, "x": 0, - "y": 55 + "y": 64 }, "id": 1008, "options": { @@ -3061,7 +3182,7 @@ "h": 1, "w": 24, "x": 0, - "y": 45 + "y": 54 }, "id": 9001, "panels": [ @@ -3088,7 +3209,7 @@ "h": 7, "w": 12, "x": 0, - "y": 46 + "y": 55 }, "id": 8, "options": { @@ -3134,7 +3255,7 @@ "h": 1, "w": 24, "x": 0, - "y": 46 + "y": 55 }, "id": 9008, "panels": [ @@ -3161,7 +3282,7 @@ "h": 10, "w": 12, "x": 0, - "y": 47 + "y": 56 }, "id": 9, "options": { @@ -3214,7 +3335,7 @@ "h": 10, "w": 12, "x": 12, - "y": 47 + "y": 56 }, "id": 10, "options": { @@ -3248,7 +3369,7 @@ "h": 1, "w": 24, "x": 0, - "y": 47 + "y": 56 }, "id": 9004, "panels": [ @@ -3275,7 +3396,7 @@ "h": 10, "w": 24, "x": 0, - "y": 48 + "y": 57 }, "id": 11, "options": { @@ -3321,7 +3442,7 @@ "h": 1, "w": 24, "x": 0, - "y": 48 + "y": 57 }, "id": 9009, "panels": [ @@ -3348,7 +3469,7 @@ "h": 8, "w": 24, "x": 0, - "y": 49 + "y": 58 }, "id": 3101, "options": { @@ -3407,7 +3528,7 @@ "h": 8, "w": 24, "x": 0, - "y": 57 + "y": 66 }, "id": 3103, "options": { @@ -3466,7 +3587,7 @@ "h": 8, "w": 24, "x": 0, - "y": 65 + "y": 74 }, "id": 3102, "options": { @@ -3512,7 +3633,7 @@ "h": 1, "w": 24, "x": 0, - "y": 49 + "y": 58 }, "id": 9010, "panels": [ @@ -3539,7 +3660,7 @@ "h": 10, "w": 12, "x": 0, - "y": 50 + "y": 59 }, "id": 3111, "options": { @@ -3595,7 +3716,7 @@ "h": 10, "w": 12, "x": 12, - "y": 50 + "y": 59 }, "id": 3114, "options": { @@ -3651,7 +3772,7 @@ "h": 10, "w": 24, "x": 0, - "y": 60 + "y": 69 }, "id": 3113, "options": { @@ -3704,7 +3825,7 @@ "h": 9, "w": 24, "x": 0, - "y": 70 + "y": 79 }, "id": 3115, "options": { @@ -3738,7 +3859,7 @@ "h": 1, "w": 24, "x": 0, - "y": 50 + "y": 59 }, "id": 9002, "panels": [ @@ -3787,7 +3908,7 @@ "h": 9, "w": 24, "x": 0, - "y": 51 + "y": 60 }, "id": 3201, "options": { @@ -3848,7 +3969,7 @@ "h": 10, "w": 24, "x": 0, - "y": 60 + "y": 69 }, "id": 3202, "options": { @@ -3924,7 +4045,7 @@ "h": 10, "w": 24, "x": 0, - "y": 70 + "y": 79 }, "id": 12, "options": { @@ -3985,7 +4106,7 @@ "h": 9, "w": 24, "x": 0, - "y": 80 + "y": 89 }, "id": 13, "options": { @@ -4031,7 +4152,7 @@ "h": 1, "w": 24, "x": 0, - "y": 51 + "y": 60 }, "id": 9007, "panels": [ @@ -4058,7 +4179,7 @@ "h": 9, "w": 12, "x": 0, - "y": 52 + "y": 61 }, "id": 14, "options": { @@ -4114,7 +4235,7 @@ "h": 9, "w": 12, "x": 12, - "y": 52 + "y": 61 }, "id": 15, "options": { @@ -4157,7 +4278,7 @@ "h": 1, "w": 24, "x": 0, - "y": 52 + "y": 61 }, "id": 9003, "panels": [ @@ -4184,7 +4305,7 @@ "h": 9, "w": 12, "x": 0, - "y": 53 + "y": 62 }, "id": 3301, "options": { @@ -4240,7 +4361,7 @@ "h": 9, "w": 12, "x": 12, - "y": 53 + "y": 62 }, "id": 3302, "options": { @@ -4296,7 +4417,7 @@ "h": 9, "w": 12, "x": 0, - "y": 62 + "y": 71 }, "id": 3303, "options": { @@ -4352,7 +4473,7 @@ "h": 9, "w": 12, "x": 12, - "y": 62 + "y": 71 }, "id": 3304, "options": { @@ -4408,7 +4529,7 @@ "h": 9, "w": 24, "x": 0, - "y": 71 + "y": 80 }, "id": 3305, "options": { @@ -4451,7 +4572,7 @@ "h": 1, "w": 24, "x": 0, - "y": 53 + "y": 62 }, "id": 9011, "panels": [ @@ -4478,7 +4599,7 @@ "h": 12, "w": 24, "x": 0, - "y": 54 + "y": 63 }, "id": 16, "options": { @@ -4531,7 +4652,7 @@ "h": 10, "w": 24, "x": 0, - "y": 66 + "y": 75 }, "id": 3311, "options": { @@ -4565,7 +4686,7 @@ "h": 1, "w": 24, "x": 0, - "y": 54 + "y": 63 }, "id": 9012, "panels": [ @@ -4593,7 +4714,7 @@ "h": 9, "w": 12, "x": 0, - "y": 55 + "y": 64 }, "id": 3501, "options": { @@ -4650,7 +4771,7 @@ "h": 9, "w": 12, "x": 12, - "y": 55 + "y": 64 }, "id": 3502, "options": { @@ -4800,7 +4921,7 @@ "h": 4, "w": 24, "x": 0, - "y": 64 + "y": 73 }, "id": 3503, "options": { @@ -4856,7 +4977,7 @@ "h": 4, "w": 4, "x": 0, - "y": 68 + "y": 77 }, "id": 3504, "options": { @@ -4946,7 +5067,7 @@ "h": 8, "w": 10, "x": 4, - "y": 68 + "y": 77 }, "id": 3505, "options": { @@ -5053,7 +5174,7 @@ "h": 8, "w": 10, "x": 14, - "y": 68 + "y": 77 }, "id": 3508, "options": { @@ -5105,7 +5226,7 @@ "h": 4, "w": 4, "x": 0, - "y": 72 + "y": 81 }, "id": 3506, "options": { @@ -5160,7 +5281,7 @@ "h": 8, "w": 24, "x": 0, - "y": 76 + "y": 85 }, "id": 3507, "options": { @@ -5203,7 +5324,7 @@ "h": 1, "w": 24, "x": 0, - "y": 55 + "y": 64 }, "id": 9005, "panels": [ @@ -5230,7 +5351,7 @@ "h": 10, "w": 12, "x": 0, - "y": 56 + "y": 65 }, "id": 17, "options": { @@ -5286,7 +5407,7 @@ "h": 10, "w": 12, "x": 12, - "y": 56 + "y": 65 }, "id": 18, "options": { @@ -5344,7 +5465,7 @@ "h": 10, "w": 12, "x": 0, - "y": 66 + "y": 75 }, "id": 19, "options": { @@ -5504,7 +5625,7 @@ "h": 10, "w": 12, "x": 12, - "y": 66 + "y": 75 }, "id": 3401, "options": { @@ -5553,7 +5674,7 @@ "h": 9, "w": 12, "x": 0, - "y": 76 + "y": 85 }, "id": 22, "options": { @@ -5609,7 +5730,7 @@ "h": 9, "w": 12, "x": 12, - "y": 76 + "y": 85 }, "id": 23, "options": { @@ -5652,7 +5773,7 @@ "h": 1, "w": 24, "x": 0, - "y": 56 + "y": 65 }, "id": 9100, "panels": [ @@ -5679,7 +5800,7 @@ "h": 10, "w": 12, "x": 0, - "y": 57 + "y": 66 }, "id": 20, "options": { @@ -5735,7 +5856,7 @@ "h": 10, "w": 12, "x": 12, - "y": 57 + "y": 66 }, "id": 21, "options": { @@ -5791,7 +5912,7 @@ "h": 10, "w": 12, "x": 0, - "y": 67 + "y": 76 }, "id": 24, "options": { @@ -5844,7 +5965,7 @@ "h": 10, "w": 12, "x": 12, - "y": 67 + "y": 76 }, "id": 25, "options": { diff --git a/scripts/build_paloalto_api_dashboard.py b/scripts/build_paloalto_api_dashboard.py index 2e494c2..b6c68eb 100644 --- a/scripts/build_paloalto_api_dashboard.py +++ b/scripts/build_paloalto_api_dashboard.py @@ -451,6 +451,62 @@ def kpi_panels(y: int) -> list[dict]: ] +# Share of the received traffic that bypasses the dataplane. The port counters +# see every frame; the dataplane ibytes (dp_in_octets) miss hardware-offloaded +# flows, so the difference estimates the offloaded traffic. Small platforms +# refresh the port counters late and in bursts: a burst first reads as a +# negative difference, then as an equal positive one in the next window, while +# real offload stays positive. The estimate is therefore the smaller of the +# current and previous window differences (delta - max(delta - previous, 0)), +# over windows of at least 2 minutes and floored at 5% of the total, which +# keeps a firewall without offload at zero. +OFFLOAD_WINDOW = "2m" +OFFLOAD_FLOOR = 0.05 + + +def offload_panel() -> dict: + panel = stacked(timeseries(3014, "Throughput Received: Dataplane vs Offloaded", f''' +window = if int(v: v.windowPeriod) > int(v: {OFFLOAD_WINDOW}) then v.windowPeriod else {OFFLOAD_WINDOW} +from(bucket: "firewalls") + |> range(start: v.timeRangeStart, stop: v.timeRangeStop) + |> filter(fn: (r) => r._measurement == "paloalto_api_interfaces" and r.hostname == "${{hostname}}" and {PHYSICAL} and (r._field == "in_octets" or r._field == "dp_in_octets")) + |> derivative(unit: 1s, nonNegative: true) + |> aggregateWindow(every: window, fn: mean, createEmpty: false) + |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value") + |> filter(fn: (r) => exists r.in_octets and exists r.dp_in_octets) + |> group(columns: ["_time"]) + |> reduce(identity: {{total: 0.0, dataplane: 0.0}}, fn: (r, accumulator) => ({{ total: accumulator.total + r.in_octets * 8.0, dataplane: accumulator.dataplane + r.dp_in_octets * 8.0 }})) + |> group() + |> sort(columns: ["_time"]) + |> map(fn: (r) => ({{ r with delta: r.total - r.dataplane, _value: r.total - r.dataplane }})) + |> difference(columns: ["_value"], keepFirst: true) + |> map(fn: (r) => {{ + offload = if exists r._value then r.delta - (if r._value > 0.0 then r._value else 0.0) else 0.0 + return {{ + _time: r._time, + "Dataplane": r.dataplane, + "Offloaded (estimate)": if offload > r.total * {OFFLOAD_FLOOR} then offload else 0.0, + "Total": r.total, + }} + }}) +''', 0, 35, 24, 9, "bps", ( + "Traffic received on the physical Ethernet ports, split between what the dataplane counted (ibytes of show " + "counter interface all) and the rest, which is mostly hardware-offloaded flows. Offloaded = port counter " + "(port/rx-bytes) minus dataplane counter. It is an estimate: it also contains frames dropped before the " + "dataplane and framing overhead. Some platforms refresh the port counters late and in bursts, so it is averaged " + "over at least 2 minutes, only counted when the previous window shows it too, and set to 0 below 5% of the " + "total. The Total line is not stacked. " + "No data means the firewall does not return the MAC-level port counters." + ))) + panel["fieldConfig"]["overrides"] = [ + override("Total", custom__stacking={"mode": "none", "group": "A"}, custom__fillOpacity=0, custom__lineWidth=2, + color={"mode": "fixed", "fixedColor": "text"}), + override("Dataplane", color={"mode": "fixed", "fixedColor": "blue"}), + override("Offloaded (estimate)", color={"mode": "fixed", "fixedColor": "orange"}), + ] + return panel + + def overview_panels() -> list[dict]: interface_load = table(3010, "Interface Load (last 5 minutes)", f''' speed = from(bucket: "firewalls") @@ -572,6 +628,7 @@ def overview_panels() -> list[dict]: |> group(columns: ["_field"]) ''', 0, 25, 14, 10, "bps", "Total throughput of the physical Ethernet ports, calculated from the MAC-level port octet counters of show counter interface all. These include hardware-offloaded flows, which the dataplane ibytes/obytes counters and the session throughput summary miss."), interface_load, + offload_panel(), guide_lines(percent_range(timeseries(3011, "Dataplane Resource Pressure", ''' from(bucket: "firewalls") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) @@ -580,7 +637,7 @@ def overview_panels() -> list[dict]: |> group(columns: ["_field"]) |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false) |> keep(columns: ["_time", "_field", "_value"]) -''', 0, 35, 8, 8, "percent", "Worst dataplane for each resource-monitor resource: session table, packet buffers, packet descriptors and software tags. Buffer or descriptor pressure precedes packet drops."))), +''', 0, 44, 8, 8, "percent", "Worst dataplane for each resource-monitor resource: session table, packet buffers, packet descriptors and software tags. Buffer or descriptor pressure precedes packet drops."))), guide_lines(percent_range(timeseries(3013, "Ingress Backlog by Dataplane", ''' from(bucket: "firewalls") |> range(start: v.timeRangeStart, stop: v.timeRangeStop) @@ -589,7 +646,7 @@ def overview_panels() -> list[dict]: |> group(columns: ["_field"]) |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false) |> keep(columns: ["_time", "_field", "_value"]) -''', 8, 35, 8, 8, "percent", "Packet-processing ingress queue usage per dataplane from show running resource-monitor ingress-backlogs. 0% means no backlog; a sustained backlog means the dataplane cannot keep up and precedes buffer exhaustion and drops. Dashed lines mark 50% and 80%.")), +''', 8, 44, 8, 8, "percent", "Packet-processing ingress queue usage per dataplane from show running resource-monitor ingress-backlogs. 0% means no backlog; a sustained backlog means the dataplane cannot keep up and precedes buffer exhaustion and drops. Dashed lines mark 50% and 80%.")), thresholds(("green", None), ("#EAB839", 50), ("red", 80))), stacked(timeseries(3012, "Global Drop Rate by Category", ''' from(bucket: "firewalls") @@ -601,7 +658,7 @@ def overview_panels() -> list[dict]: |> group(columns: ["_time", "_field"]) |> sum() |> group(columns: ["_field"]) -''', 16, 35, 8, 8, "pps", "Packets dropped per second by the dataplane, stacked by PAN-OS counter category so the total drop rate is the top of the stack. Details are in the drop counter sections.")), +''', 16, 44, 8, 8, "pps", "Packets dropped per second by the dataplane, stacked by PAN-OS counter category so the total drop rate is the top of the stack. Details are in the drop counter sections.")), ] diff --git a/telegraf/paloalto_api_collector.py b/telegraf/paloalto_api_collector.py index 2450ba3..55c358f 100644 --- a/telegraf/paloalto_api_collector.py +++ b/telegraf/paloalto_api_collector.py @@ -627,9 +627,14 @@ def parse_interface_counters(result: ET.Element) -> list[tuple[dict, dict]]: # ibytes/obytes and ipackets/opackets are maintained by the dataplane and # miss hardware-offloaded flows; the MAC-level port counters see every # frame on the wire, like SNMP ifHCInOctets. Prefer them when present. + # The dataplane octets are kept as dp_*_octets only next to a port + # counter, so the dashboard can estimate the offloaded share as + # port minus dataplane without mistaking the fallback for zero offload. for direction, octets, packets in (("rx", "in_octets", "in_packets"), ("tx", "out_octets", "out_packets")): value = _number(entry.findtext(f"port/{direction}-bytes")) if value is not None: + if octets in fields: + fields[f"dp_{octets}"] = fields[octets] fields[octets] = value frames = [ _number(entry.findtext(f"port/{direction}-{kind}")) diff --git a/tests/test_paloalto_api_collector.py b/tests/test_paloalto_api_collector.py index f099ce5..9a8fa19 100644 --- a/tests/test_paloalto_api_collector.py +++ b/tests/test_paloalto_api_collector.py @@ -344,6 +344,9 @@ def test_port_mac_counters_include_offloaded_traffic(self): self.assertEqual(tags, {"interface": "ethernet1/11"}) self.assertEqual(fields["in_octets"], 34276928149555) self.assertEqual(fields["out_octets"], 26526647391272) + # The dataplane counters stay available to estimate the offloaded share. + self.assertEqual(fields["dp_in_octets"], 13923248051098) + self.assertEqual(fields["dp_out_octets"], 23137800987779) self.assertEqual(fields["in_packets"], 91353449553 + 95833 + 16) self.assertEqual(fields["out_packets"], 39630544405 + 10598 + 49) self.assertEqual(fields["in_errors"], 0) @@ -351,6 +354,16 @@ def test_port_mac_counters_include_offloaded_traffic(self): self.assertEqual(fields["out_errors"], 0) self.assertEqual(fields["link_down_count"], 0) + def test_dataplane_octets_need_the_port_counter_of_the_same_direction(self): + result = ET.fromstring( + "ethernet1/110" + "2030" + ) + self.assertEqual( + parse_interface_counters(result), + [({"interface": "ethernet1/1"}, {"in_octets": 30, "dp_in_octets": 10, "out_octets": 20})], + ) + def test_partial_port_frame_counters_keep_dataplane_packets(self): result = ET.fromstring( "ethernet1/110" diff --git a/tests/test_paloalto_api_dashboard.py b/tests/test_paloalto_api_dashboard.py index 376475c..19e006a 100644 --- a/tests/test_paloalto_api_dashboard.py +++ b/tests/test_paloalto_api_dashboard.py @@ -55,6 +55,28 @@ def test_operational_sections_are_collapsible(self): self.assertEqual(panel["repeatDirection"], "h") self.assertEqual(panel["gridPos"]["w"], 24) + def test_offload_estimate_compares_port_and_dataplane_counters(self): + for dashboard in (self.dashboard, self.chassis_dashboard): + panel = next( + panel for panel in dashboard["panels"] + if panel["title"] == "Throughput Received: Dataplane vs Offloaded" + ) + query = panel["targets"][0]["query"] + # Both counters of the same physical ports, and only ports that + # report both, so a missing port block never reads as zero offload. + self.assertIn('r._field == "dp_in_octets"', query) + self.assertIn("exists r.in_octets and exists r.dp_in_octets", query) + # Burst-refreshed port counters are smoothed over at least 2 minutes + # and small differences are floored to zero. + self.assertIn("int(v: v.windowPeriod) > int(v: 2m)", query) + self.assertIn("r.total * 0.05", query) + # A late port-counter refresh reads negative then positive; the + # smaller of two consecutive windows drops it, steady offload stays. + self.assertIn('difference(columns: ["_value"], keepFirst: true)', query) + self.assertEqual(panel["fieldConfig"]["defaults"]["custom"]["stacking"]["mode"], "normal") + total = next(item for item in panel["fieldConfig"]["overrides"] if item["matcher"]["options"] == "Total") + self.assertIn({"id": "custom.stacking", "value": {"mode": "none", "group": "A"}}, total["properties"]) + def test_active_physical_interfaces_get_repeated_api_panels(self): rows = { panel["title"]: panel