[tbortolossi]

Summary

Lessons from two real cases, genericized: a PA-3250 HA pair whose active member silently stopped transmitting with links up (11.1.6), and a PA-VM high-DP-CPU case (11.1.10).

SKILL.md

• New section "DP CPU - onset and attribution":
  • date the onset against HA state changes;
  • VM-Series sizing: the licence caps the cores;
  • why a VM runs pktmmap instead of DPDK, per boot (pan_vm_plugin.log; pair NIC twins by MAC; pktmmapvmconfig in sysd.log is not proof);
  • compute-bound vs buffer-bound (PBP and ingress-backlog state in sdb.txt);
  • per-process 94-100 % for pktproc in dp-monitor is busy-polling on PA-3200, not load: prove it with the passive peer and quote resource-monitor;
  • in HA, an error on the suspect node is a cause only if the peer lacks it.
• panio blocks also embed show session info and the global counters with a per-second rate.
• PA-3250 on 11.1 behaves like the PA-1400: bcm_g_cntr_stats broken (Python SyntaxError), dpc_nica_stats "NICA FPGA not available".

TSF-GUIDE.md

• debug dataplane pow performance row.
• Per-problem row "active node black-holes traffic, links up, no failover":
  • the passive peer's pkt_recv tracks the active's ha_msg_sent;
  • known noise to compare between peers (DSSD, FAP counter read, cumulative session_install_error).
• Row "SFP model / optical levels / PAUSE / MAC errors": sdb.txt sys.s1.pN.phy|detail|cfg|stats, with brdagent insertion history.
• Row "where frames are lost": CPU vs MAC counters of show counter interface all on both ends of HA2:
  • 32-bit CPU counters unwrapped, cross-checked with 64-bit bytes plus 4 B FCS;
  • MAC rx - CPU rx = receive errors;
  • matched against show high-availability state-synchronization.

Evidence

Platform	PAN-OS	What was checked	Result
PA-3250	11.1.6	dp-monitor --- processes on a node passive for 4 days vs show running resource-monitor	all_pktproc_* 94-100 % in all 2076 snapshots while resource-monitor read 1-3 %
PA-3250	11.1.6	dp-monitor --- panio blocks	global counters embedded as :name value rate; bcm_g_cntr_stats SyntaxError; dpc_nica_stats NICA FPGA not available
PA-3250	11.1.6	show counter interface all on both HA peers, HA2 port, two TSFs one day apart	MAC rx - CPU rx = receive errors exactly; CPU tx 2743547 vs MAC tx 0 during a failback; frame loss matched state-sync sent-received within 1 %
PA-3250	11.1.6	tmp/cli/logs/sdb.txt sys.s1.pN.phy / .detail / .cfg / .stats	SFP vendor-name, part number, serial, DOM rx/tx power; snmpDot3InPauseFrames, FCS, E2EHOLDropPkts present
PA-3250	11.1.6	ha_agent.log, pan_dssd.log, brdagent.log per day on both HA peers	DSSD host node not available 48/day, load configuration failed and FAP counter read failed identical on both
PA-VM	11.1.10	pan_vm_plugin.log, brdagent.log, sdb.txt across two boots	per-boot DPDK/pktmmap decision; NIC driver twins paired by MAC
PA-VM	11.1.10	show system info vm-license and vm-cpu-count vs resource-monitor	only licensed DP cores print a column
PA-VM	11.1.10	sdb.txt packet-buffer-protection, pbp-zones, cfg.session.ingress_backlogs_*	PBP and ingress-backlog state present; no ingress-backlogs output in the TSF
PA-VM	11.1.10	debug dataplane pow performance	per-function total-us splits ctd, flow and zip time

Changes

File	Change	Lines
plugins/read-tsf/skills/read-tsf/SKILL.md	modify
plugins/read-tsf/skills/read-tsf/TSF-GUIDE.md	modify

Checks run before sending

• Customer-data patterns — addresses outside the documentation ranges, e-mail, serials, MAC, config values, secrets, domain names — on 127 new line(s) and the text above: no hit.
• 6 identifier(s) of the analyzed TSF (hostname, serial, domain, addresses): no hit.
• Personal denylist: 18 entries, no hit.

Received by e-mail from T. Bortolossi, made against read-tsf 1.2.0; applied with tools/intake.sh.