Skip to content

feat(diagnose): add system and v2-data-engine subcommands - #687

Open
derekbit wants to merge 1 commit into
longhorn:masterfrom
derekbit:memory-diag
Open

derekbit wants to merge 1 commit into
longhorn:masterfrom
derekbit:memory-diag

Conversation

@derekbit

Copy link
Copy Markdown
Member

Which issue(s) this PR fixes:

Issue longhorn/longhorn#14589

What this PR does / why we need it:

Special notes for your reviewer:

Additional documentation or context

@mergify

mergify Bot commented Nov 18, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue). If it does not match the queue conditions yet, it enters the queue once it does.

  • Queue this pull request

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.60961% with 672 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.71%. Comparing base (9bdf955) to head (576fb2a).

Files with missing lines Patch % Lines
pkg/local/spdk/layout.go 24.46% 281 Missing ⚠️
pkg/remote/spdk/getter.go 76.13% 142 Missing ⚠️
pkg/local/spdk/getter.go 0.00% 97 Missing ⚠️
pkg/local/spdk/mem.go 84.30% 62 Missing ⚠️
cmd/local/subcmd/diagnose.go 0.00% 41 Missing ⚠️
cmd/remote/subcmd/diagnose.go 53.73% 31 Missing ⚠️
pkg/local/spdk/rpc.go 88.32% 16 Missing ⚠️
cmd/local/longhornctl-local.go 0.00% 1 Missing ⚠️
cmd/local/subcmd/get.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master     #687       +/-   ##
===========================================
+ Coverage    7.59%   22.71%   +15.11%     
===========================================
  Files          57       63        +6     
  Lines        4462     6163     +1701     
===========================================
+ Hits          339     1400     +1061     
- Misses       4123     4763      +640     
Flag Coverage Δ
unittests 22.71% <60.60%> (+15.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mergify

mergify Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request is now in conflict. Could you fix it @derekbit? 🙏

Restructure `longhornctl diagnose` into a `<verb> <noun>` command group
consistent with other `longhornctl` commands:

- `longhornctl diagnose system`: moves the existing Kubernetes and
  Longhorn control-plane, CSI, instance-manager, and volume health
  checks under the `system` subcommand.
- `longhornctl diagnose v2-data-engine` (and `longhornctl-local diagnose
  v2-data-engine`): inspects live V2 Data Engine `spdk_tgt` processes
  across nodes via a temporary read-only DaemonSet without pausing the
  SPDK reactor.

`diagnose v2-data-engine` combines non-blocking SPDK JSON-RPC queries
with ELF/DWARF-guided `/proc/<pid>/mem` inspection to collect and
correlate:
- Global `iobuf` pool capacities, live DPDK `rte_ring` free counts,
  per-thread wait queues, and per-module lockless caches.
- Shared `blobstore` and per-blob `spdk_bs_channel` request-set pools
  (`freeReqs`, per-lvol `need_cluster_alloc` queues, and in-flight
  `cluster_cow` / `esnap_cow` operations).
- Queued `bdev` `nomem_io` requests (resolved to Longhorn lvol names)
  and active `bdev_no_mem_poller` counts.
- Cross-node NVMe-oF external snapshot read dependencies, transitive
  stall propagation, and distributed deadlock cycles.

Results support `-o yaml` (default), `-o json`, and `-o text`. YAML and
JSON outputs surface the synthesized cluster summary, deadlock cycles,
and cross-node dependencies ahead of the full raw per-node forensic
data, and the command exits with a non-zero status after DaemonSet
cleanup if any SPDK target is stalled or deadlocked.

longhorn/longhorn#14589

Signed-off-by: Derek Su <derek.su@suse.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant