Skip to content

osdtrace: print detailed per-op info in single mode - #186

Open
taodd wants to merge 1 commit into
mainfrom
osdtrace-single-mode-op-detail
Open

taodd wants to merge 1 commit into
mainfrom
osdtrace-single-mode-op-detail

Conversation

@taodd

@taodd taodd commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Summary

Single mode (-s) previously emitted only exit-time latency histograms. This PR extends the single PrimaryLogPG::log_op_stats probe (uprobe_log_op_stats_v2) so each op prints a detailed line — target pg id, object name, decoded osd op list (incl. cls calls), client id, tid, total latency — while keeping the histograms:

osd 0 pg 1.11 op_w size 4096 client 4446 tid 8 object benchmark_data_..._object7 osd_ops [set-alloc-hint,write] op_lat 1942
osd 0 pg 1.7  op_w size 0    client 4446 tid 55268 object benchmark_data_..._object27627 osd_ops [delete] op_lat 1730
osd 0 pg 1.1d op_r size 24   client 4446 tid 27640 object benchmark_last_metadata osd_ops [read] op_lat 220

No new probes are attached: everything is read at log_op_stats time from the message via new cast:MOSDOp varpaths, reusing the existing capture_decoded_osd_ops() machinery and rodata constants.

Changes

  • Varpath list: drop the unused inb/outb entries (read from PT_REGS_PARM3/4 by both program variants) and add pg (MOSDOp::pgid), object name (MOSDOp::hobj.oid.name) and MOSDOp::ops vector-bounds chains — filling the 10-varid budget exactly.
  • BPF: populate the new fields only after the -l threshold gate, so the in-kernel fast path is unchanged (~670 ns). New read_hprobe_varfield_opt degrades silently to zeroed fields when the varids are absent (DWARF JSON exported before this change) instead of dropping events or spamming bpf_printk.
  • Classification: read/write is now derived from decoded opcode mode bits — zero-payload delete and omap-only class methods are correctly labeled op_w — with wb>0 as the fallback for degraded DWARF data.
  • dwarf_parser: resolve_type_name (used by cast:) now strips typedef/cv wrappers and keeps scanning CUs for a complete definition. Needed because MOSDOp is now using MOSDOp = _mosdop::MOSDOp<std::vector<OSDOp>> — the class lives in a namespace the type cache does not descend into, so the first cached hit is a typedef that can lead to a declaration-only DIE.
  • Hardening: fill_map_hprobes (osdtrace + radostrace) errors out on >10 varpaths per function or >10 fields per varpath instead of silently corrupting adjacent varids / overflowing VarField_Kernel.fields.

Overhead (vstart, tentacle dev build, kernel 6.17, via bpftool prog stats)

path before after
-s per-op body ~2.4 µs ~3.0 µs (391 ms / 130k ops)
-s -l <ms> fast path ~670 ns unchanged (structurally untouched)
full mode (15 probes) ~18 µs/op unchanged

Test plan

  • rados bench 4K write + cleanup: op_w lines with [set-alloc-hint,write], [delete], correct pg/object/client/tid, op_lat consistent with bench-reported latency
  • seq read bench: op_r [read] lines
  • Ctrl-C still prints the size-range latency histograms
  • full-mode regression: 15 probes attach, per-op output identical to before (varids 90/91 shared with v1 are unchanged)
  • degraded-DWARF path covered by design: absent varids leave fields zeroed (pg 0.0, object -), events still emitted

🤖 Generated with Claude Code

https://claude.ai/code/session_01SQMVEJezhKCMbVMWFpNi5B

Single mode (-s) previously emitted only exit-time latency histograms.
Extend the single-probe uprobe_log_op_stats_v2 to capture and print one
line per op with the target pg id, object name, decoded osd op list
(including cls calls), client id, tid and total latency, while keeping
the histograms:

  osd 0 pg 1.11 op_w size 4096 client 4446 tid 8 \
    object benchmark_data_..._object7 \
    osd_ops [set-alloc-hint,write] op_lat 1942

- Rework the PrimaryLogPG::log_op_stats varpath list: drop the unused
  inb/outb entries (both programs read them from PT_REGS_PARM3/4) and
  add pg, object-name and MOSDOp::ops chains via cast:MOSDOp, filling
  the 10-varid budget exactly.
- Populate the new fields in uprobe_log_op_stats_v2 only after the -l
  threshold gate, keeping the in-kernel fast path unchanged (~670ns);
  the enriched body costs ~3.0us/op vs ~2.4us before (measured via
  bpftool prog stats on a vstart cluster).
- Reads of the new varids degrade silently to zeroed fields when
  tracing with DWARF JSON exported before this change
  (read_hprobe_varfield_opt), instead of dropping events or spamming
  bpf_printk.
- Classify read/write by decoded opcode mode bits, which labels
  zero-payload deletes and omap-only class methods correctly; fall
  back to wb>0 when detail ops are unavailable.
- Teach resolve_type_name to strip typedef/cv wrappers and keep
  scanning CUs for a complete definition: MOSDOp is now a using-alias
  for the templated _mosdop::MOSDOp<std::vector<OSDOp>> living in a
  namespace the type cache does not descend into, so the first cached
  hit may be a typedef leading to a declaration-only DIE.
- Harden fill_map_hprobes against varid budget/field capacity
  overflows in both osdtrace and radostrace.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant