Skip to content

osdtrace: add allocator tracing mode (-A) - #181

Open
taodd wants to merge 7 commits into
mainfrom
feat/alloc-tracing
Open

taodd wants to merge 7 commits into
mainfrom
feat/alloc-tracing

Conversation

@taodd

@taodd taodd commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

Adds osdtrace -A, an allocator tracing mode: one line per call into the BlueStore free-space allocator (Allocator::allocate() overrides — stupid/bitmap/avl/btree/btree2 and the hybrid wrappers, across the class-name variants used by different releases). BlueFS allocators go through the same method, so RocksDB/BlueFS file growth shows up next to object-data allocations.

osd 2 alloc block      want 0x1000 unit 0x1000 -> 0x1000 exts 1 [0xe1b1000~0x1000] lat 12us comm tp_osd_tp
osd 0 alloc bluefs-wal want 0x1200000 unit 0x100000 hint 0x16300000 -> 0x1200000 exts 1 [0x16300000~0x1200000] lat 9us comm bstore_kv_sync

Per call: allocator instance name, want / unit / hint, bytes actually allocated (SHORT, ENOSPC flagged), returned physical extents (first 6 — a split request is a direct fragmentation signal), latency (lock wait + free-space search) and calling thread. -l <ms> filters on latency; -A -b composes with the BlueStore latency probes.

Rebased onto current main (over #156/#163/#165/#175/#177): the probes now go through ATTACH_LIST (with a ret flag for the shared uretprobe) instead of the pre-refactor ad-hoc attach block.

Commits

  1. dwarf_parser: resolve inherited members; skip unresolvable probesfind_class_member() resolves declaration-only base-class DIEs through the type cache and adds the base's offset within the derived class to the member offset (an inherited member previously came back with a base-relative offset). handle_function() warns and drops the probes on that function instead of assert()ing when a varpath doesn't resolve on some build.
  2. osdtrace: add allocator tracing mode (-A) — entry uprobes stash the request per {thread, nesting depth}; one shared uretprobe computes latency and walks the returned PExtentVector with _M_start/_M_finish offsets from DWARF (the mempool allocator puts two pointers ahead of them, so they sit at 16/24, not 0/8). Nested calls (hybrid → its internal bitmap) are matched by depth and tagged (nested dN). The allocator name is read via a never-attached varpath on {AllocatorBase,Allocator}::get_name (its SocketHook is only fully defined in that CU) reused at every allocate() entry.
  3. osdtrace: look up BPF programs by name, not skeleton indexfunc_progid mapped functions to a position in the skeleton's progs array; adding a program anywhere but last silently shifted every later index (rebasing the allocator probes onto _txc_add_transaction did exactly that — every allocator id was off by one). Now a function → BPF program name map resolved with bpf_object__find_program_by_name(), plus a check at skeleton open that every mapped name exists.
  4. gitignore: ignore generated src/ceph_btf_local.h — Makefile-generated, removed by make clean, like embedded_dwarf_data.h.
  5. dwarf: regenerate Ubuntu 20.2.0 osdtrace references (amd64, amd64v3) — exported with osdtrace -j on hosts whose ceph-osd build-ids match the two references (the amd64v3 file is the one the regenerate workflow deliberately skips).
  6. dwarf: normalize workflow-written references to the tool's native format — 39 files the old workflow had re-serialized with indent=4, rewritten as the tools emit them (2-space); whitespace only, every file parses to the identical document. One-time consequence of ci: make regenerate-all-dwarf run end-to-end and commit tool output verbatim #180's "commit tool output verbatim".
  7. dwarf: regenerate osdtrace references with the allocator probes — output of a Regenerate all DWARF references dispatch on this branch (run 32146786374 — first full end-to-end run of the ci: make regenerate-all-dwarf run end-to-end and commit tool output verbatim #180 workflow: 52/52 jobs; 3 ~cloud0 jobs needed a rerun after a 4 h apt-mirror hang). All 43 osdtrace references now carry the allocate()/get_name entries present in each binary (5 on Octopus, 6 on Quincy/Reef, 9 on Squid+); the 20.2.0 amd64 file it produced is byte-identical to the one exported on the live host in commit 5. Radostrace references are unchanged and no existing value moved anywhere.

Compatibility

  • Default/-s/-b modes are unchanged; existing DWARF JSONs and embedded data keep working for them.
  • Every embedded reference now includes the allocator probes, so -A works out of the box; with an older user-supplied JSON (-i) it fails to attach with a hint (-A needs DWARF data that includes the allocator probes … regenerate with -j).

Verification (Ceph 20.2.0, Ubuntu 26.04 VM, live osd.2 with debug symbols)

  • osdtrace -j on the live OSD exports all 9 allocator functions present in that binary plus the get_name name donor; _M_start/_M_finish resolve at 16/24 (inherited-member path).
  • osdtrace -A -i <that json> under rados bench 4K writes: 16 probes attached (8 entry + 8 return), 10,747 alloc events for 10,739 writes, sequential 4K extents from block, latency p50 19 µs / p99 358 µs / max 3.7 ms; BlueFS calls appear from bstore_kv_sync.
  • -A -b -l 1: alloc lines all ≥ 1 ms (20) alongside 40 k bluestore lines; -A against the embedded reference prints the hint and exits.
  • Default mode regression: embedded DWARF path, 15 probes attached, 7,200 op rows, no warnings.
  • Final binary (regenerated references embedded): osdtrace -A without -i on the same OSD — embedded match, 16 probes attached, 11,729 alloc events for 11,721 writes.
  • make -j all warning-free, make test, tox -e test pass; clang-tidy runs in CI.

taodd added 7 commits August 18, 2026 22:37
find_class_member: a DW_TAG_inheritance in the probed CU usually points
at a declaration-only DIE of the base class, so the BFS searched an
empty DIE and gave up.  Resolve declaration bases through the type
cache and add the base's offset within the derived class to the member
offset (an inherited member previously got a base-relative offset).

handle_function: warn and drop the probes on that function instead of
assert()ing when one varpath fails to resolve on some build, so a single
unresolvable probe no longer aborts the whole tool.
Uprobe the concrete Allocator::allocate() overrides (stupid, bitmap,
avl, btree, btree2, hybrid variants across releases) and report, per
call: allocator instance name, want/unit/hint, bytes allocated (SHORT
and ENOSPC flagged), returned extents (first 6), latency and calling
thread.  BlueFS allocators go through the same method, so RocksDB file
growth shows up next to object-data allocations.

Entry probes stash the request per {thread, nesting depth}; one shared
uretprobe computes latency and walks the returned PExtentVector using
_M_start/_M_finish offsets from DWARF (the mempool allocator adds two
pointers ahead of them, so they are not at 0/8).  Nested calls (hybrid
falling back to bitmap) are matched by depth and tagged.  The allocator
name comes from a never-attached varpath on {AllocatorBase,Allocator}::
get_name, reused at every allocate() entry ("this" in the same
register); it needs the inherited-member support from the previous
commit.

Existing DWARF JSONs / embedded data predate these probes; -A then
fails to attach with a hint to regenerate (-j).
func_progid mapped each Ceph function to a position in the skeleton's
progs array, which is the order programs appear in the object.  Adding
a program anywhere but last silently shifts every later index and
attaches the wrong program (rebasing the allocator probes onto
_txc_add_transaction did exactly that).  Map to the BPF program name
instead and resolve it with bpf_object__find_program_by_name(); verify
all mapped names exist when the skeleton is opened.
The Makefile generates it from the running kernel's ceph.ko BTF and
'make clean' removes it, like the already-ignored embedded_dwarf_data.h.
Exported with osdtrace -j on the matching binaries so the references
carry the allocator probes; the diff is purely additive.
39 references had been re-serialized by the old regenerate workflow with
json.dumps(indent=4); osdtrace/radostrace -j write 2-space JSON and the
workflow now commits their output verbatim (#180).  Rewrite those files
in the tool's format so future regenerations diff only on content.
Whitespace only: every file parses to the identical document.
Output of the Regenerate all DWARF references workflow run on this
branch (plus the amd64v3 file exported on a matching host in the earlier
commit).  41 osdtrace references gain the allocate()/get_name entries
present in each binary; radostrace references are unchanged and no
existing value moved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant