fix(osdtrace): make the ops map an LRU hash so leaked entries can't stop tracing

[ray5273]

Summary

In full mode (the default), osdtrace can silently stop emitting events on a live OSD and never recover until it is restarted. The ops map is a plain 8192-entry hash, and several ordinary OSD code paths finish an op without reaching any of the completion probes that delete its entry. Once enough entries have leaked, bpf_map_update_elem() in uprobe_enqueue_op fails with -E2BIG for every new op. No op is tracked after that, every completion lookup misses, and the output goes quiet while the process still looks healthy.

This PR:

1. Plugs two remaining BPF-side leaks: uprobe_log_subop_stats and uprobe_repop_commit returned early on a failed length read after the lookup had succeeded, skipping the delete. This is the same pattern perf(osdtrace): submit ring buffer events with BPF_RB_NO_WAKEUP #188 fixed for the ring buffer reserve failure path.
2. Switches ops to BPF_MAP_TYPE_LRU_HASH, so a full map evicts entries instead of rejecting inserts. The OSD-side leaks listed below are not something the BPF program can see, so this keeps them from taking the tracer down.

---

Where the entries leak

uprobe_enqueue_op inserts every MSG_OSD_OP / MSG_OSD_REPOP. The entry is only deleted by log_op_stats, log_subop_stats, repop_commit or ec_submit_transaction. From the Ceph (squid) source, these paths finish an op without reaching any of them:

Path	Where
Failed reads (e.g. -ENOENT stat/read on a missing object)	PrimaryLogPG::complete_read_ctx only calls log_op_stats when result >= 0
Failed writes	execute_ctx → record_write_error() → close_op_ctx() → return
Early rejects	reply_op_error() call sites in do_op and friends (-EINVAL, -EPERM, -EBLOCKLISTED, -EAGAIN, ...)
Dup / resent ops answered from the pg log	do_op: already_complete(version) → reply_op_error(...)
Discarded ops	do_request: can_discard_request(op) → return

Client tids are monotonic, so the orphan handling added in uprobe_enqueue_op never sees these keys again. A workload with a steady stream of reads or stats on missing objects (e.g. RBD reads of image regions that were never written, which librbd turns into zeros) fills 8192 entries within seconds; in the measurements below it took under a second.

Why LRU, and what it costs

• A queue or stack map does not work here: completion probes need a keyed lookup by (pid, owner, tid), and ops do not complete in insertion order.
• In a BPF LRU hash, lookups set a reference bit and eviction prefers entries whose bit is not set. Ops still moving through the probes (dequeue_op, execute_ctx, mark_flag_point, ...) keep getting referenced. Leaked entries are looked up once right after insert and never again, so they drop to the inactive list and are evicted first.
• Trade-off: an op that stays untouched long enough while the map is under pressure can still be evicted and will be missing from the output. The typical case is an op still waiting in the op queue. Because of the per-CPU free lists (target_free = clamp(max_entries / ncpus / 2, 1, 128)), eviction can also begin before max_entries is reached, and if nothing unreferenced is left the kernel force-evicts regardless of the reference bit. In practice this was negligible in the measurements below (traced/sent stayed at 1.00 during the flood; the patched build counted 0 and 109 missed completion lookups out of ~1.1–1.5M canary ops).
• max_entries is unchanged (8192). The map is shared by every OSD traced by one osdtrace process.

---

Validation

Environment: single-OSD cephadm (podman) cluster, test pool size=1 / 32 PGs, RHEL 9.6 (kernel 5.14.0-570.12.1.el9_6), AMD EPYC 9334 32-Core (64 CPUs). Measured against two ceph-osd builds:

• A: the official Ceph 19.2.3 container image (c92aebb, el9 build 2:19.2.3-0.el9), osdtrace using its embedded DWARF data
• B: a custom Ceph 19.2.3-9-g8af27555182 image, DWARF imported with osdtrace -j / -i

Binaries: measured with the downstream osdtrace build this PR was extracted from (v1.7-based), unpatched (HASH) vs patched (LRU_HASH). The patched build additionally carries diagnostic counters (ops lookup miss / insert failure / ring buffer reserve failure) that are not part of this PR; its lookup-miss counter is quoted below. Scripts: ray5273/cephtrace@pr192-bench.

Reproducing the stall (full mode, osdtrace --id 0)

One process stats 64 existing objects in a tight loop (the canary, ~10–12k ops/s). After a 20 s baseline, 16 processes stat missing objects for 120 s (A: ~110k ops/s, B: ~73k ops/s), then tracing continues for another 40 s with only the canary running. "Traced / sent" is osdtrace op rows divided by canary ops sent in the same window (first 2 s of each phase skipped).

	A: main	A: this PR	B: main	B: this PR
baseline: traced / sent	0.995	0.994	0.993	0.995
baseline: ops entries	0–3	0–3	0–10	2–11
flood: traced / sent	0.000	1.001	0.000	1.002
after flood stopped: traced / sent	0.000	1.000	0.000	0.998
ops entries after the flood started	8192 (full) until exit	–	8192 (full) until exit	–
output stalled	~1 s after flood start, never recovered	no	~1 s after flood start, never recovered	no
completion lookups that missed (patched build counter, whole run)	–	0	–	109
canary ops sent / missing-object stats sent	1.45M / 13.2M	1.46M / 13.1M	1.10M / 8.7M	1.12M / 8.8M

• With the canary alone the map stays near empty on both builds, so the leak comes from the missing-object stats, not from normal traffic.
• On main the map is full within the first second of the flood and output stops for good, even after the flood ends while the canary keeps running.
• With this PR the canary is still fully traced during the flood (traced/sent 1.00). The absolute row rate drops during the flood because the single OSD is saturated (canary throughput drops by the same amount), not because ops are lost.
• ops entry counts for the LRU runs are not shown: sampled while ~70–110k ops/s churned through the map, the count even exceeded max_entries (9908), so it is not reliable.

Performance

rados bench 30 write -b 4096 -t 16 on the same pool, full mode, bpf_stats run_time / run_cnt summed over 2 rounds per variant (order: main, PR, main, PR; 30 s cooldown).

Program (ns/hit)	A: main	A: this PR	Δ	B: main	B: this PR	Δ
uprobe_enqueue_op	2345	2191	-6.6%	2346	2322	-1.0%
uprobe_dequeue_op	2859	2871	+0.4%	3177	3088	-2.8%
uprobe_execute_ctx	1385	1407	+1.6%	1454	1496	+2.9%
uprobe_log_op_stats	2976	3033	+1.9%	3290	3320	+0.9%
uprobe_mark_flag_point*	774	786	+1.6%	856	875	+2.2%
all osdtrace programs, per enqueued op	21697	21835	+0.6%	23628	23689	+0.3%

Client IOPS per round:

	no tracer	main r1 / r2	this PR r1 / r2
A	42372	35024 / 32573	30462 / 32167
B	29939	25431 / 23426	24303 / 23516

Total in-kernel BPF time per op is flat across rounds (A: 21692/21703 ns on main vs 21651/22008 ns with this PR; B: 23535/23729 vs 23770/23605). Client IOPS with this PR was lower in 3 of 4 paired rounds (−13.0%, −1.2%, −4.4%, +0.4%); the −13% round had the highest IOPS stddev of all cells (3000 vs 1093–2046) and the other three differences are within one stddev. Two rounds on a single saturated OSD are not enough to rule out a small throughput cost.

---

Notes / follow-ups

• The test cluster is a single OSD with a size=1 pool, so the replica paths (repop_commit, log_subop_stats, including the leak fix in the first commit) were not exercised.
• The underlying leaks are still there; probing the error-reply paths so those ops are completed (and reported) properly would be the real fix. That needs new probe points and DWARF data, so it is left out of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BMjwcq7yX2AQPvEn4LyyKX