Skip to content

perf monitoring for CBT runs - #362

Open
harriscr wants to merge 2 commits into
ceph:mainfrom
harriscr:ch_wip_perf_monitoring
Open

harriscr wants to merge 2 commits into
ceph:mainfrom
harriscr:ch_wip_perf_monitoring

Conversation

@harriscr

Copy link
Copy Markdown
Contributor

Code to provide monitoring using the perf tool of CBT runs already exists in the codebase. It was only set up to monitor OSDs and had some requirements that meant it would only run on clusters that used vstart.
#357 split the top monitoring, which was in the same format, into 2 separate classes: One for generic top monitoring, and one specifically for OSDs using vstart (so as not to break the existing behaviour). We can apply the same division of responsibility to perf monitoring.

This also adds checking to monitoring, so that if a monitoring tool requested in the yaml is not installed then CBT will exit with an error. The exception is collectl, which has a default set of options. If collectl is not inslaeed then a warning os written and the CBT run will continue.

Additional unit tests were added for the new code where appropriate.

This has been tested by running on a local system:

ls cbt_runs/results/00000000/id-94a6db48/seq32kread.cephalasquad3.ssd.hursley.ibm.com/rbdfio/numjobs-001/iodepth-000192/
collectl       json_output.3  json_output.7        output.0_iops.1.log  output.1_bw.1.log    output.1_slat.1.log  output.2_iops.1.log  output.3_bw.1.log    output.3_slat.1.log  output.4_iops.1.log  output.5_bw.1.log    output.5_slat.1.log  output.6_iops.1.log  output.7_bw.1.log    output.7_slat.1.log
json_output.0  json_output.4  output.0             output.0_lat.1.log   output.1_clat.1.log  output.2             output.2_lat.1.log   output.3_clat.1.log  output.4             output.4_lat.1.log   output.5_clat.1.log  output.6             output.6_lat.1.log   output.7_clat.1.log  perf
json_output.1  json_output.5  output.0_bw.1.log    output.0_slat.1.log  output.1_iops.1.log  output.2_bw.1.log    output.2_slat.1.log  output.3_iops.1.log  output.4_bw.1.log    output.4_slat.1.log  output.5_iops.1.log  output.6_bw.1.log    output.6_slat.1.log  output.7_iops.1.log  top
json_output.2  json_output.6  output.0_clat.1.log  output.1             output.1_lat.1.log   output.2_clat.1.log  output.3             output.3_lat.1.log   output.4_clat.1.log  output.5             output.5_lat.1.log   output.6_clat.1.log  output.7             output.7_lat.1.log   visualisation

and

ls cbt_runs/results/00000000/id-94a6db48/seq32kread.cephalasquad3.ssd.hursley.ibm.com/rbdfio/numjobs-001/iodepth-000192/perf/
perf.data

Helped by IBM Bob 2.0.1

@harriscr
harriscr requested a review from lee-j-sanders August 25, 2026 16:16
@harriscr harriscr self-assigned this Aug 25, 2026
@harriscr
harriscr force-pushed the ch_wip_perf_monitoring branch from f5e50ab to 722666c Compare September 15, 2026 12:49
Chris Harris added 2 commits September 15, 2026 19:49
Fixes for perf and blktrace monitoring to align it with the new class structures in monitoring.

Helped by IBM Bob 2.0.1

Signed-off-by: Chris Harris <harriscr@uk.ibm.com>
Running top and perf monitoring o a remote system was a blocking process, meaning that the perf data was taken before any I/O was running.
Fix this so they are non-blocking and collect at the same tme as I/O runs. This is the same behaviour as collectl has today.

Ibm Bob 2.0.3 was used to help with this change

Signed-off-by: Chris Harris <harriscr@uk.ibm.com>
@harriscr
harriscr force-pushed the ch_wip_perf_monitoring branch from 722666c to 1c1e076 Compare September 15, 2026 18:49
@harriscr

Copy link
Copy Markdown
Contributor Author

Just rebased on the code in main and force-pushed.

All the unit tests still pass, as does a manual run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant