feat(rocdbgapi): introduce clusters of workgroups by aktemur · Pull Request #7992 · ROCm/rocm-systems

aktemur · 2026-06-30T05:37:55Z

Motivation

Add a new level of hierarchy, "clusters", to the layering of wave -> workgroup -> cluster -> dispatch.

Technical Details

Define clusters as a new layer, similar to how workgroups are defined.

JIRA ID: AIROCGDB-77

Test Plan

To be done covered by GDB tests.

Test Result

Not yet available.

therock-pr-bot · 2026-06-30T05:38:44Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like `test_<name>.py` / `test_<name>.cpp` (or `<name>_test.`). Current:* code file(s) changed: `projects/rocdbgapi/src/architecture.cpp`, `projects/rocdbgapi/src/architecture.h`, `projects/rocdbgapi/src/cluster.cpp`, `projects/rocdbgapi/src/cluster.h`, `projects/rocdbgapi/src/dispatch.cpp` (+11 more); no test file found
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-06-30T05:38:45Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

lancesix

A first quick pass, this looks promising already, thanks!

In the commit msg for "rocdbgapi: create clusters of workgroups"

"We have to consider take the cluster id" either "consider" or "take" is too much.

aktemur · 2026-07-03T10:06:54Z

The update above has two major differences wrt to the previous revision:

Use first-of-cluster/last-of-cluster to determine if a wave is in a cluster.
Define an API to query cluster sizes (similar to workgroup sizes and grid sizes).

aktemur · 2026-07-07T13:37:31Z

I addressed some of Lancelot's comments. There are still a few opens that need to be clarified.

lancesix

(review for patch 3)

lancesix

(patch 4)

aktemur · 2026-07-16T12:23:17Z

The latest push is only a rebase.

aktemur · 2026-07-16T13:37:33Z

The update above addresses the comments received so far.

aktemur · 2026-07-31T10:33:24Z

Regarding AMD_DBGAPI_WORKGROUP_INFO_WORKGROUP_IN_CLUSTER_COORD:
The client (e.g. GDB) should be able to calculate the coordinate within the cluster based on cluster and workgroup sizes, which is available in another query. Would that be satisfactory?

I updated the GDB PR accordingly:
https://github.com/ROCm/ROCgdb/compare/2d1a5590c84d07c0fa53493c475cdc91d53e6af0..1c0dea6e74d0b586f3eafaabdcde8509e567ca37

lancesix · 2026-08-03T09:54:42Z

 private:
+  /* If the cluster to which this workgroup belongs has non-empty
+     coordinates, the group ids give the coordinates of this workgroup
+     within the cluster.  Otherwise it is the coordinates within the


Didn't this change?

We could also keep internal to dbgapi the workgroup in cluster, and only expose the workgroup in grid coordinates to the client.

Didn't this change?

Yes, sorry. I'll fix this, too.

We could also keep internal to dbgapi the workgroup in cluster, and only expose the workgroup in grid coordinates to the client.

In HIP, blockIdx still refers to the coordinates within the grid, regardless of being in cluster mode. Hence, it makes some sense to use grid-wise index, but I don't have a strong connection to this choice.

having blockIdx be grid-relative is probably the argument to go this way as well!

lancesix

I think we are almost there.

aktemur · 2026-08-07T20:48:32Z

@lancesix, the last force-push updates the CLUSTER_SIZES query. Please check.

aktemur · 2026-08-12T14:10:43Z

The force-push above introduces queries to get workgroup ids within the cluster.

In two places, we get a workgroup's ids by calling `group_ids ()`, which returns an optional tuple, and if the optional is non-empty, we call `group_ids ()` again. Avoid the duplicated call.

A workgroup is alive as long as it has a wave. Set the epoch mark unconditionally, whenever the mark of any wave of the workgroup is being set. In practice, this should not change the epoch value of a workgroup. This change is a refactoring to obtain clearer and simpler code.

Starting with gfx12-5, workgroups may belong to clusters. Add methods to read cluster ids from a CWSR record. The cluster and group ids are read from TTMP registers. The meaning of the registers depend on whether the wave is in cluster mode. If not in a cluster, group ids are obtained from TTMP7 and TTMP9 as in previous gfx12 architecture. If in a cluster, what we read from TTMP7 and TTMP9 are cluster ids. The group ids in this case are read from TTMP6. To figure out if the wave is in cluster mode, we keep track of the "first_of_cluster" and "last_of_cluster" bits in the compute relaunch wave register. All CWSR records created for waves between the first and the last of cluster (inclusive) are marked to be in a cluster. Bug: AIROCGDB-77

Add a first definition for clusters, which are groups of workgroups. Bug: AIROCGDB-77

Starting with gfx12-5, workgroups may be grouped into clusters. When creating workgroups, take this into account. Note that being part of a cluster is not mandatory. In that case, the cluster id would be std::nullopt. We nevertheless create clusters and associate workgroups to them, to avoid having to handle non-cluster mode separately. In cluster mode, a cluster id would be a valid 3D coordinate. In this case, a workgroup's group id is not necessarily unique. We have to take also the cluster id into account to identify workgroups uniquely. This patch adds clusters into the wave -> workgroup -> cluster -> dispatch hierarchy. API queries are added in the next patch. Bug: AIROCGDB-77

Define new enums in the API to query the cluster coordinate of a wave and a workgroup, and other cluster-related information. Bug: AIROCGDB-77

Add API query to obtain cluster sizes. To be in consistence with the existing queries that return workgroup sizes and grid sizes (i.e. AMD_DBGAPI_DISPATCH_INFO_WORKGROUP_SIZES and AMD_DBGAPI_DISPATCH_INFO_GRID_SIZES), the new query (i.e. AMD_DBGAPI_DISPATCH_INFO_CLUSTER_SIZES) returns the sizes in units of work-items. Bug: AIROCGDB-77

aktemur · 2026-08-17T11:51:01Z

The latest push above is only a rebase.

…uster Add a workgroup query and a wave query to get the workgroup coordinates within the cluster the workgroup belongs to.

aktemur · 2026-08-17T11:54:35Z

The push above removes two more "... in units of ..." sentences.

aktemur requested a review from a team as a code owner June 30, 2026 05:37

github-actions Bot added the project: rocdbgapi label Jun 30, 2026

therock-pr-bot Bot added the Not ready to Review label Jun 30, 2026

aktemur commented Jun 30, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

systems-assistant Bot added the organization: ROCm label Jun 30, 2026

aktemur changed the title ~~rocdbgapi: introduce clusters of workgroups~~ feat(rocdbgapi): introduce clusters of workgroups Jun 30, 2026

lancesix reviewed Jun 30, 2026

View reviewed changes

aktemur removed the Not ready to Review label Jun 30, 2026

aktemur force-pushed the users/aktemur/cluster branch from c6dff6d to 5c4c116 Compare July 3, 2026 10:02

therock-pr-bot Bot added the Not ready to Review label Jul 3, 2026

aktemur assigned lancesix Jul 3, 2026

lancesix reviewed Jul 7, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 5c4c116 to 6c615bc Compare July 7, 2026 13:29

aktemur force-pushed the users/aktemur/cluster branch 2 times, most recently from 8fd0911 to e9ea232 Compare July 16, 2026 07:14

aktemur requested a review from lancesix July 16, 2026 07:17

lancesix reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

lancesix reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/cluster.h Outdated

lumachad reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

Comment thread projects/rocdbgapi/src/architecture.cpp Outdated

Comment thread projects/rocdbgapi/src/process.cpp

Comment thread projects/rocdbgapi/src/queue.cpp Outdated

Comment thread projects/rocdbgapi/CHANGELOG.md Outdated

aktemur force-pushed the users/aktemur/cluster branch from e9ea232 to 5ccf7d9 Compare July 16, 2026 12:16

aktemur assigned aktemur and unassigned lancesix Jul 16, 2026

aktemur force-pushed the users/aktemur/cluster branch from 5ccf7d9 to d171615 Compare July 16, 2026 13:36

aktemur assigned lancesix Jul 16, 2026

aktemur mentioned this pull request Jul 31, 2026

refactor(rocdbgapi): set a workgroup's mark unconditionally #9428

Closed

lancesix reviewed Aug 3, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.h Outdated

lancesix reviewed Aug 3, 2026

View reviewed changes

lancesix assigned aktemur and unassigned lancesix Aug 3, 2026

aktemur force-pushed the users/aktemur/cluster branch from 972adda to 62cfcaf Compare August 3, 2026 11:08

aktemur assigned lancesix and unassigned aktemur Aug 3, 2026

lancesix reviewed Aug 5, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 62cfcaf to 60892d9 Compare August 7, 2026 20:46

aktemur force-pushed the users/aktemur/cluster branch from 60892d9 to 42d3a6e Compare August 12, 2026 14:09

lancesix reviewed Aug 17, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 42d3a6e to 18c37dd Compare August 17, 2026 11:19

aktemur added 8 commits August 17, 2026 06:50

doc(rocdbgapi): fix a documentation typo

08ede9b

refactor(rocdbgapi): avoid calling workgroup::group_ids () twice

d2df2a8

In two places, we get a workgroup's ids by calling `group_ids ()`, which returns an optional tuple, and if the optional is non-empty, we call `group_ids ()` again. Avoid the duplicated call.

feature(rocdbgapi): add cluster.h and cluster.cpp

3e5e389

Add a first definition for clusters, which are groups of workgroups. Bug: AIROCGDB-77

feature(rocdbgapi): add API definitions to query cluster information

88e1e25

Define new enums in the API to query the cluster coordinate of a wave and a workgroup, and other cluster-related information. Bug: AIROCGDB-77

aktemur force-pushed the users/aktemur/cluster branch from 18c37dd to 95fde49 Compare August 17, 2026 11:50

feature(rocdbgapi): add API definition to query workgroup coord in cl…

5a105b3

…uster Add a workgroup query and a wave query to get the workgroup coordinates within the cluster the workgroup belongs to.

aktemur force-pushed the users/aktemur/cluster branch from 95fde49 to 5a105b3 Compare August 17, 2026 11:54

aktemur mentioned this pull request Aug 17, 2026

feat(rocdbgapi): introduce clusters of workgroups (alternative) #10241

Open

Uh oh!

Conversation

aktemur commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Uh oh!

therock-pr-bot Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 7, 2026

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 16, 2026

Uh oh!

aktemur commented Jul 16, 2026

Uh oh!

aktemur commented Jul 31, 2026

Uh oh!

Uh oh!

Uh oh!

lancesix Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

aktemur Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

lancesix Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jun 30, 2026 •

edited

Loading

therock-pr-bot Bot commented Jun 30, 2026 •

edited

Loading

therock-pr-bot Bot commented Jun 30, 2026 •

edited

Loading