Skip to content

fix: align DRA driver DaemonSet/DeviceClass with AutoPartition (k8s-gpu-dra-driver#88) #648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
spraveenio merged 6 commits into from
Sep 23, 2026
Merged

fix: align DRA driver DaemonSet/DeviceClass with AutoPartition (k8s-gpu-dra-driver#88) #648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
eb7604f
fix: align DRA driver DaemonSet/DeviceClass with AutoPartition
yansun1996 Sep 22, 2026
45e40b4
chore: regenerate all generated artifacts, fix pensando registry leak
yansun1996 Sep 22, 2026
942d77e
chore: restore AMD copyright headers stripped by mockgen/controller-g…
yansun1996 Sep 22, 2026
9079cbe
fix: align OpenShift's handleDeviceClass with the AutoPartition Devic…
yansun1996 Sep 22, 2026
1a5004e
fix: grant dra-driver RBAC to delete NodeModulesConfig for KMM-manage…
yansun1996 Sep 23, 2026
095839d
chore: propagate dra-driver NodeModulesConfig RBAC into the OLM bundl…
yansun1996 Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 9 additions & 2 deletions bundle/manifests/amd-gpu-operator.clusterserviceversion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ metadata:
]
capabilities: Seamless Upgrades
categories: AI/Machine Learning,Monitoring
containerImage: registry.test.pensando.io:5000/amd-gpu-operator:dev
createdAt: "2026-07-09T09:40:15Z"
containerImage: docker.io/rocm/amd-gpu-operator:dev
createdAt: "2026-09-23T02:56:40Z"
description: |-
Operator responsible for deploying AMD GPU kernel drivers, device plugin, device test runner and device metrics exporter
For more information, visit [documentation](https://instinct.docs.amd.com/projects/gpu-operator/en/latest/)
Expand Down Expand Up @@ -1476,6 +1476,13 @@ spec:
- nodes
verbs:
- get
- apiGroups:
- kmm.sigs.x-k8s.io
resources:
- nodemodulesconfigs
verbs:
- get
- delete
serviceAccountName: amd-gpu-operator-dra-driver
- rules:
- apiGroups:
Expand Down
28 changes: 17 additions & 11 deletions bundle/manifests/amd.com_deviceconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -261,9 +262,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -354,9 +356,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -457,9 +460,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -719,9 +723,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -1438,7 +1443,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This is an alpha field and requires enabling the
This field depends on the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -1528,9 +1533,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -1672,9 +1678,8 @@ spec:
node.
type: string
timeAdded:
description: |-
TimeAdded represents the time at which the taint was added.
It is only written for NoExecute taints.
description: TimeAdded represents the time at which the
taint was added.
format: date-time
type: string
value:
Expand Down Expand Up @@ -1819,9 +1824,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down
28 changes: 17 additions & 11 deletions config/crd/bases/amd.com_deviceconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,9 +143,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -257,9 +258,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -350,9 +352,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -453,9 +456,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -715,9 +719,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -1434,7 +1439,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This is an alpha field and requires enabling the
This field depends on the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -1524,9 +1529,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down Expand Up @@ -1668,9 +1674,8 @@ spec:
node.
type: string
timeAdded:
description: |-
TimeAdded represents the time at which the taint was added.
It is only written for NoExecute taints.
description: TimeAdded represents the time at which the
taint was added.
format: date-time
type: string
value:
Expand Down Expand Up @@ -1815,9 +1820,10 @@ spec:
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators).
type: string
tolerationSeconds:
description: |-
Expand Down
7 changes: 7 additions & 0 deletions config/rbac/dra_driver_cluster_role.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,10 @@ rules:
- nodes
verbs:
- get
- apiGroups:
- kmm.sigs.x-k8s.io
resources:
- nodemodulesconfigs
verbs:
- get
- delete
11 changes: 11 additions & 0 deletions hack/k8s-patch/metadata-patch/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,17 @@ draDriver:
deviceClass:
# -- Create the gpu.amd.com DeviceClass resource. Set to false if managing the DRA driver independently.
create: true
# -- Maps classic extended-resource requests (e.g. amd.com/gpu) to a DeviceClass.
# Set "" to omit the field. When autoPartition is true, this is applied to a
# second class, gpu.amd.com-spx, constrained to whole-GPU (spx) devices only, so
# classic requests can't resolve to a leftover partition fraction. When
# autoPartition is false, it stays on the single gpu.amd.com class as before.
extendedResourceName: amd.com/gpu
# -- Set to true when DeviceConfig.spec.draDriver.cmdLineArguments enables the
# DRA driver's AutoPartition feature gate on the matching nodes. Must be kept in
# sync with that feature-gates value; it only controls this chart's DeviceClass
# rendering and cannot detect the DeviceConfig's setting automatically.
autoPartition: false
kmmModuleLoader:
serviceAccount:
annotations: {}
Expand Down
32 changes: 32 additions & 0 deletions hack/k8s-patch/template-patch/dra-driver-deviceclass.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,36 @@ spec:
selectors:
- cel:
expression: "device.driver == 'gpu.amd.com'"
{{- if and .Values.draDriver.deviceClass.extendedResourceName (not .Values.draDriver.deviceClass.autoPartition) }}
extendedResourceName: {{ .Values.draDriver.deviceClass.extendedResourceName }}
{{- end }}
{{- if .Values.draDriver.deviceClass.autoPartition }}
---
# When AutoPartition is enabled on the DRA driver, gpu.amd.com's device pool
# includes fractional partition devices (cpx/dpx/qpx) alongside whole GPUs (spx),
# all matched by the same selector above. extendedResourceName moves to this
# second, spx-constrained class instead of staying on the general one, so a
# classic amd.com/gpu request (which carries no selector of its own -- it's
# translated by the apiserver from resources.limits, not authored as a
# ResourceClaim) can only ever resolve to a whole GPU, never a partition
# fraction left over from another claim.
apiVersion: {{ $draApi }}
kind: DeviceClass
metadata:
name: gpu.amd.com-spx
labels:
app.kubernetes.io/component: amd-gpu
app.kubernetes.io/part-of: amd-gpu
{{- include "helm-charts-k8s.labels" . | nindent 4 }}
spec:
selectors:
- cel:
expression: >-
device.driver == 'gpu.amd.com' &&
(!has(device.attributes["gpu.amd.com"].computePartition) ||
device.attributes["gpu.amd.com"].computePartition == "spx")
{{- with .Values.draDriver.deviceClass.extendedResourceName }}
extendedResourceName: {{ . }}
{{- end }}
{{- end }}
{{- end }}
7 changes: 7 additions & 0 deletions hack/k8s-patch/template-patch/dra-driver-rbac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ rules:
- update
- patch
- delete
- apiGroups:
- kmm.sigs.x-k8s.io
resources:
- nodemodulesconfigs
verbs:
- get
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
Expand Down
2 changes: 1 addition & 1 deletion helm-charts-k8s/Chart.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ dependencies:
repository: file://./charts/remediation-crds
version: v1.0.0
digest: sha256:0806f6b6d7aa21be77bf1c91e720ae3238338a16f107df450a53b02ef940db1b
generated: "2026-07-09T09:40:11.647230235Z"
generated: "2026-09-23T02:56:24.120766399Z"
4 changes: 3 additions & 1 deletion helm-charts-k8s/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ For the most detailed and up-to-date documentation please visit our Instinct Doc

## Introduction

AMD GPU Operator simplifies the deployment and management of AMD Instinct GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications.
AMD GPU Operator simplifies the deployment and management of AMD Instinct™ and AMD Radeon™ GPU accelerators within Kubernetes clusters. This project enables seamless configuration and operation of GPU-accelerated workloads, including machine learning, Generative AI, and other GPU-intensive applications.

## Components

Expand Down Expand Up @@ -280,7 +280,9 @@ Kubernetes: `>= 1.29.0-0`
| deviceConfig.spec.testRunner.tolerations | list | `[]` | test runner tolerations |
| deviceConfig.spec.testRunner.upgradePolicy.maxUnavailable | int | `1` | the maximum number of Pods that can be unavailable during the update process |
| deviceConfig.spec.testRunner.upgradePolicy.upgradeStrategy | string | `"RollingUpdate"` | the type of daemonset upgrade, RollingUpdate or OnDelete |
| draDriver.deviceClass.autoPartition | bool | `false` | Set to true when DeviceConfig.spec.draDriver.cmdLineArguments enables the DRA driver's AutoPartition feature gate on the matching nodes. Must be kept in sync with that feature-gates value; it only controls this chart's DeviceClass rendering and cannot detect the DeviceConfig's setting automatically. |
| draDriver.deviceClass.create | bool | `true` | Create the gpu.amd.com DeviceClass resource. Set to false if managing the DRA driver independently. |
| draDriver.deviceClass.extendedResourceName | string | `"amd.com/gpu"` | Maps classic extended-resource requests (e.g. amd.com/gpu) to a DeviceClass. Set "" to omit the field. When autoPartition is true, this is applied to a second class, gpu.amd.com-spx, constrained to whole-GPU (spx) devices only, so classic requests can't resolve to a leftover partition fraction. When autoPartition is false, it stays on the single gpu.amd.com class as before. |
| installdefaultNFDRule | bool | `true` | Default NFD rule will detect amd gpu based on pci vendor ID |
| kmm.enabled | bool | `true` | Set to true/false to enable/disable the installation of kernel module management (KMM) operator subchart |
| kmm.watch | bool | `true` | Set to true/false to enable/disable GPU operator watching and using KMM resources |
Expand Down
Loading
Loading