add support for default nvidiadriver and migration from clusterpolicy to nvidiadriver by rahulait · Pull Request #2518 · NVIDIA/gpu-operator

rahulait · 2026-06-04T19:24:59Z

Fixes:

Summary

This PR adds support for a default NVIDIADriver CR and enables controlled migration from legacy ClusterPolicy driver management to NVIDIADriver-based driver management.

When driver.nvidiaDriverCRD.enabled=true and driver.nvidiaDriverCRD.deployDefaultCR=true, Helm renders a default NVIDIADriver CR. This CR is identified by spec.default=true:

spec:
  default: true

The CR name is not semantically important. Helm renders it as default, but users can create an arbitrarily named NVIDIADriver and mark it as the fallback driver by setting spec.default=true.

Behavior

Non-default NVIDIADriver CRs own nodes selected by their spec.nodeSelector.
The default NVIDIADriver acts as fallback for GPU nodes that do not match any non-default driver.
If no NVIDIADriver has spec.default=true, fallback ownership is skipped.
If more than one NVIDIADriver has spec.default=true, reconciliation fails closed.
A default NVIDIADriver cannot define spec.nodeSelector; the fallback always applies to remaining GPU nodes that do not match non-default drivers.
If spec.default is changed to false, that CR becomes a normal user-defined NVIDIADriver and participates in nodeSelector conflict detection.
User-defined driver conflicts do not relabel nodes to the default driver or clear existing owner labels.
NVIDIADriver node selectors cannot include the operator-managed owner label nvidia.com/gpu-operator.driver.owner.
In NVIDIADriver mode, NVIDIADriver ownership conflicts mark ClusterPolicy as notReady and surface the conflict in its status conditions.

Migration Support

This PR supports migration from ClusterPolicy driver management to NVIDIADriver driver management by:

rendering a default NVIDIADriver from Helm values when CRD mode is enabled
preserving legacy ClusterPolicy driver fields when driver.nvidiaDriverCRD.enabled=false
routing GPU nodes to NVIDIADriver ownership through the operator-managed node label nvidia.com/gpu.driver.owner
cleaning up ClusterPolicy-owned driver daemonsets after NVIDIADriver ownership takes over
preserving running driver pods during migration by orphaning old ClusterPolicy driver daemonsets before NVIDIADriver rolls replacement pods

Migration requires the target GPU nodes to be matched by at least one NVIDIADriver CR. If driver.nvidiaDriverCRD.enabled=true but no matching NVIDIADriver exists, the controller has no driver owner to assign to those nodes, so fallback ownership is skipped until a matching CR is created.

Upgrade Example

When upgrading from a release that does not contain this migration support, migration from ClusterPolicy driver management to NVIDIADriver driver management must be performed in two upgrades. This applies only to clusters that are currently using ClusterPolicy for driver management and are switching to NVIDIADriver. It applies to releases older than v26.7.0.

Clusters that are already using NVIDIADriver in an older release do not need this two-phase migration flow.

Starting with v26.7.0, the migration support is already present in the running controller. Clusters already running v26.7.0 or newer can switch to NVIDIADriver mode directly by upgrading or updating configuration with driver.nvidiaDriverCRD.enabled=true, as long as the target GPU nodes are matched by NVIDIADriver CRs. The recommended path is to also set driver.nvidiaDriverCRD.deployDefaultCR=true so Helm renders the default fallback CR. Alternatively, users can create their own NVIDIADriver CRs and mark one as the default fallback with spec.default=true. See the Behavior section for details on default and non-default ownership.

Do not jump directly from an old release to the new release with driver.nvidiaDriverCRD.enabled=true. The old controller does not know about the controlled migration flow. If the old controller exits while the new release is already configured for NVIDIADriver mode, it can tear down all existing driver pods immediately before the new controller has a chance to take ownership.

Use this sequence instead:

Upgrade to the new release with ClusterPolicy driver management still enabled.

Example Helm settings:
```
driver:
  enabled: true
  nvidiaDriverCRD:
    enabled: false
    deployDefaultCR: false
```
This gets the new controller logic running while the existing ClusterPolicy-owned driver pods remain in place.
Upgrade again to the same new release, this time enabling NVIDIADriver mode.

Example Helm settings:
```
driver:
  enabled: true
  nvidiaDriverCRD:
    enabled: true
    deployDefaultCR: true
```
This renders the default NVIDIADriver and performs the controlled migration from ClusterPolicy ownership to NVIDIADriver ownership.

Issues Found And Fixed

During testing, we found several edge cases:

Treating the literal name default as special was too restrictive and could conflict with user-created CRs. The default driver is now identified by spec.default=true instead.
If multiple default drivers existed, reconciliation could behave ambiguously. The controller now fails closed when multiple defaults are found.
Allowing the default driver to carry a node selector made fallback behavior ambiguous. Default drivers now reject non-empty spec.nodeSelector.
On user-driver nodeSelector conflicts, nodes could be relabeled away from their existing owner, causing existing driver pods to disappear. Owner assignment now preflights conflicts and preserves existing owner labels on conflict.
User-supplied spec.nodeSelector could include nvidia.com/gpu-operator.driver.owner and override the operator-managed owner selector used by driver daemonsets. The controller, admission validation, and daemonset nodeSelector construction now reject or defensively preserve that invariant.
ClusterPolicy reconciliation previously attempted NVIDIADriver owner assignment even when NVIDIADriver mode was disabled. Owner assignment and orphaned-pod migration are now gated on driver.enabled=true and driver.nvidiaDriverCRD.enabled=true.
Moving default ownership is now a spec change through spec.default, so it bumps .metadata.generation and appears in the kubectl get nvidiadriver Default print column.
Conflict messages did not identify the other conflicting drivers. They now include the first conflicting node and the conflicting NVIDIADriver names.

E2E Coverage

The NVIDIADriver e2e flow now covers:

Helm renders a default NVIDIADriver only when driver.nvidiaDriverCRD.enabled=true and driver.nvidiaDriverCRD.deployDefaultCR=true.
Helm does not render a default CR when CRD mode is disabled.
Helm does not render a default CR when deployDefaultCR=false.
Operator installs in NVIDIADriver CRD mode.
Default driver can use an arbitrary name as long as it has spec.default=true.
User-defined NVIDIADriver can take ownership of GPU nodes.
Driver image/version update works through a user-defined NVIDIADriver.
Custom driver pod labels update through a user-defined NVIDIADriver.
Setting spec.default=false makes that CR a normal driver and conflict detection applies.
Conflicts preserve existing node owner labels.
Multiple default drivers are rejected or fail closed without disrupting existing ownership.
The ClusterPolicy e2e workflow now performs an in-place migration to NVIDIADriver mode before teardown. It verifies the legacy driver daemonset is removed, the existing legacy pod is orphaned, the default NVIDIADriver takes ownership, the orphaned pod is deleted by the upgrade flow within 5 minutes, and the replacement driver pod becomes ready.
Helm uninstall remains covered by the existing e2e teardown.

Checklist

No secrets, sensitive information, or unrelated changes
Lint checks passing (make lint)
Generated assets in-sync (make validate-generated-assets)
Go mod artifacts in-sync (make validate-modules)
Test cases are added for new code paths

Testing

Added unit-tests, e2e tests and also did manual testing on a 3 node cluster with multiple nvidiadriver CRs. Everything worked as expected even with migration.

copy-pr-bot · 2026-06-10T13:20:16Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

rajathagasthya

Overall looks solid, just a few questions.

cdesiniotis · 2026-06-10T22:27:23Z

+		if changed {
+			return ctrl.Result{RequeueAfter: time.Second}, nil
+		}
+	}


I am trying to reason about why we are doing this. Does the below statement (from the PR description) capture the reasoning?

ClusterPolicy status did not consistently move to notReady for early NVIDIADriver ownership failures. It now sets status.state to notReady and records the concrete conflict message in status conditions.

It feels wrong to call AssignOwners() here in the ClusterPolicy controller when it is already being called in the NVIDIADriver controller (and I would argue it should only be called there). If the goal is to mark ClusterPolicy as notReady when any NVIDIADriver CR is notReady could we not check the status of all the NVIDIADriver CRs in the cluster instead? And even then, I question whether we want the ClusterPolicy status to take the NVIDIADriver CR statuses into account.

@rahulait it looks like my comment only got added to L164. I meant to attach this to the entire code block.

These changes were for migration from clusterpolicy to nvidiadriver. I can see these are causing confusion, let me see if I can optimize them further and only let nvidiadriver add ownership labels.

Regarding status field of clusterpolicy: how should we model ClusterPolicy status when driver management is moved to NVIDIADriver?
With NVIDIADriver enabled, ownership of the driver operand moves out of ClusterPolicy, while the remaining operands are still managed by ClusterPolicy. During a node driver upgrade, the driver pod is replaced first, and then other operands may restart or go through init again.
Do we want ClusterPolicy status to reflect only the operands it still owns, which may cause it to become Ready during driver uninstall/replacement and then flip back to NotReady when dependent operands restart? Or should ClusterPolicy continue to represent the combined health of the overall GPU Operator stack, including NVIDIADriver-managed driver state?

As of today and even with this PR, the clusterpolicy state keeps fluctuating when driver is upgraded by nvidiadriver. I have filed a separate task to fix it in a separate followup PR.

tariq1890 · 2026-06-11T15:54:27Z

+		return false, fmt.Errorf("failed to list GPU nodes: %w", err)
+	}
+
+	for i := range nodes.Items {


The AssignOwners method will be called during every reconcile of a NVIDIADriver CR and each reconciler involves a loop through the entire list of gpu nodes. How will the performance be in large clusters ?

The loop is O(number of GPU nodes × number of nvidiadriver CRs), but the CPU cost is not the main issue. The main cost at large scale is listing/copying thousands of Node objects and patching changed nodes. In an already converged/synced cluster, patches are skipped, so impact should be modest, cost is mostly large cache reads. The worst case is convergence after rollout or selector changes where multiple nodes will be patched together, API writes and watch fan-out dominate. We can mitigate by reusing a single GPU-node list across validation/assignment and rate-limiting node patches per reconcile. I would consider it an enhancement we can do to ease load in case of large clusters.

Created a task for this so that we can do enhancements in future PRs.

Anything we can add to the instrumentation so that we can closely monitor this? Or are the current metrics we expose sufficient?

I think we should. controller-runtime metrics might tell us how much time a reconcile is taking, or if the queue is backing up, but it won't tell us how much time assignOwners is taking or any other call. We should at the very minimum atleast add logs which can be enabled to view time taken during each call. I'll add this requirement to the rate-limit request so that we do both at the same time. Let me know if you think we should add it to current PR and I'll add accordingly.

cdesiniotis

LGTM. Thanks @rahulait for the hard work and patience on this one!

tariq1890 · 2026-06-15T16:32:39Z

+		return false, fmt.Errorf("failed to list GPU nodes: %w", err)
+	}
+
+	for i := range nodes.Items {


Anything we can add to the instrumentation so that we can closely monitor this? Or are the current metrics we expose sufficient?

tariq1890 · 2026-06-15T21:08:17Z

+// included even for delete events so the NotFound reconcile path can clear
+// stale node owner labels.
+func (r *NVIDIADriverReconciler) enqueueNVIDIADriverReconcilers(ctx context.Context, driver *nvidiav1alpha1.NVIDIADriver) []reconcile.Request {
+	requests := r.enqueueAllNVIDIADrivers(ctx)


Instead of doing multiple list-alls of NVIDIADriver CR, how about creating a struct field (a map/set) in the NVIDIADriverReconciler struct which holds the unique set of all NVIDIADriver CRs? We'd add a new NVIDIADriver CR to the set if it's processed for the first time. We can remove the NVIDIADriver CR from this set when it is marked for deletion.

This will create a second source of truth beside the controller-runtime cache. We'll have to deal with startup correctness, delete correctness, concurrency, etc. The current r.List(ctx, list) is normally served from the informer cache, not a live API-server request, so it is much cheaper than it looks. It is also much safer because it reflects the cache’s current view after watch/list reconciliation. Are you seeing any performance issues with current approach? Just want to be clear on the issue you might be seeing with current approach.

While it may not be an issue in clusters with a few NVIDIADrivers. This becomes a problem in large clusters where it's also possible to have several NVIDIADriver CRs to loop through. In these scenarios, we are bound to hit scaling problems.

I think we should move the assignOwners() method to its own controller, that way it can reconcile on its own when needed and add ownership label. Then, we don't need to redo those calculations again and again on each reconcile in nvidiadriver CR and it can focus just on driver enablement. I'll add ownership-reconciler in a follow up PR which will improve things a lot in large clusters with lots of nvidiadrivers. I'll open follow up PRs for this.

@rahulait in addition to moving assignOwners() to its own controller, what are your thoughts on moving the other label operations, like labelNodesWithOrphanDriverPods(), to this separate controller? Would that be possible / desirable? I am also thinking about whether it makes sense to move all of the node labeling that GPU Operator does (primarily in the clusterpolicy controller) to a dedicated controller.

rahulait · 2026-06-16T03:34:52Z

/ok to test c7d6ba0

… to nvidiadriver changes include: * added default nvidiadriver which doesn't conflict with other user-specified nvidiadrivers * added migration support from clusterpolicy to nvidiadriver * added/updated e2e tests for nvidiadriver workflow * use new field to mark a nvidiadriver as default instead of using a label * don't allow adding nodeselector to default nvidiadriver, it should cover entire cluster * move default NVIDIADriver ownership helpers into internal/nvidiadriver and re-use * requeue after node owner label changes so later reconcile steps read updated cache state * use patch instead of update for node label updates Signed-off-by: Rahul Sharma <rahulsharm@nvidia.com>

rahulait · 2026-06-16T18:26:44Z

/ok to test 53770fc

tariq1890

Thank you @rahulait, excellent work on this! This is good to merge. Any other optimisations and improvements can be made in follow-up PRs

rajathagasthya

LGTM!

rahulait force-pushed the default-nvidiadriver branch 7 times, most recently from cf23d79 to 4adfc71 Compare June 5, 2026 17:12

rahulait marked this pull request as ready for review June 5, 2026 17:14

rahulait requested review from cdesiniotis, karthikvetrivel, rajathagasthya, shivamerla and tariq1890 as code owners June 5, 2026 17:14

rahulait force-pushed the default-nvidiadriver branch 2 times, most recently from 8125f47 to b21a20b Compare June 5, 2026 20:12

rahulait mentioned this pull request Jun 5, 2026

[WIP] : Adds support for migrating from clusterpolicy to nvidiadriver and vice-versa #2117

Closed

5 tasks

rahulait force-pushed the default-nvidiadriver branch from b21a20b to a1dea10 Compare June 10, 2026 13:20

rajathagasthya reviewed Jun 10, 2026

View reviewed changes

rahulait force-pushed the default-nvidiadriver branch 2 times, most recently from 2e803be to 452812c Compare June 10, 2026 22:02

cdesiniotis reviewed Jun 10, 2026

View reviewed changes

Comment thread internal/nvidiadriver/nvidiadriver.go Outdated

tariq1890 reviewed Jun 10, 2026

View reviewed changes

Comment thread controllers/clusterpolicy_controller.go Outdated

Comment thread controllers/object_controls.go Outdated

Comment thread internal/consts/consts.go Outdated

Comment thread internal/state/driver.go Outdated

Comment thread controllers/clusterpolicy_controller.go Outdated

rajathagasthya reviewed Jun 11, 2026

View reviewed changes

Comment thread controllers/nvidiadriver_controller.go Outdated

tariq1890 reviewed Jun 11, 2026

View reviewed changes

Comment thread tests/scripts/update-nvidiadriver.sh Outdated

rahulait force-pushed the default-nvidiadriver branch 3 times, most recently from 375e329 to 56f5a7d Compare June 11, 2026 23:18

rahulait force-pushed the default-nvidiadriver branch from 56f5a7d to 66c5d29 Compare June 12, 2026 03:55

cdesiniotis approved these changes Jun 15, 2026

View reviewed changes

tariq1890 reviewed Jun 15, 2026

View reviewed changes

rahulait force-pushed the default-nvidiadriver branch from 66c5d29 to cb154ca Compare June 15, 2026 18:49

tariq1890 reviewed Jun 15, 2026

View reviewed changes

rahulait force-pushed the default-nvidiadriver branch from cb154ca to 7e5c414 Compare June 15, 2026 22:19

tariq1890 reviewed Jun 16, 2026

View reviewed changes

rahulait force-pushed the default-nvidiadriver branch 2 times, most recently from 7eec877 to c7d6ba0 Compare June 16, 2026 03:24

tariq1890 reviewed Jun 16, 2026

View reviewed changes

Comment thread controllers/state_manager.go Outdated

rahulait force-pushed the default-nvidiadriver branch from c7d6ba0 to 53770fc Compare June 16, 2026 18:00

tariq1890 approved these changes Jun 16, 2026

View reviewed changes

rajathagasthya approved these changes Jun 16, 2026

View reviewed changes

rahulait merged commit 29ecb7d into NVIDIA:main Jun 16, 2026
19 checks passed

Conversation

rahulait commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes:

Summary

Behavior

Migration Support

Upgrade Example

Issues Found And Fixed

E2E Coverage

Checklist

Testing

Uh oh!

copy-pr-bot Bot commented Jun 10, 2026

Uh oh!

rajathagasthya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cdesiniotis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

rahulait commented Jun 4, 2026 •

edited

Loading

tariq1890 left a comment •

edited

Loading