[Shivkumar13]

Description

During migration to an NVIDIADriver with upgradePolicy.autoUpgrade: false, node labeling requests an upgrade for orphaned driver pods while the upgrade controller removes that request. Check the owning driver's effective upgrade policy in the migration flow before setting upgrade-required, keeping driver ownership independent of upgrade eligibility.

Emit a ManualDriverPodReplacementRequired Warning event on the NVIDIADriver, with the orphaned pod as the related object and the node/pod names in the message. Enabled and defaulted policies retain automatic migration; disabled policies leave pod replacement to the administrator.

Fixes #3063. This is a separate alternative to #3082 with manual-action feedback and repeated-reconcile coverage.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets unaffected (no API, CRD, RBAC, Helm, or generated-file changes)
• Go mod artifacts unaffected (no dependency or vendor changes)
• Test cases are added for new code paths

Testing

• Regression fails on the unmodified implementation and passes with the fix.
• make fmt, make license-check, make unit-test (23 packages, including build), and make lint pass on Linux/amd64 with Go 1.26.8. Lint reports zero issues.
• Unit coverage includes ClusterPolicy and GPUCluster routing, disabled/enabled/defaulted policies in mixed pools, three alternating node-label/upgrade reconciles, preservation and cleanup of existing upgrade states, manual events, and resuming migration when auto-upgrade is enabled.
• Live validation on Kubernetes 1.35 (API server v1.35.10, kubelets v1.35.9), using the normal ClusterPolicy path without DRA. A harmless pause pod with the driver component label was orphaned by deleting its DaemonSet. The actual controllers were run against the real API with deterministic manual scheduling: baseline main @ 7622af8 added/removed upgrade-required in each of three cycles; the fixed build kept the label absent and recorded the real Warning event. The orphan pod kept its UID throughout.
• Cluster configuration and controlled node metadata were restored and verified against snapshots. GPU capacity and all 31 existing GPU operand pod UIDs were preserved; all nodes were Ready and ClusterPolicy returned to ready.

The live test isolates this label-policy bug. It does not exercise containerized-driver installation/replacement or controller watch delivery; the BCM host driver remained loaded.

---

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[coderabbitai]

Walkthrough

The node-labeling controller now resolves the live NVIDIADriver owner for an orphaned driver pod. If automatic upgrades are disabled, it emits a warning and skips migration. Tests cover disabled, enabled, and default policies, warning events, preserved labels and pods, and upgrade-state changes.

Severity of issue fixed: Medium

Merge Risk: Minimal · up to ab12a

Policy changes trigger migration reconciliation. The warning-event test should check its object references; no merge-blocking runtime risk was identified.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[coderabbitai]

Follow-up Comments

Assert the warning event’s object references.

111-116: Trivial · Maintainability & Code Quality · Quick win

Assert the warning event’s object references.

These assertions pass if Eventf receives nil instead of &pod. With Verbose disabled, events.FakeRecorder omits both object references from its event string. Capture the Eventf arguments and assert that regarding is the NVIDIADriver and related is the orphan pod. (raw.githubusercontent.com)

As per path instructions: “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Open the source location

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA/gpu-operator
• Branch: main
• File: .coderabbit.yaml
• Commit: 7622af8
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 2fdcf5e8-8050-4fce-a859-b3d5225b2794

Commits

Reviewing files that changed from the base of the PR and between 7622af8 and ab12ab5.

Files selected for processing: 3

• controllers/nodelabeling_controller.go
• controllers/nodelabeling_controller_test.go
• controllers/nodelabeling_migration_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.