You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Orphaned driver pod migration sets upgrade-required on NVIDIADriver pools with autoUpgrade disabled #3063
Describe the bug labelNodesWithOrphanedDriverPods (controllers/nodelabeling_controller.go:581) sets the upgrade state label to upgrade-required (line 625) on any node owned by a live NVIDIADriver that runs an ownerless driver pod. It does not check that NVIDIADriver's upgradePolicy.autoUpgrade.
For an NVIDIADriver with autoUpgrade: false, reconcileNVIDIADriverUpgrades removes the same label (controllers/upgrade_controller.go:262). Each later node-labeling reconcile adds it back, so the label flaps and the orphaned pod is never replaced.
To Reproduce
Found by code review; not yet reproduced. Expected: migrate from a ClusterPolicy-managed driver to an NVIDIADriver with upgradePolicy.autoUpgrade: false, leaving the old driver pods orphaned; watch the node's upgrade state label across reconciles.
Expected behavior
Skip nodes whose NVIDIADriver has auto-upgrade disabled, and report (event/condition) that the orphaned pod needs manual replacement.
Environment (please provide the following information):
GPU Operator Version: main @ f806d6b (code inspection)
Thanks @Shivkumar13. Heads-up that there are now two PRs linked to this issue: #3082 (@pujitha24) and #3164. It may be worth picking one so review effort isn't split. I'm happy to test whichever goes forward.
Activity
Shivkumar13 commentedon Nov 3, 2026
I have opened a PR to fix this issue: #3164
PTAL @tariq1890 @kvalliyurnatt
CloneOfAlex commentedon Nov 5, 2026
Thanks @Shivkumar13. Heads-up that there are now two PRs linked to this issue: #3082 (@pujitha24) and #3164. It may be worth picking one so review effort isn't split. I'm happy to test whichever goes forward.