Skip to content

[Bug]: Orphaned driver pod migration sets upgrade-required on NVIDIADriver pools with autoUpgrade disabled #3063

Description

@CloneOfAlex

Describe the bug
labelNodesWithOrphanedDriverPods (controllers/nodelabeling_controller.go:581) sets the upgrade state label to upgrade-required (line 625) on any node owned by a live NVIDIADriver that runs an ownerless driver pod. It does not check that NVIDIADriver's upgradePolicy.autoUpgrade.

For an NVIDIADriver with autoUpgrade: false, reconcileNVIDIADriverUpgrades removes the same label (controllers/upgrade_controller.go:262). Each later node-labeling reconcile adds it back, so the label flaps and the orphaned pod is never replaced.

To Reproduce
Found by code review; not yet reproduced. Expected: migrate from a ClusterPolicy-managed driver to an NVIDIADriver with upgradePolicy.autoUpgrade: false, leaving the old driver pods orphaned; watch the node's upgrade state label across reconciles.

Expected behavior
Skip nodes whose NVIDIADriver has auto-upgrade disabled, and report (event/condition) that the orphaned pod needs manual replacement.

Environment (please provide the following information):

  • GPU Operator Version: main @ f806d6b (code inspection)

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Oct 13, 2026
  2. Shivkumar13 commented on Nov 3, 2026

    @Shivkumar13
    Collaborator

    I have opened a PR to fix this issue: #3164

    PTAL @tariq1890 @kvalliyurnatt

  3. CloneOfAlex commented on Nov 5, 2026

    @CloneOfAlex
    Author

    Thanks @Shivkumar13. Heads-up that there are now two PRs linked to this issue: #3082 (@pujitha24) and #3164. It may be worth picking one so review effort isn't split. I'm happy to test whichever goes forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions