Skip to content

[Bug]: ClusterPolicy precompiled DaemonSets collapse kernels that differ only by arch #3124

Open

Description

@SebTardif

Describe the bug

With driver.usePrecompiled=true on a ClusterPolicy, precompiledDriverDaemonsets walks every raw feature.node.kubernetes.io/kernel-version.full. The DaemonSet name is built with getSanitizedKernelVersion, which strips x86_64, aarch64, and _64k. The node selector and the image tag keep the raw kernel.

Two kernels that differ only by architecture or page size therefore share one name. DaemonSet updates that object, and the last loop iteration wins. The other kernel's nodes get no driver. Go map order is random, so which kernel wins can change on the next reconcile.

This is the ClusterPolicy driver path. driver.nvidiaDriverCRD.enabled defaults to false. Pull request #3099 changes internal/state/nodepool.go only.

To Reproduce

The inputs are the ones TestGetSanitizedKernelVersion already treats as equal:

  • 5.14.0-427.37.1.el9_4.x86_64
  • 5.14.0-427.37.1.el9_4.aarch64

Those strings are the shape of a RHEL uname -r, which is what Node Feature Discovery stores in feature.node.kubernetes.io/kernel-version.full.

On current main, one reconcile creates nvidia-driver-daemonset-5.14.0-427.37.1.el9.4-ubuntu22.04 for the first kernel and then updates that same object for the second kernel. The log line is DaemonSet is different, updating, and one DaemonSet remains. Its node selector and image are the second kernel.

The regression test is TestPrecompiledDriverDaemonSetsStayDistinctWhenKernelsSanitizeTogether. It drives ClusterPolicyController.step() with usePrecompiled: true and those two node labels. On main it fails with 1 DaemonSet instead of 2.

This is a controller test, not a live cluster capture.

Expected behavior

One DaemonSet per raw kernel. Each object keeps that kernel in its node selector and in the precompiled image tag. Kernels that do not sanitize together, such as 5.4.0-generic and 5.4.135-generic, keep the names they have today.

Environment (please provide the following information):

  • GPU Operator Version: main (controller unit test)
  • OS: any OS whose NFD kernel label includes the architecture, for example RHEL 5.14.0-427.37.1.el9_4.x86_64
  • Kernel Version: two kernels that differ only by .x86_64 / .aarch64 or _64k
  • Container Runtime Version: n/a
  • Kubernetes Distro and Version: n/a

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions