You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: ClusterPolicy precompiled DaemonSets collapse kernels that differ only by arch #3124
With driver.usePrecompiled=true on a ClusterPolicy, precompiledDriverDaemonsets walks every raw feature.node.kubernetes.io/kernel-version.full. The DaemonSet name is built with getSanitizedKernelVersion, which strips x86_64, aarch64, and _64k. The node selector and the image tag keep the raw kernel.
Two kernels that differ only by architecture or page size therefore share one name. DaemonSet updates that object, and the last loop iteration wins. The other kernel's nodes get no driver. Go map order is random, so which kernel wins can change on the next reconcile.
This is the ClusterPolicy driver path. driver.nvidiaDriverCRD.enabled defaults to false. Pull request #3099 changes internal/state/nodepool.go only.
To Reproduce
The inputs are the ones TestGetSanitizedKernelVersion already treats as equal:
5.14.0-427.37.1.el9_4.x86_64
5.14.0-427.37.1.el9_4.aarch64
Those strings are the shape of a RHEL uname -r, which is what Node Feature Discovery stores in feature.node.kubernetes.io/kernel-version.full.
On current main, one reconcile creates nvidia-driver-daemonset-5.14.0-427.37.1.el9.4-ubuntu22.04 for the first kernel and then updates that same object for the second kernel. The log line is DaemonSet is different, updating, and one DaemonSet remains. Its node selector and image are the second kernel.
The regression test is TestPrecompiledDriverDaemonSetsStayDistinctWhenKernelsSanitizeTogether. It drives ClusterPolicyController.step() with usePrecompiled: true and those two node labels. On main it fails with 1 DaemonSet instead of 2.
This is a controller test, not a live cluster capture.
Expected behavior
One DaemonSet per raw kernel. Each object keeps that kernel in its node selector and in the precompiled image tag. Kernels that do not sanitize together, such as 5.4.0-generic and 5.4.135-generic, keep the names they have today.
Environment (please provide the following information):
GPU Operator Version: main (controller unit test)
OS: any OS whose NFD kernel label includes the architecture, for example RHEL 5.14.0-427.37.1.el9_4.x86_64
Kernel Version: two kernels that differ only by .x86_64 / .aarch64 or _64k
Activity