Repository navigation
How to make NVIDIA MIG Manager restart custom (pre-installed) device plugin daemonset during MIG reconfiguration? #1704
Description
Hello,
I have a pre-installed NVIDIA device plugin running as a daemonset in my Kubernetes cluster, and now I'm trying to perform some MIG operations.
Current Behavior:
When MIG configuration changes, MIG Manager automatically restarts GPU operator components
MIG Manager also restarts system services defined in default-gpu-clients ConfigMap
However, my pre-installed device plugin daemonset doesn't get restarted
What I've Tried:
Added labels to match GPU operator:
labels:
app: nvidia-device-plugin
nvidia.com/device-plugin: "true"
Question:
How can I make MIG Manager restart my custom device plugin daemonset when MIG configuration changes?
Activity
[-]How to make NVIDIA MIG Manager restart custom device plugin daemonset during MIG reconfiguration?[/-][+]How to make NVIDIA MIG Manager restart custom (pre-installed) device plugin daemonset during MIG reconfiguration?[/+]cdesiniotis commentedon Nov 4, 2025
@shachijun we don't currently have a solution for this. It may make sense for mig-manager to use a common label, i.e.
nvidia.com/management-container, to identify which pods are GPU management pods (like device-plugins, monitoring agents, etc) and therefore need to be restarted across MIG reconfigurations.shivamerla commentedon Nov 14, 2025
@shachijun one workaround would be to deploy the device-plugin helm chart using the node-selector as
nvidia.com/gpu.deploy.device-plugin: true.shachijun commentedon Nov 14, 2025
Thank you so much @cdesiniotis and @shivamerla for letting me know!
francisguillier commentedon Feb 3, 2026
@shachijun
does the workaround mentioned by @shivamerla work in your environment?
cdesiniotis commentedon Nov 4, 2026
We have introduced a generic
"nvidia.com/gpu.deploy.client": "true"label to designate pods that are GPU clients and therefore need to be restarted across driver upgrades and MIG reconfigurations. Refer to #2607 and NVIDIA/mig-parted#426. This feature was introduced in GPU Operator 26.7.0 / MIG Manager 0.15.0. Closing this issue as fixed.