Steps to reproduce
Hello
I am trying to setup dstack on a fleet of machines that have both an AMD iGPU (Ryzen 9 9800X3D) and an NVIDIA dGPU (RTX 5090) and I've encounterd difficulties having dstack detect my NVIDIA card.
I am running dstack 0.20.29 on Ubuntu 24.04.
Steps to reproduce
- Add a host with both an AMD iGPU and an NVIDIA dGPU to an SSH fleet
- Check dstack fleet: the GPU resource is reported using the AMD device
Result of dstack fleet
NAME NODES GPU SPOT BACKEND PRICE STATUS CREATED
pilot-fleet 1 gpu=AMD Ryzen 9 9950X3D 16-Core Processor:2GB:1 - ssh (remote) - idle ...
and host_info.json
{"gpu_vendor":"amd","gpu_name":"AMD Ryzen 9 9950X3D 16-Core Processor","gpu_memory":2048,"gpu_count":1,...}
After unloading amdgpu and restarting the shim, dstack fleet correctly showed gpu=RTX5090:32GB:1.
Actual behaviour
Only the AMD iGPU gets detected. The current GetGpuVendor logic goes through vendors in a fixed order, beginning with AMD: it checks for /dev/kfd, which exists as soon as the amdgpu module is loaded and never checks NVIDIA if that check succeeds.
I managed to get around it by unloading the amdgpu module. With /dev/kfd gone, detection correctly falls through to the NVIDIA check. I guess disabling the iGPU in BIOS would also work, since the module would never load in the first place.
Expected behaviour
Possible solutions include checking for NVIDIA first (since it's less probable anyone would need to detect an AMD card if there is an nvidia one) or adding an optional gpu_vendor field in the fleet yaml definition to force a desired vendor.
dstack version
0.20.29
Server logs
Additional information
Happy to provide more logs or test a fix if useful.
Thanks for your help!
Steps to reproduce
Hello
I am trying to setup dstack on a fleet of machines that have both an AMD iGPU (Ryzen 9 9800X3D) and an NVIDIA dGPU (RTX 5090) and I've encounterd difficulties having dstack detect my NVIDIA card.
I am running dstack 0.20.29 on Ubuntu 24.04.
Steps to reproduce
Result of
dstack fleetand
host_info.jsonAfter unloading
amdgpuand restarting the shim,dstack fleetcorrectly showedgpu=RTX5090:32GB:1.Actual behaviour
Only the AMD iGPU gets detected. The current GetGpuVendor logic goes through vendors in a fixed order, beginning with AMD: it checks for /dev/kfd, which exists as soon as the amdgpu module is loaded and never checks NVIDIA if that check succeeds.
I managed to get around it by unloading the amdgpu module. With /dev/kfd gone, detection correctly falls through to the NVIDIA check. I guess disabling the iGPU in BIOS would also work, since the module would never load in the first place.
Expected behaviour
Possible solutions include checking for NVIDIA first (since it's less probable anyone would need to detect an AMD card if there is an nvidia one) or adding an optional
gpu_vendorfield in the fleet yaml definition to force a desired vendor.dstack version
0.20.29
Server logs
Additional information
Happy to provide more logs or test a fix if useful.
Thanks for your help!