Skip to content

[Bug]: GPU vendor detection hardcodes AMD priority via /dev/kfd, never falls back to NVIDIA #4085

Description

@crqs

Steps to reproduce

Hello

I am trying to setup dstack on a fleet of machines that have both an AMD iGPU (Ryzen 9 9800X3D) and an NVIDIA dGPU (RTX 5090) and I've encounterd difficulties having dstack detect my NVIDIA card.

I am running dstack 0.20.29 on Ubuntu 24.04.

Steps to reproduce

  • Add a host with both an AMD iGPU and an NVIDIA dGPU to an SSH fleet
  • Check dstack fleet: the GPU resource is reported using the AMD device

Result of dstack fleet

NAME           NODES  GPU                                              SPOT  BACKEND       PRICE  STATUS  CREATED
pilot-fleet    1      gpu=AMD Ryzen 9 9950X3D 16-Core Processor:2GB:1  -     ssh (remote)  -      idle    ...

and host_info.json

{"gpu_vendor":"amd","gpu_name":"AMD Ryzen 9 9950X3D 16-Core Processor","gpu_memory":2048,"gpu_count":1,...}

After unloading amdgpu and restarting the shim, dstack fleet correctly showed gpu=RTX5090:32GB:1.

Actual behaviour

Only the AMD iGPU gets detected. The current GetGpuVendor logic goes through vendors in a fixed order, beginning with AMD: it checks for /dev/kfd, which exists as soon as the amdgpu module is loaded and never checks NVIDIA if that check succeeds.

I managed to get around it by unloading the amdgpu module. With /dev/kfd gone, detection correctly falls through to the NVIDIA check. I guess disabling the iGPU in BIOS would also work, since the module would never load in the first place.

Expected behaviour

Possible solutions include checking for NVIDIA first (since it's less probable anyone would need to detect an AMD card if there is an nvidia one) or adding an optional gpu_vendor field in the fleet yaml definition to force a desired vendor.

dstack version

0.20.29

Server logs

Additional information

Happy to provide more logs or test a fix if useful.

Thanks for your help!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions