You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] Windows selects CUDA llama.cpp runtime when nvidia-smi.exe exists but no NVIDIA GPU is available #386
Modly appears to select the CUDA build of its bundled llama.cpp runtime when nvidia-smi.exe exists on the system's PATH, even if the executable cannot communicate with an NVIDIA driver and no NVIDIA GPU is available.
My system uses an AMD Radeon RX 9070 XT. Windows correctly detects the card, and its PCI device ID is 7550, corresponding to gfx1201.
However, Modly's bundled runtime contains ggml-cuda.dll and CPU backend DLLs, but no ggml-vulkan.dll or AMD HIP backend. Consequently, llama-server.exe cannot detect a usable GPU.
Suspected root cause
In api/services/llm_server.py, has_nvidia_gpu() uses:
def has_nvidia_gpu() -> bool:
return shutil.which("nvidia-smi") is not None
This checks whether the executable exists, rather than verifying that a working NVIDIA GPU is available.
The _pick_assets() function then selects the Windows CUDA build when this check returns True. If _driver_cuda_version() cannot determine the driver version, the selection logic can fall back to the oldest available CUDA build instead of selecting Vulkan.
Thanks for the precise report — your diagnosis was right: has_nvidia_gpu() only checked that nvidia-smi existed, so a leftover nvidia-smi.exe was enough to get the CUDA build.
Modly now runs nvidia-smi -L and only picks CUDA when it succeeds and lists a GPU. Otherwise it installs the Vulkan build, which works on your RX 9070 XT.
Since your machine already has the CUDA build, there is a new Settings › Agent › Engine › Repair button: it re-detects the GPU and replaces the engine with the right build.
Until the release ships, you can get the same result manually: close Modly, delete %USERPROFILE%\.modly\llm\bin, restart and click Install engine — with the fix in place it will pick Vulkan. (On the current release it would pick CUDA again as long as nvidia-smi.exe is on your PATH, so either wait for the update or remove that leftover nvidia-smi.exe first.)
Feel free to reopen if llama-server still doesn't see your GPU after the update.
Activity
Lorchie commentedon Oct 26, 2026
Thanks for the precise report — your diagnosis was right:
has_nvidia_gpu()only checked thatnvidia-smiexisted, so a leftovernvidia-smi.exewas enough to get the CUDA build.Fix in #388:
nvidia-smi -Land only picks CUDA when it succeeds and lists a GPU. Otherwise it installs the Vulkan build, which works on your RX 9070 XT.Until the release ships, you can get the same result manually: close Modly, delete
%USERPROFILE%\.modly\llm\bin, restart and click Install engine — with the fix in place it will pick Vulkan. (On the current release it would pick CUDA again as long asnvidia-smi.exeis on your PATH, so either wait for the update or remove that leftovernvidia-smi.exefirst.)Feel free to reopen if llama-server still doesn't see your GPU after the update.
lightningpixel commentedon Oct 28, 2026
PR validated
fix(llm): pick CUDA only when an NVIDIA GPU actually answers, add eng…