Skip to content

[Bug] Windows selects CUDA llama.cpp runtime when nvidia-smi.exe exists but no NVIDIA GPU is available #386

Closed

Description

@Neiker45

Describe the bug

Modly appears to select the CUDA build of its bundled llama.cpp runtime when nvidia-smi.exe exists on the system's PATH, even if the executable cannot communicate with an NVIDIA driver and no NVIDIA GPU is available.

My system uses an AMD Radeon RX 9070 XT. Windows correctly detects the card, and its PCI device ID is 7550, corresponding to gfx1201.

However, Modly's bundled runtime contains ggml-cuda.dll and CPU backend DLLs, but no ggml-vulkan.dll or AMD HIP backend. Consequently, llama-server.exe cannot detect a usable GPU.

Suspected root cause

In api/services/llm_server.py, has_nvidia_gpu() uses:

def has_nvidia_gpu() -> bool:
return shutil.which("nvidia-smi") is not None

This checks whether the executable exists, rather than verifying that a working NVIDIA GPU is available.

The _pick_assets() function then selects the Windows CUDA build when this check returns True. If _driver_cuda_version() cannot determine the driver version, the selection logic can fall back to the oldest available CUDA build instead of selecting Vulkan.

Relevant source:
https://github.com/lightningpixel/modly/blob/f11a4d2/api/services/llm_server.py#L244-L299

Steps to reproduce

Run Modly on Windows with an AMD GPU and no functioning NVIDIA GPU/driver.

Have an nvidia-smi.exe executable available on PATH.

Configure Modly's local LLM engine and load a model.

Observe which llama.cpp runtime is installed and inspect the available devices.

Modly version

0.4.3

Operating system

Windows 10

Screenshots & logs

Running the bundled executable:

& '...\models\agent\bin\llama-server.exe' --list-devices

Produces:

E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
Available devices:

The bundled directory contains ggml-cuda.dll and CPU backend DLLs, but no Vulkan or AMD HIP backend.

The NVIDIA utility exists at C:\Windows\System32\nvidia-smi.exe, but running it returns:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Confirmation

  • I have searched existing issues and this has not been reported yet.
  • I can reproduce this on the latest version of Modly.

Activity

  1. Lorchie commented on Oct 26, 2026

    @Lorchie
    Collaborator

    Thanks for the precise report — your diagnosis was right: has_nvidia_gpu() only checked that nvidia-smi existed, so a leftover nvidia-smi.exe was enough to get the CUDA build.

    Fix in #388:

    • Modly now runs nvidia-smi -L and only picks CUDA when it succeeds and lists a GPU. Otherwise it installs the Vulkan build, which works on your RX 9070 XT.
    • Since your machine already has the CUDA build, there is a new Settings › Agent › Engine › Repair button: it re-detects the GPU and replaces the engine with the right build.

    Until the release ships, you can get the same result manually: close Modly, delete %USERPROFILE%\.modly\llm\bin, restart and click Install engine — with the fix in place it will pick Vulkan. (On the current release it would pick CUDA again as long as nvidia-smi.exe is on your PATH, so either wait for the update or remove that leftover nvidia-smi.exe first.)

    Feel free to reopen if llama-server still doesn't see your GPU after the update.

  2. lightningpixel commented on Oct 28, 2026

    @lightningpixel
    Owner

    PR validated

  3. added a commit that references this issue on Nov 1, 2026
    6bae496
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions