xpu-smi --query-gpu during Vulkan llama.cpp inference causes VK_ERROR_DEVICE_LOST on Arc B390 (Windows)

[mschuettlerTNG]

Hi!

While developing a feature to display GPU utilization in https://github.com/intel/AI-Playground/ we stumbled upon an issue where querying the information via xpu-smi during inference via llama.cpp (Vulkan / Windows) caused the device to crash. Below is an AI generated summary and reproducer. Let me know if you need more info or if this is too far outside the xpu-smi use-case.

Best,
Markus

Summary

Polling GPU metrics with xpu-smi --query-gpu=… while llama.cpp is running a Vulkan decode on the same Intel Arc GPU causes a Vulkan device lost. llama.cpp then fails with:

E ggml_vulkan: device lost on Vulkan0
E ggml_vulkan: getFenceStatus at ggml/src/ggml-vulkan/ggml-vulkan.cpp:2646
E srv  update_slots: decode() failed: vk::Device::getFenceStatus: ErrorDeviceLost

xpu-smi itself keeps running and prints plausible CSV. The inference process is the one that dies.

The same inference completes if xpu-smi is not running. Windows PDH GPU Engine counters (\GPU Engine(*)\Utilization Percentage) polled on the same cadence also do not trigger the loss — only the Level Zero xpu-smi query does.

No Windows System TDR (event 4101) is logged. The loss is visible to Vulkan only.

Environment

OS	Windows 11 Pro 10.0.26200
GPU	Intel Arc B390
Graphics driver	32.0.101.8974
xpu-smi	2.1.0.20250225 (8389eee7)
Level Zero	1.32.0
llama.cpp	win-vulkan-x64, 0.3.0-dev build 10666 (4e97ac86e)
Model	Qwen3.5-9B-Q4_K_M.gguf, --gpu-layers 999 --ctx-size 32000 -fa on --no-mmap --cache-ram 16384

How to reproduce

Two processes only: stock llama-server and a 2-second xpu-smi loop. No other app.

1. Start llama.cpp (Vulkan):

llama-server.exe --model Qwen3.5-9B-Q4_K_M.gguf --port 39110 --ctx-size 32000 --gpu-layers 999 --log-prefix --jinja --no-mmap -fa on --cache-ram 16384 --host 127.0.0.1

2. Wait until GET http://127.0.0.1:39110/health returns {"status":"ok"}.

3. In another terminal, poll every 2s (this is the query this xpu-smi 2.1 build accepts):

:loop
xpu-smi.exe --query-gpu=index,name,utilization.gpu,memory.used,memory.total,power.draw,clocks.current.graphics --format=csv,noheader,nounits
timeout /t 2 /nobreak >nul
goto loop

4. Send a generation long enough to overlap several polls (~12s / ~200 tokens is enough):

curl http://127.0.0.1:39110/v1/chat/completions -H "Content-Type: application/json" -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Write a complete single-file HTML and JavaScript tic-tac-toe game.\"}],\"max_tokens\":1024,\"temperature\":0.7}"

Expected: completion finishes (~19 t/s on this box).

Actual: around 160–190 generated tokens (~12s into decode) llama-server returns HTTP 500:

{"error":{"code":500,"message":"decode() failed: vk::Device::getFenceStatus: ErrorDeviceLost","type":"server_error"}}

Isolation

Same llama-server, same prompt, 2s poll interval:

Concurrent sampler	Result
none (control)	OK, 512 tokens @ ~19.6 t/s
Windows PDH \GPU Engine(*)\Utilization Percentage only	OK, 1024 tokens
xpu-smi --query-gpu=… on the first request after a fresh server	device-lost at ~160–190 tokens
xpu-smi --query-gpu=… on a later request	device-lost

While the query is running, xpu-smi reports the card busy (example: util ~14%, power ~24 W, clock 2500 MHz, memory.used ~35 GiB on this iGPU). After the Vulkan loss, later PDH engine rows drop to 0.

[savery42]

Hi @mschuettlerTNG thanks for the report!

This is reasonably within xpu-smi scope to debug. I'll take a look at reproduction later today / early next week.

[savery42]

This does seem to be caused by "metrics" (--query-gpu='metric_group.metric_name') evaluating all possible metrics always instead of lazily evaluating only the requested ones.

eu.* metrics goes through Intel's Observation Architecture (OA) h/w which will re-assign the OA context to xpu-smi and cause the llama-owned-context to be flushed.

I have been working on Linux-related changes to EU sampling, and seems like this is also a necessary candidate to fall-back to PDH on Windows if the h/w context is managed by another process.

I should be able to get a fix in for our next release in ~2 weeks. For the interim, running xpu-smi --query-gpu=... before launching llama should provide reasonable data for the requested metrics. I can provide a rc build for Windows next week, if needed, as I can't release it without the file being Signed and the credentials to do so are managed in a timezone that is already on their weekend.

[mschuettlerTNG]

Hi @savery42,
thanks for looking into it!

No worries, we are good with the regular release.

[savery42]

Hi @mschuettlerTNG

I should have thought of this beforehand, but the device_lost is primarily caused by metrics collection via powershell loop / re-launch of the process. Using [--loop (seconds) | --loop-ms (milliseconds)] directly with --query-gpu should fix the issue without any code changes.

xpu-smi.exe --query-gpu=index,name,utilization.gpu,memory.used,memory.total,power.draw,clocks.current.graphics --format=csv,noheader,nounits [ --loop 2 | --loop-ms 2000 ]

Realized that during internal validation runs for related code changes. With xpu-smi.exe -v 2.1.0 release and driver version 32.0.101.8993 I observed no failures with --loop-ms 250 over 100 runs, and 72 failures with the powershell loop set to 2 seconds.

So, there will be some related code changes but the main fix is using the internal loop method instead of the re-launch/shell loop method.