Vulkan: PTQ1_0 kernel hangs GPU (device lost, xe driver job timeout) on Intel Arc/Battlemage at longer prompts #192

Activity

1. mamama1 commented on Sep 18, 2026

   

   mamama1

   on Sep 18, 2026

   More actions

   This is from my personal documentation because I ran into what was the same issue, I think. If that's the case, then it is a xe driver issue and you need to recompile the xe module with increased timeouts because the max timeout the driver allows is not high enough. After I applied that fix, I NEVER hit that issue again:

   7. Raise the GPU job timeout (rebuild xe.ko)

   Without this, long attention kernels (>10 s at ~250k context) trigger Engine reset: engine_class=ccs → Timedout job → UR_RESULT_ERROR_DEVICE_LOST. job_timeout_max is capped at 10000 ms by CONFIG_DRM_XE_JOB_TIMEOUT_MAX.

   7.1 Source tree

   sudo dnf install rpm-build rpmdevtools dnf-plugins-core kernel-devel-$(uname -r)
   sudo dnf builddep kernel
   rpmdev-setuptree                      # as your normal user, not root
   dnf download --source kernel
   rpm -ivh kernel-$(uname -r | sed 's/\.x86_64//').src.rpm
   rpmbuild -bp ~/rpmbuild/SPECS/kernel.spec --target=x86_64
   
   export SRC=$(find ~/rpmbuild/BUILD -maxdepth 4 -name 'linux-*.fc*.x86_64' -type d | head -1)
   cd "$SRC"

   7.2 Configure

   cp /boot/config-$(uname -r) .config
   make olddefconfig
   ./scripts/config --set-val DRM_XE_JOB_TIMEOUT_MAX 60000
   grep DRM_XE_JOB_TIMEOUT .config          # must show 60000

   Do not run make olddefconfig again after --set-val.

   7.3 Fix version strings, then build

   cp /usr/src/kernels/$(uname -r)/Module.symvers .     # modules_prepare doesn't make this
   echo "-$(uname -r | cut -d- -f2-)" > localversion    # e.g. -200.fc44.x86_64
   rm -f include/generated/utsrelease.h
   make modules_prepare
   make -j$(nproc) M=drivers/gpu/drm/xe modules
   
   modinfo drivers/gpu/drm/xe/xe.ko | grep vermagic
   modinfo xe | grep vermagic

   Both vermagic strings must match exactly, including the preempt variant. If yours lacks the -200.fc44.x86_64 suffix, localversion was created after utsrelease.h — remove the header and rerun make modules_prepare.

   7.4 Install

   Fedora modules are xz with a small dictionary. Plain xz -9 uses a 64 MiB dictionary that the in-kernel decompressor rejects — the module silently fails to load with decompression failed with status 6.

   sudo cp -a /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz \
              /boot/xe.ko.xz.stock-$(uname -r)
   
   cp drivers/gpu/drm/xe/xe.ko /tmp/xe.ko
   strip --strip-debug /tmp/xe.ko                       # 164 MB -> ~10 MB
   xz -f --check=crc32 --lzma2=dict=1MiB -c /tmp/xe.ko > /tmp/xe.ko.xz
   
   sudo cp /tmp/xe.ko.xz /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz
   sudo depmod -a
   sudo dracut -f                                       # xe lives in the initramfs

   dracut -f is mandatory — the initramfs copy loads first and shadows /lib/modules otherwise.

   7.5 Verify

   sudo modprobe -r xe 2>/dev/null; sudo modprobe xe
   sudo dmesg | tail -20                                # no "decompression failed"
   sudo /usr/local/sbin/gpu-bar-resize.sh
   cat /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_max
   # 60000

   Rollback: sudo cp /boot/xe.ko.xz.stock-* /lib/modules/$(uname -r)/kernel/drivers/gpu/drm/xe/xe.ko.xz && sudo depmod -a && sudo dracut -f

   ---

   8. Apply the timeout at boot

   /etc/udev/rules.d/99-xe-job-timeout.rules:

   ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", DRIVERS=="xe", \
     RUN+="/bin/sh -c 'echo 60000 > /sys$devpath/device/tile0/gt0/engines/ccs/job_timeout_ms'"
   

   sudo udevadm control --reload
   sudo reboot

   ---

   9. Final verification

   cat /proc/cmdline
   lspci -s 25:00.0 -vv | grep -E 'Kernel driver|current size' 
   sudo dmesg | grep -i 'small bar'                     # must be empty
   cat /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_ms   # 60000
   journalctl -u gpu-bar-resize -b
   sycl-ls

   Note: the reported link (LnkSta: 2.5GT/s Width x1) on the GPU function is a normal Arc reporting quirk — the real link is on the upstream switch port (0000:23:00.0), which shows 8.0 GT/s x8.
2. flatlinebb commented on Sep 22, 2026

   

   flatlinebb

   on Sep 22, 2026

   Author

   More actions

   Tested the job_timeout_max/job_timeout_ms theory from the comment above against my original repro, without doing a full xe.ko rebuild — my host runs a custom Debian/Proxmox kernel (pve-kernel, currently 7.0.14-19-pve), so I wanted to rule this in/out with a cheap, reversible test first.

   Turns out the runtime knobs were already there: /sys/class/drm/card1/device/tile0/gt0/engines/ccs/job_timeout_ms was set to 5000ms, below the compiled job_timeout_max of 10000ms on this build. So I could test the whole hypothesis with a plain sysfs write, no kernel rebuild needed.

   • job_timeout_ms=5000 (original): repro fails cleanly at ~25s wall time — HTTP 500, decode() failed: vk::Queue::submit: ErrorDeviceLost. dmesg shows the expected Timedout job on ccs.
   • job_timeout_ms=10000 (raised to the compiled ceiling): still hits Timedout job in dmesg — same hang, just later, and this time it took the process down harder (an uncaught vk::DeviceLostError C++ exception killed llama-server outright, instead of the handled 500 I got at 5000ms).

   Doubling the timeout to the driver's max didn't let the job complete or even meaningfully change the outcome — it just moved the same reset a bit further out and made the failure mode worse. If this were purely "needs more wall-clock at this batch size," I'd expect raising the ceiling to actually help. Since it doesn't, I think this points to a real hang/bug in the PTQ1_0 Vulkan kernel itself on Intel/ANV (consistent with #185 — only ever validated on AMD RDNA3.5/RADV) rather than a timeout-configuration issue, at least on this GPU/driver combo. Not planning to pursue an xe.ko rebuild on this box on the strength of this result — seems like it would just delay the identical crash.

   Environment unchanged from the original report (Arc B580 / BMG, Mesa ANV, xe DRM driver, Debian/Proxmox host kernel now at 7.0.14-19-pve, same prism branch commit).
3. bri-prism commented on Sep 24, 2026

   

   bri-prism

   on Sep 24, 2026

   Collaborator

   More actions

   Thanks for the detailed repro and for testing the timeout theory. Raising the timeout not helping suggests a single submission genuinely takes too long rather than hitting a marginal limit. The PTQ1_0 Vulkan mat-vec on Xe2 GPUs, including the B580, now uses a much faster integer-dot kernel (#238), which is in the new release prism-b10735-842b188. Could you retry your ~1900-token prompt on that build? If it still hangs, a run with a smaller -ub (for example -ub 128) would tell us whether it's tied to batch size.
4. flatlinebb commented on Sep 25, 2026

   

   flatlinebb

   on Sep 25, 2026

   Author

   More actions

   Retested on prism-b10735-842b188 (built from the tag, same flags: -DGGML_VULKAN=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release), same Arc B580 / Mesa ANV / xe / 7.0.14-19-pve host, same model and server command as the original report. job_timeout_ms is back at the stock 5000.

   It's fixed on my end:

   Prompt tokens processed	Result	Prefill	Decode
   3,232	200 OK, 39.6s	94.8 tok/s	9.0 tok/s
   3,232 (cache_prompt: false, full re-prefill)	200 OK, 39.4s	101.3 tok/s	n/a
   8,752 total (~5.5k new, rest from prompt cache)	200 OK, 83.0s	76.2 tok/s	7.9 tok/s

   No Timedout job / devcoredump in dmesg for any of these, and the process stayed healthy throughout. I didn't need the -ub 128 run since nothing hung. For the same prompt, the previous build (1a07bfa) failed every time with ErrorDeviceLost.

   Thanks for the quick turnaround on #238. Fine to close from my side.