Post your Arc results

[Torchit1]

Running the arc-b580 branch? Post your card, driver, OS, context size and the numbers you get (llama-bench tg128/pp512, or the server's t/s). Results from other setups help most.

[lavdrimismajili]

SYCL Flash Attention on Arc 140T (Xe-LPG): oneDNN SDPA returns wrong results for quantized KV with permuted views

Summary

On Intel Arc 140T (Xe-LPG+, Arrow Lake-H), the SYCL backend's oneDNN SDPA path returns numerically incorrect results when the KV cache is quantized, the layout is non-contiguous (permute=[0,2,1,3]), and the tensor is a padded view (kv_view=1). The native TILE kernel handles the same shape correctly. This matches upstream issue ggml-org#27769.

Separately, MTP + ngram-mod speculative decoding on SYCL leaks device memory and crashes with OUT_OF_RESOURCES at first prompt. Both are backend defects, not configuration errors.

Hardware / Software

• GPU: Intel Arc 140T (12GB shared), Xe-LPG+
• Driver: Level Zero 1.15.39183+4, sycl-ls reports 12.74
• oneAPI: 2026.1.1 (DPC++ 20260101)
• oneDNN: 3.13.3 (built from source, pointed via DNNL_ROOT / DNNL_DIR)
• llama.cpp: ebf104ca2 (build 10767), PrismML / Mintplex fork with SYCL ternary kernels
• Backend: SYCL, GGML_SYCL_F16=ON, GGML_SYCL_DNN=ON
• Model: Ternary-Bonsai 2 27B PTQ1_0 (Qwen3.8-based, 1.75 bpw ternary, group 128)

test-backend-ops Results

With oneDNN and MKL FA enabled

OK   hsk=256,hsv=256,nh=2,nr23=[16,2],kv=1025,nb=1, q8_0, permute=[0,1,2,3]
FAIL hsk=256,hsv=256,nh=2,nr23=[16,1],kv=1025,nb=64, q8_0, permute=[0,2,1,3]
     ERR = 1.183620451 > 0.000500000
OK   hsk=256,hsv=256,nh=4,nr23=[6,1],kv=1025,nb=512, q4_0, permute=[0,1,2,3]
OK   hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1025,nb=64,  q4_0, permute=[0,1,2,3]
OK   hsk=256,hsv=256,nh=4,nr23=[6,1],kv=1025,nb=512, q8_0, permute=[0,1,2,3]
OK   hsk=128,hsv=128,nh=8,nr23=[4,1],kv=1025,nb=64,  q8_0, permute=[0,1,2,3]

5/6 tests passed

With oneDNN and MKL FA disabled

6/6 tests passed

Same binary, same model, same flags. Only difference is GGML_SYCL_FA_ONEDNN=0 and GGML_SYCL_ENABLE_MKL_FA=0.

Reproduction

set GGML_SYCL_ENABLE_VMM=0
set GGML_SYCL_FA_ONEDNN=1
set GGML_SYCL_ENABLE_MKL_FA=1
build-sycl\bin\test-backend-ops.exe test -o FLASH_ATTN_EXT -b SYCL0 -p "kv=1025,"

Expected: 5/6, one FAIL with ERR ≈ 1.2–1.5
With both disabled: 6/6 PASS

Observed Error Magnitudes Across Runs

oneDNN version	ERR
oneAPI bundled (3.11)	1.26
3.13.3 (source build)	1.33
3.13.3	1.28
3.13.3	1.48
3.13.3	1.18
3.13.3	1.23

Upgrading oneDNN from 3.11 to 3.13.3 did not fix the bug. Error stays in the 1.2–1.5 range, roughly 2500× the 0.0005 tolerance.

llama-bench Results

Scope: Ternary-Bonsai 2 27B PTQ1_0 only. These numbers apply to this specific model on this specific fork. They should not be extrapolated to other models or other builds.

KV	ONEDNN	MKL	pp512	tg128
f16	0	0	167.39 ± 0.89	7.90 ± 0.03
q8_0	0	0	170.46 ± 1.33	7.85 ± 0.03
q8_0	1	1	174.03 ± 0.06	7.88 ± 0.01
f16	1	1	212.10 ± 0.84	7.93 ± 0.04

The F16 + oneDNN row is the fastest correct configuration for this model. F16 removes the quantized-KV precondition, so the broken kernel path is not taken.

The q8_0 + oneDNN row is fast but produces wrong attention output per the test-backend-ops failure above. It should not be used for correctness-sensitive work.

Server Findings

MTP memory leak

llama-server with --spec-type draft-mtp,ngram-mod crashes at ggml-sycl.cpp:2853 with UR_RESULT_ERROR_OUT_OF_RESOURCES on the first task launch. This persists with GGML_SYCL_ENABLE_VMM=0. It is separate from the FA bug.

VMM

GGML_SYCL_ENABLE_VMM=0 is required to avoid an earlier OUT_OF_RESOURCES failure at graph reserve. This aligns with the VMM disable-by-default behavior in PR ggml-org#24673 and PR ggml-org#27689.

ngram-mod behavior

ngram-mod only fires when the model echoes a long contiguous block from the prompt. In one session:

ngram-mod: #calls(b,g,a) = 3  356  2
#gen drafts = 2
#acc drafts = 2
#gen tokens = 512
#acc tokens = 301
mean acc len = 151.50

Hit rate 2/356 calls. When it fires, it carries 300+ tokens in 2 drafts. When it does not, MTP alone carries generation at ~4–7 t/s.

ngram-mod window size

n_max	Behavior
256	Catches long verbatim copies; mean accepted length up to 36
48	Catches short targeted edits; mean accepted length up to 25, acceptance up to 89%

Neither wins universally. A middle value (128) may capture both patterns.

Conclusion

1. The oneDNN SDPA path is numerically broken on Xe-LPG for quantized KV + permute=[0,2,1,3] + kv_view=1. Matches issue SYCL: flash attention returns a silently wrong answer for a quantised, non-contiguous KV view (Arc 140T, Xe-LPG) ggml-org/llama.cpp#27769, not fixed by oneDNN 3.13.3.
2. F16 KV avoids the bug and is the fastest correct configuration for this model on this fork.
3. MTP + ngram-mod on SYCL has a separate memory leak causing OUT_OF_RESOURCES.
4. GGML_SYCL_ENABLE_VMM=0 is required to avoid an earlier allocation failure.

Generated with DeepSeek <3

[Torchit1]

Thanks for the detailed write-up, it was really useful. I reproduced the test-backend-ops failures on my B580 too: FLASH_ATTN_EXT with a quantized KV cache in the permuted layout fails from about 1K keys up, and it only passes with both oneDNN and MKL turned off. So it's not specific to the 140T.

I found the cause. The test builds its KV view with a gap between tokens (it allocates twice the heads and views half of them), and the oneDNN and MKL paths work out the token stride from the head count instead of reading it. llama.cpp's real cache has no gap, so actual models weren't affected. With Bonsai 27B, q4_0 KV and flash attention, perplexity at 4K context is the same on all three attention paths (oneDNN 8.727, MKL 8.731, the native TILE kernel 8.733), and needle-in-a-haystack recall is exact at 32K, 64K and 120K. I've added a guard so a layout with a gap falls back to the correct kernel. The full FLASH_ATTN_EXT suite now passes (4137/4137), plus a new sweep of the real cache layout from 1K to 8K keys. That fix will be in the next push. F16 KV is still a fine choice on the 140T if you have the memory for it.

On the MTP crash, I suspect memory rather than a leak. The draft context needs its own KV and compute buffers, and 12 GB of shared memory gets tight fast. Could you post your exact llama-server command? On the B580 I use LLAMA_ARG_SPEC_DRAFT_UBATCH=512 plus -ctkd q4_0 -ctvd q4_0 for the draft context, and on the 140T I'd start at -c 16384 and work up from there.

Also, the ternary XMX path only runs on Xe2 (B-series, Lunar Lake). On the 140T it switches itself off, so around 8 t/s is about what that memory bandwidth allows for a 27B model.

[lavdrimismajili]

Thanks for the detailed write-up, it was really useful. I reproduced the test-backend-ops failures on my B580 too: FLASH_ATTN_EXT with a quantized KV cache in the permuted layout fails from about 1K keys up, and it only passes with both oneDNN and MKL turned off. So it's not specific to the 140T.

I found the cause. The test builds its KV view with a gap between tokens (it allocates twice the heads and views half of them), and the oneDNN and MKL paths work out the token stride from the head count instead of reading it. llama.cpp's real cache has no gap, so actual models weren't affected. With Bonsai 27B, q4_0 KV and flash attention, perplexity at 4K context is the same on all three attention paths (oneDNN 8.727, MKL 8.731, the native TILE kernel 8.733), and needle-in-a-haystack recall is exact at 32K, 64K and 120K. I've added a guard so a layout with a gap falls back to the correct kernel. The full FLASH_ATTN_EXT suite now passes (4137/4137), plus a new sweep of the real cache layout from 1K to 8K keys. That fix will be in the next push. F16 KV is still a fine choice on the 140T if you have the memory for it.

On the MTP crash, I suspect memory rather than a leak. The draft context needs its own KV and compute buffers, and 12 GB of shared memory gets tight fast. Could you post your exact llama-server command? On the B580 I use LLAMA_ARG_SPEC_DRAFT_UBATCH=512 plus -ctkd q4_0 -ctvd q4_0 for the draft context, and on the 140T I'd start at -c 16384 and work up from there.

Also, the ternary XMX path only runs on Xe2 (B-series, Lunar Lake). On the 140T it switches itself off, so around 8 t/s is about what that memory bandwidth allows for a 27B model.

Is it possible to run the ternary XMX for Xe-LPG+?
Since i can run qwen 3.8 27b at q4 K M at a cool 10t/s avg (12t/s) coding but takes all my ram to do it on Vulkan

[Torchit1]

Good question. Xe-LPG+ does have XMX (it's the main thing Arrow Lake-H added over Meteor Lake), so it's possible in principle, but not with the current code. The ternary and attention kernels are written for Xe2's XMX: 16-lane sub-groups and Xe2's 2D block loads. Xe-LPG+'s XMX comes from Alchemist (8-lane DPAS), so it would need separate kernel variants, and I don't have an Xe-LPG+ machine to build or test them on, so it's not something I'm planning. That's why the path switches itself off on the 140T.

A correction to what I said before, too: 8 t/s isn't the bandwidth ceiling. If you get 10-12 t/s from a ~16 GB Q4_K_M 27B, Bonsai at 5.85 GB should do much better than 8, so the plain SYCL kernels are leaving a lot on the table on the 140T. Worth trying the Vulkan build of this branch with the same Bonsai file (steps in the guide under "Build and run (Vulkan)") and posting both numbers here. One heads-up: PrismML lists PTQ1_0 on Intel Vulkan as having been slow and occasionally hanging after about 1,900 tokens, so if it stalls, that's the likely cause.

If you or anyone with Xe-LPG+ hardware wants to have a go at the XMX side, the Xe2 check is ggml_sycl_device_is_xe2() and the kernels are in ggml/src/ggml-sycl/. I'm happy to review a PR.

[lavdrimismajili]

disabling VMM made it possible to run the model whereas before it didn't even start hence the crashes, I would like to see and try to fix the SYCL kernel but i don't know where to start since i don't know, maybe if you could guide me on the start, i could do something or find something.

[Torchit1]

Happy to point you in. One thing first: for generation speed, XMX probably isn't where the win is. Generating a token reads every weight once, so it's limited by memory bandwidth, and XMX mainly helps prompt reading and MTP draft checks. On the 140T the bigger opportunity is the plain PTQ1_0 mat-vec kernel it already uses, which is also much easier to work on.

Where to start:

1. Get a baseline. Run test-backend-ops perf -o MUL_MAT -b SYCL0 -p ptq1_0 (kernel only) and llama-bench -m -p 0 -n 128 (whole model). Work out the GB/s the kernel reaches from its timing and matrix size, and compare it with your RAM's bandwidth. That gap is what you're trying to close.

2. The kernel is in ggml/src/ggml-sycl/mmvq.cpp: the PTQ1_0 reorder path around line 800 (reorder_vec_dot_q_sycl<GGML_TYPE_PTQ1_0>) and the launch code around lines 1150 and 1460. Things worth trying on Xe-LPG: sub-group size (8 vs 16), rows per work-group, and making each work-item load weights in wider, contiguous chunks.

3. After every change, check it's still correct: test-backend-ops test -o MUL_MAT -b SYCL0 must pass, and a quick llama-perplexity run should match before and after.

4. Posting the Vulkan numbers for the same Bonsai file would give a useful reference point.

If you do want to try XMX later, the gate is ggml_sycl_device_is_xe2() in ggml/src/ggml-sycl/ptq1-t2.cpp (line 67), and those kernels live in ptq1-t2.cpp and ternsycl/. They'd need 8-lane DPAS variants for Xe-LPG+. I can't test on that hardware, but I'm happy to review whatever you come up with.

[PotatoNut]

System info:
Asrock B580 12 GB - Graphics Driver 32.0.101.9030
CPU: Ryzen 7 7700
RAM 32 GB 6000 mt/s cl 30
OS Windows 11

Context size:
Unknown, the prompt incorrectly marks this text as 4 tokens:

You have two sand timers, which can show 4 minutes and 7 minutes respectively. Use both the sand timers(at a time or one after other or any other combination) and measure a time of 9 minutes.

In most local models I have tried, this task end up getting brute forced and often is in an infinite loop.

Result from the Terminal:

9.32.264.104 I slot release: id 0 | task 391 | stop processing: n_tokens = 6385, truncated = 0
9.32.264.147 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 0.059
9.32.383.713 I slot launch_slot_: id 0 | task 637 | processing task, is_child = 0
9.35.515.056 I slot print_timing: id 0 | task 637 | n_gen = 168, tg = 55.00 t/s, tg_3s = 55.33 t/s
9.38.545.317 I slot print_timing: id 0 | task 637 | n_gen = 332, tg = 54.56 t/s, tg_3s = 54.12 t/s
9.41.549.918 I slot print_timing: id 0 | task 637 | n_gen = 518, tg = 56.99 t/s, tg_3s = 61.91 t/s
9.44.593.857 I slot print_timing: id 0 | task 637 | n_gen = 691, tg = 56.95 t/s, tg_3s = 56.83 t/s
9.47.642.063 I slot print_timing: id 0 | task 637 | n_gen = 862, tg = 56.78 t/s, tg_3s = 56.10 t/s
9.50.655.480 I slot print_timing: id 0 | task 637 | n_gen = 1029, tg = 56.56 t/s, tg_3s = 55.42 t/s
9.53.671.737 I slot print_timing: id 0 | task 637 | n_gen = 1202, tg = 56.67 t/s, tg_3s = 57.36 t/s
9.56.720.684 I slot print_timing: id 0 | task 637 | n_gen = 1350, tg = 55.65 t/s, tg_3s = 48.54 t/s
9.59.748.650 I slot print_timing: id 0 | task 637 | n_gen = 1515, tg = 55.52 t/s, tg_3s = 54.49 t/s
10.02.756.850 I slot print_timing: id 0 | task 637 | n_gen = 1680, tg = 55.45 t/s, tg_3s = 54.85 t/s
10.05.764.248 I slot print_timing: id 0 | task 637 | n_gen = 1866, tg = 56.03 t/s, tg_3s = 61.85 t/s
10.09.312.509 I slot print_timing: id 0 | task 637 | n_gen = 2056, tg = 55.79 t/s, tg_3s = 53.55 t/s
10.12.340.111 I slot print_timing: id 0 | task 637 | n_gen = 2231, tg = 55.94 t/s, tg_3s = 57.80 t/s
10.15.891.599 I slot print_timing: id 0 | task 637 | n_gen = 2419, tg = 55.70 t/s, tg_3s = 52.94 t/s
10.18.932.694 I slot print_timing: id 0 | task 637 | n_gen = 2581, tg = 55.54 t/s, tg_3s = 53.27 t/s
10.21.969.807 I slot print_timing: id 0 | task 637 | n_gen = 2752, tg = 55.59 t/s, tg_3s = 56.30 t/s
10.25.025.671 I slot print_timing: id 0 | task 637 | n_gen = 2917, tg = 55.49 t/s, tg_3s = 53.99 t/s
10.28.509.225 I slot print_timing: id 0 | task 637 | n_gen = 3102, tg = 55.35 t/s, tg_3s = 53.11 t/s
10.31.516.060 I slot print_timing: id 0 | task 637 | n_gen = 3289, tg = 55.69 t/s, tg_3s = 62.19 t/s
10.34.572.797 I slot print_timing: id 0 | task 637 | n_gen = 3469, tg = 55.85 t/s, tg_3s = 58.89 t/s
10.37.605.463 I slot print_timing: id 0 | task 637 | n_gen = 3622, tg = 55.60 t/s, tg_3s = 50.45 t/s
10.40.637.741 I slot print_timing: id 0 | task 637 | n_gen = 3759, tg = 55.14 t/s, tg_3s = 45.18 t/s
10.43.685.465 I slot print_timing: id 0 | task 637 | n_gen = 3922, tg = 55.07 t/s, tg_3s = 53.48 t/s
10.46.719.495 I slot print_timing: id 0 | task 637 | n_gen = 4110, tg = 55.35 t/s, tg_3s = 61.96 t/s
10.49.769.676 I slot print_timing: id 0 | task 637 | n_gen = 4303, tg = 55.66 t/s, tg_3s = 63.27 t/s
10.52.773.426 I slot print_timing: id 0 | task 637 | n_gen = 4465, tg = 55.60 t/s, tg_3s = 53.93 t/s
10.55.787.927 I slot print_timing: id 0 | task 637 | n_gen = 4614, tg = 55.37 t/s, tg_3s = 49.43 t/s
10.58.802.819 I slot print_timing: id 0 | task 637 | n_gen = 4750, tg = 55.01 t/s, tg_3s = 45.11 t/s
11.01.829.267 I slot print_timing: id 0 | task 637 | n_gen = 4931, tg = 55.18 t/s, tg_3s = 59.81 t/s
11.04.860.209 I slot print_timing: id 0 | task 637 | n_gen = 5098, tg = 55.17 t/s, tg_3s = 55.10 t/s
11.08.293.106 I slot print_timing: id 0 | task 637 | n_gen = 5270, tg = 54.99 t/s, tg_3s = 50.10 t/s
11.11.318.003 I slot print_timing: id 0 | task 637 | n_gen = 5396, tg = 54.58 t/s, tg_3s = 41.65 t/s
11.14.371.772 I slot print_timing: id 0 | task 637 | n_gen = 5567, tg = 54.63 t/s, tg_3s = 56.00 t/s
11.17.401.228 I slot print_timing: id 0 | task 637 | n_gen = 5699, tg = 54.31 t/s, tg_3s = 43.57 t/s
11.20.433.590 I slot print_timing: id 0 | task 637 | n_gen = 5868, tg = 54.35 t/s, tg_3s = 55.73 t/s
11.21.715.172 I slot print_timing: id 0 | task 637 | prompt eval time = 95.04 ms / 4 tokens ( 23.76 ms per token, 42.09 tokens per second)
11.21.715.178 I slot print_timing: id 0 | task 637 | eval time = 109236.30 ms / 5972 tokens ( 18.29 ms per token, 54.66 tokens per second)
11.21.715.179 I slot print_timing: id 0 | task 637 | total time = 109331.34 ms / 5976 tokens
11.21.715.179 I slot print_timing: id 0 | task 637 | graphs reused = 3952
11.21.715.267 I slot print_timing: id 0 | task 637 | draft acceptance = 0.37763 ( 4220 accepted / 11175 generated), mean len = 3.43
11.21.715.364 I slot release: id 0 | task 637 | stop processing: n_tokens = 6353, truncated = 0
18.41.181.604 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.867 (> 0.100 thold), f_keep = 0.059
18.41.291.246 I slot launch_slot_: id 0 | task 4353 | processing task, is_child = 0
18.44.649.408 I slot print_timing: id 0 | task 4353 | n_gen = 146, tg = 47.94 t/s, tg_3s = 48.27 t/s

[Torchit1]

Thanks for testing, and for including the full log. 55 t/s sustained across almost 6,000 generated tokens on Windows is a solid result. The "4 tokens" isn't wrong, it's prompt caching. Your prompt was 381 tokens and 377 of them were already in the server's cache from the previous turn (f_sim_best = 1.000), so only 4 new ones needed processing. That conversation used about 6,350 tokens of context (n_tokens = 6353). The limit is set by -c, which is 131,072 (128K) in run-bonsai.bat, and the server prints it at startup as n_ctx. A couple of questions so I can read the numbers properly: did you use run-bonsai.bat as is, or your own command? The .bat starts with thinking turned off, so the model does its working in the answer itself. And did it get the timer puzzle right (it's 9 minutes: start both, flip the 7 when it runs out at 7, then flip it again when the 4 runs out at 8, and it finishes at 9)?

[PotatoNut]

Thanks for the explanation.
I used Ternary-Bonsai-2-27B-PTQ1_0-mtp-lean with the run-bonsai.bat unchanged. No thinking.

The puzzle is particularly hard for the AI models. Gemma, Ornith, Ling, Laguna, those all failed the task, but were great at many other tests I tried with similar difficulty. Ternary-Bonsai end up the answering with this:

I think the answer is 9 minutes can be measured by:

Start the 4-minute timer.
When it finishes, start the 7-minute timer.
When the 7-minute timer finishes (11 min), flip the 4-minute timer.
When the 4-minute timer finishes (15 min), flip the 7-minute timer.
When the 7-minute timer finishes (22 min), stop.
I think I have to conclude that I cannot solve this correctly.

While it didn't produce the correct answer, it did think of the correct combo for the solution (4+4+1) Flip both, 4 runs out flip again, 7 runs out - flip, 4 runs out - flip the 7 again as 1 min is on the other end.

I think I have to accept that my manual calculation is flawed. Let's think logically:

We need 9 minutes.
Possible combinations:

7 + 2
4 + 4 + 1
4 + 5
7 + 4 - 2
4 + 7 - 2

With the flag {\"enable_thinking\":true}" the model now produces the correct solution at the first try. Here is the Terminal result:

0.28.078.851 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.32.497.094 I slot print_timing: id 0 | task 0 | n_gen = 125, tg = 41.17 t/s, tg_3s = 41.50 t/s
0.35.504.700 I slot print_timing: id 0 | task 0 | n_gen = 243, tg = 40.20 t/s, tg_3s = 39.23 t/s
0.38.516.855 I slot print_timing: id 0 | task 0 | n_gen = 360, tg = 39.75 t/s, tg_3s = 38.84 t/s
0.41.561.645 I slot print_timing: id 0 | task 0 | n_gen = 468, tg = 38.67 t/s, tg_3s = 35.47 t/s
0.44.573.647 I slot print_timing: id 0 | task 0 | n_gen = 609, tg = 40.30 t/s, tg_3s = 46.81 t/s
0.47.620.590 I slot print_timing: id 0 | task 0 | n_gen = 723, tg = 39.81 t/s, tg_3s = 37.41 t/s
0.50.620.848 I slot print_timing: id 0 | task 0 | n_gen = 839, tg = 39.65 t/s, tg_3s = 38.66 t/s
0.53.622.580 I slot print_timing: id 0 | task 0 | n_gen = 1020, tg = 42.22 t/s, tg_3s = 60.30 t/s
0.53.771.588 I slot print_timing: id 0 | task 0 | prompt eval time = 1406.25 ms / 414 tokens ( 3.40 ms per token, 294.40 tokens per second)
0.53.771.593 I slot print_timing: id 0 | task 0 | eval time = 24286.36 ms / 1031 tokens ( 23.58 ms per token, 42.41 tokens per second)
0.53.771.594 I slot print_timing: id 0 | task 0 | total time = 25692.61 ms / 1445 tokens
0.53.771.594 I slot print_timing: id 0 | task 0 | graphs reused = 500
0.53.771.668 I slot print_timing: id 0 | task 0 | draft acceptance = 0.26190 ( 528 accepted / 2016 generated), mean len = 2.05
0.53.771.730 I slot release: id 0 | task 0 | stop processing: n_tokens = 1446, truncated = 0

[Torchit1]

Thanks for running both, that's a really useful comparison. Thinking on is the right setting for puzzles like this. It's slower per token (42 vs 55 t/s, since fewer of the drafted tokens get accepted while it's reasoning), but it needed about 1,000 tokens instead of 6,000, so the answer actually came back about 4x sooner. The .bat starts with thinking off because that's faster for coding, but I'll add a note to the guide about switching it on for reasoning tasks. Nice to hear it beat Gemma and the others on this one.

[rt-tom]

Tested the Windows arc-b580-v0.5 build on an Intel Arc Pro B60 24 GB (Xe2).

System:

• Windows 11
• Intel Arc Pro B60 24 GB
• Intel driver 32.0.101.8993
• ONEAPI_DEVICE_SELECTOR=level_zero:0

Builds:

• Torchit1 arc-b580: 8160e3b (build 1)
• Comparison SYCL build: 77748bd43 (build 10817)

I tested a regular dense GGUF rather than Bonsai first:

Gemma 4 31B QAT Q4
gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
16.09 GiB, 30.7B params

Settings:

• full GPU offload
• Flash Attention on
• b=2048
• ub=2048
• KV cache q8_0/q8_0

Results on the same B60:

• pp512: current SYCL 117.14 t/s → Torchit1 XMX 404.20 t/s
• pp2048: current SYCL 144.16 t/s → Torchit1 XMX 728.22 t/s
• pp8192: current SYCL 136.74 t/s → Torchit1 XMX 687.57 t/s

Decode was essentially identical:

• tg128 @ d0: current SYCL 19.50 t/s → XMX 19.36 t/s
• tg128 @ d16384: current SYCL 13.71 t/s → XMX 13.68 t/s
• tg128 @ d32768: current SYCL 11.49 t/s → XMX 11.53 t/s

So on the B60, long-context decode performance for this regular Q4 dense model is effectively the same as my newer SYCL build, while prompt processing is dramatically faster.

I also tested llama-server with a real ~17.5K-token prompt. The XMX build sustained roughly 460+ t/s during real prefill, versus about 137 t/s with the comparison SYCL build on the same workload.

Gemma 4 MTP speculative decoding also works. I tested the QAT Q4 model with:

• draft-mtp
• n-max=2
• 44K context
• q8/q8 KV cache

Reasoning budget works as well. For Gemma 4 I needed --reasoning-format deepseek so that reasoning was separated into reasoning_content instead of being returned as normal content.

I haven't run Bonsai 2 on the B60 yet, but that's next.

[Torchit1]

Thanks for testing on the B60, and with Gemma 4 31B too. Good to see the prompt-side kernels carry over to a non-Bonsai model, and that its MTP drafting works.

[alpha2beta]

arc-b580 on 258v (Arc 140v): crashes on every model
Built clean with your oneAPI 2026.0 (-DGGML_SYCL_TARGET=INTEL), then:

• PQ2_0 bench: core dump at first decode — No device image found for external symbol _Z36intel_sub_group_i8_i4_matrix_mad_k32, thrown at ggml-sycl.cpp:3493.
• Persists with GGML_SYCL_MMVQ_CHUNK_MAX=1, -fa 0, and no XMX env vars (the extern is compiled into the mul_mat device image unconditionally).
• Reproduces with a standard Q3_K_M model, so it is not PQ2_0-specific.
• Reproduces via ONEAPI_DEVICE_SELECTOR=opencl:gpu too — both Level Zero and OpenCL IGC here lack the builtin for Lunar Lake.
  So the branch's Xe2-only DPAS builtin resolves on B580's IGC but not on this LNL iGPU's IGC (same NEO 26.35). No runtime flag avoids it; it needs either an IGC/driver that exposes the builtin for LNL or a code change guarding those kernels out of the default bundle.

[Torchit1]

Thanks for the report, that's a real bug on my side. Two of the XMX kernels use an IGC builtin (intel_sub_group_i8_i4_matrix_mad_k32) that your driver doesn't provide, and in a JIT build that breaks loading for every kernel, so it isn't specific to the model. I'm replacing it with inline vISA so it no longer depends on the builtin. The Vulkan build of the branch should work in the meantime. I'll post here when the fix is up, if you're able to retest.

[Torchit1]

The fix is in: f31a4f6 on arc-b580 replaces the intel_sub_group_i8_i4_matrix_mad_k32 builtin with inline vISA, so the build no longer references that symbol at all. If you pull arc-b580 and rebuild the same way, it should load on the 140V. There's also a Windows build in v0.6. I can't test on Lunar Lake myself, so could you rerun the same bench and let me know how it goes?

[Torchit1]

Quick update: v0.6 still had other builtins of that kind in the Bonsai matrix path (the ternary DPAS and 2D block loads), so it may still fail the same way on the 140V. v0.7 removes all of them: https://github.com/Torchit1/llama.cpp/releases/tag/arc-b580-v0.7. If you get a chance, that's the one to retest.

[alpha2beta]

Thanks for the update. It works more or less the same as the local fix that made by Gemini. I got about 210 t/s pp and 12.7 t/s tg (without MTP) on 140v

[bndkhr]

Hi. Created this github account to say thank you for this fork.

I've tested on B570 (linux 7.2.9, gentoo). (Each test was performed at least 2 times in a row.)

For Gemma4, llama-bench is 755/36 (p/g) for kv q8_0, 755/38 for kv q4_0, 875/37 for f16, against 230/38 for kv f16 SYCL on llama-cpp 0.6.0 (pp improved significantly). Token generation stays at 55-65 with MTP when using web-ui.

As for Tern-Bonsai, it will not run on 0.6.0, so I can't compare directly - but both prompt proccessing and token generation is better than "standard" Qwen-3.8-27B on any quantization:
1163/38 (kv q8_0), 1175/39 (kv q4_0), 1314/39 (kv f16).