Skip to content

fix(cuda/msvc): add CUDA 13.4 CUB fixes, Win32 CreateFileW, and graceful tensor name fallback #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vrwallace wants to merge 4 commits into
base: prism
Choose a base branch
from
Open

fix(cuda/msvc): add CUDA 13.4 CUB fixes, Win32 CreateFileW, and graceful tensor name fallback #300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9159cee
Sync Bonsai 2 Hadamard ternary kernels (PTQ1_0/PQ2_0) + CUDA 13.4 & M…
vrwallace Oct 13, 2026
0dc7771
Add WinSW service manager and Bonsai2/Gemma4 service configs (80K q8_…
vrwallace Oct 13, 2026
28a2d81
Replace presence-penalty 1.5 with repeat-penalty, frequency-penalty, …
vrwallace Oct 13, 2026
6754cba
Fix WinSW stop/start race condition when restarting active llama-server
vrwallace Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions Bonsai2Service.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<service>
<id>Bonsai2Service</id>
<name>Ternary Bonsai 2 27B llama.cpp Service</name>
<description>Native Windows Service (WinSW) for PrismML Ternary Bonsai 2 27B (PQ2_0) with CUDA 13.4 GPU acceleration</description>
<executable>C:\Users\vonwa\.gemini\antigravity\scratch\prism_cuda\bin\llama-server.exe</executable>
<arguments>-m "C:\Users\vonwa\.gemini\antigravity\scratch\llama_gemma\models\Ternary-Bonsai-2-27B-PQ2_0.gguf" --host 0.0.0.0 --port 8080 -c 81920 -ctk q8_0 -ctv q8_0 -ngl 99 -fa on -t 16 -b 2048 -ub 1024 --reasoning off --reasoning-budget 0 --temp 0.7 --top-p 0.80 --top-k 20 --min-p 0.0 --repeat-penalty 1.10 --frequency-penalty 0.3 --presence-penalty 0.2 --dry-multiplier 0.8</arguments>
<workingdirectory>C:\Users\vonwa\.gemini\antigravity\scratch\prism_cuda</workingdirectory>
<logpath>C:\Users\vonwa\.gemini\antigravity\scratch\prism_cuda\logs</logpath>
<log mode="roll"></log>
<startmode>Automatic</startmode>
<onfailure action="restart" delay="10 sec"/>
</service>
12 changes: 12 additions & 0 deletions Gemma4Service.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<service>
<id>Gemma4Service</id>
<name>Gemma 4 12B llama.cpp Service</name>
<description>Native Windows Service (WinSW) for Google Gemma 4 12B LLM inference engine with CUDA 13.4 GPU acceleration</description>
<executable>C:\Users\vonwa\.gemini\antigravity\scratch\llama_gemma\bin\llama-server.exe</executable>
<arguments>-m "C:\Users\vonwa\.gemini\antigravity\scratch\llama_gemma\models\gemma-4-12B-it-Q4_K_M.gguf" --host 0.0.0.0 --port 8081 -c 16384 -ngl 99 -fa 1 -t 16 -b 2048 -ub 512 -sps 0.5 --reasoning-budget 0 --spec-type ngram-simple</arguments>
<workingdirectory>C:\Users\vonwa\.gemini\antigravity\scratch\llama_gemma</workingdirectory>
<logpath>C:\Users\vonwa\.gemini\antigravity\scratch\llama_gemma\logs</logpath>
<log mode="roll"></log>
<startmode>Manual</startmode>
<onfailure action="restart" delay="10 sec"/>
</service>
2 changes: 1 addition & 1 deletion ggml/src/ggml-cuda/argsort.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

#ifdef GGML_CUDA_USE_CUB
# include <cub/cub.cuh>
# if (CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSION >= 1)
# if (CCCL_MAJOR_VERSION == 3 && CCCL_MINOR_VERSION == 1)
# define STRIDED_ITERATOR_AVAILABLE
# include <cuda/iterator>
# endif
Expand Down
2 changes: 1 addition & 1 deletion ggml/src/ggml-cuda/top-k.cu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

#ifdef GGML_CUDA_USE_CUB
# include <cub/cub.cuh>
# if (CCCL_MAJOR_VERSION >= 3 && CCCL_MINOR_VERSION >= 2)
# if (CCCL_MAJOR_VERSION == 3 && CCCL_MINOR_VERSION == 2)
# define CUB_TOP_K_AVAILABLE
# include <cuda/iterator>
using namespace cub;
Expand Down
488 changes: 488 additions & 0 deletions servicemanager.ps1
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/llama-arch.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1022,7 +1022,7 @@ LLM_TN_IMPL::LLM_TN_IMPL(llm_arch arch, llm_tensor tensor, const char * suffix,

std::string LLM_TN_IMPL::str() const {
if (LLM_TENSOR_NAMES.find(tensor) == LLM_TENSOR_NAMES.end()) {
GGML_ABORT("unknown tensor name for tensor id %d", static_cast<int>(tensor));
return "unknown";
}

std::string name = ::format(LLM_TENSOR_NAMES.at(tensor), bid, xid);
Expand Down
4 changes: 2 additions & 2 deletions vendor/cpp-httplib/httplib.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1517,8 +1517,8 @@ bool mmap::open(const char *path) {
if (wpath.empty()) { return false; }

hFile_ =
::CreateFile2(wpath.c_str(), GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE, OPEN_EXISTING, NULL);
::CreateFileW(wpath.c_str(), GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);

if (hFile_ == INVALID_HANDLE_VALUE) { return false; }

Expand Down