Skip to content

Add adopt-only llama.cpp engine and OpenAI proxy - #41

Open
XxSilentStormIxX wants to merge 13 commits into
NVIDIA:mainfrom
XxSilentStormIxX:feat/llamacpp-engine
Open

XxSilentStormIxX wants to merge 13 commits into
NVIDIA:mainfrom
XxSilentStormIxX:feat/llamacpp-engine

Conversation

@XxSilentStormIxX

Copy link
Copy Markdown

Summary

  • Add a first-class llamacpp engine that adopts an already-running official llama-server (default probe port 8082). PAIR does not install, spawn, stop, or load GGUFs.
  • Add llamacpp-proxy on 8084 with OpenAI /v1 routes. Cluster discovery key is lc.
  • List the full /v1/models catalog, but route only to models that are already loaded on the chosen node. Catalog-only ids return 502 and are never forwarded, so PAIR cannot trigger a GGUF swap (safe while ComfyUI holds the local GPU).
  • A node with nothing loaded locally can still forward to a paired Mac/laptop that has the model loaded.

Test Plan

  • go test -count=1 in nvpair-engine-manager, llamacpp-proxy, nvpair-ui-broker, nvpair-job-scheduler, nvpair-manual-nodes, nvpair-tui, shared, and scripts/inference-dispatcher
  • Desktop npx vitest run tests/modular/llamacpp-engine.test.ts --project unit and npm run typecheck
  • npm run service-contracts:check
  • Do not start a live 27B GGUF while ComfyUI is using the GPU
  • Optional: with llama-server already up on 8082, PAIR shows llama.cpp adopted; apps use http://127.0.0.1:8084/v1

P-DLE added 13 commits September 6, 2026 07:39
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
Signed-off-by: P-DLE <P-DLE@users.noreply.local>
@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator

Thank you for opening this up! We're working on bringing in llama.cpp as well and want to make sure we bring it in fully and correctly based on all the engagement around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants