Skip to content

feat(engine-manager): llama.cpp as a PAIR-installed engine via llama-server router mode - #36

Draft
Mayimbe07 wants to merge 1 commit into
NVIDIA:mainfrom
Mayimbe07:feat/llamacpp-installed-engine
Draft

Mayimbe07 wants to merge 1 commit into
NVIDIA:mainfrom
Mayimbe07:feat/llamacpp-installed-engine

Conversation

@Mayimbe07

Copy link
Copy Markdown

Implements #35 — opened as a draft pending maintainer alignment per CONTRIBUTING; happy to reshape it around #9's engine table or #18's proxy once there is a direction.

What

  • manifests/llamacpp.json: PAIR-installed llama.cpp from the pinned ggml-org release b10819 per platform (SHA-256 from the release digests), run in llama-server router mode (--models-dir {install_dir}/models --models-max 1), identity/ready/health on /health, actions list_models / loaded_models / load_model / unload_model / chat against the router's /models API.
  • Engine manager: result matching accepts a dotted field path (status.value) because the router reports residency as a nested object. Literal top-level keys still win; tests cover both.
  • TestBundledManifestsMerge pins the shipped manifest to router mode and a checksummed fetch on every platform.
  • Docs: engine-lifecycle.mdx llama.cpp section, architecture.mdx port map row. versions.json engine-manager 0.17.4 → 0.18.0.

Out of scope (deliberately)

Proxy routing for the engine, the desktop EngineTypes row, and pull_model — see #35 for why.

Validation

  • go test ./... in services/nvpair-engine-manager green; node scripts/spdx-headers.mjs clean.
  • Live on Windows 11 x64 (Vulkan build) through the engine manager's stdio JSON-RPC: engine:install (download + sha256 + extract) → engine:start (router up on 8080, identity probe passed) → list_models (GGUF reported status.value: unloaded, loaded_models empty) → load_modelchat (system_fingerprint: b10819-…) → unload_modelengine:stop; loadedByEngine.llamacpp went one → empty.
  • Note for callers: the router's model id is the file stem without .gguf; take ids from list_models.

Signed-off (DCO) on the commit.

…server router mode

Adds manifests/llamacpp.json: per-platform install from the pinned ggml-org
release b10819 with SHA-256 from the release digests, runtime in router mode
(--models-dir {install_dir}/models --models-max 1), identity/ready/health on
/health, and list_models / loaded_models / load_model / unload_model / chat
actions against the router's /models API.

Result matching accepts a dotted field path (status.value) because the router
reports residency as a nested object; literal top-level keys still win.
TestBundledManifestsMerge pins the shipped manifest to router mode and to a
checksummed fetch on every platform. Docs: engine-lifecycle llama.cpp section,
architecture port map row. Engine-manager 0.17.4 -> 0.18.0.

Validated live on Windows 11 x64 through the stdio JSON-RPC: install,
start, list, load, chat, unload, stop.

Signed-off-by: Joseph Pelaez <Joseph@Bodegaone.ai>
@Noah-Tervalon-Nvidia

Copy link
Copy Markdown
Collaborator

Thank you for opening this up! We're working on bringing in llama.cpp as well and want to make sure we bring it in fully and correctly based on all the engagement around it.

@Mayimbe07

Copy link
Copy Markdown
Author

Thanks Noah. Understood that you'd rather land llama.cpp on your own terms.

Two things from our side that may save you a cycle: the engine here is router-mode llama-server pinned to the ggml-org release archives with their published SHA-256 digests (no custom build), and it has been live-validated end-to-end on Windows/Linux Vulkan and CUDA. Happy to rebase this onto whatever engine-manager shape you settle on, or to close it in favour of yours — either way, if a draft design goes up on #35 we'll test against it early.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants