Conversation
…server router mode
Adds manifests/llamacpp.json: per-platform install from the pinned ggml-org
release b10819 with SHA-256 from the release digests, runtime in router mode
(--models-dir {install_dir}/models --models-max 1), identity/ready/health on
/health, and list_models / loaded_models / load_model / unload_model / chat
actions against the router's /models API.
Result matching accepts a dotted field path (status.value) because the router
reports residency as a nested object; literal top-level keys still win.
TestBundledManifestsMerge pins the shipped manifest to router mode and to a
checksummed fetch on every platform. Docs: engine-lifecycle llama.cpp section,
architecture port map row. Engine-manager 0.17.4 -> 0.18.0.
Validated live on Windows 11 x64 through the stdio JSON-RPC: install,
start, list, load, chat, unload, stop.
Signed-off-by: Joseph Pelaez <Joseph@Bodegaone.ai>
Collaborator
|
Thank you for opening this up! We're working on bringing in llama.cpp as well and want to make sure we bring it in fully and correctly based on all the engagement around it. |
Author
|
Thanks Noah. Understood that you'd rather land llama.cpp on your own terms. Two things from our side that may save you a cycle: the engine here is router-mode |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #35 — opened as a draft pending maintainer alignment per CONTRIBUTING; happy to reshape it around #9's engine table or #18's proxy once there is a direction.
What
manifests/llamacpp.json: PAIR-installed llama.cpp from the pinned ggml-org releaseb10819per platform (SHA-256 from the release digests), run inllama-serverrouter mode (--models-dir {install_dir}/models --models-max 1), identity/ready/health on/health, actionslist_models/loaded_models/load_model/unload_model/chatagainst the router's/modelsAPI.status.value) because the router reports residency as a nested object. Literal top-level keys still win; tests cover both.TestBundledManifestsMergepins the shipped manifest to router mode and a checksummed fetch on every platform.engine-lifecycle.mdxllama.cpp section,architecture.mdxport map row.versions.jsonengine-manager 0.17.4 → 0.18.0.Out of scope (deliberately)
Proxy routing for the engine, the desktop
EngineTypesrow, andpull_model— see #35 for why.Validation
go test ./...inservices/nvpair-engine-managergreen;node scripts/spdx-headers.mjsclean.engine:install(download + sha256 + extract) →engine:start(router up on 8080, identity probe passed) →list_models(GGUF reportedstatus.value: unloaded,loaded_modelsempty) →load_model→chat(system_fingerprint: b10819-…) →unload_model→engine:stop;loadedByEngine.llamacppwent one → empty..gguf; take ids fromlist_models.Signed-off (DCO) on the commit.