diff --git a/AGENTS.md b/AGENTS.md index e1d73809..c9eb28ac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ change belongs in a Go service instead. - Electron starts **only** `nvpair-ui-broker` from `desktop/cli-bin/`. - The broker supervises the other Go workers: discovery, proxies, engines, cluster, settings, manual nodes, workloads, errors, and the scheduler. -- The broker spawns all 11 workers at startup. Only the scanner is required; the +- The broker spawns all 12 workers at startup. Only the scanner is required; the rest are optional and non-fatal. - `nvpair-tui` is bundled but never supervised. It owns its own broker. - Interprocess communication is newline-delimited JSON-RPC 2.0 over stdio @@ -72,7 +72,7 @@ Never edit `desktop/docs/services-api.md` by hand. It is generated by ## Services (`services/`) -Thirteen Go binaries. Each component is its own module, with its tests beside its +Fourteen Go binaries. Each component is its own module, with its tests beside its source and a `README.md` describing its JSON-RPC surface. Shared packages live in `shared/`, and `tests/` holds cross-process tests that drive real binaries. Prefer the Go source when a README disagrees with it. diff --git a/README.md b/README.md index 0f0a7242..2e9a5f6d 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ one, and both report live GPU and memory use throughout. | **Architectures** | x64 and arm64 on all three. Windows on ARM is experimental. | | **Installers** | Windows `.exe`; Linux `.deb`; macOS `.dmg`. On other Linux distributions, [build from source](docs/building.mdx). | | **Mixing nodes** | Windows, Linux, and macOS nodes can all be paired with each other | -| **Inference engines** | Ollama and LM Studio | +| **Inference engines** | Ollama, LM Studio, and llama.cpp (adopt-only) | **PAIR running on a machine does not mean an engine will.** PAIR itself runs on any supported Windows, Linux, or macOS machine. Each engine sets its own requirements diff --git a/desktop/docs/architecture.md b/desktop/docs/architecture.md index e7c72ab7..70fa5b29 100644 --- a/desktop/docs/architecture.md +++ b/desktop/docs/architecture.md @@ -44,6 +44,7 @@ The canonical runtime inventory is | `nvpair-ui-broker` | Electron | Worker supervision and control-plane relay | | `ollama-proxy` | Broker | Ollama-compatible proxy and cluster routing | | `lmstudio-proxy` | Broker, optional | LM Studio OpenAI-compatible proxy | +| `llamacpp-proxy` | Broker, optional | llama.cpp OpenAI-compatible proxy | | `nvpair-node-scanner` | Broker | LAN discovery and announcement | | `nvpair-node-info` | Broker | Node metadata and telemetry endpoint | | `nvpair-workload-manager` | Broker, optional | Workload replication | @@ -110,7 +111,7 @@ subscribes to broker relays after `app:ready`, and converts backend responses into stable UI contracts. Electron reports the service connected after broker `app:ready`. The -broker-owned Ollama and LM Studio proxies remain asynchronous capabilities; a +broker-owned Ollama, LM Studio, and llama.cpp proxies remain asynchronous capabilities; a late or failed proxy does not misreport the broker startup as failed. If `app:ready` does not arrive within the startup deadline, Overview opens Settings @@ -195,16 +196,18 @@ Engine lifecycle and model operations flow through the broker's `engine:*` relay to `nvpair-engine-manager`. The renderer identifies engines with the closed `EngineType` union and narrows external strings with `isEngineType()`. -The Ollama and LM Studio proxies are cluster-aware. For model-bearing inference, -each proxy first keeps only nodes whose per-engine discovery inventory advertises -the requested model. Empty and non-matching inventories are excluded; an empty -owner set returns a local `502`. Routing precedence within the eligible set is: +The Ollama, LM Studio, and llama.cpp proxies are cluster-aware. For +model-bearing inference, each proxy first keeps only nodes whose per-engine +discovery inventory advertises the requested model. llama.cpp uses the **loaded** +set, not the on-disk catalog. Empty and non-matching inventories are excluded; an +empty owner set returns a local `502`. Routing precedence within the eligible set +is: 1. a user-selected manual node; 2. the priority list emitted by `nvpair-job-scheduler`; 3. the proxy's deterministic default ordering. -The scheduler combines total pending (queued and running) workload across both +The scheduler combines total pending (queued and running) workload across all engines with a smoothed 0–3 pressure derived from the busiest GPU. Missing, invalid, or older-than-10-second telemetry has neutral pressure. It emits the order, pending count, and pressure, reranking on meaningful workload, discovery, @@ -266,6 +269,9 @@ cannot yet be reported are centralized in `src/shared/constants/modular-runtime.ts`. - Ollama-compatible clients use the proxy port reported by the broker. +- llama.cpp clients use the OpenAI-compatible proxy at `http://127.0.0.1:8084/v1` + by default. PAIR adopts an already-running `llama-server` (default probe + `8082`) and does not install or load GGUFs. - Cluster pairing currently uses port `14321`. - Node telemetry is read from `/v1/node-info` at each discovered node's advertised port. diff --git a/desktop/docs/macos-privileged-helper.md b/desktop/docs/macos-privileged-helper.md index 78f78938..360bc9c2 100644 --- a/desktop/docs/macos-privileged-helper.md +++ b/desktop/docs/macos-privileged-helper.md @@ -83,11 +83,11 @@ The set in `native/PrivilegedHelper/main.swift` must stay in sync with the (`src/shared/constants/modular-binaries.ts`) and the manual uninstaller. `npm run service-contracts:check` fails when the Swift list differs from that canonical set: -`ollama-proxy`, `lmstudio-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, -`nvpair-workload-manager`, `nvpair-errors`, `nvpair-cluster-manager`, -`nvpair-engine-manager`. +`ollama-proxy`, `lmstudio-proxy`, `llamacpp-proxy`, `nvpair-node-info`, +`nvpair-node-scanner`, `nvpair-workload-manager`, `nvpair-errors`, +`nvpair-cluster-manager`, `nvpair-engine-manager`. -> These eight mirror the per-program/per-port `netsh` rules in +> These nine mirror the per-program/per-port `netsh` rules in > `scripts/build/installer.nsh` on Windows. macOS's Application Firewall is > per-application and inbound-only, so one `--add`/`--unblockapp` per binary > collapses Windows's per-port rules. diff --git a/desktop/docs/service-contract-exceptions.json b/desktop/docs/service-contract-exceptions.json index 516fb2be..5de693e2 100644 --- a/desktop/docs/service-contract-exceptions.json +++ b/desktop/docs/service-contract-exceptions.json @@ -1,7 +1,7 @@ { "$comment": "Intentionally unintegrated JSON-RPC methods for desktop ↔ services contract checks. Each ignoredMethods entry needs a current technical reason. Edit by hand; run npm run service-contracts:write to refresh docs/services-api.md after services/ changes.", "ignoredMethods": { - "schedule:priority": "Broker-internal. nvpair-job-scheduler emits a per-engine priority snapshot ({engine, nodes, ranks?} — ordered node ids plus each node's pending count) to nvpair-ui-broker, which forwards it to ollama-proxy and lmstudio-proxy via node/set-priority; each proxy then layers its own optimistic burst reservations on that baseline. PAIR consumes the routing result rather than this notification, and must not mirror the pending/reservation accounting.", + "schedule:priority": "Broker-internal. nvpair-job-scheduler emits a per-engine priority snapshot ({engine, nodes, ranks?} — ordered node ids plus each node's pending count) to nvpair-ui-broker, which forwards it to ollama-proxy, lmstudio-proxy, and llamacpp-proxy via node/set-priority; each proxy then layers its own optimistic burst reservations on that baseline. PAIR consumes the routing result rather than this notification, and must not mirror the pending/reservation accounting.", "engine:restore-enabled": "Broker-internal startup restoration. nvpair-ui-broker emits engine:restore-enabled directly to its supervised engine-manager after the managed Ollama port gate and on manager respawn; it is not a renderer/UI notification.", "proxy:ready": "Consumed, not missing: the broker relays it and normalizeBrokerProxy (modular-supervisor.ts) strips the `proxy:` prefix, so the bridge handles the de-prefixed `ready` (sets proxyPort). The literal `proxy:ready` is intentionally absent from our TS — extractor limitation, not a gap.", "node/selection-changed": "Automatic routing has no selected-node UI, so PAIR deliberately does not consume proxy selection changes.", @@ -11,7 +11,7 @@ "workload:started": "Proxy-to-broker lifecycle event translated by the broker into workloads:upsert, which PAIR consumes.", "workload:completed": "Proxy-to-broker terminal lifecycle event translated by the broker into workloads:upsert.", "workload:errored": "Proxy-to-broker terminal lifecycle event translated by the broker into workloads:upsert.", - "node/activity": "Proxy-to-broker liveness evidence, consumed entirely inside the backend. ollama-proxy and lmstudio-proxy raise it (coalesced to one report per node per 2s by nvpair-shared/nodeactivity) whenever a peer's engine returns inference response bytes; nvpair-ui-broker relays it to nvpair-node-scanner as discovery:node-activity, where it keeps a node that is busy serving inference from being evicted for failing a liveness probe it had no spare CPU to answer. PAIR consumes the result — the node staying in the discovery snapshot — not this per-request notification, which carries no state a UI could render. Not currently surfaced by the extractor either (it is emitted through a shared noderec constant, like nodeinfo:observed-addresses); this entry records the intent regardless.", + "node/activity": "Proxy-to-broker liveness evidence, consumed entirely inside the backend. ollama-proxy, lmstudio-proxy, and llamacpp-proxy raise it (coalesced to one report per node per 2s by nvpair-shared/nodeactivity) whenever a peer's engine returns inference response bytes; nvpair-ui-broker relays it to nvpair-node-scanner as discovery:node-activity, where it keeps a node that is busy serving inference from being evicted for failing a liveness probe it had no spare CPU to answer. PAIR consumes the result — the node staying in the discovery snapshot — not this per-request notification, which carries no state a UI could render. Not currently surfaced by the extractor either (it is emitted through a shared noderec constant, like nodeinfo:observed-addresses); this entry records the intent regardless.", "errors:report": "Producer-to-broker event demultiplexed into broker-owned nvpair-errors. PAIR consumes the resulting errors:update snapshot.", "errors:clear": "Producer-to-broker event demultiplexed into broker-owned nvpair-errors. PAIR consumes the resulting errors:update snapshot.", "cluster:trust-changed": "Broker-internal. nvpair-cluster-manager announces it from its trust store after a pin write lands, and nvpair-ui-broker answers it by calling discovery:reload-trust on nvpair-node-scanner so the scanner re-derives each peer's trusted annotation. PAIR consumes the result — the refreshed AvailableNode.trusted carried on the discovery snapshot — not this notification. It carries no payload by design: recipients re-read the cluster dir rather than trusting a diff on the wire.", diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.md index 590e7264..660c74c3 100644 --- a/desktop/docs/services-api.md +++ b/desktop/docs/services-api.md @@ -16,6 +16,8 @@ - none ✅ ### Requests the backend handles but the bridge never calls (unused capability) +- ⚠️ llamacpp-proxy → node/selected +- ⚠️ llamacpp-proxy → node/set-local-backend - ⚠️ lmstudio-proxy → node/selected - ⚠️ lmstudio-proxy → node/set-local-backend - ⚠️ nvpair-engine-manager → engine:describe @@ -45,6 +47,31 @@ ### Backend binaries not listed in `modular-binaries.ts` - none ✅ +## llamacpp-proxy + +| Method | Direction | In bridge? | +|---|---|---| +| `error` | notification (we consume) | ✅ yes | +| `errors:clear` | notification (we consume) | ✅ yes | +| `errors:report` | notification (we consume) | ✅ yes | +| `node/discovered` | notification (we consume) | ✅ yes | +| `node/removed` | notification (we consume) | ✅ yes | +| `node/selection-changed` | notification (we consume) | ➖ ignored | +| `node/updated` | notification (we consume) | ✅ yes | +| `proxy/request` | notification (we consume) | ✅ yes | +| `proxy/request-started` | notification (we consume) | ➖ ignored | +| `ready` | notification (we consume) | ✅ yes | +| `node/add-manual` | request (we call) | ✅ yes | +| `node/remove-manual` | request (we call) | ✅ yes | +| `node/select` | request (we call) | ✅ yes | +| `node/selected` | request (we call) | ⚠️ not called | +| `node/set-local-backend` | request (we call) | ⚠️ not called | +| `node/set-priority` | request (we call) | ✅ yes | +| `nodes/list` | request (we call) | ✅ yes | + +**Dynamic / unresolved notify sites (verify by hand — `npm run service-contracts` prints the line numbers):** +- `method (var) (proxy.go)` + ## lmstudio-proxy | Method | Direction | In bridge? | @@ -263,6 +290,7 @@ - `proxy:* (broker.go)` - `method (var) (clustermanager.go)` - `method (var) (errors.go)` +- `llamacpp-proxy:* (llamacppproxy.go)` - `lmstudio-proxy:* (lmstudioproxy.go)` - `method (var) (proxy.go)` - `method (var) (rpcworker.go, 2 sites)` diff --git a/desktop/docs/services-backend.md b/desktop/docs/services-backend.md index 0d54c4c9..a183d496 100644 --- a/desktop/docs/services-backend.md +++ b/desktop/docs/services-backend.md @@ -23,6 +23,7 @@ broker supervises every worker and relays its control plane. | `nvpair-ui-broker` | Worker supervision and relay | | `ollama-proxy` | Ollama-compatible routing proxy with cluster-mTLS ingress | | `lmstudio-proxy` | LM Studio routing proxy with cluster-mTLS ingress | +| `llamacpp-proxy` | llama.cpp routing proxy with cluster-mTLS ingress | | `nvpair-node-scanner` | Discovery and node announcement | | `nvpair-node-info` | Node metadata and telemetry | | `nvpair-manual-nodes` | User-managed node entries | @@ -45,7 +46,7 @@ flowchart TB Broker["nvpair-ui-broker"] Scanner["nvpair-node-scanner"] NodeInfo["nvpair-node-info"] - Proxies["ollama-proxy / lmstudio-proxy"] + Proxies["ollama-proxy / lmstudio-proxy / llamacpp-proxy"] Engines["nvpair-engine-manager"] Cluster["nvpair-cluster-manager"] Settings["nvpair-node-settings"] @@ -90,8 +91,8 @@ engine, workload, cluster, and error relays. The bridge then emits renderer push events from backend notifications. Connector readiness follows the broker contract: `app:ready` establishes the -service connection, while Ollama and LM Studio proxy readiness remains an -asynchronous capability signal. Personal AI Router waits up to the canonical +service connection, while Ollama, LM Studio, and llama.cpp proxy readiness +remains an asynchronous capability signal. Personal AI Router waits up to the canonical startup deadline in `src/shared/constants/modular-runtime.ts` for `app:ready`; an outright failure or stalled broker startup is surfaced in Settings > Service with retry and log access. If a stalled broker reports ready @@ -117,7 +118,7 @@ reserved for inference clients. | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `app:ready` | Complete broker startup and refresh snapshots | `state:request-refresh` | | `discovery:nodes-changed` | Replace discovery snapshot and diff nodes | `discovery:nodes-changed`, `nodes:upsert`, `nodes:remove` | -| `proxy:ready` / `lmstudio-proxy:ready` | Record engine proxy port | `engines:state-changed` | +| `proxy:ready` / `lmstudio-proxy:ready` / `llamacpp-proxy:ready` | Record engine proxy port | `engines:state-changed` | | proxy `node/*` | Update per-engine node presence; the advertised port is the peer's promoted proxy port (not the engine's private loopback port) | node and engine pushes | | `engine:ready` / `engine:state-changed` | Update engine facts and models | `engines:state-changed` | | `engine:install-progress` / `engine:remote-progress` | Update operation progress | engine progress pushes | @@ -128,7 +129,7 @@ reserved for inference clients. | `nodes:changed` | Replace membership snapshot | `nodes:changed` | | `workloads:upsert` / `workloads:remove` | Update workload catalog | workload pushes | -`nvpair-job-scheduler` combines queued and running work across both engines with +`nvpair-job-scheduler` combines queued and running work across all engines with a smoothed 0–3 pressure from the busiest GPU. Invalid, missing, or older-than-10-second telemetry receives neutral pressure. It emits `schedule:priority` with order, pending count, and pressure; the broker applies @@ -150,8 +151,11 @@ waiting for authoritative state. Pending state clears on matching engine state, progress, or error pushes. Local engine operations include install, start, stop, uninstall, update, port -changes, and model actions. Remote cluster operations use the engine manager's -remote control surface where supported. +changes, and model actions. llama.cpp is adopt-only: PAIR probes an +already-running `llama-server` (default `8082`) and does not install, spawn, or +load GGUFs. The app endpoint is `http://127.0.0.1:8084/v1`; routing requires the +model **loaded** on the serving node. Remote cluster operations use the engine +manager's remote control surface where supported. `engine:stop` (and its cluster `ec` equivalent) reclaims an orphan a prior run left on the engine's own managed port, terminating it only when that PID is @@ -263,7 +267,7 @@ resolved back to the hostname the entry was keyed by. An NVPAIR-launched engine binds to loopback only and is never directly LAN-reachable. Each node fronts its engine with its `ollama-proxy` / -`lmstudio-proxy`, whose LAN ingress is gated by cluster mTLS: only a pinned +`lmstudio-proxy` / `llamacpp-proxy`, whose LAN ingress is gated by cluster mTLS: only a pinned cluster member can send it work. Discovery advertises the promoted **proxy** port (never the engine port), and the broker hands the private loopback engine to the local proxy via `node/set-local-backend`. Every cluster-scoped worker derives diff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.md index 93d13290..64ccb4c0 100644 --- a/desktop/docs/services-parity.md +++ b/desktop/docs/services-parity.md @@ -24,6 +24,7 @@ history. | Manual nodes | Complete with local persistence | Broker owns probing and proxy registration; Electron persists entries for replay | | Ollama routing | Complete | Broker relay and backend scheduler drive proxy routing | | LM Studio routing | Complete | Parallel broker relay and scheduler path | +| llama.cpp routing | Complete | Adopt-only `llamacpp-proxy` on `8084`; loaded-model eligibility only | | Local engine lifecycle | Complete | Install, start, stop, uninstall, update, and port configuration | | Remote engine lifecycle | Partial | Remote install, start, stop, status, and model pull are supported | | Engine models | Partial | Core list, pull, load, unload, and supported delete actions are wired | @@ -41,6 +42,7 @@ history. - `ollama-proxy`; - `lmstudio-proxy`; +- `llamacpp-proxy`; - `nvpair-node-scanner`; - `nvpair-node-info`; - `nvpair-manual-nodes`; @@ -99,15 +101,16 @@ they survive worker restarts. ## Routing and inference -Both text-engine proxies are broker-owned and cluster-aware: +The text-engine proxies are broker-owned and cluster-aware: - `ollama-proxy` serves the Ollama-compatible surface; -- `lmstudio-proxy` serves the LM Studio/OpenAI-compatible surface. +- `lmstudio-proxy` serves the LM Studio/OpenAI-compatible surface; +- `llamacpp-proxy` serves the llama.cpp/OpenAI-compatible surface (`8084`; loaded models only). Routing precedence is manual selection, scheduler priority, then deterministic proxy ordering. Personal AI Router leaves proxies in automatic mode. -`nvpair-job-scheduler` combines total queued and running workload across both +`nvpair-job-scheduler` combines total queued and running workload across all engines with a smoothed 0–3 GPU-pressure signal. The backend scanner and manual node worker provide maximum-GPU utilization, while invalid, missing, or older-than-10-second samples receive neutral pressure. The scheduler emits order, diff --git a/desktop/native/PrivilegedHelper/main.swift b/desktop/native/PrivilegedHelper/main.swift index a0c4e2e9..ae9b68ab 100644 --- a/desktop/native/PrivilegedHelper/main.swift +++ b/desktop/native/PrivilegedHelper/main.swift @@ -38,6 +38,7 @@ enum Firewall { static let networkedBinaries = [ "ollama-proxy", "lmstudio-proxy", + "llamacpp-proxy", "nvpair-node-info", "nvpair-node-scanner", "nvpair-workload-manager", diff --git a/desktop/scripts/build/installer.nsh b/desktop/scripts/build/installer.nsh index 9cb87ed7..eac941b0 100644 --- a/desktop/scripts/build/installer.nsh +++ b/desktop/scripts/build/installer.nsh @@ -19,6 +19,7 @@ nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-tui.exe"' nsExec::ExecToLog 'taskkill /F /T /IM "ollama-proxy.exe"' nsExec::ExecToLog 'taskkill /F /T /IM "lmstudio-proxy.exe"' + nsExec::ExecToLog 'taskkill /F /T /IM "llamacpp-proxy.exe"' nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-node-info.exe"' nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-node-scanner.exe"' nsExec::ExecToLog 'taskkill /F /T /IM "nvpair-manual-nodes.exe"' @@ -122,6 +123,7 @@ DetailPrint "Adding Personal AI Router firewall rules..." nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Ollama Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router LM Studio Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet' + nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router llama.cpp Proxy" dir=in action=allow program="$INSTDIR\resources\cli-bin\llamacpp-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Node Info" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Node Scanner" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Workload Manager" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet' @@ -130,6 +132,7 @@ nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router Engine Manager" dir=in action=allow program="$INSTDIR\resources\cli-bin\nvpair-engine-manager.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS LM Studio Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet' + nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS llama.cpp Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\llamacpp-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Node Info (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Node Scanner (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="Personal AI Router mDNS Workload Manager (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\resources\cli-bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet' @@ -143,6 +146,7 @@ DetailPrint "Removing Personal AI Router firewall rules..." nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Ollama Proxy"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router LM Studio Proxy"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router llama.cpp Proxy"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Node Info"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Node Scanner"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Workload Manager"' @@ -151,6 +155,7 @@ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Engine Manager"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS LM Studio Proxy (UDP 5353)"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS llama.cpp Proxy (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Info (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Scanner (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Workload Manager (UDP 5353)"' diff --git a/desktop/scripts/build/macos/uninstall.sh b/desktop/scripts/build/macos/uninstall.sh index ce88d22b..fc1bfefa 100644 --- a/desktop/scripts/build/macos/uninstall.sh +++ b/desktop/scripts/build/macos/uninstall.sh @@ -58,6 +58,7 @@ for proc in \ "nvpair-tui" \ "ollama-proxy" \ "lmstudio-proxy" \ + "llamacpp-proxy" \ "nvpair-node-info" \ "nvpair-node-scanner" \ "nvpair-manual-nodes" \ @@ -74,7 +75,7 @@ sleep 1 FW=/usr/libexec/ApplicationFirewall/socketfilterfw if [ -x "$FW" ]; then - for bin in ollama-proxy lmstudio-proxy nvpair-node-info nvpair-node-scanner \ + for bin in ollama-proxy lmstudio-proxy llamacpp-proxy nvpair-node-info nvpair-node-scanner \ nvpair-workload-manager nvpair-errors nvpair-cluster-manager nvpair-engine-manager; do "$FW" --remove "$APP_PATH/Contents/Resources/cli-bin/$bin" >/dev/null 2>&1 || true done diff --git a/desktop/src/electron/model-hub/index.ts b/desktop/src/electron/model-hub/index.ts index d1fda727..570923bf 100644 --- a/desktop/src/electron/model-hub/index.ts +++ b/desktop/src/electron/model-hub/index.ts @@ -52,7 +52,7 @@ export async function getEngineHubModels(engineType: EngineType): Promise -export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] +export type ProxyEngine = Extract +export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio', 'llamacpp'] /** Map a proxy node source onto the engine it describes. */ const PROXY_SOURCE_ENGINE: Record = { 'ollama-proxy': 'ollama', - 'lmstudio-proxy': 'lm-studio' + 'lmstudio-proxy': 'lm-studio', + 'llamacpp-proxy': 'llamacpp' +} + +function proxySourceForEngine(engine: EngineType): ProxyNodeSource { + if (engine === 'ollama') return 'ollama-proxy' + if (engine === 'lm-studio') return 'lmstudio-proxy' + return 'llamacpp-proxy' } /** Per-engine presence on a node — each proxy reports its own engine. */ @@ -55,7 +62,7 @@ interface EnginePresence { /** * The node's promoted inference **proxy** port for this engine, as * advertised in discovery. Under secure inference the broker registers the - * `ol`/`lm` service at the proxy's port — never the engine's own port, which + * `ol`/`lm`/`lc` service at the proxy's port — never the engine's own port, which * is loopback-private and reachable by peers only through that proxy's * cluster-mTLS ingress. The engine's real server port is not in discovery; * it comes from `engine:remote-get-installed` facts (a peer) or @@ -164,19 +171,16 @@ function emptyPresence(): EnginePresence { } function emptyEngines(): Record { - return { ollama: emptyPresence(), 'lm-studio': emptyPresence() } + return { ollama: emptyPresence(), 'lm-studio': emptyPresence(), llamacpp: emptyPresence() } } -/** Immutably set one engine's presence, preserving the other. */ +/** Immutably set one engine's presence, preserving the others. */ function setEngine( engines: Record, engine: ProxyEngine, presence: EnginePresence ): Record { - return { - ollama: engine === 'ollama' ? presence : engines.ollama, - 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'] - } + return { ...engines, [engine]: presence } } /** @@ -390,7 +394,7 @@ export function parseWorkloadsInitial(value: JsonValue | undefined): Workload[] /** True for an engine fronted by a broker-supervised reverse proxy. */ export function isProxyEngine(engine: EngineType): engine is ProxyEngine { - return engine === 'ollama' || engine === 'lm-studio' + return engine === 'ollama' || engine === 'lm-studio' || engine === 'llamacpp' } const PENDING_OP_IDLE_TIMEOUT_MS = 90_000 @@ -728,7 +732,7 @@ function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): Mod } return { id, - sources: [engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'], + sources: [proxySourceForEngine(engine)], // `Node.Host` is the hostname; empty for the self-bridge manual node, // in which case the broker discovery entry supplies the display name on // merge (see mergeNode). Never fall back to the UUID id here. @@ -899,8 +903,9 @@ class ModularBridgeState { private logs: LogEntry[] = [] // Per-engine bound proxy port reported by the broker. 0 = not reported yet; // we never fabricate a default — an unknown port surfaces as null, not a - // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`. - private proxyPorts: Record = { ollama: 0, 'lm-studio': 0 } + // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`, + // `llamacpp` is the `llamacpp-proxy`. + private proxyPorts: Record = { ollama: 0, 'lm-studio': 0, llamacpp: 0 } private selfId: string | null = null /** * Authoritative local-engine facts from `nvpair-engine-manager`, keyed by @@ -1710,7 +1715,7 @@ class ModularBridgeState { * the renderer renders the peer engine as unavailable rather than as an * installed-but-off toggle. * - * The peer's promoted **proxy** port IS carried in discovery (the `ol`/`lm` + * The peer's promoted **proxy** port IS carried in discovery (the `ol`/`lm`/`lc` * advertisement points at the proxy), so it is surfaced as `proxyPort` from * that per-engine presence regardless of facts. * The peer's engine port stays private (loopback) and comes only from facts; @@ -2287,6 +2292,10 @@ class ModularBridgeState { this.handleProxyNotification(notification, 'lm-studio') return } + if (notification.source === 'llamacpp-proxy') { + this.handleProxyNotification(notification, 'llamacpp') + return + } if (notification.source === 'broker') { this.handleBrokerNotification(notification) } @@ -2332,7 +2341,7 @@ class ModularBridgeState { if (notification.method === 'node/discovered' || notification.method === 'node/updated') { const node = parseProxyNode(notification.params, engine) if (!node) return - this.upsertNode(node, engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy') + this.upsertNode(node, proxySourceForEngine(engine)) } } @@ -2344,7 +2353,7 @@ class ModularBridgeState { private clearNodeEngine(nodeId: string, engine: ProxyEngine): void { const existing = this.nodes.get(nodeId) if (!existing) return - const source: BrokerNodeSource = engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' + const source: BrokerNodeSource = proxySourceForEngine(engine) const sources = removeSource(existing.sources, source) if (sources.length === 0 && !existing.nodeInfoUp) { this.removeNodeEntry(nodeId) @@ -2537,7 +2546,7 @@ class ModularBridgeState { // install/running state; discovery only fills in models. A remote node // has no local engine-manager, so its status comes from authoritative // peer facts or its advertisement, and is omitted when neither is known. - // Push per proxy-engine (Ollama + LM Studio) so both light up per node. + // Push per proxy-engine (Ollama, LM Studio, llama.cpp) so each lights up per node. const isSelf = merged.id === this.selfId for (const engine of PROXY_ENGINES) { if (!isSelf) { @@ -2592,7 +2601,7 @@ class ModularBridgeState { } } - // A proxy source (ollama-proxy / lmstudio-proxy): refresh only that + // A proxy source (ollama-proxy / lmstudio-proxy / llamacpp-proxy): refresh only that // engine's presence; keep the other engine, telemetry, and node-info. const engine = PROXY_SOURCE_ENGINE[source] return { diff --git a/desktop/src/electron/service-bridge/modular-supervisor.ts b/desktop/src/electron/service-bridge/modular-supervisor.ts index 943d0f84..3f809437 100644 --- a/desktop/src/electron/service-bridge/modular-supervisor.ts +++ b/desktop/src/electron/service-bridge/modular-supervisor.ts @@ -299,12 +299,22 @@ function engineManagerId(engine: ProxyEngine): string { function proxyEngineFromManagerId(id: string): ProxyEngine | null { if (id === 'ollama') return 'ollama' if (id === 'lmstudio') return 'lm-studio' + if (id === 'llamacpp') return 'llamacpp' return null } /** The broker relay namespace fronting an engine's reverse proxy. */ function proxyRelayPrefix(engine: ProxyEngine): string { - return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy' + if (engine === 'ollama') return 'proxy' + if (engine === 'lm-studio') return 'lmstudio-proxy' + return 'llamacpp-proxy' +} + +function proxyEngineFromRelaySource(source: string): ProxyEngine | null { + if (source === 'proxy') return 'ollama' + if (source === 'lmstudio-proxy') return 'lm-studio' + if (source === 'llamacpp-proxy') return 'llamacpp' + return null } /** @@ -315,11 +325,11 @@ function proxyRelayPrefix(engine: ProxyEngine): string { * * - The `nvpair-ui-broker` is the **only** Electron-spawned binary and is itself the * parent of every broker-owned worker (`ollama-proxy`, `lmstudio-proxy`, - * `nvpair-node-scanner`, `nvpair-node-info`, `nvpair-workload-manager`, + * `llamacpp-proxy`, `nvpair-node-scanner`, `nvpair-node-info`, `nvpair-workload-manager`, * `nvpair-cluster-manager`, `nvpair-node-settings`, `nvpair-manual-nodes`, * `nvpair-engine-manager`, `nvpair-errors`, `nvpair-job-scheduler`). Electron passes their resolved paths to * the broker (see `brokerStartupArgs`) and reaches each through a broker relay: - * `proxy:` / `lmstudio-proxy:` for the two engine proxies, `engine:` for the + * `proxy:` / `lmstudio-proxy:` / `llamacpp-proxy:` for the engine proxies, `engine:` for the * engine-manager, `errors:` for the error pipeline, `node/*` for manual nodes, * `settings/*` and `cluster:` for the rest. Local inference jobs arrive on the * broker's `workloads:subscribe` stream. @@ -827,6 +837,7 @@ class ModularSupervisor { passPath('--node-info-path', 'node-info') passPath('--proxy-path', 'proxy') passPath('--lmstudio-proxy-path', 'lmstudio-proxy') + passPath('--llamacpp-proxy-path', 'llamacpp-proxy') passPath('--workload-manager-path', 'workload-manager') passPath('--cluster-manager-path', 'cluster-manager') passPath('--settings-path', 'node-settings') @@ -878,6 +889,7 @@ class ModularSupervisor { await subscribe('discovery:subscribe', 'subscribe to broker discovery') await subscribe('proxy:subscribe', 'subscribe to broker ollama-proxy relay') await subscribe('lmstudio-proxy:subscribe', 'subscribe to broker lmstudio-proxy relay') + await subscribe('llamacpp-proxy:subscribe', 'subscribe to broker llamacpp-proxy relay') // Engine events are opt-in and replay no baseline — subscribe then hydrate. await subscribe('engine:subscribe', 'subscribe to broker engine relay') await subscribe('workloads:subscribe', 'subscribe to broker workloads stream') @@ -1079,7 +1091,7 @@ class ModularSupervisor { const obj = objectValue(result) if (obj && booleanValue(obj.ready)) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyRelayPrefix(engine), method: 'ready', params: { port: numberValue(obj.port) } }) @@ -1100,7 +1112,7 @@ class ModularSupervisor { if (!obj || !Array.isArray(obj.nodes)) return for (const node of obj.nodes) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyRelayPrefix(engine), method: 'node/discovered', params: node }) @@ -1266,12 +1278,7 @@ class ModularSupervisor { this.scheduleRemoteEngineStatusRefresh() } - const proxyEngine: ProxyEngine | null = - event.source === 'proxy' - ? 'ollama' - : event.source === 'lmstudio-proxy' - ? 'lm-studio' - : null + const proxyEngine = proxyEngineFromRelaySource(event.source) if (proxyEngine && event.method === 'ready') { // A (re)bound proxy starts with an empty manual-node set, so forget // what we think we bridged and re-push the local node if applicable. @@ -1316,9 +1323,16 @@ class ModularSupervisor { this.readinessWaiters.clear() } - /** Rewrite broker `proxy:`/`lmstudio-proxy:` relay frames into proxy-source events. */ + /** Rewrite broker `proxy:`/`lmstudio-proxy:`/`llamacpp-proxy:` relay frames into proxy-source events. */ private normalizeBrokerProxy(notification: JsonRpcNotification): JsonRpcNotification { if (notification.source !== 'broker') return notification + if (notification.method.startsWith('llamacpp-proxy:')) { + return { + source: 'llamacpp-proxy', + method: notification.method.slice('llamacpp-proxy:'.length), + params: notification.params + } + } if (notification.method.startsWith('lmstudio-proxy:')) { return { source: 'lmstudio-proxy', diff --git a/desktop/src/shared/constants/engines.ts b/desktop/src/shared/constants/engines.ts index 3a1472c8..f4a87c09 100644 --- a/desktop/src/shared/constants/engines.ts +++ b/desktop/src/shared/constants/engines.ts @@ -4,28 +4,33 @@ import { EngineType, ModelExpiry } from '@/shared/types/engines' // The engines `nvpair-engine-manager` ships a manifest for, and therefore the -// only ones PAIR can install, run or route to. llama-cpp, whisper-cpp, -// piper-tts, sherpa-onnx-tts and stable-diffusion-cpp were carried here as -// never-enabled placeholders; they were removed with the chat window, which was -// their only in-app consumer. Adding an engine back means shipping its manifest -// first -- an engine row without one renders commands that fail with `-32000`. -export const EngineTypes = ['ollama', 'lm-studio'] as const +// only ones PAIR can install, run or route to. whisper-cpp, piper-tts, +// sherpa-onnx-tts and stable-diffusion-cpp were carried here as never-enabled +// placeholders; they were removed with the chat window, which was their only +// in-app consumer. Adding an engine back means shipping its manifest first -- +// an engine row without one renders commands that fail with `-32000`. +export const EngineTypes = ['ollama', 'lm-studio', 'llamacpp'] as const // Kept as a distinct export so a future engine can ship behind it rather than // appearing the moment its type exists. -export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio'] as const +export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio', 'llamacpp'] as const export const EngineSources = ['bundled', 'detected', 'installed'] as const export const EngineDisplayNames: Record = { ollama: 'Ollama', - 'lm-studio': 'LM Studio' + 'lm-studio': 'LM Studio', + llamacpp: 'llama.cpp' } as const /** Default docs/install URLs for built-in backends. Single source of truth for UI and adapter buildInfo(). */ export const EngineDefaultLinks: Record = { ollama: { docsUrl: 'https://docs.ollama.com/', installUrl: 'https://ollama.com/download' }, - 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' } + 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' }, + llamacpp: { + docsUrl: 'https://github.com/ggml-org/llama.cpp', + installUrl: 'https://github.com/ggml-org/llama.cpp' + } } as const export const ModelItemStatuses = ['idle', 'loading', 'loaded', 'ejecting', 'pulling'] as const diff --git a/desktop/src/shared/constants/modular-binaries.ts b/desktop/src/shared/constants/modular-binaries.ts index 8f24c8d7..cb24f092 100644 --- a/desktop/src/shared/constants/modular-binaries.ts +++ b/desktop/src/shared/constants/modular-binaries.ts @@ -6,6 +6,7 @@ import type { SupportedPlatform } from '@/shared/types/platform' export type ModularProcessName = | 'proxy' | 'lmstudio-proxy' + | 'llamacpp-proxy' | 'broker' | 'node-info' | 'scanner' @@ -75,6 +76,19 @@ export const MODULAR_RUNTIME_BINARIES: ModularRuntimeBinary[] = [ needsFirewallAccess: true, optional: true }, + { + // llama.cpp reverse proxy — the llama.cpp counterpart of `lmstudio-proxy`, + // supervised the same way and relayed under the `llamacpp-proxy:` + // namespace (`--llamacpp-proxy-path`). Like the other inference proxies + // it binds its HTTP listener on all interfaces, so it needs firewall + // access to be reachable. + processName: 'llamacpp-proxy', + baseName: 'llamacpp-proxy', + args: [], + launchOwner: 'broker', + needsFirewallAccess: true, + optional: true + }, { processName: 'scanner', baseName: 'nvpair-node-scanner', diff --git a/desktop/src/shared/types/inference-demo.ts b/desktop/src/shared/types/inference-demo.ts index 246addf1..b9f5bb39 100644 --- a/desktop/src/shared/types/inference-demo.ts +++ b/desktop/src/shared/types/inference-demo.ts @@ -40,8 +40,8 @@ export const DEMO_REQUEST_TIMEOUT_SECONDS = 120 /** * Engines the demo can drive, paired with the proxy each one sits behind. * - * Ports are deliberately absent. The broker owns `ollama-proxy` and - * `lmstudio-proxy` and reports their bound listeners; PAIR never fabricates a + * Ports are deliberately absent. The broker owns `ollama-proxy`, + * `lmstudio-proxy`, and `llamacpp-proxy` and reports their bound listeners; PAIR never fabricates a * port (see the note at the top of `@/shared/constants/modular-runtime`). The * demo resolves each port at start via `getProxyPort()` and skips any engine * whose proxy has not reported. diff --git a/desktop/src/shared/types/inference-dispatcher.ts b/desktop/src/shared/types/inference-dispatcher.ts index 2abe35ae..9c6d7a45 100644 --- a/desktop/src/shared/types/inference-dispatcher.ts +++ b/desktop/src/shared/types/inference-dispatcher.ts @@ -10,7 +10,7 @@ * reads its `--list-models` inventory. */ -export type DispatcherBackend = 'ollama' | 'lmstudio' +export type DispatcherBackend = 'ollama' | 'lmstudio' | 'llamacpp' /** One entry from the binary's `--list-models` JSON inventory. */ export interface DispatcherModel { diff --git a/desktop/src/ui/components/BackendRow/BackendHeader.tsx b/desktop/src/ui/components/BackendRow/BackendHeader.tsx index cceebe31..db0f9698 100644 --- a/desktop/src/ui/components/BackendRow/BackendHeader.tsx +++ b/desktop/src/ui/components/BackendRow/BackendHeader.tsx @@ -114,7 +114,7 @@ export function BackendHeader({ e.stopPropagation() handleCopy() }} - title={`Copy ${backend.displayName} API http://127.0.0.1:${backend.proxyPort}`} + title={`Copy ${backend.displayName} API ${proxyUrl}`} style={{ padding: '2px 6px', minWidth: 'auto' }} aria-label={`Copy ${backend.displayName} API URL`} > diff --git a/desktop/src/ui/components/EngineIcon.tsx b/desktop/src/ui/components/EngineIcon.tsx index 4ee81367..a5f71755 100644 --- a/desktop/src/ui/components/EngineIcon.tsx +++ b/desktop/src/ui/components/EngineIcon.tsx @@ -21,23 +21,39 @@ export default function EngineIcon({ type, size = 32 }: { type: EngineType; size overflow: 'hidden' } - if (type === 'ollama') { - return ( -
- Ollama -
- ) + switch (type) { + case 'ollama': + return ( +
+ Ollama +
+ ) + case 'lm-studio': + imgStyle.objectFit = 'cover' + return ( +
+ LM Studio +
+ ) + case 'llamacpp': + return ( +
+ + cpp + +
+ ) } - - if (type === 'lm-studio') { - imgStyle.objectFit = 'cover' - - return ( -
- LM Studio -
- ) - } - - return null } diff --git a/desktop/src/ui/constants/engine-capabilities.ts b/desktop/src/ui/constants/engine-capabilities.ts index 77e6f2bf..3ae1241f 100644 --- a/desktop/src/ui/constants/engine-capabilities.ts +++ b/desktop/src/ui/constants/engine-capabilities.ts @@ -43,5 +43,18 @@ export const EngineCapabilities: Record = { // server. Deleting therefore interrupts inference and needs a warning. restartsOnModelDelete: true, engineHub: { label: 'LM Studio', url: 'https://lmstudio.ai/models' } + }, + llamacpp: { + hasExpiry: false, + hasEject: false, + hasInstall: [], + hasEnginePort: true, + hasInstallPath: false, + hasProxyWebUI: false, + hasPreferredNode: false, + hasCrashAlert: false, + hasModelSearchOnlyWhenRunning: true, + modelOpsWhenStopped: false, + hasDeleteModel: false } } diff --git a/desktop/src/ui/constants/welcome.ts b/desktop/src/ui/constants/welcome.ts index 4f9fa04a..71755562 100644 --- a/desktop/src/ui/constants/welcome.ts +++ b/desktop/src/ui/constants/welcome.ts @@ -13,7 +13,8 @@ export const WELCOME_STEP_SUB_HEADINGS = ['', 'You can update later by clicking export const WELCOME_ENGINE_DEFAULT_SELECTED: Record = { ollama: true, - 'lm-studio': true + 'lm-studio': true, + llamacpp: false } export function getWelcomeEngineCandidates(os: PlatformDisplayName): EngineType[] { diff --git a/desktop/src/ui/utils/format-model-display-name.ts b/desktop/src/ui/utils/format-model-display-name.ts index e16282ed..dc1cd1e7 100644 --- a/desktop/src/ui/utils/format-model-display-name.ts +++ b/desktop/src/ui/utils/format-model-display-name.ts @@ -40,6 +40,7 @@ export function formatModelDisplayName(name: string, engineType?: string | null) switch (engineType) { case 'lm-studio': + case 'llamacpp': return isHfModel ? formatted : formatLmStudioModelName(formatted) case 'ollama': diff --git a/desktop/src/ui/utils/gateway-inference-paths.ts b/desktop/src/ui/utils/gateway-inference-paths.ts index 688dd9ec..4e67e517 100644 --- a/desktop/src/ui/utils/gateway-inference-paths.ts +++ b/desktop/src/ui/utils/gateway-inference-paths.ts @@ -8,7 +8,13 @@ export function gatewayEndpointDisplayUrl( proxyPort: number, inferenceType: EngineType ): string | null { - void inferenceType if (proxyPort <= 0) return null - return `http://127.0.0.1:${proxyPort}` + const base = `http://127.0.0.1:${proxyPort}` + switch (inferenceType) { + case 'ollama': + return base + case 'lm-studio': + case 'llamacpp': + return `${base}/v1` + } } diff --git a/desktop/tests/modular/llamacpp-engine.test.ts b/desktop/tests/modular/llamacpp-engine.test.ts new file mode 100644 index 00000000..4296c9a7 --- /dev/null +++ b/desktop/tests/modular/llamacpp-engine.test.ts @@ -0,0 +1,29 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from 'vitest' +import { EngineDisplayNames, EnabledEngineTypes, EngineTypes } from '@/shared/constants/engines' +import { EngineCapabilities } from '@/ui/constants/engine-capabilities' +import { MODULAR_RUNTIME_BINARIES } from '@/shared/constants/modular-binaries' + +describe('llamacpp engine', () => { + it('is a enabled engine type', () => { + expect(EngineTypes).toContain('llamacpp') + expect(EnabledEngineTypes).toContain('llamacpp') + expect(EngineDisplayNames.llamacpp).toBe('llama.cpp') + }) + it('cannot install, load, eject, or delete', () => { + const caps = EngineCapabilities.llamacpp + expect(caps.hasInstall).toEqual([]) + expect(caps.hasEject).toBe(false) + expect(caps.hasDeleteModel).toBe(false) + expect(caps.hasEnginePort).toBe(true) + expect(caps.engineHub).toBeUndefined() + }) + it('ships llamacpp-proxy as a broker-owned binary', () => { + const bin = MODULAR_RUNTIME_BINARIES.find(b => b.processName === 'llamacpp-proxy') + expect(bin?.baseName).toBe('llamacpp-proxy') + expect(bin?.launchOwner).toBe('broker') + expect(bin?.optional).toBe(true) + }) +}) diff --git a/docs/architecture.mdx b/docs/architecture.mdx index f7e7ffed..47ae3b44 100644 --- a/docs/architecture.mdx +++ b/docs/architecture.mdx @@ -20,18 +20,22 @@ flowchart TB NodeB["Node"] EngineA1["Engine
Ollama"] EngineA2["Engine
LM Studio"] + EngineA3["Engine
llama.cpp"] EngineB1["Engine
Ollama"] ModelA1["Models
present on this engine"] ModelA2["Models"] + ModelA3["Models"] ModelB1["Models"] Cluster --- NodeA Cluster --- NodeB NodeA --- EngineA1 NodeA --- EngineA2 + NodeA --- EngineA3 NodeB --- EngineB1 EngineA1 --- ModelA1 EngineA2 --- ModelA2 + EngineA3 --- ModelA3 EngineB1 --- ModelB1 ``` @@ -40,9 +44,9 @@ flowchart TB one cluster. - A **node** is one machine. Nodes are peers where each runs the same services, and each can both serve requests and route them elsewhere. -- An **engine** is an inference server on a node, Ollama or LM Studio. A node can - run both, one, or neither, and a node with no running engine is not eligible to - serve. +- An **engine** is an inference server on a node: Ollama, LM Studio, or + llama.cpp. A node can run any combination, or none, and a node with no running + engine is not eligible to serve. - **Models** belong to an engine on a specific node. Nothing is shared. The same model on two nodes is two independent copies and that duplication is what makes the two nodes interchangeable for a request. @@ -314,8 +318,8 @@ ordering reaches a proxy in three steps: The ranking combines *pending work and GPU pressure*. A workload counts as pending while it is queued or running, and it is attributed to the node it was -placed on. Both engines count together, so Ollama load affects LM Studio ordering -and vice versa. +placed on. All engines count together, so Ollama load affects LM Studio and llama.cpp +ordering and vice versa. GPU pressure is deliberately coarse. The scheduler smooths the busiest GPU's utilization, maps it to 0–3 pressure units at 40%, 70%, and 85%, and uses lower @@ -358,6 +362,8 @@ When an inference request contains a non-empty model, only nodes whose current inventory for that engine advertises the model enter the failover list. An empty inventory and an inventory that lists other models are both ineligible. Ollama's implicit `:latest` tag is normalized; LM Studio model IDs match exactly. +llama.cpp is stricter: only models **loaded** on the serving node are eligible, +not ids that are merely present on disk. If no advertised owner is routable, the proxy returns an actionable local `502` without sending the request to an engine. It does not broaden the candidate list @@ -418,20 +424,22 @@ the same node. are the same unit of pending work, so "fewest jobs" is not "least busy." A node running one enormous request looks more idle than a node running two trivial ones. -**Model load state is not considered.** Eligibility asks whether a node *has* the -model, not whether it is already loaded in memory. PAIR knows which models are -loaded, and the interface shows it, but routing does not use it, so a request can -be sent to a node that must cold-load the model while a node holding it warm sits -one place lower in the order. +**Model load state is not considered for Ollama and LM Studio.** Eligibility +asks whether a node *has* the model, not whether it is already loaded in memory. +PAIR knows which models are loaded, and the interface shows it, but those two +engines do not use load state for routing, so a request can be sent to a node +that must cold-load the model while a node holding it warm sits one place lower +in the order. llama.cpp is the exception: routing requires the model already +loaded, so PAIR never triggers a GGUF swap. **Only work PAIR routed contributes to pending counts.** Inference sent straight to an engine's own port is absent from workload events. GPU-heavy external work can still raise pressure, but CPU-only work and queued demand remain invisible. -**Both engines are counted as one pool.** Ollama and LM Studio load is summed, -and maximum GPU pressure applies to the whole node. That is conservative on a -typical single-GPU machine and can underuse a multi-GPU node where the engines -occupy different devices. +**All engines are counted as one pool.** Ollama, LM Studio, and llama.cpp load is +summed, and maximum GPU pressure applies to the whole node. That is conservative +on a typical single-GPU machine and can underuse a multi-GPU node where the +engines occupy different devices. **Every node ranks from its own view, and views lag.** There is no shared schedule. Two nodes dispatching at the same moment can briefly steer work to the @@ -457,18 +465,22 @@ reports from real deployments are more useful than guesses. Refer to `nvpair-engine-manager` owns everything about a local engine except serving inference. It finds the engine, installs it, starts and stops it, and chooses the -port it listens on. +port it listens on. llama.cpp is adopt-only: PAIR probes an already-running +`llama-server` and never installs, starts, stops, or loads GGUFs. ### Finding an Engine PAIR does not assume it installed the engine. Detection checks the manifest's -known install locations for each engine, so an Ollama or LM Studio you installed -yourself is found where it already is. "Installing" an engine that is already -present downloads nothing and reports it as installed. +known install locations for each engine, so an Ollama, LM Studio, or llama.cpp +`llama-server` you installed yourself is found where it already is. "Installing" +an engine that is already present downloads nothing and reports it as installed. +llama.cpp has no install path: PAIR only reports it installed when a binary or a +healthy adopt probe (default port `8082`) is present. Starting is similarly deferential. If something is already serving the engine's port, PAIR **adopts** that instance instead of spawning a second copy, and reports -it as running even though it did not start it. +it as running even though it did not start it. For llama.cpp, adopt is the only +start path. Adoption is a real distinction, not a label. PAIR cannot stop or move a process it did not start, so it refuses operations that need process ownership rather @@ -484,7 +496,8 @@ listened somewhere else, every tool would need reconfiguring to gain anything, s PAIR inverts it: the **proxy** takes the port the engine would normally use, and the engine moves behind it — Ollama to `11435` and upwards, LM Studio to `1235` and upwards. Existing clients keep working untouched and transparently gain the -cluster. +cluster. llama.cpp is different: the proxy listens on `8084` and `llama-server` +stays on its adopt port (default `8082`). PAIR never binds the adopt port. This is also what makes the engine unreachable from outside. Engines PAIR starts bind to loopback, so the only network-facing listener is the proxy, which is where @@ -522,6 +535,7 @@ A default installation listens on these ports: | --- | --- | | `11434` | Ollama-compatible proxy (Ollama itself moves to `11435`+) | | `1234` | OpenAI-compatible proxy (LM Studio moves to `1235`+) | +| `8084` | OpenAI-compatible proxy for llama.cpp (`llama-server` stays on its adopt port, default `8082`) | | `14318` | Node hardware and model inventory | | `14319` | Service-error synchronization between nodes | | `14320` | Workload propagation between nodes | @@ -550,8 +564,8 @@ browsing themselves. The scanner advertises one `_nvpair-node._tcp` multicast DNS (mDNS) record for each host, and that record carries two kinds of content: -- The ports its sibling services registered: node-info, both proxies, errors, - workloads, cluster manager, and engine manager +- The ports its sibling services registered: node-info, the inference proxies, + errors, workloads, cluster manager, and engine manager - The node's identity: `uuid=`, `cluster-uuid=` after clustering, and where to reach it — `ip=` for the address the node ranks first, and `ips=` for the whole ranked list @@ -707,7 +721,7 @@ desktop build compiles the sibling tree directly. | Build In | Output | Contents | | --- | --- | --- | | `desktop/` | `desktop/cli-bin/` | What the app supervises, one OS/arch | -| `services/` | `services/build/bin/` | All 13 executables, for standalone use | +| `services/` | `services/build/bin/` | All 14 executables, for standalone use | `desktop/scripts/build-modular-binaries.ts` compiles the runtime inventory for a selected target and writes a manifest recording the source identity, versions, diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index 81bf01d9..290b78d9 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -6,10 +6,11 @@ SPDX-License-Identifier: Apache-2.0 # Managing Engines in NVIDIA Personal AI Router An **engine** is the local inference runtime Personal AI Router (PAIR) uses to -run models. Today that means Ollama or LM Studio on a given machine. PAIR can -install and run those engines for you, or work with a copy you already have. -This page explains what you can expect when you install, start, stop, update, or -remove an engine. +run models. Today that means Ollama, LM Studio, or llama.cpp on a given machine. +PAIR can install and run Ollama and LM Studio for you, or work with a copy you +already have. llama.cpp is adopt-only: PAIR never installs it, never starts or +stops `llama-server`, and never loads GGUFs. This page explains what you can +expect when you install, start, stop, update, or remove an engine. For first-time setup, refer to [Getting started](getting-started.mdx). If something does not start, refer to [Troubleshooting](troubleshooting.mdx). @@ -77,6 +78,12 @@ When you install an engine, consider the following: - If Ollama or LM Studio is already running on the machine, PAIR can **adopt** that install instead of downloading another copy. Adoption helps when the usual engine port is already in use. +- llama.cpp is always adopted. PAIR probes an already-running `llama-server` + (default port `8082`) and does not download, spawn, or load anything. There is + no install, update, uninstall, load, or eject action for it. Applications talk + to PAIR at `http://127.0.0.1:8084/v1`. Routing requires the requested model to + already be **loaded** on the serving node. Cluster peers need this PAIR fork to + advertise llama.cpp. To download models: @@ -99,7 +106,8 @@ engine. An engine must be **running** before **Endpoints** or your applications can use its models. If **Endpoints** says **No engines are running**, start an engine -first. +first. For llama.cpp, start `llama-server` yourself; PAIR's switch only reports +whether the adopt probe succeeded. ## Updating and Uninstalling an Engine @@ -128,6 +136,9 @@ controlled: the engine in its own application first if you want PAIR to manage it fully. - **LM Studio** publishes an official stop command, so PAIR can stop an adopted instance that way and restart it on the port you chose. +- **llama.cpp** is never started or stopped by PAIR. Change its port on + `llama-server` itself; PAIR's adopt probe follows the configured probe port + (default `8082`). Refer to [Engines and Ports](architecture.mdx#engines-and-ports) in the architecture guide. @@ -181,7 +192,8 @@ working with **models** (download, load, unload, and delete): | Remove a downloaded model | **Delete**, where the engine supports it | **Eject** removes a model from memory only. The download stays, and **Load** -brings it back without fetching it again. +brings it back without fetching it again. llama.cpp has no Load, Eject, or +Delete in PAIR: load GGUFs in `llama-server` itself. ![An engine's model list showing the Load, Eject, and Delete actions against a downloaded model.](assets/onboarding/engine-lifecycle/04-model-actions.png) diff --git a/docs/getting-started.mdx b/docs/getting-started.mdx index cb7e7e34..dad91684 100644 --- a/docs/getting-started.mdx +++ b/docs/getting-started.mdx @@ -33,7 +33,10 @@ inference; two or more on the same local network let you try pairing and routing Nothing else has to be in place first: - **Engines.** PAIR installs and starts Ollama or LM Studio for you in step 4. If - an engine is already installed, PAIR detects and uses it instead. + an engine is already installed, PAIR detects and uses it instead. llama.cpp is + adopt-only: start `llama-server` yourself (PAIR probes port `8082` by default) + and PAIR exposes `http://127.0.0.1:8084/v1`. It does not install llama.cpp or + load GGUFs. Cluster peers need this PAIR fork to route llama.cpp. - **Models.** PAIR downloads models for you in step 4. A request needs only one eligible node, so a single node holding the model is enough. Prepare the same model on additional nodes when you want any of them to be able to serve it. @@ -157,8 +160,10 @@ For each node that should serve a model: ![An engine's model list on the node card, with one model pulling and its progress shown.](assets/onboarding/getting-started/07-add-model.png) A node is eligible for a request only when it is online, a compatible engine is -running, and the requested model is available there. To test routing across -multiple nodes, prepare the same model on each of those nodes. +running, and the requested model is available there. For llama.cpp the model +must already be **loaded** on that node's `llama-server`; PAIR does not load +GGUFs. To test routing across multiple nodes, prepare the same model on each of +those nodes. Refer to [Managing engines](engine-lifecycle.mdx) for details about: @@ -191,6 +196,8 @@ engine would normally listen on: - `11434` for Ollama. - `1234` for LM Studio. +- `8084` for llama.cpp (`llama-server` itself stays on its adopt port, default + `8082`). Tools already pointed at those ports keep working without reconfiguration, and the engine moves to the next free port. @@ -219,15 +226,16 @@ The engine therefore has to understand the style you send. | --- | --- | --- | | Ollama | Works | Works | | LM Studio | Works | Not available | +| llama.cpp | Works | Not available | -**If you are unsure, send the OpenAI-style request.** It works with either engine, -and most tools and SDKs use it. Use the Ollama style +**If you are unsure, send the OpenAI-style request.** It works with any of these +engines, and most tools and SDKs use it. Use the Ollama style only when something you already have is written against Ollama's `/api/...` API. The choice does not affect routing. Both styles are routed across your cluster the same way, and neither one changes which node serves the request. -### OpenAI-Style Request (Works With Either Engine) +### OpenAI-Style Request (Works With Any Engine) ```bash curl /v1/chat/completions \ @@ -290,11 +298,14 @@ free port. | --- | --- | | Ollama-compatible proxy | `11434` | | LM Studio / OpenAI-compatible proxy | `1234` | +| llama.cpp / OpenAI-compatible proxy | `8084` | When PAIR takes one of those ports, the engine behind it moves: - Ollama moves to `11435` or higher. - LM Studio moves to `1235` or higher. +- llama.cpp does not move: `llama-server` stays on its adopt port (default + `8082`). PAIR never binds that port. **Endpoints** is the authoritative source for the URL to use. @@ -408,6 +419,7 @@ What that base URL serves depends on the engine behind it: | --- | --- | --- | | Ollama | `http://127.0.0.1:11434` | Ollama's own API — `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags` — and the OpenAI-compatible `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models` | | LM Studio | `http://127.0.0.1:1234` | The OpenAI-compatible paths only | +| llama.cpp | `http://127.0.0.1:8084/v1` | The OpenAI-compatible paths only. Completions require the model **loaded** on the serving node; a catalog-only id returns `502`. | The distinction matters when you fill in an application's settings. A client written against the OpenAI API usually wants the `/v1` included, as in @@ -555,8 +567,10 @@ downloading again. Updating keeps your settings, logs, cluster identity, and cluster membership, so a node stays paired across an update. Model weights are untouched as well, because they belong to the engine rather than to PAIR. Updating PAIR does not -update Ollama or LM Studio — engines are updated separately from **Engine -settings**, described in [Managing engines](engine-lifecycle.mdx). +update Ollama, LM Studio, or llama.cpp — Ollama and LM Studio are updated +separately from **Engine settings**, described in +[Managing engines](engine-lifecycle.mdx). PAIR does not update llama.cpp. PAIR does not +update llama.cpp. You update each machine from that machine. PAIR never updates a peer for you, and it cannot be driven remotely, so a cluster is updated one node at a time. diff --git a/docs/overview.mdx b/docs/overview.mdx index 95ea262c..bbb08a73 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -35,9 +35,10 @@ These terms have specific meanings in PAIR: is no server, controller, or primary node. - **Cluster** — the set of nodes you have paired together. A node belongs to at most one cluster, and it must leave before it can join another. -- **Engine** — the local inference server that runs models: Ollama or LM Studio. - PAIR can install, start, stop, and update an engine, or adopt one you already - run yourself. +- **Engine** — the local inference server that runs models: Ollama, LM Studio, + or llama.cpp. PAIR can install, start, stop, and update Ollama and LM Studio, + or adopt an engine you already run yourself. llama.cpp is adopt-only: PAIR + never installs it or loads GGUFs. - **Model** — what you prepare on each node. Nodes do not share models, so a node can serve a request only for a model it already holds. Preparing the same model on several nodes is what makes those nodes interchangeable. @@ -121,7 +122,8 @@ For the trust boundaries in detail, refer to - A local endpoint for compatible AI applications and development tools. - LAN discovery plus manually configured nodes. -- Ollama-compatible and LM Studio/OpenAI-compatible routing proxies. +- Ollama-compatible and OpenAI-compatible (LM Studio and llama.cpp) routing + proxies. - Pairing and cluster membership managed by the background services. - Model-aware, workload-informed routing of independent requests. - Encrypted routing between machines: a request sent to another node travels over @@ -163,10 +165,11 @@ It stages the generated binaries in `desktop/cli-bin/`. ### `services/` -The Go tree contains 13 build outputs: +The Go tree contains 14 build outputs: - `nvpair-ui-broker`: service entry point, worker supervisor, and JSON-RPC API -- `ollama-proxy` and `lmstudio-proxy`: compatible inference proxies +- `ollama-proxy`, `lmstudio-proxy`, and `llamacpp-proxy`: compatible inference + proxies - `nvpair-node-scanner` and `nvpair-node-info`: discovery and host telemetry - `nvpair-manual-nodes`: user-specified nodes - `nvpair-engine-manager`: local engine and model lifecycle diff --git a/docs/superpowers/plans/2026-09-06-llamacpp-engine.md b/docs/superpowers/plans/2026-09-06-llamacpp-engine.md new file mode 100644 index 00000000..fac7a5c6 --- /dev/null +++ b/docs/superpowers/plans/2026-09-06-llamacpp-engine.md @@ -0,0 +1,1186 @@ +# llama.cpp Engine Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a first-class `llamacpp` engine to PAIR that adopts an already-running `llama-server`, exposes OpenAI `/v1` on PAIR port 8084, lists the full catalog, and routes only to nodes that already have the requested model loaded. + +**Architecture:** Clone `lmstudio-proxy` into `llamacpp-proxy` (discovery key `lc`, facade 8084). Engine-manager gains `runtime.mode: "adopt"` and a `llamacpp.json` manifest (default probe 8082). Routing uses `LoadedByEngine`, never catalog ids, so PAIR cannot trigger a GGUF swap. This node can still forward to paired Mac/laptop PAIR nodes while ComfyUI holds the local GPU. + +**Tech Stack:** Go 1.25 services, Electron/TypeScript desktop, newline-delimited JSON-RPC, existing `nvpair-shared` discovery. + +**Spec:** `docs/superpowers/specs/2026-09-05-llamacpp-engine-design.md` + +**Constraints:** No live GGUF loads. No PAIR installer. Do not bind 8082. Do not spawn or kill llama-server. Commits use `git commit -s`. Every new file gets the two-line SPDX header. + +--- + +## File map + +| Path | Role | +|---|---| +| `services/shared/noderec/noderec.go` | `ServiceLlamaCpp = "lc"`; `EngineLoadedModels` | +| `services/nvpair-engine-manager/models.go` | Dotted `status.value` match | +| `services/nvpair-engine-manager/registry.go` | Validate `mode: "adopt"` | +| `services/nvpair-engine-manager/lifecycle.go` | Adopt-only Start (no spawn) | +| `services/nvpair-engine-manager/setport.go` | Adopt-mode port = probe override, no rebind | +| `services/nvpair-engine-manager/manifests/llamacpp.json` | Engine manifest | +| `services/llamacpp-proxy/` | New OpenAI proxy (clone of lmstudio-proxy) | +| `services/nvpair-ui-broker/` | Spawn, advertise `lc`, relay `llamacpp-proxy:` | +| `services/nvpair-job-scheduler/schedule.go` | `schedulerEngines` += `llamacpp` | +| `services/nvpair-manual-nodes/` | Probe llama.cpp on adopt port | +| `desktop/src/shared/constants/engines.ts` | Engine type `llamacpp` | +| `desktop/src/shared/constants/modular-binaries.ts` | `llamacpp-proxy` binary | +| `desktop/src/electron/service-bridge/modular-supervisor.ts` | `--llamacpp-proxy-path` | +| `desktop/src/electron/service-bridge/modular-state.ts` | Proxy source mapping | +| `desktop/src/ui/constants/engine-capabilities.ts` | No install/load/delete | +| `services/nvpair-tui/ui/proxies.go` | Third proxy row | +| `scripts/inference-dispatcher/config.go` | `--backend llamacpp` | +| `services/versions.json`, `services/build.bat`, `services/build.sh` | Build the 14th binary | + +Do **not** copy LM Studio's managed-facade port steal (`lmstudioport.go`). llama.cpp must never take 8082. + +--- + +### Task 1: Discovery key `lc` and loaded-model helper + +**Files:** +- Modify: `services/shared/noderec/noderec.go` +- Modify: `services/shared/noderec/noderec_test.go` + +- [ ] **Step 1: Write the failing test** + +Add to `services/shared/noderec/noderec_test.go` after `TestEngineModels`: + +```go +func TestEngineLoadedModels(t *testing.T) { + n := DirectoryNode{ + Models: []string{"catalog-a", "catalog-b"}, + ModelsByEngine: map[string][]string{ + "llamacpp": {"catalog-a", "catalog-b"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {"catalog-a"}, + }, + } + got := n.EngineLoadedModels("llamacpp") + if !reflect.DeepEqual(got, []string{"catalog-a"}) { + t.Fatalf("EngineLoadedModels(llamacpp) = %v, want [catalog-a]", got) + } + if got := n.EngineLoadedModels("ollama"); len(got) != 0 { + t.Fatalf("EngineLoadedModels(missing) = %v, want empty", got) + } + legacy := DirectoryNode{Models: []string{"catalog-a"}} + if got := legacy.EngineLoadedModels("llamacpp"); len(got) != 0 { + t.Fatalf("nil LoadedByEngine must not fall back to catalog, got %v", got) + } +} + +func TestServiceLlamaCppKey(t *testing.T) { + if ServiceLlamaCpp != "lc" { + t.Fatalf("ServiceLlamaCpp = %q, want lc", ServiceLlamaCpp) + } + found := false + for _, k := range serviceKeyOrder { + if k == ServiceLlamaCpp { + found = true + break + } + } + if !found { + t.Fatal("ServiceLlamaCpp missing from serviceKeyOrder") + } + if ServiceLlamaCpp.Transport() != TransportPlain { + t.Fatal("lc transport must be TransportPlain (same as ol/lm)") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run from `services/shared`: + +``` +go test ./noderec -count=1 -run "TestEngineLoadedModels|TestServiceLlamaCppKey" +``` + +Expected: FAIL (`EngineLoadedModels` undefined and/or `ServiceLlamaCpp` undefined). + +- [ ] **Step 3: Write minimal implementation** + +In `services/shared/noderec/noderec.go` add after `ServiceLMStudio`: + +```go + ServiceLlamaCpp ServiceKey = "lc" +``` + +Add `ServiceLlamaCpp` to `serviceKeyOrder` immediately after `ServiceLMStudio`. + +Add after `EngineModels`: + +```go +// EngineLoadedModels returns models currently resident in memory for one +// engine. It never falls back to Models or ModelsByEngine: a missing +// LoadedByEngine report means nothing is loaded, so a router cannot treat +// catalog ids as eligible. +func (n DirectoryNode) EngineLoadedModels(engine string) []string { + if n.LoadedByEngine == nil { + return nil + } + return n.LoadedByEngine[engine] +} +``` + +- [ ] **Step 4: Run the tests and make sure they pass** + +``` +go test ./noderec -count=1 -run "TestEngineLoadedModels|TestServiceLlamaCppKey|TestEngineModels" +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/shared/noderec/noderec.go services/shared/noderec/noderec_test.go +git commit -s -m "feat(noderec): add lc service key and EngineLoadedModels" +``` + +--- + +### Task 2: Dotted ResultMatch for `status.value` + +**Files:** +- Modify: `services/nvpair-engine-manager/models.go` +- Modify: `services/nvpair-engine-manager/models_test.go` + +llama.cpp `/v1/models` rows look like `{"id":"...","status":{"value":"loaded"}}`. Existing `ResultMatch.In` only reads a top-level string field. + +- [ ] **Step 1: Write the failing test** + +Append a case to the `extractStrings` table in `models_test.go`: + +```go + { + name: "dotted status.value keeps only loaded llama.cpp rows", + raw: `{"data":[{"id":"a","status":{"value":"loaded"}},{"id":"b","status":{"value":"unloaded"}},{"id":"c"},{"id":"d","status":{"value":"loaded"}}]}`, + spec: &ActionResult{Array: "data", Field: "id", Match: &ResultMatch{Field: "status.value", In: []string{"loaded"}}}, + want: []string{"a", "d"}, + }, + { + name: "dotted match: missing status is unloaded", + raw: `{"data":[{"id":"a","status":{"value":"loaded"}},{"id":"b"}]}`, + spec: &ActionResult{Array: "data", Field: "id", Match: &ResultMatch{Field: "status.value", In: []string{"loaded"}}}, + want: []string{"a"}, + }, +``` + +- [ ] **Step 2: Run test to verify it fails** + +``` +cd services/nvpair-engine-manager +go test -count=1 -run TestExtractStrings +``` + +Expected: FAIL on the dotted `status.value` cases (field lookup misses nested object). + +- [ ] **Step 3: Write minimal implementation** + +Replace `matchRow` in `models.go` so `Field` may be a dotted path. Keep nonempty and `In` behavior: + +```go +func lookupField(el map[string]json.RawMessage, field string) (json.RawMessage, bool) { + obj := el + parts := strings.Split(field, ".") + for i, part := range parts { + fv, ok := obj[part] + if !ok { + return nil, false + } + if i == len(parts)-1 { + return fv, true + } + next := map[string]json.RawMessage{} + if err := json.Unmarshal(fv, &next); err != nil { + return nil, false + } + obj = next + } + return nil, false +} + +func matchRow(el map[string]json.RawMessage, m *ResultMatch) bool { + fv, ok := lookupField(el, m.Field) + if !ok { + return false + } + if m.Nonempty { + var arr []json.RawMessage + if err := json.Unmarshal(fv, &arr); err != nil { + return false + } + return len(arr) > 0 + } + var s string + if err := json.Unmarshal(fv, &s); err != nil { + return false + } + for _, want := range m.In { + if s == want { + return true + } + } + return false +} +``` + +Add `"strings"` to `models.go` imports if missing. + +- [ ] **Step 4: Run the tests and make sure they pass** + +``` +go test -count=1 -run TestExtractStrings +``` + +Expected: PASS, including existing LM Studio nonempty cases. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-engine-manager/models.go services/nvpair-engine-manager/models_test.go +git commit -s -m "feat(engine-manager): match nested JSON fields like status.value" +``` + +--- + +### Task 3: Adopt-only runtime mode + +**Files:** +- Modify: `services/nvpair-engine-manager/registry.go` (`modeOrDefault` validation) +- Modify: `services/nvpair-engine-manager/lifecycle.go` (`doStart`) +- Modify: `services/nvpair-engine-manager/registry_test.go` +- Create: `services/nvpair-engine-manager/adopt_test.go` + +- [ ] **Step 1: Write the failing tests** + +In `registry_test.go` add a validation case: `runtime.mode: "adopt"` with `ready` set and empty `bin`/`start` must load; `adopt` with a `bin` must fail; `adopt` without `ready` must fail. + +In `adopt_test.go`: + +```go +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +func TestAdoptModeStartsWithoutSpawning(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/models" { + http.NotFound(w, r) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "object": "list", + "data": []map[string]any{{"id": "m1", "status": map[string]string{"value": "unloaded"}}}, + }) + })) + defer srv.Close() + u, _ := url.Parse(srv.URL) + port, _ := strconv.Atoi(u.Port()) + + m := adoptManifest(port) + ex := newTestExecutor(t, m) + if err := ex.Start(context.Background(), "llamacpp"); err != nil { + t.Fatalf("adopt start: %v", err) + } + st, err := ex.Status("llamacpp") + if err != nil { + t.Fatal(err) + } + if !st.Running || !st.Healthy || st.Port != port { + t.Fatalf("status = %+v", st) + } + state, _ := ex.state("llamacpp") + state.mu.Lock() + proc := state.proc + state.mu.Unlock() + if proc != nil { + t.Fatal("adopt mode spawned a process") + } +} + +func TestAdoptModeDoesNotSpawnWhenDown(t *testing.T) { + m := adoptManifest(1) // nothing listens on :1 + ex := newTestExecutor(t, m) + err := ex.Start(context.Background(), "llamacpp") + if err == nil { + t.Fatal("expected start error when probe fails") + } + st, _ := ex.Status("llamacpp") + if st.Running { + t.Fatal("must not mark running") + } +} +``` + +`newTestExecutor(t, m *Manifest)` already exists in `executor_test.go` (same package `main`). `adoptManifest(port int) *Manifest` must return `Engine: "llamacpp"`, `DisplayName: "llama.cpp"`, platforms for `runtime.GOOS/runtime.GOARCH` with `Runtime.Mode: "adopt"`, `Runtime.Port: port`, `Runtime.Ready.HTTP: "http://127.0.0.1:{port}/v1/models"`. + +- [ ] **Step 2: Run test to verify it fails** + +``` +go test -count=1 -run "TestAdoptMode" +``` + +Expected: FAIL (mode `adopt` rejected by validate, or Start tries to spawn). + +- [ ] **Step 3: Write minimal implementation** + +`registry.go` `Platform.validate`: + +```go + case "adopt": + if strings.TrimSpace(p.Runtime.Bin) != "" { + return fmt.Errorf("platform %q: runtime.bin is forbidden in adopt mode", key) + } + if len(p.Runtime.Start) != 0 { + return fmt.Errorf("platform %q: runtime.start is forbidden in adopt mode", key) + } + if p.Runtime.Ready == nil { + return fmt.Errorf("platform %q: runtime.ready is required in adopt mode", key) + } +``` + +`lifecycle.go` `doStart`, after the `presence.Identified` adopt-success block and before `bringUp`: + +```go + if rt.modeOrDefault() == "adopt" { + return fmt.Errorf("cannot start engine %q: nothing is serving on port %d (PAIR will not launch llama-server)", engine, port) + } +``` + +Do not call `bringUp` for adopt mode. + +- [ ] **Step 4: Run the tests and make sure they pass** + +``` +go test -count=1 -run "TestAdoptMode|TestManifestValidate" +go test -count=1 +``` + +Expected: new tests PASS; existing executor tests still PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-engine-manager/registry.go services/nvpair-engine-manager/lifecycle.go services/nvpair-engine-manager/registry_test.go services/nvpair-engine-manager/adopt_test.go +git commit -s -m "feat(engine-manager): add adopt-only runtime mode" +``` + +--- + +### Task 4: Adopt-mode set-port probes only + +**Files:** +- Modify: `services/nvpair-engine-manager/setport.go` +- Modify: `services/nvpair-engine-manager/setport_test.go` + +Spec: changing the adopt port persists and changes which loopback port PAIR probes. It must not stop llama-server or spawn on the new port. + +- [ ] **Step 1: Write the failing test** + +In `setport_test.go` (create if missing; same package): + +```go +func TestSetPortAdoptModeDoesNotStopListener(t *testing.T) { + hits := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits++ + w.WriteHeader(200) + _, _ = w.Write([]byte(`{"object":"list","data":[]}`)) + })) + defer srv.Close() + u, _ := url.Parse(srv.URL) + oldPort, _ := strconv.Atoi(u.Port()) + + ex := newTestExecutor(t, adoptManifest(oldPort)) + if err := ex.Start(context.Background(), "llamacpp"); err != nil { + t.Fatal(err) + } + before := hits + _, err := ex.SetPort(context.Background(), "llamacpp", oldPort+1) + if err != nil { + t.Fatalf("set-port adopt: %v", err) + } + if !srvOpen(srv) { + t.Fatal("set-port stopped the foreign listener") + } + st, _ := ex.Status("llamacpp") + if st.Port != oldPort+1 { + t.Fatalf("port = %d, want %d", st.Port, oldPort+1) + } + if hits < before { + t.Fatal("listener should still be reachable on the old port") + } +} +``` + +`srvOpen` can be a 1-line `http.Get(srv.URL + "/v1/models")` that expects no connection refused. + +- [ ] **Step 2: Run test to verify it fails** + +``` +go test -count=1 -run TestSetPortAdoptModeDoesNotStopListener +``` + +Expected: FAIL because current `SetPort` refuses adopted engines or stops them. + +- [ ] **Step 3: Write minimal implementation** + +At the top of `SetPort`, after loading `st` and `wasRunning`/`adopted`: + +```go + if st.plat.Runtime.modeOrDefault() == "adopt" { + if err := e.persistPort(engine, port); err != nil { + return EngineStatus{}, err + } + st.mu.Lock() + st.port = port + if st.plat != nil { + st.plat.Runtime.Port = port + } + st.mu.Unlock() + if wasRunning { + _ = e.doStart(ctx, st, engine, startOpts{}) + } + return e.snapshot(engine, st), nil + } +``` + +`doStart` in adopt mode only re-probes; it must not call `doStop` first. Skip the existing “refuse adopted process-mode” branch when mode is `adopt`. + +`snapshot(engine, st)` is the existing SetPort return helper in `setport.go`. + +- [ ] **Step 4: Run the tests and make sure they pass** + +``` +go test -count=1 -run "TestSetPort" +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-engine-manager/setport.go services/nvpair-engine-manager/setport_test.go +git commit -s -m "feat(engine-manager): adopt-mode set-port only changes the probe" +``` + +--- + +### Task 5: `llamacpp.json` manifest + +**Files:** +- Create: `services/nvpair-engine-manager/manifests/llamacpp.json` +- Modify: `services/nvpair-engine-manager/registry_test.go` (bundled manifest load already walks `manifests/`) + +- [ ] **Step 1: Write the failing test** + +If bundled-manifest tests already load every JSON in `manifests/`, adding the file is enough. Add an explicit test: + +```go +func TestLlamaCppManifestIsAdoptOnly(t *testing.T) { + raw, err := os.ReadFile("manifests/llamacpp.json") + if err != nil { + t.Fatal(err) + } + var m Manifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + if m.Engine != "llamacpp" || m.DisplayName != "llama.cpp" { + t.Fatalf("identity = %s %s", m.Engine, m.DisplayName) + } + if _, ok := m.Actions["pull_model"]; ok { + t.Fatal("pull_model must not exist") + } + if _, ok := m.Actions["load_model"]; ok { + t.Fatal("load_model must not exist") + } + for key, p := range m.Platforms { + if p.Runtime.modeOrDefault() != "adopt" { + t.Fatalf("%s mode = %q", key, p.Runtime.Mode) + } + if p.Runtime.Port != 8082 { + t.Fatalf("%s port = %d", key, p.Runtime.Port) + } + if p.Runtime.Ready == nil || !strings.Contains(p.Runtime.Ready.HTTP, "/v1/models") { + t.Fatalf("%s ready probe must be /v1/models", key) + } + loaded := m.Actions["loaded_models"] + if loaded.Result == nil || loaded.Result.Match == nil || loaded.Result.Match.Field != "status.value" { + t.Fatal("loaded_models must match status.value") + } + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +``` +go test -count=1 -run TestLlamaCppManifestIsAdoptOnly +``` + +Expected: FAIL (file missing). + +- [ ] **Step 3: Write the manifest** + +Create `services/nvpair-engine-manager/manifests/llamacpp.json`: + +```json +{ + "engine": "llamacpp", + "display_name": "llama.cpp", + "manifest_version": 1, + "runtime": { + "mode": "adopt", + "bind": "127.0.0.1", + "port": 8082, + "ready": { "http": "http://127.0.0.1:{port}/v1/models", "status": 200, "timeout_s": 5 }, + "health": { "http": "http://127.0.0.1:{port}/v1/models", "status": 200, "interval_s": 5 } + }, + "platforms": { + "windows/amd64": { + "detect": [ + "%LOCALAPPDATA%\\Microsoft\\WinGet\\Packages\\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\\llama-server.exe", + "C:\\Users\\P-DLE\\Desktop\\AI Playground\\vendor\\llama.cpp-official\\bin\\llama-server.exe" + ] + }, + "windows/arm64": { + "detect": [ + "%LOCALAPPDATA%\\Microsoft\\WinGet\\Packages\\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\\llama-server.exe" + ] + }, + "darwin/arm64": { "detect": ["/opt/homebrew/bin/llama-server", "/usr/local/bin/llama-server"] }, + "darwin/amd64": { "detect": ["/usr/local/bin/llama-server"] }, + "linux/amd64": { "detect": ["/usr/local/bin/llama-server", "/usr/bin/llama-server"] }, + "linux/arm64": { "detect": ["/usr/local/bin/llama-server", "/usr/bin/llama-server"] } + }, + "actions": { + "list_models": { + "description": "List every model id advertised by llama-server.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "loaded_models": { + "description": "List model ids currently loaded in memory.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { + "array": "data", + "field": "id", + "match": { "field": "status.value", "in": ["loaded"] } + } + } + } +} +``` + +Each platform still needs a `runtime` object after merge. Confirm `registry.go` deep-merges top-level `runtime` onto platforms (Ollama does this). If a platform with only `detect` fails validate, copy the top-level runtime into each platform block. + +- [ ] **Step 4: Run the tests and make sure they pass** + +``` +go test -count=1 -run "TestLlamaCppManifest|TestLoadBundled" +go test -count=1 +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-engine-manager/manifests/llamacpp.json services/nvpair-engine-manager/registry_test.go +git commit -s -m "feat(engine-manager): add adopt-only llamacpp manifest" +``` + +--- + +### Task 6: `llamacpp-proxy` (clone + loaded-only eligibility) + +**Files:** +- Create: `services/llamacpp-proxy/` (copy of `services/lmstudio-proxy/`) +- Modify after copy: `go.mod`, `portstore.go`, `discovery.go`, `proxy.go` (`subscribedToNode`), `README.md` +- Create: `services/llamacpp-proxy/loaded_eligibility_test.go` + +- [ ] **Step 1: Copy the module** + +From `services/`: + +``` +Copy-Item -Recurse lmstudio-proxy llamacpp-proxy +``` + +In `llamacpp-proxy/go.mod` set `module llamacpp-proxy`. + +In `portstore.go`: + +```go +const proxyPortFile = "llamacpp-proxy-port.json" +const defaultProxyPort = 8084 +``` + +Remove `legacyDefaultProxyPort` special-case for 1235; `chooseStartupPort` should honor persisted 8084 and ignore nothing except invalid values. + +Replace every `noderec.ServiceLMStudio` with `noderec.ServiceLlamaCpp`. +Replace engine string `"lmstudio"` with `"llamacpp"` in workload tags and `EngineModels` calls. +Replace JSON-RPC log names `lmstudio-proxy` with `llamacpp-proxy`. + +- [ ] **Step 2: Write the failing eligibility test** + +`loaded_eligibility_test.go`: + +```go +func TestSubscribedToNodeUsesLoadedNotCatalog(t *testing.T) { + n := noderec.DirectoryNode{ + Name: "box", + HostUUID: "uuid-1", + IP: "127.0.0.1", + Services: map[noderec.ServiceKey]noderec.ServiceStatus{ + noderec.ServiceLlamaCpp: {Port: 8084}, + }, + ModelsByEngine: map[string][]string{ + "llamacpp": {"ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp", "other"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {"ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp"}, + }, + } + node, ok := subscribedToNode(n) + if !ok { + t.Fatal("expected lc node") + } + if !nodeAdvertisesModel(node, "ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp") { + t.Fatal("loaded id must be eligible") + } + if nodeAdvertisesModel(node, "other") { + t.Fatal("catalog-only id must not be eligible") + } +} +``` + +- [ ] **Step 3: Run test to verify it fails** + +``` +cd services/llamacpp-proxy +go test -count=1 -run TestSubscribedToNodeUsesLoadedNotCatalog +``` + +Expected: FAIL because `subscribedToNode` still copies `EngineModels` (catalog). + +- [ ] **Step 4: Fix `subscribedToNode`** + +In `proxy.go` `subscribedToNode`: + +```go + svc, ok := n.Services[noderec.ServiceLlamaCpp] + // ... + Models: append([]string(nil), n.EngineLoadedModels("llamacpp")...), +``` + +- [ ] **Step 5: Add a proxy test that catalog-only chat is 502 with zero upstream hits** + +Follow `proxy_test.go` patterns: httptest backend that increments `hits` on `POST /v1/chat/completions`. Seed a node whose `Models` is empty (nothing loaded) but whose backend `/v1/models` would list ids. POST chat with `{"model":"other","messages":[]}`. Expect HTTP 502 and `hits == 0`. + +If existing tests assume `EngineModels`, update those in the clone to use loaded lists. + +- [ ] **Step 6: Run all proxy tests** + +``` +go test ./... -count=1 +``` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +``` +git add services/llamacpp-proxy +git commit -s -m "feat: add llamacpp-proxy with loaded-only routing" +``` + +--- + +### Task 7: Broker spawn, advertise, relay + +**Files:** +- Create: `services/nvpair-ui-broker/llamacppproxy.go` (mirror `lmstudioproxy.go`, no facade steal) +- Modify: `services/nvpair-ui-broker/advertiser.go` (add `runAutoAdvertiseLlamaCpp`) +- Modify: `services/nvpair-ui-broker/broker.go` (path flag, spawn, shutdown, `llamacpp-proxy:` relay) +- Modify: `services/nvpair-ui-broker/advertiser_test.go` +- Create: `services/nvpair-ui-broker/llamacpp_advertise_test.go` + +Do **not** port `lmstudioport.go` managed facade. + +- [ ] **Step 1: Write the failing advertise test** + +```go +func TestReconcileAdvertiseLlamaCppRegistersProxyPort(t *testing.T) { + // Fake engine-manager status: llamacpp running on 8082. + // Fake proxy listen port 8084. + // Expect registerService(lc, 8084) and set-local-backend port 8082. +} +``` + +Mirror `advertiser_test.go` LM Studio cases. Also test: engine down → unregister `lc`; equal proxy/engine ports → do not register. + +- [ ] **Step 2: Run test to verify it fails** + +``` +cd services/nvpair-ui-broker +go test -count=1 -run TestReconcileAdvertiseLlamaCpp +``` + +Expected: FAIL (no `runAutoAdvertiseLlamaCpp`). + +- [ ] **Step 3: Implement** + +`llamacppproxy.go`: copy `getLMStudioProxy` / `spawnLMStudioProxy` / relay helpers, rename to LlamaCpp, default startup port **8084**, worker name `llamacpp-proxy`, engine id `llamacpp`. + +`advertiser.go`: add `defaultLlamaCppPort = 8082` and `runAutoAdvertiseLlamaCpp` cloned from LM Studio, using `noderec.ServiceLlamaCpp` and `b.getLlamaCppProxy()`. Health check: `GET http://127.0.0.1:{enginePort}/v1/models`. + +`broker.go`: +- Add `llamaCppProxyPath string` and `--llamacpp-proxy-path` (same parsing as `--lmstudio-proxy-path`). +- Spawn when path is set (optional, warn and continue if missing). +- Relay `llamacpp-proxy:` the same way as `lmstudio-proxy:`. +- On `schedule:priority` for engine `llamacpp`, `node/set-priority` to the llama.cpp proxy. +- Shutdown: stop llama.cpp proxy with LM Studio (before engine-manager is fine). + +Do not bridge manual llama.cpp nodes in this task. Task 9 adds `llamacpp_up` and the broker `node/add-manual` call together. + +- [ ] **Step 4: Run broker tests** + +``` +go test -count=1 -run "LlamaCpp|LMStudio|Advertise" +go test -count=1 +``` + +Expected: PASS. Existing LM Studio tests unchanged. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-ui-broker +git commit -s -m "feat(broker): supervise llamacpp-proxy and advertise lc" +``` + +--- + +### Task 8: Job scheduler + +**Files:** +- Modify: `services/nvpair-job-scheduler/schedule.go` (`schedulerEngines`) +- Modify: `services/nvpair-job-scheduler/schedule_test.go` (loops already use `schedulerEngines`; add one explicit `llamacpp` upsert if a test hard-codes two engines) + +- [ ] **Step 1: Write the failing test** + +```go +func TestSchedulerEnginesIncludesLlamaCpp(t *testing.T) { + found := false + for _, e := range schedulerEngines { + if e == "llamacpp" { + found = true + } + } + if !found { + t.Fatal("schedulerEngines missing llamacpp") + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +``` +cd services/nvpair-job-scheduler +go test -count=1 -run TestSchedulerEnginesIncludesLlamaCpp +``` + +Expected: FAIL. + +- [ ] **Step 3: Implement** + +```go +var schedulerEngines = []string{"ollama", "lmstudio", "llamacpp"} +``` + +- [ ] **Step 4: Run tests** + +``` +go test -count=1 +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-job-scheduler/schedule.go services/nvpair-job-scheduler/schedule_test.go +git commit -s -m "feat(scheduler): rank llamacpp alongside ollama and lmstudio" +``` + +--- + +### Task 9: Manual-node llama.cpp probe + +**Files:** +- Modify: `services/nvpair-manual-nodes/manager.go` +- Modify: `services/nvpair-manual-nodes/manager_test.go` +- Modify: `services/nvpair-ui-broker/` manual-node bridge (add-manual into llamacpp-proxy) + +- [ ] **Step 1: Write the failing test** + +Extend the existing probe test (or add `TestProbeLlamaCpp`): a httptest server on a chosen port serving `GET /v1/models` with one loaded id must set `llamacpp_up=true`, `llamacpp_port=`, and `llamacpp_models` containing only loaded ids. + +Default probe port is 8082. Tests should pass an explicit port to the probe helper rather than binding 8082. + +- [ ] **Step 2: Run test to verify it fails** + +``` +cd services/nvpair-manual-nodes +go test -count=1 -run TestProbeLlamaCpp +``` + +Expected: FAIL (no llamacpp fields). + +- [ ] **Step 3: Implement** + +Add to the discovered-node payload: + +```go +LlamaCppUp bool `json:"llamacpp_up"` +LlamaCppPort int `json:"llamacpp_port,omitempty"` +LlamaCppModels []string `json:"llamacpp_models,omitempty"` +``` + +Probe `GET http://{addr}:{port}/v1/models` (port from node override, else 8082). Parse `data[].id` and `data[].status.value`. Loaded ids go to `llamacpp_models` used for routing; keep a separate catalog slice only if the existing Ollama/LM Studio pattern stores full lists on the event. Match LM Studio: liveness and model list from the same `/v1/models` call. + +Broker: when `llamacpp_up`, call `node/add-manual` on `getLlamaCppProxy()` with host/port from the event. + +- [ ] **Step 4: Run tests** + +``` +go test -count=1 +cd ../nvpair-ui-broker +go test -count=1 -run Manual +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-manual-nodes services/nvpair-ui-broker +git commit -s -m "feat(manual-nodes): probe llama.cpp and bridge into llamacpp-proxy" +``` + +--- + +### Task 10: Desktop engine type, capabilities, supervisor + +**Files:** +- Modify: `desktop/src/shared/constants/engines.ts` +- Modify: `desktop/src/ui/constants/engine-capabilities.ts` +- Modify: `desktop/src/shared/constants/modular-binaries.ts` +- Modify: `desktop/src/electron/service-bridge/modular-supervisor.ts` +- Modify: `desktop/src/electron/service-bridge/modular-state.ts` +- Create: `desktop/tests/modular/llamacpp-engine.test.ts` + +- [ ] **Step 1: Write the failing test** + +`desktop/tests/modular/llamacpp-engine.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { EngineDisplayNames, EnabledEngineTypes, EngineTypes } from '@/shared/constants/engines' +import { EngineCapabilities } from '@/ui/constants/engine-capabilities' +import { MODULAR_RUNTIME_BINARIES } from '@/shared/constants/modular-binaries' + +describe('llamacpp engine', () => { + it('is a enabled engine type', () => { + expect(EngineTypes).toContain('llamacpp') + expect(EnabledEngineTypes).toContain('llamacpp') + expect(EngineDisplayNames.llamacpp).toBe('llama.cpp') + }) + it('cannot install, load, eject, or delete', () => { + const caps = EngineCapabilities.llamacpp + expect(caps.hasInstall).toEqual([]) + expect(caps.hasEject).toBe(false) + expect(caps.hasDeleteModel).toBe(false) + expect(caps.hasEnginePort).toBe(true) + expect(caps.engineHub).toBeUndefined() + }) + it('ships llamacpp-proxy as a broker-owned binary', () => { + const bin = MODULAR_RUNTIME_BINARIES.find(b => b.processName === 'llamacpp-proxy') + expect(bin?.baseName).toBe('llamacpp-proxy') + expect(bin?.launchOwner).toBe('broker') + expect(bin?.optional).toBe(true) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +``` +cd desktop +npx vitest run tests/modular/llamacpp-engine.test.ts +``` + +Expected: FAIL (`llamacpp` not in EngineTypes). + +- [ ] **Step 3: Implement** + +`engines.ts`: + +```ts +export const EngineTypes = ['ollama', 'lm-studio', 'llamacpp'] as const +export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio', 'llamacpp'] as const +export const EngineDisplayNames: Record = { + ollama: 'Ollama', + 'lm-studio': 'LM Studio', + llamacpp: 'llama.cpp' +} as const +export const EngineDefaultLinks: Record = { + ollama: { docsUrl: 'https://docs.ollama.com/', installUrl: 'https://ollama.com/download' }, + 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' }, + llamacpp: { docsUrl: 'https://github.com/ggml-org/llama.cpp', installUrl: 'https://github.com/ggml-org/llama.cpp' } +} +``` + +`engine-capabilities.ts` add: + +```ts + llamacpp: { + hasExpiry: false, + hasEject: false, + hasInstall: [], + hasEnginePort: true, + hasInstallPath: false, + hasProxyWebUI: false, + hasPreferredNode: false, + hasCrashAlert: false, + hasModelSearchOnlyWhenRunning: true, + modelOpsWhenStopped: false, + hasDeleteModel: false + } +``` + +`modular-binaries.ts`: add `'llamacpp-proxy'` to `ModularProcessName` and a broker-owned optional entry (`baseName: 'llamacpp-proxy'`, `needsFirewallAccess: true`). + +`modular-supervisor.ts`: +- `engineManagerId`: `llamacpp` stays `llamacpp` +- `proxyEngineFromManagerId`: `id === 'llamacpp' → 'llamacpp'` +- `proxyRelayPrefix`: `llamacpp` → `'llamacpp-proxy'` +- `brokerStartupArgs`: `passPath('--llamacpp-proxy-path', 'llamacpp-proxy')` + +`modular-state.ts`: extend `ProxyNodeSource` with `'llamacpp-proxy'` and map it to engine `'llamacpp'`. Replace remaining `engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'` ternaries with a three-way helper: + +```ts +function proxySourceForEngine(engine: EngineType): ProxyNodeSource { + if (engine === 'ollama') return 'ollama-proxy' + if (engine === 'lm-studio') return 'lmstudio-proxy' + return 'llamacpp-proxy' +} +``` + +Endpoint copy: wherever LM Studio shows `http://127.0.0.1:${proxyPort}/v1`, llama.cpp uses the same pattern (proxy default 8084). Do not hardcode 8082 as the app endpoint. + +- [ ] **Step 4: Run tests and typecheck** + +``` +npx vitest run tests/modular/llamacpp-engine.test.ts +npm run typecheck +``` + +Expected: PASS. Fix every `EngineType` exhaustiveness error the compiler reports (switch statements, records). That is required, not optional. + +- [ ] **Step 5: Commit** + +``` +git add desktop/src desktop/tests/modular/llamacpp-engine.test.ts +git commit -s -m "feat(desktop): enable llamacpp engine and llamacpp-proxy binary" +``` + +--- + +### Task 11: TUI and inference dispatcher + +**Files:** +- Modify: `services/nvpair-tui/ui/proxies.go` +- Modify: `services/nvpair-tui/ui/health.go` +- Modify: `scripts/inference-dispatcher/config.go` +- Modify: `scripts/inference-dispatcher/dispatcher_test.go` +- Modify: `desktop/src/shared/types/inference-dispatcher.ts` + +- [ ] **Step 1: Write failing tests** + +TUI: if `proxies.go` has a table of engines, add a test that the table includes `{label: "llama.cpp", prefix: "llamacpp-proxy"}`. + +Dispatcher: + +```go +func TestBackendLlamaCpp(t *testing.T) { + cfg, err := parseConfig([]string{"--backend", "llamacpp", "--prompt", "hi"}, io.Discard) + if err != nil { + t.Fatal(err) + } + if cfg.Backend != "llamacpp" { + t.Fatalf("backend = %q", cfg.Backend) + } + if defaultPort(cfg) != 8084 { + t.Fatalf("port = %d, want 8084", defaultPort(cfg)) + } +} +``` + +Use the real function names from `config.go` (`defaultLMStudioPort` analogue). + +- [ ] **Step 2: Run tests to verify they fail** + +``` +cd services/nvpair-tui +go test ./ui -count=1 -run LlamaCpp +cd ../../scripts/inference-dispatcher +go test -count=1 -run TestBackendLlamaCpp +``` + +Expected: FAIL. + +- [ ] **Step 3: Implement** + +`proxies.go`: add `{label: "llama.cpp", prefix: "llamacpp-proxy", table: newTable(nil)}` next to LM Studio. Handle `llamacpp-proxy:` in the method switch the same as `lmstudio-proxy:`. + +`health.go`: append `"llamacpp-proxy"` to the worker list. + +`config.go`: allow `--backend llamacpp`; default port 8084; OpenAI chat path like LM Studio (`/v1/chat/completions`). Reuse LM Studio list/parse helpers when `backend == "llamacpp"` (same `/v1/models`). + +`inference-dispatcher.ts`: `DispatcherBackend = 'ollama' | 'lmstudio' | 'llamacpp'`. + +- [ ] **Step 4: Run tests** + +``` +go test ./... -count=1 +``` + +from each module. Expected: PASS. + +- [ ] **Step 5: Commit** + +``` +git add services/nvpair-tui scripts/inference-dispatcher desktop/src/shared/types/inference-dispatcher.ts +git commit -s -m "feat: expose llamacpp in TUI and inference dispatcher" +``` + +--- + +### Task 12: Versions, build scripts, docs, contracts + +**Files:** +- Modify: `services/versions.json` +- Modify: `services/build.bat` +- Modify: `services/build.sh` +- Modify: `services/bom.md` +- Modify: `docs/overview.mdx`, `docs/architecture.mdx`, `docs/engine-lifecycle.mdx`, `docs/getting-started.mdx` +- Modify: `desktop/docs/architecture.md`, `desktop/docs/services-backend.md` +- Run: `npm run service-contracts:write` from `desktop/` after broker RPC surface changes + +Version bumps (VERSIONING.md, MINOR = additive IPC/HTTP): + +| Component | From | To | Why | +|---|---|---|---| +| `llamacpp-proxy` | (new) | `0.1.0` | new binary | +| `nvpair-engine-manager` | `0.17.4` | `0.18.0` | adopt mode + manifest | +| `nvpair-ui-broker` | `0.40.2` | `0.41.0` | new worker + `lc` advertise | +| `nvpair-job-scheduler` | `0.4.1` | `0.5.0` | new engine | +| `nvpair-manual-nodes` | `0.11.1` | `0.12.0` | llamacpp probe fields | +| `nvpair-tui` | `0.7.2` | `0.8.0` | third proxy | +| `product` / `installer` | `0.91.7` | `0.92.0` | user-visible engine | + +Do not bump `desktop/package.json` unless cutting an Electron release. + +- [ ] **Step 1: Add `llamacpp-proxy` to `versions.json` and both build scripts** + +`build.bat` after the lmstudio-proxy `for /f` line: + +``` +for /f "delims=" %%V in ('jq -r --arg k "llamacpp-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_LCPROXY=%%V" +``` + +Add a `go build` for `llamacpp-proxy` next to `lmstudio-proxy`, ldflag `-X main.Version=%V_LCPROXY%`. Mirror in `build.sh` (`V_LCPROXY`, echo, build). Change comments that say “thirteen” binaries to “fourteen”. + +- [ ] **Step 2: Docs** + +In overview/architecture/engine-lifecycle/getting-started: engine list becomes Ollama, LM Studio, or llama.cpp. State: PAIR adopts llama-server; it does not install or load GGUFs; app endpoint is `http://127.0.0.1:8084/v1`; cluster peers need this fork; routing requires the model loaded on the serving node. + +- [ ] **Step 3: Service contracts** + +``` +cd desktop +npm run service-contracts:write +npm run service-contracts:check +``` + +Expected: check PASS. + +- [ ] **Step 4: Full verification (no live GGUF)** + +``` +cd services/nvpair-engine-manager ; go test -count=1 +cd ../llamacpp-proxy ; go test -count=1 +cd ../nvpair-ui-broker ; go test -count=1 +cd ../nvpair-job-scheduler ; go test -count=1 +cd ../nvpair-manual-nodes ; go test -count=1 +cd ../nvpair-tui ; go test -count=1 +cd ../shared ; go test ./... -count=1 +cd ../../desktop ; npm run typecheck ; npm run test:unit +``` + +Expected: all PASS. Do not start llama-server. Do not run the PAIR installer. + +- [ ] **Step 5: Commit** + +``` +git add services/versions.json services/build.bat services/build.sh services/bom.md docs desktop/docs desktop/docs/services-api.md +git commit -s -m "chore: version, build, and document the llamacpp engine" +``` + +--- + +## Spec coverage (self-review) + +| Spec requirement | Task | +|---|---| +| Adopt-only, no spawn/kill/install | 3, 5 | +| Default adopt port 8082, per-node probe override | 4, 5 | +| PAIR facade 8084, never bind adopt port | 6, 7 | +| Discovery `lc` | 1, 7 | +| Wire id `llamacpp`, display `llama.cpp` | 5, 10 | +| Full catalog list | 5 `list_models`, 6 GET `/v1/models` | +| Loaded-only routing; catalog-only 502; zero upstream hits | 1, 2, 6 | +| Missing `status` = unloaded | 2 | +| Local unloaded + peer loaded = forward | 6, 7 | +| Manual nodes | 9 | +| Scheduler | 8 | +| Desktop/TUI/dispatcher | 10, 11 | +| No ComfyUI detection, no GGUF load | 5 (no load action), 6 (no forward) | +| Tests use fake `/v1`, not 8082/27B | all test steps | +| Build 14th binary + versions | 12 | + +No placeholders. Types: engine id is `llamacpp` everywhere except desktop display `llama.cpp` and LM Studio’s existing `lm-studio` hyphen type. diff --git a/docs/superpowers/specs/2026-09-05-llamacpp-engine-design.md b/docs/superpowers/specs/2026-09-05-llamacpp-engine-design.md new file mode 100644 index 00000000..c79de3f6 --- /dev/null +++ b/docs/superpowers/specs/2026-09-05-llamacpp-engine-design.md @@ -0,0 +1,266 @@ + + +# llama.cpp Engine for NVIDIA PAIR + +**Date:** 2026-09-05 +**Updated:** 2026-09-06 +**Status:** Approved +**Repo:** NVIDIA Personal AI Router (local fork) +**Approach:** First-class `llamacpp` engine plus a new OpenAI-compatible proxy, adopt-only against an already-running `llama-server`. + +## Goal + +PAIR routes OpenAI-compatible inference to already-running official llama.cpp `llama-server` instances the same way it routes to Ollama and LM Studio, including across paired Windows, macOS, and Linux nodes on the LAN. PAIR does not install, launch, stop, or load GGUFs. On this Windows machine the AI Playground owns `llama-server` on port 8082. ComfyUI currently holds that GPU, so PAIR must never cause a local model load or swap. Requests may still go to a Mac or laptop that has the model loaded. + +## Decisions (locked) + +| Decision | Choice | +|---|---| +| Engine surface | Official `llama-server` HTTP API (`/v1/...`) | +| Lifecycle | Adopt already-running server only. No install, no spawn, no kill | +| Default adopt port | **8082** (this machine's AI Playground official llama.cpp / Hermes / DeepSeek Harness) | +| Per-node adopt port | Configurable in Engine settings. Changes which port PAIR **probes**. PAIR never rebinds a foreign llama-server. A Mac on stock `:8080` sets 8080 on that Mac's PAIR | +| PAIR facade port | **8084** (new `llamacpp-proxy`; must not bind the adopt port) | +| Discovery key | `lc` | +| Wire engine id | `llamacpp` (Go, JSON-RPC, scheduler, desktop `EngineType`) | +| Display name | `llama.cpp` | +| Catalog | Full `GET /v1/models` id list | +| Routing eligibility | Only ids with `status.value == "loaded"` **on that node** | +| Unloaded on every node | HTTP **502**, never forwarded, never loaded | +| Unloaded locally, loaded on a peer | Forward to that peer. Not a 502 | +| Preferred model | `ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp` (docs/default label only, not a filter) | +| Cluster | PAIR on each participating Mac/laptop, same LAN, pair with PIN. This fork, not stock NVIDIA 0.1.1 | +| Manual nodes | Probe llama.cpp as well as Ollama/LM Studio (port = that node's adopt port) | +| Other playground ports on this PC | Out of scope as extra engines (Bonsai 8080, ik_llama 8083). 8080 remains valid as another node's adopt port | +| ComfyUI | Not detected. Safety = never trigger a load | + +## Architecture + +PAIR gains a third engine beside `ollama` and `lmstudio`. + +``` +App on any PAIR node + │ + ▼ +local llamacpp-proxy :8084 + │ + ├─ GET /v1/models → merge catalog from every lc node + └─ chat/completions + │ + ├─ this node has id loaded? → loopback llama-server (adopt port) + ├─ a paired Mac/laptop has it loaded? → that node's PAIR :8084 (mTLS) + └─ nobody has it loaded? → 502 (do not load) +``` + +Two ports per node, same split as LM Studio: apps and peers talk to **8084**. The local `llama-server` stays on that node's adopt port (8082 here, often 8080 on a Mac). Hermes, DeepSeek Harness, and the llama.cpp Web UI on this PC keep talking to **8082** directly. + +Engine-manager today can only start an engine by spawning a binary (process mode) or running CLI commands (command mode). llama.cpp needs a third **adopt-only** runtime: Start probes 8082 and never launches a process. If the probe fails, the engine is not running. + +Readiness and health probes are `GET http://127.0.0.1:{port}/v1/models` with status 200 (same idea as LM Studio). That stays green when the router is up with an empty loaded set. PAIR does not use `/health` as the probe, because some llama.cpp builds return 503 when no model is loaded, which would hide a running unloaded server. + +If a `/v1/models` entry has no `status.value`, treat it as **unloaded**. Missing status must not be treated as loaded. + +## Components + +### 1. `llamacpp-proxy` (new Go service) + +Clone of `lmstudio-proxy`. Own module under `services/llamacpp-proxy/`. + +- Default listen port **8084**, persisted like the other proxies, `--ignore-persisted-port` for broker-managed start. +- Subscribe to discovery `lc`. +- Forward `POST /v1/chat/completions`, `POST /v1/completions`, `POST /v1/embeddings`. +- `GET /v1/models` merges the full catalog across candidate nodes. +- Inference eligibility uses **loaded** ids only. Catalog-only ids are 502 and produce zero upstream requests. +- Cluster mTLS ingress, CORS, failover, `node/set-local-backend`, `node/set-priority`, and activity reports match LM Studio. +- Workloads tagged `llamacpp`. +- Failover only to nodes that advertise the requested id as loaded. + +Depends on: `nvpair-shared` (noderec, cors, clustertrust, schedulerwire). Does not own llama-server. + +### 2. Engine-manager: manifest + adopt-only runtime + +New `services/nvpair-engine-manager/manifests/llamacpp.json`. + +- `engine`: `llamacpp` +- `display_name`: `llama.cpp` +- `runtime.port`: 8082 (default adopt probe; overridable per node, not a rebind of llama-server) +- `runtime.bind`: `127.0.0.1` (PAIR talks loopback; playground may still bind `0.0.0.0` itself) +- `runtime.ready` / `runtime.health`: `GET http://127.0.0.1:{port}/v1/models` status 200 +- No `install`, `uninstall`, `pull_model`, `load_model`, `unload_model`, `delete_model` +- `list_models`: `GET /v1/models`, result array `data`, field `id` +- `loaded_models`: same endpoint, field `id`, match `status.value` equals `"loaded"` + +Engine-manager changes: + +- New runtime mode `runtime.mode: "adopt"`. Start probes ready; success marks running with `adopted=true` and no child process. Failure returns an actionable error. No spawn path. Process mode still requires `bin`; command mode still requires `start`; adopt mode requires `ready` and forbids `bin` and `start`. +- Action result `match` must support equality on a nested field (`status.value == "loaded"`). Today LM Studio only has `nonempty`. Add equality rather than a llama.cpp special case in Go. +- Detect: WinGet `ggml.llamacpp` `llama-server.exe`, `llama-server` on `PATH`, AI Playground official binary `vendor\llama.cpp-official\bin\llama-server.exe`, **or** a healthy 8082. Binary present + 8082 down = installed, not running. No binary + 8082 down = not installed. + +Stop/restart against a foreign listener stay declined (existing adoption rules). Setting the adopt port is allowed: it only changes which loopback port PAIR probes and hands to `set-local-backend`. It must not send a bind change to llama-server. PAIR never sends load/unload HTTP to llama-server. + +### 3. Shared discovery (`nvpair-shared/noderec`) + +- `ServiceLlamaCpp ServiceKey = "lc"` +- Include `lc` in `serviceKeyOrder` +- Transport policy same as `ol` / `lm`: advertised proxy port is cluster mTLS ingress; local engine is loopback plaintext + +### 4. Manual nodes + +`nvpair-manual-nodes` today probes Ollama `:11434` and LM Studio `:1234`. Add a llama.cpp probe (`GET /v1/models`) on the node's adopt port (default 8082). Emit `llamacpp_up` / `llamacpp_port` / `llamacpp_models` (loaded ids for routing, full catalog for listing). The broker bridges a reachable manual llama.cpp node into `llamacpp-proxy` the same way it bridges LM Studio into `lmstudio-proxy`. + +### 5. Broker, scheduler, desktop, TUI + +**Broker (`nvpair-ui-broker`)** + +- Spawn `llamacpp-proxy` via `--llamacpp-proxy-path` (parallel to `--lmstudio-proxy-path`) +- Advertise loop: healthy engine + proxy up + ports differ → register `lc` with **proxy** port; `node/set-local-backend` to the adopt port (8082 here) +- Otherwise unregister `lc` and clear the local backend +- Relay namespace `llamacpp-proxy:` + +**Scheduler** + +- `schedulerEngines` includes `llamacpp` + +**Desktop** + +- `EngineTypes` / `EnabledEngineTypes` include `llamacpp` +- Display name `llama.cpp` +- Capabilities: no install, uninstall, delete, load, eject, or engine-hub pull. `hasEnginePort: true`. Endpoints show `http://127.0.0.1:8084/v1` +- Map `llamacpp` through the same proxy-bridge path as LM Studio (`proxyEngineFromManagerId`, `proxyRelayPrefix`, `MODULAR_RUNTIME_BINARIES`) + +**TUI** + +- Third proxy row (label `llama.cpp`, prefix `llamacpp-proxy`) +- Health list includes `llamacpp-proxy` + +**Inference dispatcher** + +- `--backend llamacpp` talks to PAIR facade port 8084, not 8082 + +### 6. Build and packaging + +- `services/versions.json`: new `llamacpp-proxy` component version `0.1.0`; bump `nvpair-engine-manager`, `nvpair-ui-broker`, `nvpair-job-scheduler`, `nvpair-tui`, and shared as required by VERSIONING.md +- `desktop/src/shared/constants/modular-binaries.ts`: add `llamacpp-proxy` +- `services/build.bat` / `build.sh`, BOM, service-contract docs (`npm run service-contracts:write`), architecture/engine-lifecycle docs +- Fourteen supervised binaries instead of thirteen + +## Data flow + +**Bring-up** + +1. Electron starts only `nvpair-ui-broker`. Broker starts `llamacpp-proxy` on 8084 and engine-manager with the `llamacpp` manifest. +2. Engine-manager probes `GET http://127.0.0.1:{adoptPort}/v1/models` (default 8082). + - 200: adopt (installed, running, healthy). No process spawned. + - down: not running. Proxy stays up. `lc` is not advertised for this node. The proxy can still route to paired peers. +3. If healthy, broker registers `lc` with port 8084 and sets the proxy local backend to `127.0.0.1:{adoptPort}`. +4. Engine-manager fills `modelsByEngine.llamacpp` (all ids) and `loadedByEngine.llamacpp` (loaded ids). Peers read this from engine-manager `em` `/v1/models`, not from mDNS. + +**Local inference** + +``` +App → http://127.0.0.1:8084/v1/chat/completions + → llamacpp-proxy + → this node has id loaded? → 127.0.0.1:{adoptPort} + → a paired node has it loaded? → that node's PAIR :8084 (mTLS) + → nobody has it loaded? → 502, never forwarded, never loaded +``` + +`GET /v1/models` on 8084 returns the merged catalog across the cluster. Completions still require the id to be **loaded on the chosen node**. + +**Cluster (Mac, laptops, this PC)** + +Install **this fork** of PAIR on each machine (stock NVIDIA 0.1.1 has no `llamacpp` engine). Pair with the PIN on the same LAN. On each machine, run `llama-server` yourself and set that node's adopt port if it is not 8082. + +A peer sends the OpenAI request over cluster mTLS to the chosen node's advertised 8084. Ingress on that node forwards only to its local llama-server. It does not hop again. Scheduler ranks `llamacpp` using each node's **loaded** inventory. + +This Windows box can be a **client only** while ComfyUI has the GPU: local 8082 down or unloaded means this node is not eligible, but 8084 still forwards to a Mac or laptop that has the model loaded. + +**ComfyUI holding the local GPU** + +Local 8082 is down, or up with an empty loaded set. This node is not advertised as a llama.cpp server (or is advertised with an empty loaded set). Completions for a model loaded on another paired node go there. Completions for a model loaded nowhere are 502. Hermes and the playground Web UI remain the only **local** loaders, and only after ComfyUI is stopped. + +**Shutdown** + +PAIR unregisters `lc` and clears the proxy backend. It does not stop llama-server. + +## Error handling + +| Condition | PAIR behavior | +|---|---| +| Local adopt port down | This node is not a llama.cpp server (`lc` unregistered). Proxy stays up. Route to paired nodes that have the id loaded. 502 only if none do. No spawn, no playground script, no install | +| Local up, nothing loaded here | This node is not eligible. Route to a peer with the id loaded. 502 only if none do. No local load | +| Requested id loaded on no node | 502. Never forwarded. Never loaded | +| Stop/restart on foreign listener | Declined. Desired-off may still persist so PAIR does not re-advertise this node; llama-server keeps running | +| Adopt port change | Persist and probe the new port. Do not rebind llama-server | +| 8084 bind failure | Existing bind-failed error. PAIR does not bind the adopt port | +| Upstream 5xx or drop mid-stream | LM Studio-style failover among nodes with the id loaded. If none, 502. No local reload | +| Health flap | Failed probe unregisters `lc` and clears local backend. Next 5s poll re-adopts if the adopt port answers `/v1/models` | +| Logs | Engine, model id, node id, job id only. No prompts, messages, or response bodies | + +## Testing + +No live GGUF loads. ComfyUI keeps the GPU. Tests use a fake OpenAI `/v1` listener on a free loopback port, never 8082 and never the ISTA Qwen3.8 file. + +**Engine-manager** + +- Fake `/v1/models` 200 → Start adopts, no child process +- Probe down → not running, Start errors, no spawn +- `list_models` returns every `id` +- `loaded_models` returns only `status.value == "loaded"` +- Missing `status` is treated as unloaded +- Stop against a foreign listener is declined + +**llamacpp-proxy** + +- `GET /v1/models` merges the full catalog +- Chat for a loaded id is forwarded to the fake backend +- Chat for a catalog-only id is 502 and the fake backend sees **zero** requests +- Failover only considers peers that advertise the id as loaded +- CORS and loopback-plaintext rules match LM Studio + +**Broker** + +- Healthy engine + proxy up → register `lc` with 8084, local backend = adopt port +- Engine down → unregister `lc`, clear backend; proxy still routes to peers +- Equal proxy/engine ports are refused +- Adopt-port override is probed without spawning or rebinding +- Unloaded locally + loaded on a fake peer → request goes to the peer, not 502 + +**Desktop** + +- `EngineTypes` includes `llamacpp` +- Capabilities: no install, delete, load, or eject +- Endpoint copy is `http://127.0.0.1:8084/v1` + +**Commands** + +- `go test` in `nvpair-engine-manager`, `llamacpp-proxy`, `nvpair-ui-broker` +- Desktop `npm run test:unit` for engine-type tests +- `npm run service-contracts:check` after JSON-RPC/docs updates + +**Not in this work** + +Real `llama-server`, ISTA Qwen3.8, Playground launchers, PAIR installer, any VRAM-using load. + +## Out of scope + +- Installing or compiling llama.cpp +- PAIR starting or stopping playground launchers (`start-llamacpp.ps1`, `connect-dsh.ps1`, …) +- Detecting ComfyUI +- Treating Bonsai 8080 or ik_llama 8083 on **this PC** as extra PAIR engines (a Mac's stock llama-server on 8080 is in scope as that node's adopt port) +- Running stock NVIDIA PAIR 0.1.1 on the Mac/laptops and expecting llama.cpp routing (those machines need this fork) +- Native llama.cpp `/completion` (non-OpenAI) +- PAIR-driven GGUF pull, load, unload, or delete +- Running `NVPAIR-Setup-0.1.1-x64.exe` until the user gives a green light +- Live inference against the 27B GGUF while ComfyUI holds the GPU + +## Success criteria + +1. With the local adopt port down, PAIR shows llama.cpp as not running on this node and never starts a process. The local 8084 proxy still accepts requests. +2. With a fake loaded model on a peer and nothing loaded locally, PAIR forwards OpenAI chat to the peer, not 502. +3. With a catalog-only (unloaded) id on every node, PAIR returns 502 and no fake backend receives a request. +4. Hermes / dsh on 8082 are untouched: PAIR does not bind 8082 and does not kill llama-server on shutdown. +5. Ollama and LM Studio paths still compile and their existing tests pass. diff --git a/scripts/inference-dispatcher/client.go b/scripts/inference-dispatcher/client.go index a759e973..da585444 100644 --- a/scripts/inference-dispatcher/client.go +++ b/scripts/inference-dispatcher/client.go @@ -111,7 +111,7 @@ func decodeObject(data []byte, target any) error { func (c *backendClient) listModels(ctx context.Context) ([]RegisteredModel, error) { var models []RegisteredModel var err error - if c.cfg.Backend == "lmstudio" { + if usesOpenAIAPI(c.cfg.Backend) { models, err = c.listLMStudioModels(ctx) } else { models, err = c.listOllamaModels(ctx) @@ -340,8 +340,12 @@ func (c *backendClient) resolveModel(ctx context.Context) (string, []RegisteredM return "", models, errors.New("no available model advertises text-generation support") } +func usesOpenAIAPI(backend string) bool { + return backend == "lmstudio" || backend == "llamacpp" +} + func (c *backendClient) inferencePath() string { - if c.cfg.Backend == "lmstudio" { + if usesOpenAIAPI(c.cfg.Backend) { return "/v1/chat/completions" } return "/api/generate" @@ -349,7 +353,7 @@ func (c *backendClient) inferencePath() string { func (c *backendClient) infer(ctx context.Context, model, prompt string) (string, error) { var payload map[string]any - if c.cfg.Backend == "lmstudio" { + if usesOpenAIAPI(c.cfg.Backend) { payload = map[string]any{ "model": model, "messages": []map[string]string{{"role": "user", "content": prompt}}, @@ -386,7 +390,7 @@ func (c *backendClient) infer(ctx context.Context, model, prompt string) (string if err != nil { return "", err } - if c.cfg.Backend == "lmstudio" { + if usesOpenAIAPI(c.cfg.Backend) { return parseLMStudioResponse(data) } var response struct { diff --git a/scripts/inference-dispatcher/config.go b/scripts/inference-dispatcher/config.go index 427d8ce5..d5a22975 100644 --- a/scripts/inference-dispatcher/config.go +++ b/scripts/inference-dispatcher/config.go @@ -23,6 +23,10 @@ const ( // PAIR's managed LM Studio backend is moved behind it starting at 1235; // pass --port explicitly to reach that directly, which bypasses routing. defaultLMStudioPort = 1234 + // llama.cpp's stock listener is :8082; PAIR's llamacpp-proxy facade + // claims :8084 (services/llamacpp-proxy/portstore.go defaultProxyPort). + // Defaulting to the engine would bypass routing, same as LM Studio. + defaultLlamaCppPort = 8084 defaultErrorLog = "inference_errors.txt" maxPromptsPerBatch = 100 ) @@ -289,7 +293,7 @@ func parseConfig(args []string, stderr io.Writer) (Config, error) { fs.SetOutput(stderr) var parsedConfigPath string fs.StringVar(&parsedConfigPath, "config", configPath, "JSON configuration file") - fs.StringVar(&cfg.Backend, "backend", cfg.Backend, "backend: ollama or lmstudio") + fs.StringVar(&cfg.Backend, "backend", cfg.Backend, "backend: ollama, lmstudio, or llamacpp") fs.StringVar(&cfg.Backend, "provider", cfg.Backend, "alias for --backend") fs.IntVar(&cfg.Port, "port", cfg.Port, "server port (backend default when omitted)") fs.StringVar(&cfg.Model, "model", cfg.Model, "model name; omitted or auto selects an available model") @@ -356,8 +360,8 @@ func parseConfig(args []string, stderr io.Writer) (Config, error) { } func validateConfig(cfg Config) error { - if cfg.Backend != "ollama" && cfg.Backend != "lmstudio" { - return errors.New("--backend must be ollama or lmstudio") + if cfg.Backend != "ollama" && cfg.Backend != "lmstudio" && cfg.Backend != "llamacpp" { + return errors.New("--backend must be ollama, lmstudio, or llamacpp") } if cfg.Port < 0 || cfg.Port > 65535 { return errors.New("--port must be between 1 and 65535") @@ -417,5 +421,8 @@ func effectivePort(cfg Config) int { if cfg.Backend == "lmstudio" { return defaultLMStudioPort } + if cfg.Backend == "llamacpp" { + return defaultLlamaCppPort + } return defaultOllamaPort } diff --git a/scripts/inference-dispatcher/dispatcher_test.go b/scripts/inference-dispatcher/dispatcher_test.go index c858fcc7..4782c9e7 100644 --- a/scripts/inference-dispatcher/dispatcher_test.go +++ b/scripts/inference-dispatcher/dispatcher_test.go @@ -334,6 +334,52 @@ func TestLMStudioDefaultPort(t *testing.T) { } } +func TestBackendLlamaCpp(t *testing.T) { + var stderr bytes.Buffer + cfg, err := parseConfig([]string{"--backend", "llamacpp", "--prompt", "hi"}, &stderr) + if err != nil { + t.Fatal(err) + } + if cfg.Backend != "llamacpp" { + t.Fatalf("backend = %q", cfg.Backend) + } + if port := effectivePort(cfg); port != 8084 { + t.Fatalf("port = %d, want 8084", port) + } +} + +func TestLlamaCppUsesOpenAIInventory(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/v1/models": + http.Error(w, "not found", http.StatusNotFound) + case "/v1/models": + _, _ = w.Write([]byte(`{"data":[{"id":"gguf-model"}]}`)) + case "/v1/chat/completions": + _, _ = w.Write([]byte(`{"choices":[{"message":{"content":"done"}}]}`)) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + + var stdout, stderr bytes.Buffer + exit := runAgainstServer( + t, + context.Background(), + []string{"--backend", "llamacpp", "--prompt", "test"}, + server.URL, + &stdout, + &stderr, + ) + if exit != 0 { + t.Fatalf("exit=%d stderr=%s", exit, stderr.String()) + } + if !strings.Contains(stdout.String(), "gguf-model") { + t.Fatalf("selected model missing from output: %s", stdout.String()) + } +} + func TestResponseTextNeverReachesStdout(t *testing.T) { const secret = "the capital of France is Paris" server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { diff --git a/scripts/wipe-app-data.sh b/scripts/wipe-app-data.sh index 335415de..b72de1ec 100755 --- a/scripts/wipe-app-data.sh +++ b/scripts/wipe-app-data.sh @@ -148,6 +148,7 @@ PAIR_PROCS=( nvpair-ui-broker ollama-proxy lmstudio-proxy + llamacpp-proxy nvpair-node-info nvpair-node-scanner nvpair-manual-nodes diff --git a/services/bom.md b/services/bom.md index 63497a36..10c5a0db 100644 --- a/services/bom.md +++ b/services/bom.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0 # Bill of Materials — Third-Party Go Libraries -Scope: dependencies linked into the thirteen shipped binaries (`ollama-proxy`, `lmstudio-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-cluster-manager`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-job-scheduler`, `nvpair-tui`). The local modules `nvpair-shared` and `eapnoob` (the EAP-NOOB implementation under `eap-noob/`, linked by `nvpair-cluster-manager`) are first-party and excluded. The `tests/`, `mdns-test/`, and `broker-test-driver/` modules are development-only and excluded. +Scope: dependencies linked into the fourteen shipped binaries (`ollama-proxy`, `lmstudio-proxy`, `llamacpp-proxy`, `nvpair-node-info`, `nvpair-node-scanner`, `nvpair-manual-nodes`, `nvpair-workload-manager`, `nvpair-errors`, `nvpair-node-settings`, `nvpair-cluster-manager`, `nvpair-ui-broker`, `nvpair-engine-manager`, `nvpair-job-scheduler`, `nvpair-tui`). The local modules `nvpair-shared` and `eapnoob` (the EAP-NOOB implementation under `eap-noob/`, linked by `nvpair-cluster-manager`) are first-party and excluded. The `tests/`, `mdns-test/`, and `broker-test-driver/` modules are development-only and excluded. `nvpair-tui` is the only component that links the Bubble Tea terminal-UI stack (`charmbracelet/bubbletea` + `lipgloss` + `bubbles`); its transitive `charmbracelet/*`, `muesli/*`, `mattn/*`, `clipperhouse/*`, `atotto/clipboard`, `aymanbagabas/go-osc52`, `lucasb-eyer/go-colorful`, `erikgeiser/coninput`, and `xo/terminfo` dependencies are unique to it. @@ -21,15 +21,15 @@ As of the mDNS dedup, `grandcat/zeroconf`, `miekg/dns`, and `golang.org/x/net` a | Library | Version | Used By | License | License URL | |---------|---------|---------|---------|-------------| -| `github.com/Microsoft/go-winio` | v0.6.2 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) | +| `github.com/Microsoft/go-winio` | v0.6.2 | ollama-proxy, lmstudio-proxy, llamacpp-proxy, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-settings, nvpair-cluster-manager, nvpair-ui-broker, nvpair-workload-manager, nvpair-errors, nvpair-engine-manager, nvpair-job-scheduler | MIT | [LICENSE](https://github.com/microsoft/go-winio/blob/main/LICENSE) | | `github.com/charmbracelet/bubbles` | v1.0.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbles/blob/master/LICENSE) | | `github.com/charmbracelet/bubbletea` | v1.3.10 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/bubbletea/blob/master/LICENSE) | | `github.com/charmbracelet/lipgloss` | v1.1.0 | nvpair-tui | MIT | [LICENSE](https://github.com/charmbracelet/lipgloss/blob/master/LICENSE) | -| `github.com/grandcat/zeroconf` | v1.0.0 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | MIT | [LICENSE](https://github.com/grandcat/zeroconf/blob/master/LICENSE) | +| `github.com/grandcat/zeroconf` | v1.0.0 | ollama-proxy, lmstudio-proxy, llamacpp-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | MIT | [LICENSE](https://github.com/grandcat/zeroconf/blob/master/LICENSE) | | `github.com/jaypipes/ghw` | v0.24.0 | nvpair-node-info | Apache-2.0 | [COPYING](https://github.com/jaypipes/ghw/blob/main/COPYING) | -| `github.com/miekg/dns` | v1.1.55 / v1.1.72 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://github.com/miekg/dns/blob/master/LICENSE) | +| `github.com/miekg/dns` | v1.1.55 / v1.1.72 | ollama-proxy, lmstudio-proxy, llamacpp-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://github.com/miekg/dns/blob/master/LICENSE) | | `github.com/shirou/gopsutil/v4` | v4.26.7 | nvpair-node-info (macOS) | BSD-3-Clause | [LICENSE](https://github.com/shirou/gopsutil/blob/master/LICENSE) | -| `golang.org/x/net` | v0.58.0 | ollama-proxy, lmstudio-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/net/+/master:LICENSE) | +| `golang.org/x/net` | v0.58.0 | ollama-proxy, lmstudio-proxy, llamacpp-proxy, nvpair-node-scanner, nvpair-errors, nvpair-cluster-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/net/+/master:LICENSE) | | `golang.org/x/sys` | v0.47.0 | nvpair-node-info, nvpair-cluster-manager, nvpair-engine-manager | BSD-3-Clause | [LICENSE](https://cs.opensource.google/go/x/sys/+/master:LICENSE) | | `howett.net/plist` | v1.0.2-0.20250314 | nvpair-node-info | BSD-2-Clause | [LICENSE](https://github.com/DHowett/go-plist/blob/main/LICENSE) | diff --git a/services/build.bat b/services/build.bat index 6311e423..c0553562 100644 --- a/services/build.bat +++ b/services/build.bat @@ -40,6 +40,7 @@ REM bare .components.nvpair-ui-broker would parse as subtraction). for /f "delims=" %%V in ('jq -r ".product" "%VERSIONS_FILE%"') do set "V_PRODUCT=%%V" for /f "delims=" %%V in ('jq -r --arg k "ollama-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_PROXY=%%V" for /f "delims=" %%V in ('jq -r --arg k "lmstudio-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_LMPROXY=%%V" +for /f "delims=" %%V in ('jq -r --arg k "llamacpp-proxy" ".components[$k]" "%VERSIONS_FILE%"') do set "V_LCPROXY=%%V" for /f "delims=" %%V in ('jq -r --arg k "nvpair-node-info" ".components[$k]" "%VERSIONS_FILE%"') do set "V_NINFO=%%V" for /f "delims=" %%V in ('jq -r --arg k "nvpair-node-scanner" ".components[$k]" "%VERSIONS_FILE%"') do set "V_NSCAN=%%V" for /f "delims=" %%V in ('jq -r --arg k "nvpair-manual-nodes" ".components[$k]" "%VERSIONS_FILE%"') do set "V_MNODES=%%V" @@ -61,6 +62,7 @@ if "%V_PRODUCT%"=="" ( echo product = %V_PRODUCT% echo ollama-proxy = %V_PROXY% echo lmstudio-proxy = %V_LMPROXY% +echo llamacpp-proxy = %V_LCPROXY% echo nvpair-node-info = %V_NINFO% echo nvpair-node-scanner = %V_NSCAN% echo nvpair-manual-nodes = %V_MNODES% @@ -79,67 +81,72 @@ echo Building all components echo ======================================== echo. -echo [1/13] Building ollama-proxy (v%V_PROXY%)... +echo [1/14] Building ollama-proxy (v%V_PROXY%)... cd /d "%ROOT%ollama-proxy" go build -ldflags "-X main.Version=%V_PROXY%" -o ollama-proxy.exe . || goto :fail echo OK -echo [2/13] Building lmstudio-proxy (v%V_LMPROXY%)... +echo [2/14] Building lmstudio-proxy (v%V_LMPROXY%)... cd /d "%ROOT%lmstudio-proxy" go build -ldflags "-X main.Version=%V_LMPROXY%" -o lmstudio-proxy.exe . || goto :fail echo OK -echo [3/13] Building nvpair-node-info (v%V_NINFO%)... +echo [3/14] Building llamacpp-proxy (v%V_LCPROXY%)... +cd /d "%ROOT%llamacpp-proxy" +go build -ldflags "-X main.Version=%V_LCPROXY%" -o llamacpp-proxy.exe . || goto :fail +echo OK + +echo [4/14] Building nvpair-node-info (v%V_NINFO%)... cd /d "%ROOT%nvpair-node-info" go build -ldflags "-X main.Version=%V_NINFO%" -o nvpair-node-info.exe . || goto :fail echo OK -echo [4/13] Building nvpair-node-scanner (v%V_NSCAN%)... +echo [5/14] Building nvpair-node-scanner (v%V_NSCAN%)... cd /d "%ROOT%nvpair-node-scanner" go build -ldflags "-X main.Version=%V_NSCAN%" -o nvpair-node-scanner.exe . || goto :fail echo OK -echo [5/13] Building nvpair-manual-nodes (v%V_MNODES%)... +echo [6/14] Building nvpair-manual-nodes (v%V_MNODES%)... cd /d "%ROOT%nvpair-manual-nodes" go build -ldflags "-X main.Version=%V_MNODES%" -o nvpair-manual-nodes.exe . || goto :fail echo OK -echo [6/13] Building nvpair-workload-manager (v%V_WLMGR%)... +echo [7/14] Building nvpair-workload-manager (v%V_WLMGR%)... cd /d "%ROOT%nvpair-workload-manager" go build -ldflags "-X main.Version=%V_WLMGR%" -o nvpair-workload-manager.exe . || goto :fail echo OK -echo [7/13] Building nvpair-errors (v%V_ERRORS%)... +echo [8/14] Building nvpair-errors (v%V_ERRORS%)... cd /d "%ROOT%nvpair-errors" go build -ldflags "-X main.Version=%V_ERRORS%" -o nvpair-errors.exe . || goto :fail echo OK -echo [8/13] Building nvpair-engine-manager (v%V_ENGMGR%)... +echo [9/14] Building nvpair-engine-manager (v%V_ENGMGR%)... cd /d "%ROOT%nvpair-engine-manager" go build -ldflags "-X main.Version=%V_ENGMGR%" -o nvpair-engine-manager.exe . || goto :fail echo OK -echo [9/13] Building nvpair-node-settings (v%V_NSETTINGS%)... +echo [10/14] Building nvpair-node-settings (v%V_NSETTINGS%)... cd /d "%ROOT%nvpair-node-settings" go build -ldflags "-X main.Version=%V_NSETTINGS%" -o nvpair-node-settings.exe . || goto :fail echo OK -echo [10/13] Building nvpair-ui-broker (v%V_BROKER%)... +echo [11/14] Building nvpair-ui-broker (v%V_BROKER%)... cd /d "%ROOT%nvpair-ui-broker" go build -ldflags "-X main.Version=%V_BROKER%" -o nvpair-ui-broker.exe . || goto :fail echo OK -echo [11/13] Building nvpair-cluster-manager (v%V_CLUMGR%)... +echo [12/14] Building nvpair-cluster-manager (v%V_CLUMGR%)... cd /d "%ROOT%nvpair-cluster-manager" go build -ldflags "-X main.Version=%V_CLUMGR%" -o nvpair-cluster-manager.exe . || goto :fail echo OK -echo [12/13] Building nvpair-job-scheduler (v%V_SCHED%)... +echo [13/14] Building nvpair-job-scheduler (v%V_SCHED%)... cd /d "%ROOT%nvpair-job-scheduler" go build -ldflags "-X main.Version=%V_SCHED%" -o nvpair-job-scheduler.exe . || goto :fail echo OK -echo [13/13] Building nvpair-tui (v%V_TUI%)... +echo [14/14] Building nvpair-tui (v%V_TUI%)... cd /d "%ROOT%nvpair-tui" go build -ldflags "-X main.Version=%V_TUI%" -o nvpair-tui.exe . || goto :fail echo OK @@ -157,6 +164,7 @@ if exist "%BIN_OUT%" rmdir /s /q "%BIN_OUT%" mkdir "%BIN_OUT%" copy /y "%ROOT%ollama-proxy\ollama-proxy.exe" "%BIN_OUT%\ollama-proxy.exe" >nul || goto :fail copy /y "%ROOT%lmstudio-proxy\lmstudio-proxy.exe" "%BIN_OUT%\lmstudio-proxy.exe" >nul || goto :fail +copy /y "%ROOT%llamacpp-proxy\llamacpp-proxy.exe" "%BIN_OUT%\llamacpp-proxy.exe" >nul || goto :fail copy /y "%ROOT%nvpair-node-info\nvpair-node-info.exe" "%BIN_OUT%\nvpair-node-info.exe" >nul || goto :fail copy /y "%ROOT%nvpair-node-scanner\nvpair-node-scanner.exe" "%BIN_OUT%\nvpair-node-scanner.exe" >nul || goto :fail copy /y "%ROOT%nvpair-manual-nodes\nvpair-manual-nodes.exe" "%BIN_OUT%\nvpair-manual-nodes.exe" >nul || goto :fail @@ -176,6 +184,7 @@ echo ======================================== echo. echo Proxy: %BIN_OUT%\ollama-proxy.exe echo LM Studio Proxy: %BIN_OUT%\lmstudio-proxy.exe +echo llama.cpp Proxy: %BIN_OUT%\llamacpp-proxy.exe echo Node Info: %BIN_OUT%\nvpair-node-info.exe echo Node Scanner: %BIN_OUT%\nvpair-node-scanner.exe echo Manual Nodes: %BIN_OUT%\nvpair-manual-nodes.exe diff --git a/services/build.sh b/services/build.sh index feadff14..b6630f82 100755 --- a/services/build.sh +++ b/services/build.sh @@ -4,7 +4,7 @@ # build.sh — NVIDIA Personal AI Router build script for Linux and macOS. # -# Mirrors build.bat. Reads versions.json with jq, builds the thirteen worker +# Mirrors build.bat. Reads versions.json with jq, builds the fourteen worker # binaries with -X main.Version=... ldflags, then copies them into the # repo-root staging bundle at: # @@ -57,6 +57,7 @@ echo V_PRODUCT=$(jq -r '.product' "$VERSIONS_FILE") V_PROXY=$( jq -r --arg k 'ollama-proxy' '.components[$k]' "$VERSIONS_FILE") V_LMPROXY=$(jq -r --arg k 'lmstudio-proxy' '.components[$k]' "$VERSIONS_FILE") +V_LCPROXY=$(jq -r --arg k 'llamacpp-proxy' '.components[$k]' "$VERSIONS_FILE") V_NINFO=$( jq -r --arg k 'nvpair-node-info' '.components[$k]' "$VERSIONS_FILE") V_NSCAN=$( jq -r --arg k 'nvpair-node-scanner' '.components[$k]' "$VERSIONS_FILE") V_MNODES=$( jq -r --arg k 'nvpair-manual-nodes' '.components[$k]' "$VERSIONS_FILE") @@ -77,6 +78,7 @@ fi printf ' product = %s\n' "$V_PRODUCT" printf ' ollama-proxy = %s\n' "$V_PROXY" printf ' lmstudio-proxy = %s\n' "$V_LMPROXY" +printf ' llamacpp-proxy = %s\n' "$V_LCPROXY" printf ' nvpair-node-info = %s\n' "$V_NINFO" printf ' nvpair-node-scanner = %s\n' "$V_NSCAN" printf ' nvpair-manual-nodes = %s\n' "$V_MNODES" @@ -97,23 +99,24 @@ echo build_subbinary() { local idx="$1" name="$2" version="$3" - echo "[$idx/13] Building $name (v$version)..." + echo "[$idx/14] Building $name (v$version)..." (cd "$ROOT/$name" && go build -ldflags "-X main.Version=$version" -o "$name" .) echo " OK" } build_subbinary 1 ollama-proxy "$V_PROXY" build_subbinary 2 lmstudio-proxy "$V_LMPROXY" -build_subbinary 3 nvpair-node-info "$V_NINFO" -build_subbinary 4 nvpair-node-scanner "$V_NSCAN" -build_subbinary 5 nvpair-manual-nodes "$V_MNODES" -build_subbinary 6 nvpair-workload-manager "$V_WLMGR" -build_subbinary 7 nvpair-errors "$V_ERRORS" -build_subbinary 8 nvpair-engine-manager "$V_ENGMGR" -build_subbinary 9 nvpair-node-settings "$V_NSETTINGS" -build_subbinary 10 nvpair-ui-broker "$V_BROKER" -build_subbinary 11 nvpair-cluster-manager "$V_CLUMGR" -build_subbinary 12 nvpair-job-scheduler "$V_SCHED" -build_subbinary 13 nvpair-tui "$V_TUI" +build_subbinary 3 llamacpp-proxy "$V_LCPROXY" +build_subbinary 4 nvpair-node-info "$V_NINFO" +build_subbinary 5 nvpair-node-scanner "$V_NSCAN" +build_subbinary 6 nvpair-manual-nodes "$V_MNODES" +build_subbinary 7 nvpair-workload-manager "$V_WLMGR" +build_subbinary 8 nvpair-errors "$V_ERRORS" +build_subbinary 9 nvpair-engine-manager "$V_ENGMGR" +build_subbinary 10 nvpair-node-settings "$V_NSETTINGS" +build_subbinary 11 nvpair-ui-broker "$V_BROKER" +build_subbinary 12 nvpair-cluster-manager "$V_CLUMGR" +build_subbinary 13 nvpair-job-scheduler "$V_SCHED" +build_subbinary 14 nvpair-tui "$V_TUI" BIN_OUT="$ROOT/build/bin" @@ -132,6 +135,7 @@ rm -rf "$BIN_OUT" mkdir -p "$BIN_OUT" cp "$ROOT/ollama-proxy/ollama-proxy" "$BIN_OUT/ollama-proxy" cp "$ROOT/lmstudio-proxy/lmstudio-proxy" "$BIN_OUT/lmstudio-proxy" +cp "$ROOT/llamacpp-proxy/llamacpp-proxy" "$BIN_OUT/llamacpp-proxy" cp "$ROOT/nvpair-node-info/nvpair-node-info" "$BIN_OUT/nvpair-node-info" cp "$ROOT/nvpair-node-scanner/nvpair-node-scanner" "$BIN_OUT/nvpair-node-scanner" cp "$ROOT/nvpair-manual-nodes/nvpair-manual-nodes" "$BIN_OUT/nvpair-manual-nodes" @@ -151,6 +155,7 @@ echo "========================================" echo printf ' Proxy: %s\n' "$BIN_OUT/ollama-proxy" printf ' LM Studio Proxy: %s\n' "$BIN_OUT/lmstudio-proxy" +printf ' llama.cpp Proxy: %s\n' "$BIN_OUT/llamacpp-proxy" printf ' Node Info: %s\n' "$BIN_OUT/nvpair-node-info" printf ' Node Scanner: %s\n' "$BIN_OUT/nvpair-node-scanner" printf ' Manual Nodes: %s\n' "$BIN_OUT/nvpair-manual-nodes" diff --git a/services/installer/linux/INSTALL.md b/services/installer/linux/INSTALL.md index 7e50420f..73456ac9 100644 --- a/services/installer/linux/INSTALL.md +++ b/services/installer/linux/INSTALL.md @@ -25,6 +25,7 @@ NVIDIA-Personal-AI-Router-/ │ ├── nvpair-ui-broker # primary entry point (JSON-RPC over stdio / IPC) │ ├── ollama-proxy │ ├── lmstudio-proxy +│ ├── llamacpp-proxy │ ├── nvpair-node-info │ ├── nvpair-node-scanner │ ├── nvpair-manual-nodes diff --git a/services/installer/nvpair-setup.nsi b/services/installer/nvpair-setup.nsi index eaeb5a0f..60fe81ad 100644 --- a/services/installer/nvpair-setup.nsi +++ b/services/installer/nvpair-setup.nsi @@ -109,6 +109,7 @@ FunctionEnd DetailPrint "Checking for running ${PRODUCT_NAME} processes..." nsExec::ExecToLog 'taskkill /F /IM "ollama-proxy.exe"' nsExec::ExecToLog 'taskkill /F /IM "lmstudio-proxy.exe"' + nsExec::ExecToLog 'taskkill /F /IM "llamacpp-proxy.exe"' nsExec::ExecToLog 'taskkill /F /IM "nvpair-node-info.exe"' nsExec::ExecToLog 'taskkill /F /IM "nvpair-node-scanner.exe"' nsExec::ExecToLog 'taskkill /F /IM "nvpair-manual-nodes.exe"' @@ -159,6 +160,10 @@ Section "Install" ; same dual-protocol port (loopback plaintext + cluster mTLS ingress); it ; listens for clients and browses mDNS, so it gets firewall rules below. File "..\build\bin\lmstudio-proxy.exe" + ; llamacpp-proxy fronts already-running llama-server instances (OpenAI API) + ; on :8084 with the same dual-protocol port; it never binds the adopt probe + ; (default :8082). + File "..\build\bin\llamacpp-proxy.exe" File "..\build\bin\nvpair-node-info.exe" File "..\build\bin\nvpair-node-scanner.exe" File "..\build\bin\nvpair-manual-nodes.exe" @@ -229,6 +234,7 @@ Section "Install" ; public networks. nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Ollama Proxy" dir=in action=allow program="$INSTDIR\bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR LM Studio Proxy" dir=in action=allow program="$INSTDIR\bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet' + nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR llama.cpp Proxy" dir=in action=allow program="$INSTDIR\bin\llamacpp-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Node Info" dir=in action=allow program="$INSTDIR\bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR Node Scanner" dir=in action=allow program="$INSTDIR\bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet' @@ -236,6 +242,7 @@ Section "Install" ; mDNS needs UDP 5353 inbound nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\ollama-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS LM Studio Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\lmstudio-proxy.exe" enable=yes profile=any remoteip=localsubnet' + nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS llama.cpp Proxy (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\llamacpp-proxy.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Node Info (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-node-info.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Node Scanner (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-node-scanner.exe" enable=yes profile=any remoteip=localsubnet' nsExec::ExecToLog 'netsh advfirewall firewall add rule name="NVPAIR mDNS Workload Manager (UDP 5353)" dir=in action=allow protocol=UDP localport=5353 program="$INSTDIR\bin\nvpair-workload-manager.exe" enable=yes profile=any remoteip=localsubnet' @@ -272,8 +279,12 @@ Section "Uninstall" ; Remove firewall exceptions nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Ollama Proxy"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR LM Studio Proxy"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR llama.cpp Proxy"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Node Info"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS (UDP 5353)"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS LM Studio Proxy (UDP 5353)"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS llama.cpp Proxy (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR Node Scanner"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS Node Info (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="NVPAIR mDNS Node Scanner (UDP 5353)"' @@ -290,6 +301,7 @@ Section "Uninstall" ; Remove files Delete "$INSTDIR\bin\ollama-proxy.exe" Delete "$INSTDIR\bin\lmstudio-proxy.exe" + Delete "$INSTDIR\bin\llamacpp-proxy.exe" Delete "$INSTDIR\bin\nvpair-node-info.exe" Delete "$INSTDIR\bin\nvpair-node-scanner.exe" Delete "$INSTDIR\bin\nvpair-manual-nodes.exe" diff --git a/services/installer_build.sh b/services/installer_build.sh index cba2c440..aadeed80 100755 --- a/services/installer_build.sh +++ b/services/installer_build.sh @@ -139,6 +139,7 @@ fi cp "$BIN_SRC/ollama-proxy" "$STAGE/bin/" cp "$BIN_SRC/lmstudio-proxy" "$STAGE/bin/" +cp "$BIN_SRC/llamacpp-proxy" "$STAGE/bin/" cp "$BIN_SRC/nvpair-node-info" "$STAGE/bin/" cp "$BIN_SRC/nvpair-node-scanner" "$STAGE/bin/" cp "$BIN_SRC/nvpair-manual-nodes" "$STAGE/bin/" diff --git a/services/llamacpp-proxy/README.md b/services/llamacpp-proxy/README.md new file mode 100644 index 00000000..13f1fe79 --- /dev/null +++ b/services/llamacpp-proxy/README.md @@ -0,0 +1,444 @@ + + +# llama.cpp Proxy + +A discovery-aware HTTP reverse proxy for llama.cpp nodes on the local network. It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lc]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket). + +> **Clone of `lmstudio-proxy`.** This proxy is a deliberate clone of [`lmstudio-proxy`](../lmstudio-proxy/README.md) so the two share identical routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lc` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags workloads `llamacpp`, and persists its port to its own file. Inference eligibility uses **loaded** models only (`EngineLoadedModels`), never the on-disk catalog. It has no `--alias-address`, so its self-forward guard covers only its own listener. + +## Build + +```bash +go build -o llamacpp-proxy . +``` + +## Usage + +``` +llamacpp-proxy [flags] +``` + +### Flags + +| Flag | Default | Description | +|------|---------|-------------| +| `--port` | `8084` | HTTP listen port for request forwarding | +| `--ignore-persisted-port` | `false` | Use `--port` even when a prior runtime port was saved | +| `--ipc` | *(empty — use stdio)* | Path to a Unix domain socket or Windows named pipe for IPC | +| `--cluster-dir` | *(empty)* | Cluster trust directory (`node.crt`/`node.key` plus trusted pins). Enables the LAN mTLS inference ingress while this node is a cluster member; empty means no ingress and no peer candidates. | +| `--log-level` | *(`$NVPAIR_LOG_LEVEL`, else `info`)* | Initial log level: `debug`, `info`, `warn`, or `error`. Changeable at runtime with `log/set-level`. | +| `--version` | | Print version and exit | + +### HTTP Reverse Proxy + +The proxy listens on `--port` (default 8084) and forwards incoming HTTP requests to the currently active llama.cpp node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory (the full backend catalog, including ids that are not currently loaded). Point your OpenAI-compatible client at `http://localhost:8084` and the proxy handles routing. + +**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to the local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart. + +**Persisted port.** A port chosen at runtime via the `set-port` request (see below) is saved as `llamacpp-proxy-port.json` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and **restored on startup**, taking precedence over `--port`/the default. A valid stored port is always honoured; there is no legacy-port special case. The broker uses `--ignore-persisted-port` while reserving the managed `8084` facade. + +Node selection: +- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose **loaded** llama.cpp inventory advertises the exact requested model ID. Catalog-only ids (present on disk but not resident in memory) are not eligible; a request for one returns a local `502` without contacting the backend. An empty or non-matching loaded set is excluded until a later discovery update. +- **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order. +- **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md). +- **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model. +- **Failover**: If the selected node disappears from the discovery set, the proxy falls back to auto-select and emits a `node/selection-changed` notification. A transport error or retryable status, including a model `404` from an advertised owner with stale inventory, steps to the next eligible owner. + +### IPC Transport + +By default the proxy communicates over **stdin/stdout** using newline-delimited JSON-RPC 2.0 (one message per line). All diagnostic logging goes to **stderr**. + +For environments where stdout may conflict with the host process (e.g. Electron), pass `--ipc` to redirect the JSON-RPC channel to a named socket or pipe. The parent process should create and listen on the endpoint before spawning the proxy. + +```bash +# Default: stdin/stdout +llamacpp-proxy + +# Unix domain socket +llamacpp-proxy --ipc /tmp/llamacpp-proxy.sock + +# Windows named pipe +llamacpp-proxy --ipc \\.\pipe\llamacpp-proxy +``` + +## JSON-RPC 2.0 Protocol + +All messages conform to the [JSON-RPC 2.0 specification](https://www.jsonrpc.org/specification). Messages are newline-delimited (one JSON object per `\n`). + +### Node Object + +Nodes are represented throughout the protocol with this shape: + +```json +{ + "id": "22222222-2222-2222-2222-222222222222", + "host": "my-workstation", + "port": 8084, + "addresses": ["192.168.1.50"], + "txt": ["uuid=22222222-2222-2222-2222-222222222222", "lc=8084"], + "models": ["qwen2.5-7b-instruct"], + "ip": "192.168.1.50" +} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `id` | string | Stable per-host UUID from the discovery record (the ID you supply, for a manual node) | +| `host` | string | Hostname, for display — routing never keys on it | +| `port` | int | llama.cpp port from the discovery record's `lc` service entry | +| `addresses` | string[] | Addresses to dial. A node fed by the discovery relay always carries exactly one canonical address; several only ever appear on a manual node | +| `txt` | string[] | The discovery record's TXT pairs, carried verbatim | +| `models` | string[] | The node's **loaded** llama.cpp model inventory from the discovery snapshot (`EngineLoadedModels`). Model-bearing inference is eligible only when this list advertises the exact requested model ID. Catalog-only ids are omitted. An omitted or empty list excludes the node from that request until a later loaded-set update; it remains available for non-inference routes and model-list aggregation | +| `ip` | string | The single canonical LAN address to dial or display, resolved from the node's `ip=` TXT if present and otherwise the best-scored advertised IPv4. Stamped onto outbound `node/*` notifications so consumers agree with the address the proxy routes to | + +--- + +### Notifications (proxy → client) + +Notifications have no `id` field and do not expect a response. + +#### `ready` + +Sent after startup, before discovery begins, and again after every successful `set-port` rebind — `port` carries the port now bound. + +```json +{"jsonrpc":"2.0","method":"ready","params":{"version":"0.1.0","port":8084}} +``` + +| Param | Type | Description | +|-------|------|-------------| +| `version` | string | Proxy version | +| `port` | int | HTTP listen port | + +#### `error` + +Sent when a fatal startup condition stops the proxy from serving — currently only a failed bind — immediately before the process exits non-zero. + +```json +{"jsonrpc":"2.0","method":"error","params":{"code":"bind-failed","message":"failed to bind port 8084: ...","port":8084}} +``` + +#### `node/discovered` + +A new llama.cpp node appeared on the network. + +```json +{"jsonrpc":"2.0","method":"node/discovered","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":8084,"addresses":["192.168.1.50"]}} +``` + +#### `node/updated` + +A previously discovered node changed its host, port, or addresses. + +```json +{"jsonrpc":"2.0","method":"node/updated","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":8084,"addresses":["192.168.1.51"]}} +``` + +#### `node/removed` + +A node is no longer present in the discovery set (it left the relay's `lc` nodes, or a manual node was removed). + +```json +{"jsonrpc":"2.0","method":"node/removed","params":{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":8084,"addresses":["192.168.1.50"]}} +``` + +#### `node/selection-changed` + +The active node selection changed (either explicitly via `node/select` or because the selected node was removed). + +```json +{"jsonrpc":"2.0","method":"node/selection-changed","params":{"id":"22222222-2222-2222-2222-222222222222"}} +``` + +An empty `id` means the proxy has reverted to auto-select mode. + +#### `proxy/request-started` + +A request has been committed to a target and its response body is about to stream. It pairs by `id` with the matching `proxy/request`, so a consumer can keep an in-flight count per node. Model-list aggregation has no single target, so it reports `"target":"cluster"` with no `node_id`; a request rejected before forwarding was never in flight and gets no started event. + +```json +{"jsonrpc":"2.0","method":"proxy/request-started","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/v1/chat/completions","target":"192.168.1.50:8084"}} +``` + +#### `proxy/request` + +A proxied request finished, or was rejected before forwarding. `duration_ms` covers the whole request; `ttfb_ms` is the time to the upstream's status line and is omitted where no response header arrived (rejection and transport-error paths). `error` carries normalized error text. This is operational metadata only — request and response bodies are never reported. + +```json +{"jsonrpc":"2.0","method":"proxy/request","params":{"id":"17","node_id":"22222222-2222-2222-2222-222222222222","method":"POST","path":"/v1/chat/completions","target":"192.168.1.50:8084","status":200,"duration_ms":6120,"ttfb_ms":95}} +``` + +#### `workload:started` / `workload:completed` / `workload:errored` + +One lifecycle transition per forwarded inference request, carrying a single `workloadInfo`. `engine` is always `llamacpp`; `originatedFrom` is left empty for the broker to stamp, and `scheduledOn` names the node that actually served (re-pointed if failover moved the request). The broker relays these to `nvpair-workload-manager`. The proxy never emits `workload:submitted` — it forwards immediately rather than queueing. + +```json +{"jsonrpc":"2.0","method":"workload:started","params":{"workloadInfo":{"id":"17","model":"qwen2.5-7b-instruct","engine":"llamacpp","runId":"3ce8a1740b62df95","state":"running","originatedFrom":"","scheduledOn":"22222222-2222-2222-2222-222222222222","createdAt":1716998400000,"startedAt":1716998400000,"completedAt":null,"error":null,"requesterId":null}}} +``` + +#### `node/activity` + +Raised while a node's engine is streaming a response back through the proxy: every successful write of upstream body bytes reports the node that produced them. The broker relays it to `nvpair-node-scanner`, which treats it as proof of life and cancels that node's eviction — a node saturated by inference cannot answer a liveness probe, but it is demonstrably alive precisely because it is streaming. Coalesced to one report per node per 2s (`nvpair-shared/nodeactivity`), since a generation writes hundreds of chunks and the scanner treats one report as good for a minute. `msSince` is the age of the observation; the broker adds its own relay delay before passing it on. + +Only bytes that came from the upstream count. The proxy's own error bodies travel through the same writer and are never reported, because they say nothing about the node. + +```json +{"jsonrpc":"2.0","method":"node/activity","params":{"hostUuid":"22222222-2222-2222-2222-222222222222","msSince":0}} +``` + +#### `errors:report` / `errors:clear` + +Entries for the `nvpair-errors` pipeline, keyed by a stable `id` so a report and its clear cannot drift. A node dropping out of the discovery set raises `llamacpp-proxy:upstream-unreachable:`; its reappearance clears the same id. `nodeId` and `timestamp` are left unset for the broker to stamp. + +```json +{"jsonrpc":"2.0","method":"errors:report","params":{"id":"llamacpp-proxy:upstream-unreachable:22222222-2222-2222-2222-222222222222","message":"Upstream node \"my-workstation\" is no longer reachable (dropped from discovery)","severity":"warning","action":"none"}} +``` + +--- + +### Requests (client → proxy) + +Requests carry an `id` and receive a response. + +#### `nodes/list` + +Returns all currently discovered nodes. + +**Request:** +```json +{"jsonrpc":"2.0","id":1,"method":"nodes/list"} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":1,"result":{"nodes":[{"id":"22222222-2222-2222-2222-222222222222","host":"my-workstation","port":8084,"addresses":["192.168.1.50"]}]}} +``` + +#### `node/select` + +Pin the proxy to route HTTP traffic to a specific node. Pass an empty `id` to return to auto-select. + +**Request:** +```json +{"jsonrpc":"2.0","id":2,"method":"node/select","params":{"id":"22222222-2222-2222-2222-222222222222"}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":2,"result":{"id":"22222222-2222-2222-2222-222222222222"}} +``` + +**Error** (node not found): +```json +{"jsonrpc":"2.0","id":2,"error":{"code":-32602,"message":"node \"xyz\" not found"}} +``` + +#### `node/selected` + +Query the currently selected node. + +**Request:** +```json +{"jsonrpc":"2.0","id":3,"method":"node/selected"} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":3,"result":{"id":"22222222-2222-2222-2222-222222222222"}} +``` + +An empty `id` means auto-select is active. + +#### `node/set-priority` + +Set the **auto-routing priority order** — an ordered list of node IDs, highest +priority first, optionally with each node's pending-work count and GPU pressure +in `ranks`. Delivered by `nvpair-ui-broker` on behalf of `nvpair-job-scheduler`, +which ranks the cluster least-loaded-first by total pending workload across +engines plus smoothed GPU pressure (see +[`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md)). The snapshot is +stored verbatim and applied at request time. A `nodes`-only payload is valid and +supplies zero pending and GPU-pressure baselines. + +**Request:** +```json +{"jsonrpc":"2.0","id":8,"method":"node/set-priority","params":{"nodes":["MY-PC","LAB-DESK-B","GPU-RIG"],"ranks":[{"id":"MY-PC","pending":0,"gpuPressure":0,"rank":0},{"id":"LAB-DESK-B","pending":1,"gpuPressure":1,"rank":1},{"id":"GPU-RIG","pending":3,"gpuPressure":3,"rank":2}]}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":8,"result":{"count":3}} +``` + +| Field | Type | Description | +|-------|------|-------------| +| `count` | int | Number of node IDs stored (the length of the accepted list) | + +Semantics: +- **Capability gate.** A model-bearing inference request first intersects the + discovery snapshot with nodes advertising that exact loaded llama.cpp model ID. + Selection, priority, reservations, and failover operate only on that + request-local owner set. If it is empty, the proxy returns `502` without + contacting an engine. +- **Auto ordering.** Within the eligible owner set, the proxy picks the listed + node carrying the least estimated load — + `pending + gpuPressure` from the last snapshot plus the proxy's own + reservations for requests it has already dispatched but whose workload feedback + has not come back yet — breaking ties by position in `nodes`. It increments the + chosen node's reservation before forwarding, so a concurrent burst spreads + instead of repeatedly choosing from the same stale snapshot. That node moves to + the front of this request's failover list and the rest keeps its order, so a + transport error or retryable status (the existing failover trigger) steps to + the next candidate. +- **Snapshot reset.** Each new snapshot replaces the pending and GPU-pressure + baselines and clears the reservations. GPU pressure is clamped to the + scheduler's 0–3 range. +- **Eligible manual pin wins.** An active `node/select` pin takes precedence when + it is in the request's owner set. An ineligible pin is ignored for that request, + so automatic reservations still apply among eligible owners. Clearing the pin + (`node/select` with an empty `id`) activates the most-recently-set list. Setting + a priority list does **not** emit `node/selection-changed` (the manual selection + is unchanged). +- **Unknown IDs are ignored.** IDs not currently in discovery are kept in the + stored list (a node may appear later) but contribute nothing until discovered. +- **Eligible unlisted nodes are a lowest-priority fallback.** An advertised owner + absent from the list stays routable, but only after every listed owner — + ordered among themselves by the default stable ID sort. This ensures an + eligible manually-added node the scheduler never saw is never stranded. +- **Empty list reverts to default.** `{"nodes":[]}` clears the scheduler's + influence and returns the proxy to its default auto ordering (eligible + discovered nodes by stable ID). + +The list persists only in memory for the proxy's lifetime; it is not saved across +restarts. On restart the proxy comes back with an empty list, and the broker +re-pushes the last order once the proxy re-announces `ready`. + +#### `set-port` + +Change the HTTP listen port at runtime and persist the choice. The proxy +binds the new port first (so a bind failure leaves the current listener +serving), starts serving on it, then closes the old listener — in-flight +connections on the old port drain naturally. The new port is saved to +`llamacpp-proxy-port.json` and a fresh `ready` notification announces it. + +**Request:** +```json +{"jsonrpc":"2.0","id":7,"method":"set-port","params":{"port":1300}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":7,"result":{"version":"0.9.0","port":1300}} +``` + +**Error** (port in use / out of range): +```json +{"jsonrpc":"2.0","id":7,"error":{"code":-32000,"message":"failed to bind port 1300: ..."}} +``` + +When supervised by `nvpair-ui-broker`, callers reach this as `proxy:set-port`, +and the broker first steers the port clear of any running engine's port +(engines take precedence) before handing it down — see the broker README. + +#### `node/add-manual` + +Add a node manually (for networks where mDNS is blocked). If the node ID already exists as a manual node, it is updated. + +**Request:** +```json +{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","host":"remote-server","port":8084,"addresses":["10.0.1.50"]}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":5,"result":{"added":true}} +``` + +The proxy emits a `node/discovered` notification (or `node/updated` if the node was already registered). Manual nodes are a separate overlay that discovery snapshots never touch — they persist until explicitly removed. + +#### `node/remove-manual` + +Remove a previously added manual node. + +**Request:** +```json +{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server"}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":6,"result":{"removed":true}} +``` + +The proxy emits a `node/removed` notification and clears the active selection if it pointed to this node. + +#### `node/set-local-backend` + +Tell the proxy which loopback engine this node's own traffic terminates on. The broker sends it once the local llama.cpp address and health are known. It is the target the cluster mTLS ingress forwards to, and the substitute used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears it, and the ingress then answers `503`. + +**Request:** +```json +{"jsonrpc":"2.0","id":9,"method":"node/set-local-backend","params":{"engine":"llamacpp","host":"127.0.0.1","port":8082,"healthy":true}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":9,"result":{"ok":true}} +``` + +#### `log/set-level` + +Change the active log level at runtime (`debug`, `info`, `warn`, `error`). Accepted as a request or a notification; as a request it responds with the resolved level and rejects an unknown one with `-32602`. + +**Request:** +```json +{"jsonrpc":"2.0","id":10,"method":"log/set-level","params":{"level":"debug"}} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":10,"result":{"level":"debug"}} +``` + +#### `shutdown` + +Gracefully shuts down the proxy. + +**Request:** +```json +{"jsonrpc":"2.0","id":4,"method":"shutdown"} +``` + +**Response:** +```json +{"jsonrpc":"2.0","id":4,"result":null} +``` + +--- + +### Error Codes + +Standard JSON-RPC 2.0 error codes apply: + +| Code | Meaning | +|------|---------| +| `-32601` | Method not found | +| `-32602` | Invalid params | +| `-32000` | The request was well-formed but could not be carried out — returned by `set-port` when the new port cannot be bound | + +--- + +## Shutdown + +The proxy shuts down gracefully on any of: + +1. **stdin EOF** — parent closes stdin (stdio mode only) +2. **`shutdown` JSON-RPC request** — programmatic shutdown +3. **SIGINT / SIGTERM** — standard OS signals + +## Discovery + +The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lc` (llama.cpp) nodes (`discovery:subscribe {services:[lc]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set. diff --git a/services/llamacpp-proxy/activity_test.go b/services/llamacpp-proxy/activity_test.go new file mode 100644 index 00000000..49a63541 --- /dev/null +++ b/services/llamacpp-proxy/activity_test.go @@ -0,0 +1,80 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// The two inference proxies are held to deliberate parity, so the liveness +// report a streaming response raises is asserted on both. See ollama-proxy's +// activity_test.go for the reasoning behind the signal itself. + +// TestStreamedBytesReportNodeActivity mirrors the ollama-proxy test of the same +// name. +func TestStreamedBytesReportNodeActivity(t *testing.T) { + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"choices":[]}`)) + })) + defer upstream.Close() + + rec := &prRec{} + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "serving-node", upstream.URL, "qwen")) + p := NewProxy(NewCodec(rec), disc, 1235) + + p.handleHTTP(httptest.NewRecorder(), + httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen"}`))) + + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) && !rec.has(`"method":"node/activity"`) { + time.Sleep(5 * time.Millisecond) + } + if !rec.has(`"method":"node/activity"`) { + t.Fatal("no node/activity was reported after the upstream streamed a response") + } + if !rec.has(`"hostUuid":"serving-node"`) { + t.Fatal("node/activity did not name the node that served the request") + } +} + +// A node that never wrote a response byte has proved nothing and must not be +// vouched for. +func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) { + rec := &prRec{} + disc := NewDiscovery() + // A port nothing is listening on: the dial fails, so no upstream byte can + // ever reach the client. It still has to advertise the requested model, or + // candidate pruning rejects the request before anything is dialled and the + // test passes without exercising the dial failure at all. + disc.AddManual(Node{ + ID: "dead-node", + Addresses: []string{"127.0.0.1"}, + Port: closedPortFor(t), + Models: []string{"qwen"}, + }) + p := NewProxy(NewCodec(rec), disc, 1235) + + p.handleHTTP(httptest.NewRecorder(), + httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen"}`))) + + if rec.has(`"method":"node/activity"`) { + t.Fatal("activity was reported for a node that never answered") + } +} + +// closedPortFor returns a port nothing is listening on, by binding and releasing +// it, so the dial is a prompt refusal rather than a timeout. +func closedPortFor(t *testing.T) int { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + port := nodeFor(t, "probe", srv.URL).Port + srv.Close() + return port +} diff --git a/services/llamacpp-proxy/choose_reachable_test.go b/services/llamacpp-proxy/choose_reachable_test.go new file mode 100644 index 00000000..1ca29b03 --- /dev/null +++ b/services/llamacpp-proxy/choose_reachable_test.go @@ -0,0 +1,271 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "net" + "net/http" + "net/http/httptest" + "path/filepath" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "nvpair-shared/clustertrust" + "nvpair-shared/clustertrusttest" + "nvpair-shared/reach" +) + +// waitForTarget polls targetURL until it settles on want. +// +// Routing never waits on a handshake — reach.Prefer answers with the node's own +// ranking and confirms behind the request — so the address a multi-homed peer +// settles on is what the requests after the first see. +func waitForTarget(t *testing.T, p *Proxy, n Node, want string) { + t.Helper() + deadline := time.Now().Add(2 * time.Second) + for { + u := p.targetURL(n) + if u != nil && u.Host == want { + return + } + if time.Now().After(deadline) { + got := "" + if u != nil { + got = u.Host + } + t.Fatalf("targetURL settled on %s, want %s", got, want) + } + time.Sleep(time.Millisecond) + } +} + +// countingChooser installs a target chooser that records how many connection +// attempts routing makes and whether they succeed, so a test can assert on +// confirmation behaviour without opening sockets. +func countingChooser(p *Proxy, accept bool) *atomic.Int32 { + var dials atomic.Int32 + p.targets = reach.NewChooser(reach.WithDial( + func(_, _ string, _ time.Duration) (net.Conn, error) { + dials.Add(1) + if !accept { + return nil, net.ErrClosed + } + c1, c2 := net.Pipe() + _ = c2.Close() + return c1, nil + })) + return &dials +} + +func TestChooseReachableFailsOverForPinnedPeer(t *testing.T) { + const peerUUID = "principal-peer" + const reachable = "192.0.2.11" + clusterDir := filepath.Join(t.TempDir(), "cluster") + clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID) + + p := testProxy(NewDiscovery(), 1235) + p.mesh = clustertrust.Open(clusterDir) + var dials atomic.Int32 + p.targets = reach.NewChooser(reach.WithDial( + func(_, address string, _ time.Duration) (net.Conn, error) { + dials.Add(1) + host, _, _ := net.SplitHostPort(address) + if host != reachable { + return nil, net.ErrClosed + } + c1, c2 := net.Pipe() + _ = c2.Close() + return c1, nil + })) + + n := Node{ + ID: "peer-a", + Port: 1234, + Addresses: []string{"192.0.2.10", "192.0.2.11"}, + ClusterUUID: peerUUID, + } + // The first request is not made to wait for the confirmation, so it uses the + // node's own top-ranked address; the ones behind it use the one that answers. + if u := p.targetURL(n); u == nil || u.Host != net.JoinHostPort("192.0.2.10", "1234") { + t.Fatalf("first selection = %v, want the published ranking without waiting", u) + } + waitForTarget(t, p, n, net.JoinHostPort(reachable, "1234")) + if dials.Load() != 2 { + t.Fatalf("pinned peer triggered %d TCP probes, want both candidates tried", dials.Load()) + } +} + +func TestChooseReachableProbesPlainMultiHomed(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + dials := countingChooser(p, true) + n := Node{ + ID: "manual-a", + Port: 1234, + Addresses: []string{"192.0.2.10", "192.0.2.11"}, + } + if u := p.targetURL(n); u == nil { + t.Fatal("targetURL returned nil") + } + deadline := time.Now().Add(2 * time.Second) + for dials.Load() == 0 { + if time.Now().After(deadline) { + t.Fatal("plain multi-homed target did not confirm reachability") + } + time.Sleep(time.Millisecond) + } +} + +// TestTargetURLFailsOverToAReachableAddress is the reported defect at the routing +// layer: a peer whose canonical address is a direct-connect link this host cannot +// reach must still be routed to, at the address that answers. +func TestTargetURLFailsOverToAReachableAddress(t *testing.T) { + const reachable = "10.172.55.129" + p := testProxy(NewDiscovery(), 1235) + p.targets = reach.NewChooser(reach.WithDial( + func(_, address string, _ time.Duration) (net.Conn, error) { + host, _, _ := net.SplitHostPort(address) + if host != reachable { + return nil, net.ErrClosed + } + c1, c2 := net.Pipe() + _ = c2.Close() + return c1, nil + })) + + n := Node{ + ID: "spark", + Port: 1234, + // The node's own ranking leads with a link only its cabled neighbour can + // reach; this host is not that neighbour. + Addresses: []string{"192.168.240.1", reachable}, + TXT: []string{"ip=192.168.240.1", "ips=192.168.240.1," + reachable}, + } + waitForTarget(t, p, n, net.JoinHostPort(reachable, "1234")) +} + +// TestNodeCandidatesKeepsPublishedOrder: the node ranked its addresses from +// evidence no observer has, so routing must try them in that order rather than +// re-sorting by address class. +func TestNodeCandidatesKeepsPublishedOrder(t *testing.T) { + n := Node{ + ID: "spark", + Port: 1234, + Addresses: []string{"192.168.240.1", "10.172.55.129"}, + TXT: []string{"ip=10.172.55.129", "ips=10.172.55.129,192.168.240.1"}, + } + got := nodeCandidates(n) + want := []string{ + net.JoinHostPort("10.172.55.129", "1234"), + net.JoinHostPort("192.168.240.1", "1234"), + } + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { + t.Fatalf("nodeCandidates = %v, want %v", got, want) + } +} + +// fakeNetwork is a chooser dialer whose accepting address can be moved, so a test +// can describe an address that stops answering and another that starts. Probes run +// on background goroutines, so both fields are read concurrently with the test. +type fakeNetwork struct { + mu sync.Mutex + accepting string + dials atomic.Int32 +} + +func (f *fakeNetwork) accept(address string) { + f.mu.Lock() + defer f.mu.Unlock() + f.accepting = address +} + +func (f *fakeNetwork) install(p *Proxy) { + p.targets = reach.NewChooser(reach.WithDial( + func(_, address string, _ time.Duration) (net.Conn, error) { + f.dials.Add(1) + host, _, _ := net.SplitHostPort(address) + f.mu.Lock() + accepting := f.accepting + f.mu.Unlock() + if host != accepting { + return nil, net.ErrClosed + } + c1, c2 := net.Pipe() + _ = c2.Close() + return c1, nil + })) +} + +// confirmedDeadPeer builds a proxy whose only routing target is a multi-homed peer +// whose confirmed address no longer answers: the address is a loopback endpoint +// whose server is already closed, so a forwarded request fails at the transport +// the way an unplugged link does, while the chooser still believes in it. +func confirmedDeadPeer(t *testing.T, replacement string) (*Proxy, Node, *fakeNetwork) { + t.Helper() + dead := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + n := nodeForModel(t, "peer-a", dead.URL, "llama") + dead.Close() + confirmed := n.Addresses[0] + n.Addresses = append(n.Addresses, replacement) + + disc := NewDiscovery() + disc.AddManual(n) + p := testProxy(disc, 1235) + fake := &fakeNetwork{accepting: confirmed} + fake.install(p) + + waitForTarget(t, p, n, net.JoinHostPort(confirmed, strconv.Itoa(n.Port))) + return p, n, fake +} + +// assertReprobed moves the accepting address and requires the selections after the +// failure to confirm again and land on the replacement. A cached winner that +// outlived the failure keeps answering with the old address and dials nothing. +func assertReprobed(t *testing.T, p *Proxy, n Node, fake *fakeNetwork, replacement string) { + t.Helper() + probesBefore := fake.dials.Load() + fake.accept(replacement) + + waitForTarget(t, p, n, net.JoinHostPort(replacement, strconv.Itoa(n.Port))) + if fake.dials.Load() == probesBefore { + t.Fatal("selection probed nothing: the failed address is still cached") + } +} + +// TestUpstreamTransportFailureReprobesTheNextSelection: a dial failure against a +// multi-homed peer must retire the confirmed address. Without that, every later +// request keeps being sent to the address that just failed, and the peer's other +// published addresses are never tried — which is the whole reason routing confirms +// reachability in the first place. +func TestUpstreamTransportFailureReprobesTheNextSelection(t *testing.T) { + const replacement = "192.0.2.11" + p, n, fake := confirmedDeadPeer(t, replacement) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + if rec.Code != http.StatusBadGateway { + t.Fatalf("status = %d, want 502 from the only, unreachable candidate", rec.Code) + } + + assertReprobed(t, p, n, fake, replacement) +} + +// TestModelListTransportFailureReprobesTheNextSelection: the aggregated model list +// reaches every candidate directly, so it learns about a dead address before any +// inference request does, and must retire it on the same evidence. +func TestModelListTransportFailureReprobesTheNextSelection(t *testing.T) { + const replacement = "192.0.2.11" + p, n, fake := confirmedDeadPeer(t, replacement) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil)) + if rec.Code != http.StatusServiceUnavailable { + t.Fatalf("status = %d, want 503 when the only inventory source is unreachable", rec.Code) + } + + assertReprobed(t, p, n, fake, replacement) +} diff --git a/services/llamacpp-proxy/cluster_trust_test.go b/services/llamacpp-proxy/cluster_trust_test.go new file mode 100644 index 00000000..39f1d188 --- /dev/null +++ b/services/llamacpp-proxy/cluster_trust_test.go @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "path/filepath" + "testing" + + "nvpair-shared/clustertrust" + "nvpair-shared/clustertrusttest" +) + +// TestResolveCandidatesFollowsLivePinSet reproduces the join ordering that made +// a freshly-joined node route to nobody. The peer is discovered BEFORE this node +// has any cluster identity, and its discovery record never changes again — which +// is the normal steady state, because the relay only re-sends a node when its +// mDNS record actually moves. Routing must still pick the peer up the moment the +// pin lands, and drop it again the moment the pin is removed, because it reads +// the live pin set rather than a trust flag cached at discovery time. +func TestResolveCandidatesFollowsLivePinSet(t *testing.T) { + const peerUUID = "principal-peer" + clusterDir := filepath.Join(t.TempDir(), "cluster") + + disc := NewDiscovery() + disc.SetSubscribed([]Node{{ + ID: "peer-a", Host: "peer-a", Port: 1234, + Addresses: []string{"192.0.2.10"}, + IP: "192.0.2.10", + ClusterUUID: peerUUID, + }}) + p := testProxy(disc, 1235) + p.mesh = clustertrust.Open(clusterDir) + + // Pre-join: no identity, no pins, so the peer is not a routable target. + if got := p.resolveCandidates(""); len(got) != 0 { + t.Fatalf("pre-join candidates = %+v, want none", got) + } + + // The cluster-manager lands the join on disk while the proxy is running. No + // new discovery snapshot arrives — the peer's mDNS record has not changed. + clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID) + + cands := p.resolveCandidates("") + if len(cands) != 1 { + t.Fatalf("post-join candidates = %+v, want the peer", cands) + } + if cands[0].id != "peer-a" || cands[0].peerUUID != peerUUID || cands[0].url.Scheme != "https" { + t.Fatalf("post-join candidate = %+v, want peer-a over https pinned to %s", cands[0], peerUUID) + } + + // Removing the peer from the cluster retires it as a target just as promptly, + // again with no discovery event involved. + clustertrusttest.RemovePeerPin(t, clusterDir, peerUUID) + if got := p.resolveCandidates(""); len(got) != 0 { + t.Fatalf("post-removal candidates = %+v, want none", got) + } +} + +// TestResolveCandidatesRejectsUnpinnedClusteredPeer keeps the isolation property +// honest now that trust is read locally: a peer that advertises a cluster +// principal we hold no pin for is not routable, even though this node is itself +// a healthy cluster member. Two clusters on one LAN must not route to each other. +func TestResolveCandidatesRejectsUnpinnedClusteredPeer(t *testing.T) { + clusterDir := filepath.Join(t.TempDir(), "cluster") + clustertrusttest.Join(t, clusterDir, "cluster-ours", "principal-self", "principal-ourpeer") + + disc := NewDiscovery() + disc.SetSubscribed([]Node{{ + ID: "stranger", Host: "stranger", Port: 1234, + Addresses: []string{"192.0.2.30"}, + IP: "192.0.2.30", + ClusterUUID: "principal-stranger", + }}) + p := testProxy(disc, 1235) + p.mesh = clustertrust.Open(clusterDir) + + if got := p.resolveCandidates(""); len(got) != 0 { + t.Fatalf("candidates = %+v, want none for a peer in another cluster", got) + } +} diff --git a/services/llamacpp-proxy/codec.go b/services/llamacpp-proxy/codec.go new file mode 100644 index 00000000..5d6fd177 --- /dev/null +++ b/services/llamacpp-proxy/codec.go @@ -0,0 +1,18 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +// The newline-delimited JSON-RPC 2.0 codec is single-sourced in +// nvpair-shared/jsonrpc. These local aliases keep this package's call sites and +// tests unchanged after removing the copy-pasted per-service codec. + +import "nvpair-shared/jsonrpc" + +type ( + Message = jsonrpc.Message + RPCError = jsonrpc.RPCError + Codec = jsonrpc.Codec +) + +var NewCodec = jsonrpc.NewCodec diff --git a/services/llamacpp-proxy/discovery.go b/services/llamacpp-proxy/discovery.go new file mode 100644 index 00000000..a8f2d30d --- /dev/null +++ b/services/llamacpp-proxy/discovery.go @@ -0,0 +1,166 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +// Discovery is the proxy's routing-target set. The proxy runs no mDNS of its +// own: routing targets are pushed down from the broker's discovery relay +// (discovery:nodes snapshots for the lc service) into the subscribed overlay, +// merged with user-added manual nodes. The proxy is itself advertised — as an lc +// service — by the node-scanner daemon's single _nvpair-node record, keyed off +// the engine port the broker's poller registers. +// +// The routable Node projection (IP / withPrimaryIP) and the manual-node +// overlay live here; request-path reachability (TCP-probe + failover) lives in +// proxy.go. + +import ( + "slices" + "sync" + + "nvpair-shared/discovery" + "nvpair-shared/netpick" +) + +// uuidFromTXT extracts a node's stable uuid= from its TXT records. Kept as a +// thin re-export of the shared helper (it was triplicated across the two proxies +// and the scanner before consolidation). +var uuidFromTXT = discovery.UUIDFromTXT + +// Node is the proxy's routable view of a node. It adds a canonical dialable IP +// field over the discovered node shape. +type Node struct { + ID string `json:"id"` + Host string `json:"host"` + Port int `json:"port"` + Addresses []string `json:"addresses"` + TXT []string `json:"txt"` + // Models is the loaded-model set from the broker's discovery snapshot + // (EngineLoadedModels), not the on-disk catalog. Model-bearing inference is + // eligible only when this list advertises the requested model; an empty + // list stays in discovery but is not an inference candidate until a later + // loaded-set update. GET /v1/models still merges the full backend catalog. + Models []string `json:"models,omitempty"` + // IP is the single canonical LAN address a consumer should dial/display for + // this node, resolved via the shared netpick ranker: the node's + // own ip= TXT if present, else the best-scored advertised IPv4. It is + // stamped onto outbound node/* notifications only (see withPrimaryIP) so a + // downstream consumer agrees on the same address the proxy routes to. + IP string `json:"ip,omitempty"` + // ClusterUUID is the relay peer's cluster principal (its mTLS cert UUID), + // carried from the discovery DirectoryNode. Non-empty only for a clustered + // peer; it is the key used to pin the peer's server cert when dialing its + // promoted proxy over cluster mTLS. Whether we actually hold that pin is + // resolved against the live mesh at routing time (resolveCandidates), never + // cached here: a cached answer goes stale the moment a peer is paired or + // removed. Internal routing metadata, not part of the proxy's outward node + // contract. + ClusterUUID string `json:"-"` +} + +// withPrimaryIP returns a copy of the node with IP resolved by the shared ranker +// (netpick.Primary over its TXT + Addresses). +func (n Node) withPrimaryIP() Node { + n.IP = netpick.Primary(n.TXT, n.Addresses) + return n +} + +// Discovery holds the proxy's routing targets: the relay-fed subscribed overlay +// and the user-added manual overlay. +type Discovery struct { + mu sync.RWMutex + manualNodes map[string]Node + // subscribedNodes are routing targets pushed down by the broker's discovery + // relay (discovery:nodes snapshots for the lc service), keyed by node ID (the + // directory instance name). + subscribedNodes map[string]Node +} + +func NewDiscovery() *Discovery { + return &Discovery{ + manualNodes: make(map[string]Node), + subscribedNodes: make(map[string]Node), + } +} + +// Nodes returns the merged subscribed + manual node set (manual entries that +// aren't also present via the relay are appended). +func (d *Discovery) Nodes() []Node { + d.mu.RLock() + defer d.mu.RUnlock() + out := make([]Node, 0, len(d.manualNodes)+len(d.subscribedNodes)) + seen := make(map[string]struct{}, len(d.subscribedNodes)) + for id, n := range d.subscribedNodes { + out = append(out, n) + seen[id] = struct{}{} + } + for id, n := range d.manualNodes { + if _, exists := seen[id]; !exists { + out = append(out, n) + } + } + return out +} + +// SetSubscribed replaces the relay-fed routing overlay with the given set and +// reports what changed versus the previous set (keyed by node ID): nodes newly +// present, nodes whose routable details changed, and nodes that dropped out. The +// broker pushes the full filtered snapshot on every change, so the overlay is +// replaced wholesale; the returned diff lets the caller emit node/discovered| +// updated|removed so a consumer (the UI) learns which peers currently run this +// engine. Manual nodes are a separate overlay and are untouched. +func (d *Discovery) SetSubscribed(nodes []Node) (discovered, updated, removed []Node) { + d.mu.Lock() + defer d.mu.Unlock() + next := make(map[string]Node, len(nodes)) + for _, n := range nodes { + next[n.ID] = n + switch prev, ok := d.subscribedNodes[n.ID]; { + case !ok: + discovered = append(discovered, n) + case !nodeEqual(prev, n): + updated = append(updated, n) + } + } + for id, prev := range d.subscribedNodes { + if _, ok := next[id]; !ok { + removed = append(removed, prev) + } + } + d.subscribedNodes = next + return discovered, updated, removed +} + +// nodeEqual reports whether two routable Nodes carry the same routing/display +// identity — the fields a consumer dials or renders. A change in any of them +// warrants a node/updated. +func nodeEqual(a, b Node) bool { + return a.ID == b.ID && a.Host == b.Host && a.Port == b.Port && a.IP == b.IP && + slices.Equal(a.Addresses, b.Addresses) && slices.Equal(a.TXT, b.TXT) && + slices.Equal(a.Models, b.Models) +} + +func (d *Discovery) AddManual(node Node) (added bool) { + d.mu.Lock() + defer d.mu.Unlock() + _, exists := d.manualNodes[node.ID] + d.manualNodes[node.ID] = node + return !exists +} + +func (d *Discovery) RemoveManual(id string) (removed bool) { + d.mu.Lock() + defer d.mu.Unlock() + _, exists := d.manualNodes[id] + if exists { + delete(d.manualNodes, id) + } + return exists +} + +func (d *Discovery) IsManual(id string) bool { + d.mu.RLock() + defer d.mu.RUnlock() + _, exists := d.manualNodes[id] + return exists +} diff --git a/services/llamacpp-proxy/e2e_test.go b/services/llamacpp-proxy/e2e_test.go new file mode 100644 index 00000000..9c914176 --- /dev/null +++ b/services/llamacpp-proxy/e2e_test.go @@ -0,0 +1,217 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "testing" + "time" +) + +// proxyBin is the real llamacpp-proxy binary, built once in TestMain so the +// e2e test exercises the shipped artifact (not just in-process handlers). +var proxyBin string + +func TestMain(m *testing.M) { + tmp, err := os.MkdirTemp("", "nvpair-lcpproxy-e2e-*") + if err != nil { + panic(err) + } + suffix := "" + if runtime.GOOS == "windows" { + suffix = ".exe" + } + proxyBin = filepath.Join(tmp, "llamacpp-proxy"+suffix) + if out, err := exec.Command("go", "build", "-o", proxyBin, ".").CombinedOutput(); err != nil { + panic("build llamacpp-proxy: " + err.Error() + "\n" + string(out)) + } + code := m.Run() + _ = os.RemoveAll(tmp) + os.Exit(code) +} + +type e2eFrame struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + Params json.RawMessage `json:"params"` + Result json.RawMessage `json:"result"` + Error json.RawMessage `json:"error"` +} + +func e2eReadFrames(r io.Reader, out chan<- e2eFrame) { + sc := bufio.NewScanner(r) + sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) + for sc.Scan() { + var f e2eFrame + if err := json.Unmarshal(sc.Bytes(), &f); err != nil { + continue + } + out <- f + } +} + +func e2eSend(t *testing.T, w io.Writer, id int, method string, params any) { + t.Helper() + msg := map[string]any{"jsonrpc": "2.0", "id": id, "method": method} + if params != nil { + msg["params"] = params + } + data, _ := json.Marshal(msg) + if _, err := w.Write(append(data, '\n')); err != nil { + t.Fatalf("send %s: %v", method, err) + } +} + +func e2eWaitResult(t *testing.T, frames <-chan e2eFrame, id string, timeout time.Duration) { + t.Helper() + deadline := time.After(timeout) + for { + select { + case f := <-frames: + if string(f.ID) != id { + continue + } + if len(f.Error) > 0 && string(f.Error) != "null" { + t.Fatalf("rpc id %s returned error: %s", id, f.Error) + } + return + case <-deadline: + t.Fatalf("timed out waiting for response id %s", id) + } + } +} + +func e2eWaitReadyPort(t *testing.T, frames <-chan e2eFrame, timeout time.Duration) int { + t.Helper() + deadline := time.After(timeout) + for { + select { + case f := <-frames: + if f.Method != "ready" { + continue + } + var p struct { + Port int `json:"port"` + } + if err := json.Unmarshal(f.Params, &p); err != nil { + t.Fatalf("parse ready params: %v", err) + } + return p.Port + case <-deadline: + t.Fatalf("timed out waiting for ready notification") + return 0 + } + } +} + +func e2eFreePort(t *testing.T) int { + t.Helper() + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer l.Close() + return l.Addr().(*net.TCPAddr).Port +} + +func e2eSplitHostPort(t *testing.T, serverURL string) (string, int) { + t.Helper() + host, portStr, err := net.SplitHostPort(strings.TrimPrefix(serverURL, "http://")) + if err != nil { + t.Fatalf("split %q: %v", serverURL, err) + } + port, err := strconv.Atoi(portStr) + if err != nil { + t.Fatalf("port %q: %v", portStr, err) + } + return host, port +} + +// TestE2EFailoverOverRealBinary spawns the real llamacpp-proxy binary and +// drives it the way the broker/UI does: register a busy (503) and a healthy +// (200) upstream as manual nodes over JSON-RPC stdio, then send a genuine +// OpenAI inference POST to the proxy's real HTTP port. It asserts the request +// fails over from the busy node to the healthy one, the original body is +// replayed, and CORS headers are present — the whole shipped path (binary + +// stdio control plane + HTTP forwarding + failover) end-to-end, no mocks. +func TestE2EFailoverOverRealBinary(t *testing.T) { + var gotBody string + busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusServiceUnavailable) + })) + defer busy.Close() + good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + gotBody = string(b) + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"ok":true}`) + })) + defer good.Close() + + port := e2eFreePort(t) + cmd := exec.Command(proxyBin, "--port", strconv.Itoa(port)) + stdin, err := cmd.StdinPipe() + if err != nil { + t.Fatal(err) + } + stdout, err := cmd.StdoutPipe() + if err != nil { + t.Fatal(err) + } + if err := cmd.Start(); err != nil { + t.Fatal(err) + } + defer func() { + _ = stdin.Close() + _ = cmd.Process.Kill() + _ = cmd.Wait() + }() + + frames := make(chan e2eFrame, 256) + go e2eReadFrames(stdout, frames) + + if got := e2eWaitReadyPort(t, frames, 10*time.Second); got != port { + t.Fatalf("ready port = %d, want %d", got, port) + } + + busyHost, busyPort := e2eSplitHostPort(t, busy.URL) + goodHost, goodPort := e2eSplitHostPort(t, good.URL) + e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}}) + e2eWaitResult(t, frames, "1", 5*time.Second) + e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}}) + e2eWaitResult(t, frames, "2", 5*time.Second) + // Select the busy node so the failover path is deterministic. + e2eSend(t, stdin, 3, "node/select", map[string]any{"id": "busy"}) + e2eWaitResult(t, frames, "3", 5*time.Second) + + resp, err := http.Post(fmt.Sprintf("http://127.0.0.1:%d/v1/chat/completions", port), "application/json", strings.NewReader(`{"model":"m"}`)) + if err != nil { + t.Fatalf("inference POST: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200 (should fail over from the 503 node)", resp.StatusCode) + } + if got := resp.Header.Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want *", got) + } + if gotBody != `{"model":"m"}` { + t.Errorf("healthy upstream got body %q, want the original request body", gotBody) + } + + e2eSend(t, stdin, 9, "shutdown", nil) + e2eWaitResult(t, frames, "9", 5*time.Second) +} diff --git a/services/llamacpp-proxy/failover_test.go b/services/llamacpp-proxy/failover_test.go new file mode 100644 index 00000000..02e5361b --- /dev/null +++ b/services/llamacpp-proxy/failover_test.go @@ -0,0 +1,600 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "io" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "strings" + "testing" + "time" +) + +// rwNop is a no-op io.ReadWriter so a Codec can be constructed in tests +// without a real transport: reads hit EOF immediately and writes are +// discarded. handleHTTP only ever writes (notifications), so this is enough. +type rwNop struct{} + +func (rwNop) Read([]byte) (int, error) { return 0, io.EOF } +func (rwNop) Write(p []byte) (int, error) { return len(p), nil } + +func testProxy(disc *Discovery, port int) *Proxy { + return NewProxy(NewCodec(rwNop{}), disc, port) +} + +// nodeFor turns an httptest server URL into a discovery Node pointing at it. +func nodeFor(t *testing.T, id, serverURL string) Node { + t.Helper() + u, err := url.Parse(serverURL) + if err != nil { + t.Fatalf("parse %q: %v", serverURL, err) + } + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split %q: %v", u.Host, err) + } + port, err := strconv.Atoi(portStr) + if err != nil { + t.Fatalf("port %q: %v", portStr, err) + } + return Node{ID: id, Addresses: []string{host}, Port: port} +} + +func nodeForModel(t *testing.T, id, serverURL, model string) Node { + t.Helper() + node := nodeFor(t, id, serverURL) + node.Models = []string{model} + return node +} + +// TestHandlePlain_OptionsPreflight: a CORS preflight is answered locally with +// 204 + permissive headers and never forwarded. +func TestHandlePlain_OptionsPreflight(t *testing.T) { + p := testProxy(NewDiscovery(), 11434) + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil) + req.RemoteAddr = "127.0.0.1:40000" + req.Header.Set("Access-Control-Request-Headers", "X-Custom-Token") + p.handlePlain(rec, req) + + if rec.Code != http.StatusNoContent { + t.Fatalf("status = %d, want 204", rec.Code) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want *", got) + } + if rec.Header().Get("Access-Control-Allow-Methods") == "" { + t.Errorf("missing Access-Control-Allow-Methods") + } + if got := rec.Header().Get("Access-Control-Expose-Headers"); got != "*" { + t.Errorf("Access-Control-Expose-Headers = %q, want *", got) + } + // The browser's requested headers are echoed so an arbitrary header clears preflight. + if got := rec.Header().Get("Access-Control-Allow-Headers"); got != "X-Custom-Token" { + t.Errorf("Access-Control-Allow-Headers = %q, want echoed X-Custom-Token", got) + } +} + +// TestHandlePlain_EngineCredentialedPreflightPreserved: when an engine opts an +// exact origin into credentialed CORS, its preflight policy reaches the browser +// instead of being replaced by the proxy's uncredentialed wildcard fallback. +func TestHandlePlain_EngineCredentialedPreflightPreserved(t *testing.T) { + preflightSeen := make(chan struct{}, 1) + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodOptions { + t.Errorf("engine method = %s, want OPTIONS", r.Method) + } + preflightSeen <- struct{}{} + w.Header().Set("Access-Control-Allow-Origin", "https://app.example") + w.Header().Set("Access-Control-Allow-Credentials", "true") + w.Header().Set("Access-Control-Allow-Methods", "POST") + w.Header().Set("Access-Control-Allow-Headers", "Content-Type") + w.WriteHeader(http.StatusNoContent) + })) + defer engine.Close() + + disc := NewDiscovery() + disc.AddManual(nodeFor(t, "engine", engine.URL)) + p := testProxy(disc, 11434) + req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil) + req.RemoteAddr = "127.0.0.1:40000" + req.Header.Set("Origin", "https://app.example") + req.Header.Set("Access-Control-Request-Method", http.MethodPost) + req.Header.Set("Access-Control-Request-Headers", "Content-Type") + rec := httptest.NewRecorder() + + p.handlePlain(rec, req) + + select { + case <-preflightSeen: + default: + t.Fatal("engine did not receive the credentialed preflight") + } + if rec.Code != http.StatusNoContent { + t.Errorf("status = %d, want %d", rec.Code, http.StatusNoContent) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" { + t.Errorf("Access-Control-Allow-Origin = %q, want the engine's exact origin", got) + } + if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" { + t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got) + } +} + +// TestHandleHTTP_EngineCORSPolicyPreserved: an engine that declares its own +// origin policy keeps it. Replacing it with the proxy's wildcard would widen +// what the user configured, and would break a credentialed response outright. +func TestHandleHTTP_EngineCORSPolicyPreserved(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Access-Control-Allow-Origin", "https://app.example") + w.Header().Set("Access-Control-Allow-Credentials", "true") + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"done":true}`) + })) + defer engine.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama")) + p := testProxy(disc, 11434) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "https://app.example" { + t.Errorf("Access-Control-Allow-Origin = %q, want the engine's own origin", got) + } + if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "true" { + t.Errorf("Access-Control-Allow-Credentials = %q, want the engine's true", got) + } +} + +// TestHandleHTTP_EngineCredentialsWithoutOriginDropped: an engine (or an +// intermediary in front of it) that sends Allow-Credentials but no origin has +// declared no policy to keep, so the proxy supplies its own. The wildcard it +// writes is invalid next to Allow-Credentials: true, and a browser rejects that +// pair, so the inherited header must not survive the forward. +func TestHandleHTTP_EngineCredentialsWithoutOriginDropped(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Access-Control-Allow-Credentials", "true") + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"done":true}`) + })) + defer engine.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "engine", engine.URL, "llama")) + p := testProxy(disc, 11434) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want the proxy's wildcard", got) + } + if got := rec.Header().Get("Access-Control-Allow-Credentials"); got != "" { + t.Errorf("Access-Control-Allow-Credentials = %q, want cleared alongside the wildcard origin", got) + } +} + +// TestHandleHTTP_HappyPathSingleNode: the common case — one healthy node +// answers directly, body forwarded, CORS present on the success response. +func TestHandleHTTP_HappyPathSingleNode(t *testing.T) { + var gotBody string + good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + gotBody = string(b) + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"done":true}`) + })) + defer good.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "good", good.URL, "llama")) + p := testProxy(disc, 11434) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200", rec.Code) + } + if gotBody != `{"model":"llama"}` { + t.Errorf("node got body %q, want the original request body", gotBody) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want * on success", got) + } +} + +// TestHandleHTTP_NoRetryOn400: a client error (400) is returned as-is and not +// failed over — retrying elsewhere would return the same error and mask it. +func TestHandleHTTP_NoRetryOn400(t *testing.T) { + hits := 0 + bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits++ + w.WriteHeader(http.StatusBadRequest) + io.WriteString(w, `{"error":"bad request"}`) + })) + defer bad.Close() + other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer other.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "bad", bad.URL, "llama")) + disc.AddManual(nodeForModel(t, "other", other.URL, "llama")) + p := testProxy(disc, 11434) + p.SetSelected("bad") + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusBadRequest { + t.Fatalf("status = %d, want 400 (client errors must not fail over)", rec.Code) + } + if hits != 1 { + t.Errorf("bad node hit %d times, want exactly 1 (no retry on 400)", hits) + } +} + +// TestHandleHTTP_RejectionHasCORS: even the no-node rejection carries CORS so a +// browser sees the real 502 instead of an opaque CORS error. +func TestHandleHTTP_RejectionHasCORS(t *testing.T) { + p := testProxy(NewDiscovery(), 11434) + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"x"}`))) + + if rec.Code != http.StatusBadGateway { + t.Fatalf("status = %d, want 502", rec.Code) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want * on rejection", got) + } +} + +// TestHandleHTTP_FailoverOn503: a busy first node (503) is skipped and the +// request is filled by the next node, with the original body replayed. +func TestHandleHTTP_FailoverOn503(t *testing.T) { + busy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusServiceUnavailable) + io.WriteString(w, `{"error":"loading model"}`) + })) + defer busy.Close() + + var gotBody string + good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + gotBody = string(b) + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"done":true}`) + })) + defer good.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "busy", busy.URL, "llama")) + disc.AddManual(nodeForModel(t, "good", good.URL, "llama")) + p := testProxy(disc, 11434) + p.SetSelected("busy") // deterministic: busy is tried first + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200 (should have failed over past the 503)", rec.Code) + } + if gotBody != `{"model":"llama"}` { + t.Errorf("failover node got body %q, want the original request body", gotBody) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want * on proxied success", got) + } +} + +// TestHandleHTTP_AllNodesDownReturnsError: when every candidate fails at the +// transport, the client gets one clean 502 (not a hang), still with CORS. +func TestHandleHTTP_AllNodesDownReturnsError(t *testing.T) { + // Two servers we immediately close so dials fail. + a := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + b := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + na := nodeForModel(t, "a", a.URL, "llama") + nb := nodeForModel(t, "b", b.URL, "llama") + a.Close() + b.Close() + + disc := NewDiscovery() + disc.AddManual(na) + disc.AddManual(nb) + p := testProxy(disc, 11434) + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusBadGateway { + t.Fatalf("status = %d, want 502 when all nodes are down", rec.Code) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want * on exhausted error", got) + } +} + +// TestHandleHTTP_404FailoverInferenceOnly: a 404 (model-not-found) on an +// inference call fails over to the next advertised owner, but a 404 on a +// non-inference path is returned as-is. +func TestHandleHTTP_404FailoverInferenceOnly(t *testing.T) { + missing := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNotFound) + io.WriteString(w, `{"error":"model not found"}`) + })) + defer missing.Close() + has := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"done":true}`) + })) + defer has.Close() + + newProxy := func() *Proxy { + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "missing", missing.URL, "llama")) + disc.AddManual(nodeForModel(t, "has", has.URL, "llama")) + p := testProxy(disc, 11434) + p.SetSelected("missing") + return p + } + + // Inference POST: 404 on first → fail over → 200. + rec := httptest.NewRecorder() + newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + if rec.Code != http.StatusOK { + t.Fatalf("inference 404: status = %d, want 200 (should fail over)", rec.Code) + } + + // An ordinary non-inference GET still returns the first node's 404. + rec = httptest.NewRecorder() + newProxy().handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/unknown", nil)) + if rec.Code != http.StatusNotFound { + t.Fatalf("non-inference 404: status = %d, want 404 (must NOT fail over)", rec.Code) + } +} + +func TestHandleHTTP_AggregatesModelList(t *testing.T) { + entered := make(chan struct{}, 2) + release := make(chan struct{}) + server := func(body string) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet || r.URL.Path != "/v1/models" { + t.Errorf("upstream request = %s %s, want GET /v1/models", r.Method, r.URL.Path) + } + if r.Header.Get("Authorization") != "" || r.Header.Get("Cookie") != "" { + t.Errorf("client credentials leaked to fan-out target") + } + entered <- struct{}{} + <-release + _, _ = io.WriteString(w, body) + })) + } + a := server(`{"object":"list","data":[{"id":"a","owned_by":"a-only"},{"id":"shared","owned_by":"first"}]}`) + defer a.Close() + b := server(`{"object":"list","data":[{"id":"shared","owned_by":"second"},{"id":"c","owned_by":"c-only"}]}`) + defer b.Close() + malformed := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, `{"object":"list","data":null}`) + })) + defer malformed.Close() + down := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + downNode := nodeFor(t, "down", down.URL) + down.Close() + + disc := NewDiscovery() + disc.AddManual(nodeFor(t, "a", a.URL)) + disc.AddManual(nodeFor(t, "b", b.URL)) + disc.AddManual(downNode) + disc.AddManual(nodeFor(t, "malformed", malformed.URL)) + p := testProxy(disc, 1234) + p.SetSelected("a") + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodGet, "/v1/models", nil) + req.Header.Set("Authorization", "Bearer client-secret") + req.Header.Set("Cookie", "session=client-secret") + done := make(chan struct{}) + go func() { + p.handleHTTP(rec, req) + close(done) + }() + + for range 2 { + select { + case <-entered: + case <-time.After(5 * time.Second): + close(release) + t.Fatal("model-list requests were not issued concurrently") + } + } + close(release) + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("aggregate request did not finish") + } + + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String()) + } + var got struct { + Object string `json:"object"` + Data []struct { + ID string `json:"id"` + OwnedBy string `json:"owned_by"` + } `json:"data"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil { + t.Fatal(err) + } + if got.Object != "list" { + t.Errorf("object = %q, want list", got.Object) + } + if len(got.Data) != 3 || got.Data[0].ID != "a" || got.Data[1].ID != "shared" || got.Data[2].ID != "c" { + t.Fatalf("models = %+v, want a, shared, c", got.Data) + } + if got.Data[1].OwnedBy != "first" { + t.Errorf("duplicate metadata = %q, want deterministic first candidate", got.Data[1].OwnedBy) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want *", got) + } +} + +func TestHandleHTTP_ModelListEmptyAndUnavailable(t *testing.T) { + empty := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, `{"object":"list","data":[]}`) + })) + emptyNode := nodeFor(t, "empty", empty.URL) + empty.Close() + + disc := NewDiscovery() + disc.AddManual(emptyNode) + rec := httptest.NewRecorder() + testProxy(disc, 1234).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil)) + if rec.Code != http.StatusServiceUnavailable { + t.Fatalf("unavailable status = %d, want 503", rec.Code) + } + + empty = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, `{"object":"list","data":[]}`) + })) + defer empty.Close() + disc = NewDiscovery() + disc.AddManual(nodeFor(t, "empty", empty.URL)) + rec = httptest.NewRecorder() + testProxy(disc, 1234).handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil)) + if rec.Code != http.StatusOK || rec.Body.String() != `{"object":"list","data":[]}` { + t.Fatalf("empty response = %d %s, want 200 native empty list", rec.Code, rec.Body.String()) + } +} + +// TestHandleHTTP_StrictModelRouting proves capability is a gate before +// selection and priority. Unknown and known-missing nodes are excluded from +// inference but remain available for non-inference routes. +func TestHandleHTTP_StrictModelRouting(t *testing.T) { + missHits, unknownHits, matchHits := 0, 0, 0 + miss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + missHits++ + w.WriteHeader(http.StatusOK) + })) + defer miss.Close() + unknown := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + unknownHits++ + w.WriteHeader(http.StatusOK) + })) + defer unknown.Close() + match := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + matchHits++ + body, _ := io.ReadAll(r.Body) + if string(body) != `{"model":"llama"}` { + t.Errorf("matching node got body %q", body) + } + w.WriteHeader(http.StatusOK) + })) + defer match.Close() + + disc := NewDiscovery() + missNode := nodeFor(t, "selected-miss", miss.URL) + missNode.Models = []string{"mistral"} + unknownNode := nodeFor(t, "a-unknown", unknown.URL) + matchNode := nodeFor(t, "z-match", match.URL) + matchNode.Models = []string{"llama"} + disc.AddManual(missNode) + disc.AddManual(unknownNode) + disc.AddManual(matchNode) + p := testProxy(disc, 11434) + p.SetSelected("selected-miss") + p.SetPriority([]string{"a-unknown", "selected-miss", "z-match"}) + candidates := p.resolveCandidates("llama") + if len(candidates) != 1 || candidates[0].id != "z-match" { + t.Fatalf("model candidates = %v, want only z-match", candidates) + } + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusOK { + t.Fatalf("status = %d, want 200", rec.Code) + } + if missHits != 0 || unknownHits != 0 || matchHits != 1 { + t.Fatalf("hits miss=%d unknown=%d match=%d, want 0/0/1", missHits, unknownHits, matchHits) + } + + // Capability filtering applies only to model-bearing inference. + rec = httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodGet, "/v1/unknown", nil)) + if rec.Code != http.StatusOK { + t.Fatalf("non-inference status = %d, want 200", rec.Code) + } + if missHits != 1 || unknownHits != 0 || matchHits != 1 { + t.Fatalf("non-inference hits miss=%d unknown=%d match=%d, want 1/0/1", missHits, unknownHits, matchHits) + } +} + +func TestHandleHTTP_NoAdvertisedModelRejectsLocally(t *testing.T) { + hits := 0 + upstream := func() *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits++ + w.WriteHeader(http.StatusOK) + })) + } + missing := upstream() + defer missing.Close() + unknown := upstream() + defer unknown.Close() + + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "missing", missing.URL, "mistral")) + disc.AddManual(nodeFor(t, "unknown", unknown.URL)) + events := &prRec{} + p := NewProxy(NewCodec(events), disc, 1234) + p.SetSelected("missing") + + rec := httptest.NewRecorder() + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + + if rec.Code != http.StatusBadGateway || + !strings.Contains(rec.Body.String(), "no available node advertises the requested model") { + t.Fatalf("response = %d %s, want actionable local 502", rec.Code, rec.Body.String()) + } + if hits != 0 { + t.Fatalf("ineligible upstreams received %d requests, want 0", hits) + } + if !events.has("no node advertises requested model") { + t.Fatalf("missing rejected request event: %s", events.b) + } +} + +// TestResolveCandidates_SelfGuard: a node resolving to the proxy's own +// listen address is dropped so we never self-forward. +func TestResolveCandidates_SelfGuard(t *testing.T) { + disc := NewDiscovery() + disc.AddManual(Node{ID: "self", Addresses: []string{"127.0.0.1"}, Port: 11434}) + disc.AddManual(Node{ID: "real", Addresses: []string{"192.0.2.10"}, Port: 11434}) + p := testProxy(disc, 11434) + + cands := p.resolveCandidates("") + var haveReal bool + for _, c := range cands { + if c.id == "self" { + t.Errorf("self-target node must be excluded, got candidate %+v", c) + } + if c.id == "real" { + haveReal = true + } + } + if !haveReal { + t.Errorf("expected the real node to survive the self-guard, candidates = %+v", cands) + } +} diff --git a/services/llamacpp-proxy/go.mod b/services/llamacpp-proxy/go.mod new file mode 100644 index 00000000..aed18868 --- /dev/null +++ b/services/llamacpp-proxy/go.mod @@ -0,0 +1,19 @@ +module llamacpp-proxy + +go 1.25.0 + +require nvpair-shared v0.0.0-00010101000000-000000000000 + +replace nvpair-shared => ../shared + +require ( + github.com/Microsoft/go-winio v0.6.2 // indirect + github.com/cenkalti/backoff v2.2.1+incompatible // indirect + github.com/grandcat/zeroconf v1.0.0 // indirect + github.com/miekg/dns v1.1.55 // indirect + golang.org/x/mod v0.17.0 // indirect + golang.org/x/net v0.58.0 // indirect + golang.org/x/sync v0.10.0 // indirect + golang.org/x/sys v0.47.0 // indirect + golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect +) diff --git a/services/llamacpp-proxy/go.sum b/services/llamacpp-proxy/go.sum new file mode 100644 index 00000000..fe333ffe --- /dev/null +++ b/services/llamacpp-proxy/go.sum @@ -0,0 +1,35 @@ +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4= +github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM= +github.com/grandcat/zeroconf v1.0.0 h1:uHhahLBKqwWBV6WZUDAT71044vwOTL+McW0mBJvo6kE= +github.com/grandcat/zeroconf v1.0.0/go.mod h1:lTKmG1zh86XyCoUeIHSA4FJMBwCJiQmGfcP2PdzytEs= +github.com/miekg/dns v1.1.27/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM= +github.com/miekg/dns v1.1.55 h1:GoQ4hpsj0nFLYe+bWiCToyrBEJXkQfOOIvFGFy0lEgo= +github.com/miekg/dns v1.1.55/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA= +golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= +golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ= +golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/services/llamacpp-proxy/ingress.go b/services/llamacpp-proxy/ingress.go new file mode 100644 index 00000000..f5d7e344 --- /dev/null +++ b/services/llamacpp-proxy/ingress.go @@ -0,0 +1,165 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "log/slog" + "net" + "net/http" + "net/http/httputil" + "net/url" + "strconv" + + "nvpair-shared/cors" +) + +const engineIdentityProbeHeader = "X-NVPAIR-Engine-Identity-Probe" + +// localBackend is the explicit loopback engine the cluster mTLS ingress +// forwards to. It is supplied by the broker over node/set-local-backend and is +// deliberately NOT sourced from the discovery overlay: a request that arrived +// over the LAN mTLS ingress can only ever be dumped on this node's own local +// engine, never re-routed to a peer, so the ingress path is strictly terminal +// and cannot recurse or amplify. +type localBackend struct { + Engine string `json:"engine"` + Host string `json:"host"` + Port int `json:"port"` + Healthy bool `json:"healthy"` +} + +// setLocalBackend records (or, with a zero port / unhealthy flag, effectively +// clears) the local engine the ingress serves. +func (p *Proxy) setLocalBackend(b localBackend) { + p.backendMu.Lock() + p.backend = b + p.backendMu.Unlock() +} + +// localBackendTarget returns the loopback URL of the current local engine, and +// false when none is set/healthy (the ingress then answers 503 rather than +// forwarding). The host defaults to 127.0.0.1 and is always loopback. +func (p *Proxy) localBackendTarget() (*url.URL, bool) { + p.backendMu.RLock() + b := p.backend + p.backendMu.RUnlock() + if b.Port <= 0 || !b.Healthy { + return nil, false + } + host := b.Host + if host == "" { + host = "127.0.0.1" + } + return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true +} + +// handlePlain is the plaintext personality: it accepts requests only from +// loopback and hands them to the full local router (handleHTTP). A non-loopback +// caller — any LAN peer — is refused; peers must use the mTLS ingress. This is +// what closes the former open-relay exposure (the listener still binds all +// interfaces for the TLS personality, but plaintext is loopback-only). +func (p *Proxy) handlePlain(w http.ResponseWriter, r *http.Request) { + if !isLoopbackRemote(r.RemoteAddr) { + // Answer a non-loopback preflight ahead of the gate. It grants no access + // on its own; the request that follows still receives the real 403. A + // loopback preflight continues into handleHTTP so an available engine's + // exact origin and credentials policy can be preserved. + if cors.WritePreflight(w, r) { + return + } + slog.Warn("rejected non-loopback plaintext request; cluster peers must use mTLS", + "remote", r.RemoteAddr, "method", r.Method, "path", r.URL.Path) + writeIngressError(w, http.StatusForbidden, "loopback-only", + "plaintext requests are accepted only from loopback; cluster peers must use the mTLS ingress") + return + } + // Engine-manager marks identity/action requests so the federated model-list + // facade can never satisfy llama.cpp's own /v1/models readiness probe. + if r.Header.Get(engineIdentityProbeHeader) == "1" { + writeIngressError(w, http.StatusConflict, "proxy-facade", "the compatibility facade is not a llama.cpp engine") + return + } + p.handleHTTP(w, r) +} + +// handleClusterIngress is the LAN mTLS personality: it authenticates the caller +// against this node's cluster pins and, once the peer is a trusted cluster +// member, forwards the request straight to the local loopback engine — exactly +// like the local plaintext path, with no route filtering. The mTLS pin is the +// sole authorization boundary (a trusted peer is treated like a local client), +// so the two personalities stay behaviorally identical toward the engine. It +// never calls resolveCandidates, so a peer request cannot be re-routed onward. +func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) { + // Re-derive membership and pins per request so a cluster left, or a peer + // paired or removed, after startup is reflected immediately without a proxy + // restart — a removed peer must stop being accepted right away, which is the + // whole point of the gate. + p.mesh.Refresh() + peer, ok := p.mesh.VerifyClientPin(r) + if !ok { + writeIngressError(w, http.StatusForbidden, "cluster-auth", + "client certificate is not a pinned member of this node's cluster") + return + } + target, ok := p.localBackendTarget() + if !ok { + writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend", + "no local inference backend is available on this node") + return + } + slog.Debug("cluster ingress forwarding to local backend", + "peer", peer, "method", r.Method, "path", r.URL.Path, "target", target.Host) + p.reverseProxyToLocal(w, r, target) +} + +// reverseProxyToLocal streams the request to the local engine, preserving +// cancellation (the request context is the proxy's root context, so a client +// disconnect or shutdown tears down the upstream call and stops generation). +func (p *Proxy) reverseProxyToLocal(w http.ResponseWriter, r *http.Request, target *url.URL) { + p.newLocalReverseProxy(target).ServeHTTP(w, r) +} + +func (p *Proxy) newLocalReverseProxy(target *url.URL) *httputil.ReverseProxy { + return &httputil.ReverseProxy{ + Director: func(req *http.Request) { + req.URL.Scheme = target.Scheme + req.URL.Host = target.Host + req.Host = target.Host + }, + Transport: p.plainHTTPTransport(), + ErrorHandler: func(ew http.ResponseWriter, _ *http.Request, err error) { + slog.Warn("cluster ingress upstream error", "target", target.Host, "err", err) + writeIngressError(ew, http.StatusBadGateway, "backend-error", "local inference backend error") + }, + } +} + +// isLoopbackRemote reports whether an http.Request RemoteAddr (host:port) is a +// loopback address (127.0.0.0/8 or ::1). An unparseable/empty RemoteAddr is not +// loopback, so it fails closed. +func isLoopbackRemote(remoteAddr string) bool { + host, _, err := net.SplitHostPort(remoteAddr) + if err != nil { + host = remoteAddr + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} + +// writeIngressError writes a small structured JSON error. It never echoes the +// request body or any generated output. CORS headers are included because these +// are the proxy's own rejections: without them a browser client cannot read the +// status or reason, and every one of them looks like a generic CORS failure. +func writeIngressError(w http.ResponseWriter, status int, code, msg string) { + cors.Apply(w.Header()) + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-Content-Type-Options", "nosniff") + w.WriteHeader(status) + body, err := json.Marshal(map[string]string{"error": msg, "code": code}) + if err != nil { + body = []byte(`{"error":"ingress error"}`) + } + _, _ = w.Write(body) +} diff --git a/services/llamacpp-proxy/ingress_test.go b/services/llamacpp-proxy/ingress_test.go new file mode 100644 index 00000000..2f1e1869 --- /dev/null +++ b/services/llamacpp-proxy/ingress_test.go @@ -0,0 +1,111 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "net/http" + "net/http/httptest" + "net/url" + "testing" +) + +// TestResolveCandidatesUnclusteredDropsRelayPeers is the core isolation +// assertion: an unclustered node (nil mesh) must not route inference to +// relay-discovered peers, only to explicit user-added manual nodes. +func TestResolveCandidatesUnclusteredDropsRelayPeers(t *testing.T) { + disc := NewDiscovery() + disc.SetSubscribed([]Node{{ + ID: "peer-a", Host: "peer-a", Port: 1234, + Addresses: []string{"192.0.2.10"}, + IP: "192.0.2.10", + ClusterUUID: "cluster-uuid-a", + }}) + disc.AddManual(Node{ + ID: "manual-x", Host: "manual-x", Port: 1234, + Addresses: []string{"192.0.2.20"}, IP: "192.0.2.20", + }) + p := testProxy(disc, 1235) // mesh nil => unclustered + + cands := p.resolveCandidates("") + if len(cands) != 1 { + t.Fatalf("unclustered candidate set = %+v, want exactly the manual node", cands) + } + if cands[0].id != "manual-x" || cands[0].peerUUID != "" || cands[0].url.Scheme != "http" { + t.Fatalf("unclustered candidate = %+v, want plaintext manual-x with no peerUUID", cands[0]) + } +} + +func TestHandlePlainRejectsNonLoopback(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", nil) + req.RemoteAddr = "192.0.2.50:40000" + rec := httptest.NewRecorder() + + p.handlePlain(rec, req) + if rec.Code != http.StatusForbidden { + t.Fatalf("non-loopback plaintext status = %d, want %d", rec.Code, http.StatusForbidden) + } + // The refusal carries CORS so a browser client reads this 403 and its reason + // instead of an opaque "CORS error" that hides why the call failed. + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want * on the refusal", got) + } +} + +// TestHandlePlainAnswersPreflightBeforeLoopbackGate: the preflight is answered +// even for a caller the gate will refuse. It authorizes nothing — the request +// that follows is still rejected — but without it the browser never sends that +// request and reports the refusal as a generic CORS failure. +func TestHandlePlainAnswersPreflightBeforeLoopbackGate(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + req := httptest.NewRequest(http.MethodOptions, "/v1/chat/completions", nil) + req.RemoteAddr = "192.0.2.50:40000" + rec := httptest.NewRecorder() + + p.handlePlain(rec, req) + if rec.Code != http.StatusNoContent { + t.Fatalf("preflight status = %d, want %d", rec.Code, http.StatusNoContent) + } + if got := rec.Header().Get("Access-Control-Allow-Origin"); got != "*" { + t.Errorf("Access-Control-Allow-Origin = %q, want *", got) + } +} + +func TestHandlePlainRejectsEngineIdentityProbe(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + req := httptest.NewRequest(http.MethodGet, "/v1/models", nil) + req.RemoteAddr = "127.0.0.1:40000" + req.Header.Set("X-NVPAIR-Engine-Identity-Probe", "1") + rec := httptest.NewRecorder() + + p.handlePlain(rec, req) + if rec.Code != http.StatusConflict { + t.Fatalf("identity probe status = %d, want %d", rec.Code, http.StatusConflict) + } +} + +func TestHandleClusterIngressUnclusteredForbids(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", nil) + rec := httptest.NewRecorder() + + p.handleClusterIngress(rec, req) + if rec.Code != http.StatusForbidden { + t.Fatalf("unclustered ingress status = %d, want %d", rec.Code, http.StatusForbidden) + } +} + +func TestLocalReverseProxyUsesSharedPlainTransport(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + shared := p.plainHTTPTransport() + target := &url.URL{Scheme: "http", Host: "127.0.0.1:1"} + rp := p.newLocalReverseProxy(target) + tr, ok := rp.Transport.(*http.Transport) + if !ok { + t.Fatalf("Transport type = %T, want *http.Transport", rp.Transport) + } + if tr != shared { + t.Fatal("ingress reverse proxy did not use the shared plain Transport") + } +} diff --git a/services/llamacpp-proxy/ipc.go b/services/llamacpp-proxy/ipc.go new file mode 100644 index 00000000..63c92ffb --- /dev/null +++ b/services/llamacpp-proxy/ipc.go @@ -0,0 +1,12 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +// The IPC transport (Unix socket / Windows named pipe) is single-sourced in +// nvpair-shared/ipc; the platform split lives there. dialIPC aliases it so call +// sites are unchanged. + +import "nvpair-shared/ipc" + +var dialIPC = ipc.Dial diff --git a/services/llamacpp-proxy/loaded_eligibility_test.go b/services/llamacpp-proxy/loaded_eligibility_test.go new file mode 100644 index 00000000..4f46ee74 --- /dev/null +++ b/services/llamacpp-proxy/loaded_eligibility_test.go @@ -0,0 +1,101 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "strings" + "testing" + + "nvpair-shared/noderec" +) + +func TestSubscribedToNodeUsesLoadedNotCatalog(t *testing.T) { + n := noderec.DirectoryNode{ + Name: "box", + HostUUID: "uuid-1", + IP: "127.0.0.1", + Services: map[noderec.ServiceKey]noderec.ServiceStatus{ + noderec.ServiceLlamaCpp: {Port: 8084}, + }, + ModelsByEngine: map[string][]string{ + "llamacpp": {"ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp", "other"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {"ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp"}, + }, + } + node, ok := subscribedToNode(n) + if !ok { + t.Fatal("expected lc node") + } + if !nodeAdvertisesModel(node, "ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF_IQ3_XXS-mtp") { + t.Fatal("loaded id must be eligible") + } + if nodeAdvertisesModel(node, "other") { + t.Fatal("catalog-only id must not be eligible") + } +} + +func TestCatalogOnlyChatDoesNotHitBackend(t *testing.T) { + hits := 0 + backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPost && r.URL.Path == "/v1/chat/completions" { + hits++ + } + w.WriteHeader(http.StatusOK) + })) + defer backend.Close() + + u, err := url.Parse(backend.URL) + if err != nil { + t.Fatalf("parse backend URL: %v", err) + } + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split backend host: %v", err) + } + port, err := strconv.Atoi(portStr) + if err != nil { + t.Fatalf("backend port: %v", err) + } + + n := noderec.DirectoryNode{ + Name: "box", + HostUUID: "uuid-1", + IP: host, + Services: map[noderec.ServiceKey]noderec.ServiceStatus{ + noderec.ServiceLlamaCpp: {Port: port}, + }, + ModelsByEngine: map[string][]string{ + "llamacpp": {"other"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {}, + }, + } + node, ok := subscribedToNode(n) + if !ok { + t.Fatal("expected lc node") + } + + disc := NewDiscovery() + disc.AddManual(node) + p := testProxy(disc, 8084) + + rec := httptest.NewRecorder() + body := strings.NewReader(`{"model":"other","messages":[{"role":"user","content":"hi"}]}`) + p.handleHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", body)) + + if rec.Code != http.StatusBadGateway { + t.Fatalf("status = %d, want 502", rec.Code) + } + if hits != 0 { + t.Fatalf("backend hits = %d, want 0 (catalog-only id must not be forwarded)", hits) + } +} diff --git a/services/llamacpp-proxy/main.go b/services/llamacpp-proxy/main.go new file mode 100644 index 00000000..3fd0dc4b --- /dev/null +++ b/services/llamacpp-proxy/main.go @@ -0,0 +1,95 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "flag" + "fmt" + "io" + "log" + "log/slog" + "os" + "os/signal" + "syscall" + + "nvpair-shared/applog" + "nvpair-shared/clustertrust" +) + +func main() { + port := flag.Int("port", defaultProxyPort, "HTTP listen port") + ignorePersistedPort := flag.Bool("ignore-persisted-port", false, "use --port even when a persisted port exists") + ipcPath := flag.String("ipc", "", "IPC endpoint: Unix domain socket path or Windows named pipe (default: stdin/stdout)") + clusterDir := flag.String("cluster-dir", "", "cluster trust directory (node.crt/key + trusted pins); enables the LAN mTLS inference ingress when this node is clustered") + showVersion := flag.Bool("version", false, "print version and exit") + resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo) + flag.Parse() + + if *showVersion { + fmt.Println(Version) + os.Exit(0) + } + + applog.Init("llamacpp-proxy", resolveLevel()) + + var transport io.ReadWriteCloser + if *ipcPath != "" { + conn, err := dialIPC(*ipcPath) + if err != nil { + log.Fatalf("failed to connect to IPC endpoint %q: %v", *ipcPath, err) + } + transport = conn + log.Printf("using IPC transport: %s", *ipcPath) + } else { + transport = newStdioTransport() + log.Print("using stdio transport") + } + defer transport.Close() + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + go func() { + select { + case sig := <-sigCh: + log.Printf("received %s, shutting down", sig) + cancel() + case <-ctx.Done(): + } + }() + + // Restore a previously chosen port (set via set-port) over the + // --port/default, so the proxy comes back up where the user last put it. + persisted, hasPersisted := loadPersistedPort() + effectivePort := chooseStartupPort(*port, *ignorePersistedPort, persisted, hasPersisted) + if hasPersisted && !*ignorePersistedPort && effectivePort == persisted { + log.Printf("restored persisted proxy port %d", persisted) + } + + codec := NewCodec(transport) + disc := NewDiscovery() + proxy := NewProxy(codec, disc, effectivePort) + // Open a live view of this node's cluster mTLS trust fabric. While unclustered + // the proxy serves only the loopback plaintext personality; once this node is + // a member the same listener also serves the pin-gated LAN mTLS ingress, and + // peers become routable candidates. The proxy needs no restart to notice + // either transition — it re-derives membership per request and on a watch. + // + // Membership is gated on an active admission or a pin, never on keypair + // presence: a left/removed node keeps its keypair by design, and would + // otherwise keep logging cluster_ingress with no cluster peers to serve. + proxy.mesh = clustertrust.Open(*clusterDir) + go proxy.mesh.Watch(ctx, func(clustered bool) { + slog.Info("cluster inference ingress switched personality", "cluster_ingress", clustered) + proxy.dropUnpinnedPeerTransports() + }) + + if err := proxy.Run(ctx); err != nil && ctx.Err() == nil { + log.Fatalf("proxy error: %v", err) + } + log.Print("shutdown complete") +} diff --git a/services/llamacpp-proxy/portstore.go b/services/llamacpp-proxy/portstore.go new file mode 100644 index 00000000..37875128 --- /dev/null +++ b/services/llamacpp-proxy/portstore.go @@ -0,0 +1,78 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + + "nvpair-shared/appdir" +) + +const proxyPortFile = "llamacpp-proxy-port.json" + +const defaultProxyPort = 8084 + +type persistedPort struct { + Port int `json:"port"` +} + +func chooseStartupPort(flagPort int, ignorePersisted bool, persisted int, hasPersisted bool) int { + if ignorePersisted || !hasPersisted { + return flagPort + } + return persisted +} + +func proxyPortPath() (string, error) { + return appdir.Path(proxyPortFile) +} + +// loadPersistedPort returns the previously chosen proxy port, if a valid one +// was saved. Any error (no file, bad JSON, out-of-range) reports "none" so +// startup falls back to the --port flag / default. +func loadPersistedPort() (int, bool) { + path, err := proxyPortPath() + if err != nil { + return 0, false + } + data, err := os.ReadFile(path) + if err != nil { + return 0, false + } + var pp persistedPort + if err := json.Unmarshal(data, &pp); err != nil { + return 0, false + } + if pp.Port < 1 || pp.Port > 65535 { + return 0, false + } + return pp.Port, true +} + +// savePersistedPort atomically writes the chosen port (tmp + rename) so a +// crash mid-write can't leave a truncated file behind. +func savePersistedPort(port int) error { + path, err := proxyPortPath() + if err != nil { + return err + } + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + return err + } + data, err := json.Marshal(persistedPort{Port: port}) + if err != nil { + return err + } + tmp := path + ".tmp" + if err := os.WriteFile(tmp, data, 0o644); err != nil { + return err + } + if err := os.Rename(tmp, path); err != nil { + _ = os.Remove(tmp) + return err + } + return nil +} diff --git a/services/llamacpp-proxy/portstore_test.go b/services/llamacpp-proxy/portstore_test.go new file mode 100644 index 00000000..29ef7812 --- /dev/null +++ b/services/llamacpp-proxy/portstore_test.go @@ -0,0 +1,145 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "context" + "fmt" + "net" + "os" + "strings" + "testing" + "time" +) + +// redirectConfigDir points os.UserConfigDir() at a temp dir for the test, so +// proxy-port.json reads/writes don't touch the real per-user config. Sets all +// three env vars os.UserConfigDir() consults across platforms: XDG_CONFIG_HOME +// on Linux, $HOME/Library on macOS, and APPDATA on Windows. Missing APPDATA +// meant the Windows-first-class path read/wrote the real %AppData% file — +// clobbering the user's saved port and making the test fail on repeat runs. +func redirectConfigDir(t *testing.T) { + t.Helper() + dir := t.TempDir() + t.Setenv("XDG_CONFIG_HOME", dir) + t.Setenv("HOME", dir) + t.Setenv("APPDATA", dir) + t.Setenv("LOCALAPPDATA", dir) +} + +func freeTCPPort(t *testing.T) int { + t.Helper() + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("reserve free port: %v", err) + } + defer l.Close() + return l.Addr().(*net.TCPAddr).Port +} + +func TestDefaultProxyPort(t *testing.T) { + if defaultProxyPort != 8084 { + t.Fatalf("default proxy port = %d, want 8084", defaultProxyPort) + } +} + +func TestChooseStartupPort(t *testing.T) { + for _, tc := range []struct { + name string + flagPort, persisted int + ignorePersisted, hasPersisted bool + want int + }{ + {"new default", 8084, 0, false, false, 8084}, + {"persisted wins", 8084, 18084, false, true, 18084}, + {"custom survives opt-out", 8084, 12400, false, true, 12400}, + {"managed flag wins", 8084, 12400, true, true, 8084}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := chooseStartupPort(tc.flagPort, tc.ignorePersisted, tc.persisted, tc.hasPersisted); got != tc.want { + t.Fatalf("chooseStartupPort() = %d, want %d", got, tc.want) + } + }) + } +} + +func TestPersistedPortRoundTrip(t *testing.T) { + redirectConfigDir(t) + + if _, ok := loadPersistedPort(); ok { + t.Fatal("expected no persisted port before any save") + } + if err := savePersistedPort(11500); err != nil { + t.Fatalf("savePersistedPort: %v", err) + } + if p, ok := loadPersistedPort(); !ok || p != 11500 { + t.Errorf("round-trip: got %d ok=%v, want 11500", p, ok) + } + + // An out-of-range stored value is treated as "none" so startup falls + // back to the flag/default rather than trying to bind port 0. + path, err := proxyPortPath() + if err != nil { + t.Fatalf("proxyPortPath: %v", err) + } + if err := os.WriteFile(path, []byte(`{"port":0}`), 0o644); err != nil { + t.Fatal(err) + } + if _, ok := loadPersistedPort(); ok { + t.Error("port 0 should be treated as none") + } +} + +// TestSetPortRebinds drives a live rebind: the proxy starts serving on one +// port, set-port moves it to another, and afterward the new port accepts +// connections, the old one doesn't, the choice is persisted, and a fresh +// ready notification carries the new port. +func TestSetPortRebinds(t *testing.T) { + redirectConfigDir(t) + + buf := &bytes.Buffer{} + codec := NewCodec(buf) + disc := NewDiscovery() + + portA := freeTCPPort(t) + proxy := NewProxy(codec, disc, portA) + + lnA, err := net.Listen("tcp", fmt.Sprintf(":%d", portA)) + if err != nil { + t.Fatalf("listen on port A: %v", err) + } + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + proxy.serveHTTP(ctx, lnA) + defer proxy.shutdown(context.Background()) + + portB := freeTCPPort(t) + if err := proxy.setPort(portB); err != nil { + t.Fatalf("setPort: %v", err) + } + + // New port is now serving. + conn, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portB), 2*time.Second) + if err != nil { + t.Fatalf("new port %d not listening after rebind: %v", portB, err) + } + conn.Close() + + // Old port stopped accepting. + if c, err := net.DialTimeout("tcp", fmt.Sprintf("127.0.0.1:%d", portA), 500*time.Millisecond); err == nil { + c.Close() + t.Errorf("old port %d should be closed after rebind", portA) + } + + // Persisted for next startup. + if p, ok := loadPersistedPort(); !ok || p != portB { + t.Errorf("persisted port: got %d ok=%v, want %d", p, ok, portB) + } + + // A fresh ready notification announced the new port. + if !strings.Contains(buf.String(), fmt.Sprintf("\"port\":%d", portB)) { + t.Errorf("expected ready notification carrying port %d, got %q", portB, buf.String()) + } +} diff --git a/services/llamacpp-proxy/priority_test.go b/services/llamacpp-proxy/priority_test.go new file mode 100644 index 00000000..6661b30b --- /dev/null +++ b/services/llamacpp-proxy/priority_test.go @@ -0,0 +1,177 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "io" + "strconv" + "sync" + "testing" +) + +// prRec is a thread-safe io.ReadWriter that records codec writes so a test can +// assert the response emitted for a request. Reads hit EOF immediately. +type prRec struct { + mu sync.Mutex + b []byte +} + +func (r *prRec) Read([]byte) (int, error) { return 0, io.EOF } + +func (r *prRec) Write(p []byte) (int, error) { + r.mu.Lock() + defer r.mu.Unlock() + r.b = append(r.b, p...) + return len(p), nil +} + +func (r *prRec) has(s string) bool { + r.mu.Lock() + defer r.mu.Unlock() + return string(r.b) != "" && contains(string(r.b), s) +} + +func contains(hay, needle string) bool { + for i := 0; i+len(needle) <= len(hay); i++ { + if hay[i:i+len(needle)] == needle { + return true + } + } + return false +} + +// prNode builds a discovery Node with a single non-local address so +// resolveCandidates resolves it deterministically (single candidate → no TCP +// probe) and it never trips the loopback rewrite or self-forward guard. The +// octet is derived from the id's first byte so each id gets a distinct valid IP. +func prNode(id string) Node { + return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234} +} + +// prProxy returns a proxy whose discovery holds the given node ids. +func prProxy(t *testing.T, ids ...string) *Proxy { + t.Helper() + disc := NewDiscovery() + for _, id := range ids { + disc.AddManual(prNode(id)) + } + return testProxy(disc, 1235) +} + +func candidateIDs(p *Proxy) []string { + return candidateIDsForModel(p, "") +} + +func candidateIDsForModel(p *Proxy, model string) []string { + cands := p.resolveCandidates(model) + out := make([]string, 0, len(cands)) + for _, c := range cands { + out = append(out, c.id) + } + return out +} + +func assertOrder(t *testing.T, got, want []string) { + t.Helper() + if len(got) != len(want) { + t.Fatalf("candidate order = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("candidate order = %v, want %v", got, want) + } + } +} + +// TestResolveCandidates_PriorityOrder: the priority list dictates auto order. +func TestResolveCandidates_PriorityOrder(t *testing.T) { + p := prProxy(t, "a", "b", "c") + p.SetPriority([]string{"c", "a", "b"}) + assertOrder(t, candidateIDs(p), []string{"c", "a", "b"}) +} + +// TestResolveCandidates_UnlistedFallback: nodes absent from the priority list +// come last, in stable ID order. +func TestResolveCandidates_UnlistedFallback(t *testing.T) { + p := prProxy(t, "a", "b", "c") + p.SetPriority([]string{"b"}) + assertOrder(t, candidateIDs(p), []string{"b", "a", "c"}) +} + +// TestResolveCandidates_UnknownIgnored: an id not in discovery is skipped. +func TestResolveCandidates_UnknownIgnored(t *testing.T) { + p := prProxy(t, "a", "b", "c") + p.SetPriority([]string{"zzz", "c"}) + assertOrder(t, candidateIDs(p), []string{"c", "a", "b"}) +} + +// TestResolveCandidates_ManualPinOverridesPriority: an explicit node/select pin +// wins over the priority list; the rest follow priority order. +func TestResolveCandidates_ManualPinOverridesPriority(t *testing.T) { + p := prProxy(t, "a", "b", "c") + p.SetPriority([]string{"a", "b", "c"}) + p.SetSelected("b") + assertOrder(t, candidateIDs(p), []string{"b", "a", "c"}) +} + +func TestResolveCandidates_FiltersBeforeSelectionAndPriority(t *testing.T) { + disc := NewDiscovery() + a, c, d := prNode("a"), prNode("c"), prNode("d") + a.Models, c.Models, d.Models = []string{"llama"}, []string{"llama"}, []string{"mistral"} + for _, n := range []Node{a, prNode("b"), c, d} { + disc.AddManual(n) + } + p := testProxy(disc, 1235) + p.SetSelected("d") + p.SetPriority([]string{"d", "b", "c", "a"}) + assertOrder(t, candidateIDsForModel(p, "llama"), []string{"c", "a"}) + + p.SetSelected("a") + assertOrder(t, candidateIDsForModel(p, "llama"), []string{"a", "c"}) +} + +// TestResolveCandidates_EmptyReverts: an empty priority list reverts to the +// default stable ID order. +func TestResolveCandidates_EmptyReverts(t *testing.T) { + p := prProxy(t, "c", "a", "b") + p.SetPriority([]string{"c", "a"}) + p.SetPriority(nil) // clear + assertOrder(t, candidateIDs(p), []string{"a", "b", "c"}) +} + +// TestSetPriority_CountAndCopy: SetPriority returns the stored length and +// PriorityList hands back an independent copy. +func TestSetPriority_CountAndCopy(t *testing.T) { + p := prProxy(t, "a") + if n := p.SetPriority([]string{"a", "b", "c"}); n != 3 { + t.Fatalf("SetPriority count = %d, want 3", n) + } + got := p.PriorityList() + got[0] = "mutated" + if again := p.PriorityList(); again[0] != "a" { + t.Fatalf("PriorityList returned an aliased slice: %v", again) + } +} + +// TestHandleSetPriority_Response: the node/set-priority request returns {count}. +func TestHandleSetPriority_Response(t *testing.T) { + rec := &prRec{} + p := NewProxy(NewCodec(rec), NewDiscovery(), 1235) + + id := json.RawMessage(`7`) + p.handleMessage(&Message{ + JSONRPC: "2.0", + ID: &id, + Method: "node/set-priority", + Params: json.RawMessage(`{"nodes":["x","y"]}`), + }) + + if !rec.has(`"count":2`) { + t.Fatalf("expected response with count=2, got: %s", rec.b) + } + if got := p.PriorityList(); len(got) != 2 || got[0] != "x" || got[1] != "y" { + t.Fatalf("stored priority = %v, want [x y]", got) + } +} diff --git a/services/llamacpp-proxy/proxy.go b/services/llamacpp-proxy/proxy.go new file mode 100644 index 00000000..1747419a --- /dev/null +++ b/services/llamacpp-proxy/proxy.go @@ -0,0 +1,2042 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "context" + "crypto/rand" + "crypto/tls" + "encoding/hex" + "encoding/json" + stderrors "errors" + "fmt" + "io" + "log" + "log/slog" + "net" + "net/http" + "net/http/httputil" + "net/url" + "sort" + "strconv" + "sync" + "sync/atomic" + "time" + + "nvpair-shared/applog" + "nvpair-shared/clustertrust" + "nvpair-shared/cors" + "nvpair-shared/errors" + "nvpair-shared/netmon" + "nvpair-shared/netpick" + "nvpair-shared/nodeactivity" + "nvpair-shared/noderec" + "nvpair-shared/reach" + "nvpair-shared/schedulerwire" + "nvpair-shared/splitlisten" +) + +// Version is stamped at build time via -ldflags "-X main.Version=...". +// See versions.json at the repo root for the source of truth. +var Version = "dev" + +type ReadyParams struct { + Version string `json:"version"` + Port int `json:"port"` +} + +// ErrorParams is sent as a JSON-RPC "error" notification when the +// proxy encounters a fatal startup-time condition it wants to surface +// to the orchestrator before exiting. Code is a short machine-readable +// tag ("bind-failed" today); Message is a human-friendly string suitable +// for an error bar. +type ErrorParams struct { + Code string `json:"code"` + Message string `json:"message"` + Port int `json:"port,omitempty"` +} + +type NodesResult struct { + Nodes []Node `json:"nodes"` +} + +type SelectParams struct { + ID string `json:"id"` +} + +type SelectedResult struct { + ID string `json:"id"` +} + +// RequestStartedEvent is emitted as a `proxy/request-started` +// notification the moment we've resolved a target node and are about +// to forward the request to it. Pairs with the existing +// `proxy/request` completion event by ID so the orchestrator can +// track in-flight requests per target — increment on start, decrement +// on the matching completion. Rejection-path requests (no active +// node) never get a started event because they were never in flight; +// they go straight to a completion event with an unmatched ID. +// +// NodeID is the chosen node's identifier in the discovery list. It's +// the authoritative way to attribute activity to a node card in the +// UI: Target (host:port) would be ambiguous whenever the proxy +// rewrote a local-interface address to 127.0.0.1 (see nodeURL), so +// multiple nodes could plausibly match the same Target string. +// Cluster model-list fan-out has no single node, so it reports an empty +// NodeID and the explicit Target "cluster". +type RequestStartedEvent struct { + ID string `json:"id"` + NodeID string `json:"node_id,omitempty"` + Method string `json:"method"` + Path string `json:"path"` + Target string `json:"target"` +} + +// RequestEvent is emitted as a `proxy/request` notification when a +// proxied request finishes (or is rejected before forwarding). The +// ID is unique within one proxy process lifetime — paired with the +// matching RequestStartedEvent so consumers can pop it from an +// in-flight map. ID is always populated even on the rejection path +// (where no Started event was emitted) so consumers don't need a +// separate code path for ID-less completions. +// +// NodeID is empty on the rejection path (no target was resolved) and +// on cluster model-list fan-out, and is the chosen node's identifier otherwise. See RequestStartedEvent +// for why attribution by NodeID is needed instead of by Target. +// +// TTFB is the time-to-first-byte: milliseconds from the moment we +// started forwarding to the node until its HTTP response status line +// came back, captured via ReverseProxy.ModifyResponse. Omitted +// (serialized as absent rather than zero) when not applicable: +// rejection path (no forward happened) and upstream-error path +// (ModifyResponse is never called on connection/dial failures). This +// is the "is the node snappy?" signal — distinct from Duration, +// which for streaming llama.cpp responses is dominated by token +// generation time and so doesn't really reflect latency at all. +type RequestEvent struct { + ID string `json:"id"` + NodeID string `json:"node_id,omitempty"` + Method string `json:"method"` + Path string `json:"path"` + Target string `json:"target"` + Status int `json:"status"` + Duration int64 `json:"duration_ms"` + TTFB int64 `json:"ttfb_ms,omitempty"` + Error string `json:"error,omitempty"` +} + +// Workload lifecycle method names (workload-manager spec 7). The proxy is +// a workload *producer*: it emits one of these per forwarded inference +// request so the broker can stamp the origin (originatedFrom) and forward it to the +// workload-manager, which broadcasts it cluster-wide. We don't emit +// workload:submitted (the proxy never queues — it forwards immediately) or +// workloads:remove (retirement is a broker concern). +const ( + workloadStartedMethod = "workload:started" + workloadCompletedMethod = "workload:completed" + workloadErroredMethod = "workload:errored" + + // workloadEngine is the opaque engine identifier carried in every + // workload this proxy produces. This proxy only ever fronts llama.cpp. + workloadEngine = "llamacpp" +) + +// inferenceEndpoints is the set of request paths that count as cluster +// workloads. Health checks, model listings (/v1/models), and other control +// traffic are deliberately excluded so we don't flood the cluster with +// non-inference noise. llama.cpp serves the OpenAI-compatible API, so these +// are the OpenAI inference routes. +var inferenceEndpoints = map[string]bool{ + "/v1/chat/completions": true, + "/v1/completions": true, + "/v1/embeddings": true, +} + +// isInferenceRequest reports whether a request should be tracked as a +// workload — a POST to one of the known inference endpoints. +func isInferenceRequest(method, path string) bool { + return method == http.MethodPost && inferenceEndpoints[path] +} + +// Workload mirrors the workload-manager spec 6 object. The proxy populates +// the fields it can observe: originatedFrom is intentionally left empty for +// the broker to stamp with the authoritative local node id (exactly like +// errors:report), while scheduledOn is set to the node this proxy actually +// routed the request to (the served candidate's node id — the same +// authoritative attribution handle), so a consumer can attribute the workload +// to where it ran rather than to where it came from. requesterId is omitted. +// Pointer fields serialize as JSON null when unset, matching the spec's +// nullable columns. +type Workload struct { + ID string `json:"id"` + Model string `json:"model"` + Engine string `json:"engine"` + RunID string `json:"runId"` + State string `json:"state"` + OriginatedFrom string `json:"originatedFrom"` + ScheduledOn string `json:"scheduledOn,omitempty"` + CreatedAt int64 `json:"createdAt"` + StartedAt *int64 `json:"startedAt"` + CompletedAt *int64 `json:"completedAt"` + Error *string `json:"error"` + RequesterID *string `json:"requesterId"` +} + +// workloadParams is the params envelope for a workload:* notification +// (spec 7.1): a single workloadInfo carrying the full Workload. +type workloadParams struct { + WorkloadInfo Workload `json:"workloadInfo"` +} + +// bufferBodyAndModel reads the request body once and returns the raw bytes +// (so each failover attempt can replay it — see the loop in handleHTTP) along +// with the JSON "model" field for workload tracking. Inference bodies are +// small (prompt + model), so full buffering is cheap. Returns (nil, "") when +// the body is absent and an empty model when none is parseable. The caller +// restores r.Body from the returned bytes before each forward attempt. +func bufferBodyAndModel(r *http.Request) ([]byte, string) { + if r.Body == nil { + return nil, "" + } + body, err := io.ReadAll(r.Body) + _ = r.Body.Close() + if err != nil { + return body, "" + } + var probe struct { + Model string `json:"model"` + } + if err := json.Unmarshal(body, &probe); err != nil { + return body, "" + } + return body, probe.Model +} + +type statusCapture struct { + http.ResponseWriter + status int + + // idle bounds how long a single write of streamed bytes to the client may + // block before it's abandoned. Zero disables the deadline. See + // idleClientWriteTimeout for the rationale (killed-client / half-open + // socket zombie jobs). + idle time.Duration + rc *http.ResponseController + // wroteErr retains the first error returned when writing the response body + // to the client (e.g. a dead client's send buffer filling and the write + // deadline tripping), so handleHTTP can mark the workload failed rather + // than misreporting the truncated stream as completed. + wroteErr error + + // upstreamAlive is called after each successful body write, but only once the + // upstream has committed — handleHTTP sets it in ModifyResponse, so it stays + // nil while the only thing this writer could carry is the proxy's own error + // body. Past that point every byte written came from the node serving the + // request, which is proof that node is working: the liveness evidence + // discovery cannot obtain for itself while the node is too busy to answer a + // probe. Called on the reverse proxy's copy goroutine, so it must be cheap. + upstreamAlive func() +} + +// Unwrap exposes the underlying ResponseWriter so http.ResponseController can +// reach the connection for SetWriteDeadline (and Flush) through this wrapper. +func (sc *statusCapture) Unwrap() http.ResponseWriter { return sc.ResponseWriter } + +func (sc *statusCapture) WriteHeader(code int) { + sc.status = code + sc.ResponseWriter.WriteHeader(code) +} + +// Write bounds each streamed write to the client with a deadline so a write +// blocked on a dead/half-open client fails promptly instead of hanging the +// reverse-proxy copy indefinitely. The deadline is cleared after every +// successful write, so a legitimately slow generation with long gaps between +// tokens is never penalized — only a write actively stuck on a gone client +// trips it. The first write error is retained (wroteErr) for the caller. +func (sc *statusCapture) Write(b []byte) (int, error) { + if sc.idle > 0 { + if sc.rc == nil { + sc.rc = http.NewResponseController(sc.ResponseWriter) + } + _ = sc.rc.SetWriteDeadline(time.Now().Add(sc.idle)) + } + n, err := sc.ResponseWriter.Write(b) + if err != nil { + if sc.wroteErr == nil { + sc.wroteErr = err + } + } else if sc.rc != nil { + _ = sc.rc.SetWriteDeadline(time.Time{}) + } + // Reported on every chunk rather than once per response so a long generation + // keeps vouching for its node for as long as it streams. The reporter + // coalesces, so the cost of calling this per chunk is a mutex and a clock + // read. + if err == nil && sc.upstreamAlive != nil { + sc.upstreamAlive() + } + return n, err +} + +// FlushError makes the streamed flush deadline-aware. For a streaming +// (chunked) upstream, ReverseProxy flushes after every write via +// http.NewResponseController(w).Flush — and because a small chunk buffers on +// Write without touching the socket, the actual network write for it happens +// here in Flush, not in Write. Without this method that flush reaches the +// underlying connection through Unwrap with no deadline and blocks unbounded on +// a stalled client (the same zombie the Write deadline guards against). So arm +// the same idle deadline around the flush, clear it on success, and retain a +// real flush error so the workload is classified failed. Implementing +// FlushError (which also satisfies the Flusher path via the ResponseController) +// means the flush routes through here instead of unwrapping past us. +func (sc *statusCapture) FlushError() error { + if sc.rc == nil { + sc.rc = http.NewResponseController(sc.ResponseWriter) + } + if sc.idle > 0 { + _ = sc.rc.SetWriteDeadline(time.Now().Add(sc.idle)) + } + err := sc.rc.Flush() + if err != nil { + // A ResponseWriter that genuinely can't flush is not a client failure; + // only retain real I/O errors (e.g. the deadline tripping on a dead + // client) so we don't misreport an unsupported-flush as a failed write. + if !stderrors.Is(err, http.ErrNotSupported) && sc.wroteErr == nil { + sc.wroteErr = err + } + return err + } + if sc.idle > 0 { + _ = sc.rc.SetWriteDeadline(time.Time{}) + } + return nil +} + +type Proxy struct { + codec *Codec + discovery *Discovery + cancel context.CancelFunc + + // httpMu guards port, the servers, the split listener, and ln across a + // live set-port rebind. The HTTP handlers never read port (they route by + // upstream node), so the only contention is set-port vs set-port + // (serialized) and the initial serveHTTP store vs a later rebind. + httpMu sync.Mutex + port int + plainSrv *http.Server + tlsSrv *http.Server + split *splitlisten.Splitter + ln net.Listener + + // mesh is this node's cluster mTLS trust fabric, loaded from --cluster-dir. + // nil = unclustered: the LAN TLS ingress accepts nothing and the node does + // only loopback-plaintext local routing. Read-only after startup. + mesh *clustertrust.Mesh + + // backendMu guards backend, the explicit loopback engine the cluster mTLS + // ingress forwards to. The broker sets/clears it via node/set-local-backend; + // it is never sourced from discovery, so an ingress request can only ever + // reach this node's own local engine and can never be re-routed to a peer. + backendMu sync.RWMutex + backend localBackend + + selectedMu sync.RWMutex + selectedID string + + // activity coalesces the liveness reports raised when a peer's engine streams + // response bytes back through us (see reportActivity). + activity *nodeactivity.Reporter + + // priorityMu guards the scheduler's authoritative baseline and the + // optimistic reservations made since that snapshot arrived. resolveCandidates + // reads priority to form the failover list; reserveCandidate atomically adds + // local dispatches before forwarding so a concurrent burst cannot repeatedly + // choose from the same stale scheduler state. + priorityMu sync.RWMutex + priority []string + priorityPending map[string]int + priorityGPUPressure map[string]int + priorityReservations map[string]int + + // targets remembers, per node, which of its published addresses accepted a + // connection, so a repeated forward costs no confirmation. An entry is + // re-confirmed when the node's candidate list changes and forgotten on an + // upstream error, so the next request fails over to another address. + targets *reach.Chooser + + // transportMu guards the long-lived HTTP transports reused across forwards + // and model-list fetches. Allocating a new http.Transport per request + // defeats connection pooling and leaks idle sockets until GC. + transportMu sync.Mutex + plainTransport *http.Transport + peerTransports map[string]*http.Transport + + // nextRequestID is a monotonic counter for tagging RequestStarted / + // RequestEvent pairs. Atomic add returns the new value, so request + // IDs start at 1 and never collide within a single proxy lifetime. + // IDs deliberately reset across restarts — they're only meaningful + // while the orchestrator's in-flight map is also alive, and a + // fresh proxy session always starts that map empty on the + // orchestrator side via the proxy:stopped → proxy:ready event + // pair. + nextRequestID atomic.Uint64 + + // runID is a per-process nonce minted at startup and stamped on every + // workload this proxy emits. It makes a workload's identity + // (originatedFrom, engine, runId, id) globally unique even though + // nextRequestID resets to 1 on restart and the Ollama proxy also counts + // from 1 — without it, two concurrent cross-engine jobs, or a reused id + // after a restart, would collide in the broker's store. + runID string +} + +func NewProxy(codec *Codec, discovery *Discovery, port int) *Proxy { + return &Proxy{ + codec: codec, + discovery: discovery, + port: port, + targets: reach.NewChooser(), + runID: newRunID(), + activity: nodeactivity.NewReporter(activityReportInterval), + } +} + +// activityReportInterval is how often a single node's streaming may raise a +// liveness report. A generation writes hundreds of chunks and the scanner treats +// a report as good for a minute, so anything finer is pure noise on the broker +// pipe. +const activityReportInterval = 2 * time.Second + +// reportActivity tells the broker a node's engine just returned response bytes, +// so discovery can keep that node even while it is too busy to answer a liveness +// probe. This is the only liveness signal that strengthens under load, which is +// exactly when the probe-based ones fail. +// +// Reports are not filtered to remote nodes here: this proxy knows targets by URL +// and port, not by whether a uuid is its own. The scanner holds that identity and +// drops its own (see noteActivity). +func (p *Proxy) reportActivity(nodeID string) { + if !p.activity.Due(nodeID) { + return + } + if err := p.codec.Notify(noderec.NotifyNodeActivity, noderec.NodeActivityParams{HostUUID: nodeID}); err != nil { + slog.Debug("failed to report node activity", "node_id", nodeID, "err", err) + } +} + +// newRunID returns a short random per-process nonce (hex). A crypto/rand read +// failure falls back to a timestamp — uniqueness matters more than +// unpredictability here. +func newRunID() string { + var b [8]byte + if _, err := rand.Read(b[:]); err != nil { + return strconv.FormatInt(time.Now().UnixNano(), 16) + } + return hex.EncodeToString(b[:]) +} + +func (p *Proxy) Run(ctx context.Context) error { + ctx, cancel := context.WithCancel(ctx) + p.cancel = cancel + defer cancel() + + // Keep the "is this address local?" set fresh as interfaces come and go, + // so the loopback rewrite in nodeURL stays correct after VPN/dock changes + // or a sleep/wake IP reassignment. + startLocalAddrWatch(ctx) + + // Bind synchronously before announcing "ready": if the port is + // already in use, we want the UI to see a real error reason + // instead of being stuck on "Proxy running" while ListenAndServe + // silently fails in a goroutine. + ln, err := p.listen() + if err != nil { + // Best-effort: notify the orchestrator with a structured + // reason. The process will exit non-zero regardless (main.go + // log.Fatalf's on a non-nil Run error), so a failed Notify + // here is not worth surfacing separately. + _ = p.codec.Notify("error", ErrorParams{ + Code: "bind-failed", + Message: fmt.Sprintf("failed to bind port %d: %v", p.port, err), + Port: p.port, + }) + return fmt.Errorf("failed to bind port %d: %w", p.port, err) + } + + if err := p.codec.Notify("ready", ReadyParams{ + Version: Version, + Port: p.port, + }); err != nil { + // Don't leave a dangling listener holding the port if we + // couldn't even tell the orchestrator about it. + _ = ln.Close() + return fmt.Errorf("failed to send ready notification: %w", err) + } + + // Routing targets come from the broker's discovery relay. Subscribe + // for lc nodes; they arrive as discovery:nodes snapshots (handled in + // handleMessage), each replacing the subscribed overlay. Non-fatal: if the + // parent isn't a relay-aware broker the proxy still routes to manual nodes. + slog.Debug("subscribing to discovery relay for routing targets", "service", string(noderec.ServiceLlamaCpp)) + if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: []noderec.ServiceKey{noderec.ServiceLlamaCpp}}); err != nil { + slog.Warn("failed to subscribe to discovery relay", "err", err) + } + + p.serveHTTP(ctx, ln) + + err = p.readLoop(ctx) + + // The app is going away (stdin closed or ctx cancelled). Stop any + // inference requests still in flight rather than letting them run to + // completion: cancelling the proxy's root context propagates to every + // in-flight request context — and thus the upstream reverse-proxy + // connection — so the target llama.cpp sees the client disconnect and stops + // generating instead of burning the GPU on a result nobody will read. + cancel() + + // srv.Shutdown then waits for the handlers to unwind (now fast, since + // their upstream calls were just cancelled). As each returns it emits its + // own terminal workload:errored, so peers don't keep showing the workload + // as a "running" ghost. A hard kill (SIGKILL) bypasses all of this. + shutCtx, shutCancel := context.WithTimeout(context.Background(), 3*time.Second) + defer shutCancel() + p.shutdown(shutCtx) + + return err +} + +// Timeouts for upstream connections. Logged at startup so they're always +// present in any captured log for post-mortem analysis. +const ( + proxyDialTimeout = 10 * time.Second + proxyKeepAlive = 30 * time.Second + proxyResponseTimeout = 120 * time.Second + proxyMaxIdleConns = 50 + proxyIdleConnTimeout = 90 * time.Second + // Inbound http.Server limits — keep IdleTimeout aligned with client + // IdleConnTimeout so idle keep-alives are reaped on both sides. + proxyReadHeaderTimeout = 10 * time.Second + proxyServerIdleTimeout = 90 * time.Second + maxModelListBytes = 16 << 20 +) + +// idleClientWriteTimeout bounds how long a single write of streamed response +// bytes to the client may block. A killed client can leave a half-open socket +// whose kernel send buffer fills and never drains; without this deadline the +// reverse-proxy copy blocks indefinitely (TCP retransmit backoff runs into +// minutes, and r.Context() never fires when no FIN/RST arrives), so the request +// handler never returns and its terminal workload event is never emitted — the +// "zombie job" left showing as running until PAIR restarts. statusCapture.Write +// resets the deadline after every successful write, so this only trips a write +// that is actively stuck on a gone client, never a slow-but-live generation. +// +// It is a var (not a const) only so a test can shorten it to exercise the +// deadline against a real socket; production never reassigns it. +var idleClientWriteTimeout = 30 * time.Second + +var modelListClient = &http.Client{ + Transport: &http.Transport{ + DialContext: (&net.Dialer{ + Timeout: proxyDialTimeout, + KeepAlive: proxyKeepAlive, + }).DialContext, + ResponseHeaderTimeout: proxyDialTimeout, + MaxIdleConns: proxyMaxIdleConns, + IdleConnTimeout: proxyIdleConnTimeout, + }, + Timeout: proxyDialTimeout, +} + +// listen binds the proxy's TCP listener synchronously so bind failures +// (EADDRINUSE and friends) can be reported through a structured error +// notification before the process exits. The caller is responsible for +// closing the returned listener if it doesn't hand it to serveHTTP. +func (p *Proxy) listen() (net.Listener, error) { + return net.Listen("tcp", fmt.Sprintf(":%d", p.port)) +} + +// serveHTTP takes the already-bound base listener and drives the two proxy +// personalities over it: a plaintext HTTP server (loopback-only, full local +// router) and a LAN mTLS ingress (pin-gated, forwards to the local engine), +// split by the connection's first byte via nvpair-shared/splitlisten. The two +// http.Servers are recorded so set-port can rebind both onto a fresh split +// without tearing the servers down. +func (p *Proxy) serveHTTP(ctx context.Context, ln net.Listener) { + base := func(_ net.Listener) context.Context { return ctx } + plainSrv := &http.Server{ + Handler: http.HandlerFunc(p.handlePlain), + BaseContext: base, + ReadHeaderTimeout: proxyReadHeaderTimeout, + IdleTimeout: proxyServerIdleTimeout, + } + tlsSrv := &http.Server{ + Handler: http.HandlerFunc(p.handleClusterIngress), + BaseContext: base, + ReadHeaderTimeout: proxyReadHeaderTimeout, + IdleTimeout: proxyServerIdleTimeout, + } + + p.httpMu.Lock() + p.plainSrv = plainSrv + p.tlsSrv = tlsSrv + p.ln = ln + p.startSplitLocked(ln) + p.httpMu.Unlock() + + slog.Info("proxy timeouts configured", + "dial_timeout", proxyDialTimeout, + "keep_alive", proxyKeepAlive, + "response_header_timeout", proxyResponseTimeout, + "max_idle_conns", proxyMaxIdleConns, + "idle_conn_timeout", proxyIdleConnTimeout, + ) + slog.Info("HTTP proxy listening", "port", p.port, "addr", ln.Addr().String(), + "cluster_ingress", p.mesh.Clustered()) +} + +// startSplitLocked wraps base in a first-byte splitter and starts both servers +// on its sub-listeners. Caller holds httpMu. Reuses the persistent plainSrv / +// tlsSrv so set-port can call it repeatedly on fresh listeners. +func (p *Proxy) startSplitLocked(base net.Listener) { + split := splitlisten.New(base) + p.split = split + go func() { + if err := p.plainSrv.Serve(split.Plain()); err != nil && err != http.ErrServerClosed { + slog.Error("plaintext HTTP server exited", "err", err) + } + }() + go p.serveTLS(split.TLS()) +} + +// serveTLS terminates cluster mTLS on the split's TLS sub-listener. The server +// certificate is resolved per handshake from the live mesh, so this one +// sub-listener covers both states: while this node is unclustered there is no +// leaf to present and the handshake is refused (it exposes no LAN inference +// surface), and the moment the node becomes a member the same sub-listener +// serves the pin-gated ingress — no rebind, and no process restart to pick up a +// freshly-minted identity. +func (p *Proxy) serveTLS(l net.Listener) { + if err := p.tlsSrv.Serve(tls.NewListener(l, p.mesh.ServerTLSConfig())); err != nil && err != http.ErrServerClosed { + slog.Error("cluster mTLS ingress exited", "err", err) + } +} + +// shutdown gracefully stops both personalities and closes the split listener. +func (p *Proxy) shutdown(ctx context.Context) { + p.httpMu.Lock() + plainSrv, tlsSrv, split := p.plainSrv, p.tlsSrv, p.split + p.httpMu.Unlock() + if plainSrv != nil { + _ = plainSrv.Shutdown(ctx) + } + if tlsSrv != nil { + _ = tlsSrv.Shutdown(ctx) + } + if split != nil { + _ = split.Close() + } + p.closeIdleTransports() +} + +// setPort live-rebinds the HTTP listener onto newPort and persists the choice +// so it survives a restart. It binds the new listener first (so a bind +// failure leaves the current one serving), starts the same server on it, then +// closes the old listener — in-flight connections on the old port drain +// naturally. A fresh `ready` notification announces the new port so the +// orchestrator/UI learn where the proxy is now listening. +func (p *Proxy) setPort(newPort int) error { + p.httpMu.Lock() + defer p.httpMu.Unlock() + + if newPort == p.port { + return nil + } + newLn, err := net.Listen("tcp", fmt.Sprintf(":%d", newPort)) + if err != nil { + return fmt.Errorf("failed to bind port %d: %w", newPort, err) + } + oldSplit := p.split + p.ln = newLn + p.port = newPort + + // Re-serve both personalities on a fresh split over the new listener, then + // close the old split (and its base listener) so in-flight connections on + // the old port drain naturally. The plaintext and mTLS personalities always + // move together as one unit. + slog.Info("HTTP proxy listening", "port", newPort, "addr", newLn.Addr().String(), + "cluster_ingress", p.mesh.Clustered()) + p.startSplitLocked(newLn) + if oldSplit != nil { + _ = oldSplit.Close() + } + + if err := savePersistedPort(newPort); err != nil { + slog.Warn("failed to persist proxy port", "port", newPort, "err", err) + } + if err := p.codec.Notify("ready", ReadyParams{Version: Version, Port: newPort}); err != nil { + slog.Warn("failed to emit ready after rebind", "err", err) + } + return nil +} + +// emitWorkload sends a workload:* lifecycle notification to the +// orchestrator. The broker stamps the origin (originatedFrom) and forwards it to the +// workload-manager; a failed write is logged but never blocks the request. +func (p *Proxy) emitWorkload(method string, w Workload) { + if err := p.codec.Notify(method, workloadParams{WorkloadInfo: w}); err != nil { + slog.Warn("failed to emit workload notification", "method", method, "err", err) + } +} + +// candidate is one forwarding target: the node's discovery ID (the +// authoritative attribution handle, stable across nodeURL's 127.0.0.1 +// rewrite) and its resolved URL. peerUUID is set for a remote cluster peer: +// the request is dialed over cluster mTLS to the peer's promoted proxy (https), +// pinned to that peer's exact server cert. Empty peerUUID means a plain-HTTP +// dial — the local backend (self) or an explicit manual node. +type candidate struct { + id string + url *url.URL + peerUUID string +} + +// candidateTransport returns the reverse-proxy / model-list transport for a +// candidate. Plain/self/manual candidates share one long-lived Transport. +// Cluster peers share one long-lived mTLS Transport per peerUUID. Callers must +// not CloseIdleConnections on the returned value except via closeIdleTransports. +func (p *Proxy) candidateTransport(c candidate) *http.Transport { + if c.peerUUID == "" { + return p.plainHTTPTransport() + } + return p.peerHTTPTransport(c.peerUUID) +} + +func newProxyTransport(tlsCfg *tls.Config) *http.Transport { + tr := &http.Transport{ + DialContext: (&net.Dialer{ + Timeout: proxyDialTimeout, + KeepAlive: proxyKeepAlive, + }).DialContext, + ResponseHeaderTimeout: proxyResponseTimeout, + MaxIdleConns: proxyMaxIdleConns, + MaxIdleConnsPerHost: proxyMaxIdleConns, + IdleConnTimeout: proxyIdleConnTimeout, + } + if tlsCfg != nil { + tr.TLSClientConfig = tlsCfg + } + return tr +} + +func (p *Proxy) plainHTTPTransport() *http.Transport { + p.transportMu.Lock() + defer p.transportMu.Unlock() + if p.plainTransport == nil { + p.plainTransport = newProxyTransport(nil) + } + return p.plainTransport +} + +func (p *Proxy) peerHTTPTransport(peerUUID string) *http.Transport { + p.transportMu.Lock() + defer p.transportMu.Unlock() + if tr, ok := p.peerTransports[peerUUID]; ok { + if p.mesh != nil && p.mesh.HasPin(peerUUID) { + return tr + } + tr.CloseIdleConnections() + delete(p.peerTransports, peerUUID) + } + if p.mesh == nil { + return newProxyTransport(nil) + } + cfg, ok := p.mesh.ClientTLSConfig(peerUUID) + if !ok { + return newProxyTransport(nil) + } + tr := newProxyTransport(cfg) + if p.peerTransports == nil { + p.peerTransports = make(map[string]*http.Transport) + } + p.peerTransports[peerUUID] = tr + return tr +} + +// dropUnpinnedPeerTransports closes idle conns for peer Transports whose pins +// are gone. Safe to call from the mesh Watch callback. +func (p *Proxy) dropUnpinnedPeerTransports() { + p.transportMu.Lock() + defer p.transportMu.Unlock() + for uuid, tr := range p.peerTransports { + if p.mesh != nil && p.mesh.HasPin(uuid) { + continue + } + tr.CloseIdleConnections() + delete(p.peerTransports, uuid) + } +} + +func (p *Proxy) closeIdleTransports() { + p.transportMu.Lock() + defer p.transportMu.Unlock() + if p.plainTransport != nil { + p.plainTransport.CloseIdleConnections() + } + for uuid, tr := range p.peerTransports { + tr.CloseIdleConnections() + delete(p.peerTransports, uuid) + } +} + +// retrySignal is returned from ModifyResponse to abort a retryable upstream +// response before its body streams to the client, so handleHTTP can fail over +// to the next candidate. It's a distinct type rather than errors.New(...) +// because this package aliases nvpair-shared/errors as `errors` (which has no New). +type retrySignal struct{} + +func (retrySignal) Error() string { return "llamacpp-proxy: retry next candidate" } + +type modelListItem struct { + key string + raw json.RawMessage +} + +type modelListResult struct { + items []modelListItem + ok bool + err error +} + +// serveModelList queries every llama.cpp candidate concurrently and returns +// the native /v1/models envelope with duplicate IDs removed. Results are +// merged in candidate order, not completion order, so duplicate metadata is +// deterministic while an unavailable peer cannot hide healthy inventories. +func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidates []candidate) (int, error) { + writeJSON := func(status int, body []byte) { + cors.Apply(w.Header()) + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-Content-Type-Options", "nosniff") + w.WriteHeader(status) + _, _ = w.Write(body) + } + results := make([]modelListResult, len(candidates)) + var wg sync.WaitGroup + for i, cand := range candidates { + target := *cand.url + target.Path = r.URL.Path + target.RawPath = r.URL.RawPath + target.RawQuery = r.URL.RawQuery + upstream, err := http.NewRequestWithContext(r.Context(), http.MethodGet, target.String(), nil) + if err != nil { + results[i].err = err + continue + } + upstream.Header.Set("Accept", "application/json") + + // A cluster-peer candidate is queried over mTLS to its promoted proxy; + // self/manual candidates use the shared plain client. + client := modelListClient + if cand.peerUUID != "" { + client = &http.Client{Timeout: modelListClient.Timeout, Transport: p.candidateTransport(cand)} + } + + wg.Add(1) + go func(i int, cand candidate, req *http.Request, client *http.Client) { + defer wg.Done() + resp, err := client.Do(req) + if err != nil { + p.targets.Forget(cand.id) + results[i].err = err + return + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + results[i].err = fmt.Errorf("upstream returned %s", resp.Status) + return + } + body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1)) + if err != nil { + results[i].err = err + return + } + if len(body) > maxModelListBytes { + results[i].err = fmt.Errorf("model list exceeds %d bytes", maxModelListBytes) + return + } + var envelope struct { + Data *[]json.RawMessage `json:"data"` + } + if err := json.Unmarshal(body, &envelope); err != nil { + results[i].err = err + return + } + if envelope.Data == nil { + results[i].err = fmt.Errorf("upstream response has no data array") + return + } + models := *envelope.Data + items := make([]modelListItem, 0, len(models)) + for _, raw := range models { + var identity struct { + ID string `json:"id"` + } + if err := json.Unmarshal(raw, &identity); err != nil { + results[i].err = fmt.Errorf("invalid model record: %w", err) + return + } + if identity.ID == "" { + results[i].err = fmt.Errorf("model record has no id") + return + } + items = append(items, modelListItem{key: identity.ID, raw: raw}) + } + results[i] = modelListResult{items: items, ok: true} + }(i, cand, upstream, client) + } + wg.Wait() + + success := false + models := make([]json.RawMessage, 0) + seen := make(map[string]bool) + for i, result := range results { + if !result.ok { + slog.Debug("model list candidate unavailable", + "node_id", candidates[i].id, "target", candidates[i].url.Host, "err", result.err) + continue + } + success = true + for _, item := range result.items { + if !seen[item.key] { + seen[item.key] = true + models = append(models, item.raw) + } + } + } + if !success { + err := fmt.Errorf("no valid model list from %d candidate(s)", len(candidates)) + writeJSON(http.StatusServiceUnavailable, []byte(`{"error":"model inventory unavailable"}`)) + return http.StatusServiceUnavailable, err + } + body, err := json.Marshal(struct { + Object string `json:"object"` + Data []json.RawMessage `json:"data"` + }{Object: "list", Data: models}) + if err != nil { + writeJSON(http.StatusInternalServerError, []byte(`{"error":"failed to encode model inventory"}`)) + return http.StatusInternalServerError, err + } + writeJSON(http.StatusOK, body) + return http.StatusOK, nil +} + +func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) { + start := time.Now() + + // Allocate the request ID up front so both code paths (rejection + // and forward) can stamp the same value into their notification. + // The rejection path never emits a Started event, so its ID won't + // appear in any orchestrator in-flight map — that's fine; the + // completion event still bumps the failed counter regardless of + // whether a matching Started was seen. + reqID := strconv.FormatUint(p.nextRequestID.Add(1), 10) + + // Parse the request's model before choosing a node. Model eligibility only + // applies to inference routes; control endpoints retain their existing + // routing behavior even when their JSON happens to contain a model field. + bodyBytes, model := bufferBodyAndModel(r) + isInf := isInferenceRequest(r.Method, r.URL.Path) + routingModel := "" + if isInf { + routingModel = model + } + candidates := p.resolveCandidates(routingModel) + if isInf && model != "" { + candidates = p.reserveCandidate(candidates) + } + if r.Method == http.MethodGet && r.URL.Path == "/v1/models" { + if len(candidates) > 0 { + p.codec.Notify("proxy/request-started", RequestStartedEvent{ + ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster", + }) + } + status, err := p.serveModelList(w, r, candidates) + errText := "" + if err != nil { + errText = err.Error() + } + p.codec.Notify("proxy/request", RequestEvent{ + ID: reqID, Method: r.Method, Path: r.URL.Path, Target: "cluster", + Status: status, Duration: time.Since(start).Milliseconds(), Error: errText, + }) + return + } + if len(candidates) == 0 { + // With no engine to consult, retain the local permissive preflight used + // for engines that do not publish a CORS policy. + if cors.WritePreflight(w, r) { + return + } + cors.Apply(w.Header()) + rejectionBody := `{"error":"no active node selected or available"}` + rejectionError := "no active node" + if isInf && model != "" { + rejectionBody = `{"error":"no available node advertises the requested model"}` + rejectionError = "no node advertises requested model" + } + slog.Warn("proxy request rejected", + "id", reqID, "method", r.Method, "path", r.URL.Path, + "remote", r.RemoteAddr, "reason", rejectionError) + http.Error(w, rejectionBody, http.StatusBadGateway) + p.codec.Notify("proxy/request", RequestEvent{ + ID: reqID, + Method: r.Method, + Path: r.URL.Path, + Status: http.StatusBadGateway, + Duration: time.Since(start).Milliseconds(), + Error: rejectionError, + }) + return + } + // shouldRetry reports whether an upstream status warrants failing over to + // the next candidate: busy/unavailable/gateway statuses, plus a 404 on an + // inference call (an advertised owner's inventory may have become stale). + // Genuine client errors (400/401/422…) are not retried — they'd fail + // identically on every node. + shouldRetry := func(code int) bool { + switch code { + case http.StatusRequestTimeout, + http.StatusTooManyRequests, + http.StatusBadGateway, + http.StatusServiceUnavailable, + http.StatusGatewayTimeout: + return true + case http.StatusNotFound: + return isInf + } + return code >= 500 + } + + var ( + servedNodeID string + servedTarget string + ttfbMs int64 + proxyErr string + finalStatus int + started bool + wl *Workload + ) + + // Emit workload:started up front, the moment we begin forwarding, naming + // the first candidate we'll try. A burst of concurrent inference requests + // must surface as job cards immediately; the upstream engine serializes + // concurrent requests on a single GPU slot, so gating "started" on the + // upstream response headers (the commit point) left every queued-but- + // forwarded job invisible until the node dequeued it — only one card at a + // time (a prior regression). If failover later commits a different + // node, the commit block re-points scheduledOn; the terminal + // completed/errored transition is emitted once at the end regardless. + if isInf && model != "" { + createdMs := start.UnixMilli() + wl = &Workload{ + ID: reqID, + Model: model, + Engine: workloadEngine, + RunID: p.runID, + State: "running", + ScheduledOn: candidates[0].id, + CreatedAt: createdMs, + StartedAt: &createdMs, + } + p.emitWorkload(workloadStartedMethod, *wl) + } + + // The terminal workload transition (completed/errored) can be reached from + // two places: the normal path after the stream copy unwinds below, and the + // disconnect watcher that fires while the copy is still blocked. terminalOnce + // guarantees exactly one is emitted; wlMu guards the shared wl fields the + // watcher (a separate goroutine) and ModifyResponse's failover re-point both + // touch; terminated suppresses a late started re-point once we've finalized. + var ( + terminalOnce sync.Once + wlMu sync.Mutex + terminated bool + ) + emitTerminal := func(state, errMsg string) { + if wl == nil { + return + } + terminalOnce.Do(func() { + now := time.Now().UnixMilli() + wlMu.Lock() + terminated = true + wl.CompletedAt = &now + wl.State = state + if errMsg != "" { + wl.Error = &errMsg + } + snapshot := *wl + wlMu.Unlock() + method := workloadCompletedMethod + if state != "completed" { + method = workloadErroredMethod + } + p.emitWorkload(method, snapshot) + }) + } + + // Watch for the client going away while the request is in flight. The + // terminal event is otherwise emitted only after the stream copy returns; + // a client that disconnects mid-stream can leave the copy blocked, so we + // emit the terminal here the moment r.Context() is cancelled instead of + // waiting for the unwind. Cancelling r.Context() (client close, or our own + // shutdown) also propagates to the ReverseProxy's upstream request, so the + // engine stops generating. terminalOnce keeps this from double-emitting + // with the normal path. The half-open case (no FIN, r.Context() never + // fires) is caught instead by statusCapture's write deadline below. + if wl != nil { + reqCtx := r.Context() + finished := make(chan struct{}) + defer close(finished) + go func() { + select { + case <-reqCtx.Done(): + emitTerminal("failed", "client disconnected before completion") + case <-finished: + } + }() + } + + // committedSC is the statusCapture of the candidate we committed to + // streaming; its wroteErr tells us after the fact whether the client write + // failed (dead/half-open client) so we can mark the workload failed. + var committedSC *statusCapture + + // Failover loop: try candidates in order until one returns a + // usable response or the list is exhausted. We can only retry before the + // first byte reaches the client; once a response starts streaming we're + // committed. proxy/request-started fires at that commit point so it names + // the node that actually serves the request, not one we failed over from; + // workload:started was already emitted above (and is re-pointed there on a + // failover). The self-forward guard lives in resolveCandidates. + for i := range candidates { + cand := candidates[i] + last := i == len(candidates)-1 + if bodyBytes != nil { + r.Body = io.NopCloser(bytes.NewReader(bodyBytes)) + } + retry := false + sc := &statusCapture{ResponseWriter: w, status: http.StatusOK, idle: idleClientWriteTimeout} + + proxy := &httputil.ReverseProxy{ + Director: func(req *http.Request) { + req.URL.Scheme = cand.url.Scheme + req.URL.Host = cand.url.Host + req.Host = cand.url.Host + }, + // A remote cluster peer is dialed over mTLS (per-peer pinned config); + // self/manual candidates use the plain transport. See candidateTransport. + Transport: p.candidateTransport(cand), + // ModifyResponse fires when the upstream's status line + headers + // have arrived but before the body streams. That's both the retry + // decision point and, on commit, the time-to-first-byte boundary. + ModifyResponse: func(resp *http.Response) error { + if !last && shouldRetry(resp.StatusCode) { + // Abort before streaming: ReverseProxy closes resp.Body and + // calls ErrorHandler with our sentinel, then we try next. + retry = true + return retrySignal{} + } + // Prefer an engine-declared preflight policy so an exact origin plus + // Allow-Credentials can pass a credentialed browser fetch. Engines + // that publish no policy retain the proxy's permissive 204 fallback. + cors.CompletePreflightFallback(resp) + // Committing to this candidate — body stream is about to begin. + ttfbMs = time.Since(start).Milliseconds() + servedNodeID = cand.id + servedTarget = cand.url.Host + proxyErr = "" // clear any error recorded from a failed-over candidate + // Arm the liveness report only now. statusCapture also carries + // the proxy's OWN error bodies — ReverseProxy's ErrorHandler + // writes a failed dial's message through it — and those bytes + // prove nothing about the node. Reaching here means the upstream + // returned a status line, so everything written from this point + // came from the node. Same goroutine as the body copy, so no + // synchronization is needed. + sc.upstreamAlive = func() { p.reportActivity(cand.id) } + // The engine may enforce its own origin policy. Honor it: + // overwriting a declared Access-Control-Allow-Origin would + // silently widen the user's policy, and a wildcard is invalid + // alongside Allow-Credentials, so it would break a credentialed + // response outright. An engine that omits the header has + // expressed nothing to preserve, so the proxy supplies its own. + if resp.Header.Get("Access-Control-Allow-Origin") == "" { + cors.Apply(resp.Header) + } + if !started { + started = true + p.codec.Notify("proxy/request-started", RequestStartedEvent{ + ID: reqID, + NodeID: cand.id, + Method: r.Method, + Path: r.URL.Path, + Target: cand.url.Host, + }) + // workload:started was already emitted up front naming the + // first candidate. If failover landed us on a different + // node, re-point scheduledOn so the card — and the terminal + // completed/errored event, which carries the same wl — name + // the node that actually served. Guarded by wlMu against the + // disconnect watcher, and skipped once terminated so a late + // re-point can't resurrect a workload we've already failed. + if wl != nil { + wlMu.Lock() + if !terminated && wl.ScheduledOn != cand.id { + wl.ScheduledOn = cand.id + snapshot := *wl + wlMu.Unlock() + p.emitWorkload(workloadStartedMethod, snapshot) + } else { + wlMu.Unlock() + } + } + } + return nil + }, + ErrorHandler: func(ew http.ResponseWriter, _ *http.Request, err error) { + if _, ok := err.(retrySignal); ok { + return // retryable status — the loop advances to the next candidate + } + // Transport/dial error (not a status-based retry): forget this + // node's confirmed address so the next request re-confirms and + // can fail over to another of its published addresses + // (multi-homed peer). The in-request failover below moves on to + // the next node. + p.targets.Forget(cand.id) + if !last { + // Transport/dial error with candidates left: fail over. + retry = true + proxyErr = err.Error() + slog.Warn("proxy upstream error, failing over", + "id", reqID, "node_id", cand.id, "target", cand.url.Host, + "path", r.URL.Path, "err", err) + return + } + // Last candidate failed at the transport: terminal, surface it. + servedNodeID = cand.id + servedTarget = cand.url.Host + if cors.WritePreflight(ew, r) { + proxyErr = "" + return + } + proxyErr = err.Error() + slog.Warn("proxy upstream error, candidates exhausted", + "id", reqID, "node_id", cand.id, "target", cand.url.Host, + "method", r.Method, "path", r.URL.Path, + "duration_ms", time.Since(start).Milliseconds(), "err", err) + body, mErr := json.Marshal(map[string]string{ + "error": "upstream error: " + err.Error(), + }) + if mErr != nil { + body = []byte(`{"error":"upstream error"}`) + } + cors.Apply(ew.Header()) + ew.Header().Set("Content-Type", "application/json") + ew.Header().Set("X-Content-Type-Options", "nosniff") + ew.WriteHeader(http.StatusBadGateway) + ew.Write(body) + }, + } + + proxy.ServeHTTP(sc, r) + if !retry { + finalStatus = sc.status + committedSC = sc + break + } + } + + slog.Debug("proxy request complete", + "id", reqID, + "node_id", servedNodeID, + "method", r.Method, + "path", r.URL.Path, + "target", servedTarget, + "status", finalStatus, + "duration_ms", time.Since(start).Milliseconds(), + "ttfb_ms", ttfbMs, + "err", proxyErr, + ) + + p.codec.Notify("proxy/request", RequestEvent{ + ID: reqID, + NodeID: servedNodeID, + Method: r.Method, + Path: r.URL.Path, + Target: servedTarget, + Status: finalStatus, + Duration: time.Since(start).Milliseconds(), + TTFB: ttfbMs, + Error: proxyErr, + }) + + // Terminal workload transition pairs with the workload:started emitted at + // the commit point above. Cancellation, an upstream/transport error, a + // failed client write (dead/half-open client), or any non-2xx status is a + // failure; a clean 2xx is a completion. The Workload carries the same id so + // the broker (and peers) can collapse the start/finish pair. Routed through + // emitTerminal so the disconnect watcher and this path emit exactly once. + if wl != nil { + switch { + case r.Context().Err() != nil: + // The request was cancelled before it finished — either the + // client disconnected or, on shutdown, we cancelled it to stop + // the in-flight inference. A mid-stream cancel never reaches + // ErrorHandler (the 200 headers are already sent), so without this + // branch it would be misreported as completed. (The watcher above + // usually beats us to it; emitTerminal makes that a no-op.) + emitTerminal("failed", "request cancelled before completion") + case committedSC != nil && committedSC.wroteErr != nil: + // The response committed but a write to (or flush toward) the + // client failed — typically the idle deadline tripping on a + // dead/half-open client. The stream is truncated, so this is a + // failure, not the completion the 200 status would otherwise + // suggest. + emitTerminal("failed", "client connection lost: "+committedSC.wroteErr.Error()) + case proxyErr != "" || finalStatus >= http.StatusBadRequest: + msg := proxyErr + if msg == "" { + msg = fmt.Sprintf("upstream returned HTTP %d", finalStatus) + } + emitTerminal("failed", msg) + default: + emitTerminal("completed", "") + } + } +} + +// resolveCandidates returns the ordered list of nodes to try for the current +// request. A model-bearing request first filters a request-local node copy to +// advertised owners. A user-selected eligible node then leads, followed by +// scheduler priority and stable ID fallback. The failover loop walks the +// resulting owner list until a node returns a usable response. +// +// A node that resolves to this proxy's own listen address is dropped +// (self-forward guard): a local llama.cpp advertisement could otherwise +// resolve back to this proxy's own port and loop. +// +// Returns an empty slice when no forwarding target is available; the caller +// treats that as the rejection path. +func (p *Proxy) resolveCandidates(model string) []candidate { + p.selectedMu.RLock() + id := p.selectedID + p.selectedMu.RUnlock() + + priority := p.PriorityList() + + p.httpMu.Lock() + selfPort := p.port + p.httpMu.Unlock() + + // Re-derive membership and pins before resolving so a cluster joined or left, + // and a peer paired or removed, since the last request is reflected without a + // restart: a removed peer stops being a routable candidate immediately, and a + // freshly-paired one becomes one. + p.mesh.Refresh() + + nodes := p.discovery.Nodes() + known := len(nodes) + if model != "" { + owners := make([]Node, 0, len(nodes)) + for _, node := range nodes { + if nodeAdvertisesModel(node, model) { + owners = append(owners, node) + } + } + nodes = owners + } + // Sort by ID so candidate order is stable across calls — Discovery.Nodes() + // iterates a map, whose order is randomized per call, which otherwise bounced + // back-to-back requests between nodes. The ID sort is also the + // fallback order for nodes the scheduler's priority list doesn't mention. + sort.Slice(nodes, func(i, j int) bool { return nodes[i].ID < nodes[j].ID }) + byID := make(map[string]Node, len(nodes)) + for _, n := range nodes { + byID[n.ID] = n + } + + out := make([]candidate, 0, len(nodes)) + // Dedup by resolved backend host: the same physical node can appear under two + // IDs (e.g. a manually-added entry and its relay-discovered record), and + // routing to the same engine twice is wasteful. + seenHost := make(map[string]bool, len(nodes)) + // placed tracks node IDs already considered so the priority-ordered and + // fallback passes don't reconsider one (scheduler ordering). + placed := make(map[string]bool, len(nodes)) + add := func(n Node) { + placed[n.ID] = true + // targetURL picks a reachable address for a multi-homed node (cached, + // TCP-probed), falling back to the first candidate; nil only when the + // node advertises no usable address. + u := p.targetURL(n) + if u == nil { + return + } + peerUUID := "" + switch { + case isSelfTarget(u, selfPort): + // Our own advertised endpoint (lc now points at this proxy). Serve + // it from the explicit local backend — the loopback engine — rather + // than dialing our own mTLS ingress, which would recurse. Ranking + // still used this node's real (discovered) model list above. + lb, ok := p.localBackendTarget() + if !ok { + slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID) + return + } + u = lb + case p.mesh.HasPin(n.ClusterUUID): + // A pinned cluster peer: reach it only over mTLS to its promoted + // proxy (the lc port now advertises the proxy, not the engine). + // The pin is read from the live mesh refreshed above, not from the + // relayed n.Trusted: that flag is the scanner's answer from whenever + // it last saw this peer's mDNS record, so a peer discovered before + // this node's pins were written stays false until its record next + // changes. It is also strictly weaker than what we hold here — the + // dial itself is gated on ClientTLSConfig finding the same pin — so + // the relayed value can only ever disagree by being stale. + u.Scheme = "https" + peerUUID = n.ClusterUUID + case p.discovery.IsManual(n.ID): + // An explicit user-added manual node: dialed plain to the address + // the user supplied (a deliberate, separately-labeled bypass). + default: + // A relay peer we don't hold a pin for (untrusted, or this node is + // unclustered). Its engine is loopback-only and its proxy refuses + // plaintext from the LAN, so it is not a routable target. + slog.Debug("resolveCandidates: dropping unpinned relay peer", + "node_id", n.ID, "cluster_uuid", n.ClusterUUID) + return + } + // Defensive: the local backend must never resolve back to this proxy. + if isSelfTarget(u, selfPort) { + slog.Debug("resolveCandidates: skipping self-target node", + "node_id", n.ID, "target", u.Host, "self_port", selfPort) + return + } + if seenHost[u.Host] { + return + } + seenHost[u.Host] = true + out = append(out, candidate{ + id: n.ID, + url: u, + peerUUID: peerUUID, + }) + } + + // Capability has already been enforced. An eligible explicit selection wins, + // then the scheduler's least-loaded order, then unlisted owners by stable ID. + if id != "" { + if n, ok := byID[id]; ok && !placed[id] { + add(n) + } + } + for _, pid := range priority { + if n, ok := byID[pid]; ok && !placed[pid] { + add(n) + } + } + for _, n := range nodes { + if !placed[n.ID] { + add(n) + } + } + + slog.Debug("resolveCandidates resolved", + "selected", id, "priority", len(priority), "candidates", len(out), + "eligible", len(nodes), "known", known) + return out +} + +// reserveCandidate atomically moves the least estimated loaded scheduler-listed +// candidate to the front of this request's failover list. The scheduler's +// pending count and GPU pressure form the authoritative baseline; reservations +// are local dispatches made since that snapshot arrived and have not necessarily +// completed the proxy→broker→scheduler→proxy feedback loop yet. +// +// Model eligibility was enforced before this function receives the list. An +// explicit node/select pin bypasses reservations, and unlisted/manual owners +// retain their existing fallback position. +func (p *Proxy) reserveCandidate(candidates []candidate) []candidate { + if len(candidates) == 0 { + return candidates + } + if selectedID := p.SelectedID(); selectedID != "" { + for _, cand := range candidates { + if cand.id == selectedID { + return candidates + } + } + } + + candidateIndex := make(map[string]int, len(candidates)) + for i, cand := range candidates { + candidateIndex[cand.id] = i + } + + p.priorityMu.Lock() + defer p.priorityMu.Unlock() + if len(p.priority) == 0 { + return candidates + } + if p.priorityReservations == nil { + p.priorityReservations = make(map[string]int) + } + + bestIndex := -1 + bestOrder := len(p.priority) + var bestLoad uint64 + for order, id := range p.priority { + index, ok := candidateIndex[id] + if !ok { + continue + } + load := uint64(p.priorityPending[id]) + + uint64(p.priorityGPUPressure[id]) + + uint64(p.priorityReservations[id]) + if bestIndex < 0 || load < bestLoad || (load == bestLoad && order < bestOrder) { + bestIndex = index + bestOrder = order + bestLoad = load + } + } + if bestIndex < 0 { + return candidates + } + + chosen := candidates[bestIndex] + p.priorityReservations[chosen.id]++ + if bestIndex > 0 { + copy(candidates[1:bestIndex+1], candidates[:bestIndex]) + candidates[0] = chosen + } + return candidates +} + +func nodeAdvertisesModel(n Node, model string) bool { + if model == "" { + return false + } + for _, available := range n.Models { + if available == model { + return true + } + } + return false +} + +// isSelfTarget reports whether u points back at this proxy's own listener. +// nodeURL has already rewritten local-interface addresses to 127.0.0.1, so a +// loopback host on our own port is us. +func isSelfTarget(u *url.URL, selfPort int) bool { + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + return false + } + port, err := strconv.Atoi(portStr) + if err != nil || port != selfPort { + return false + } + switch host { + case "127.0.0.1", "::1", "localhost": + return true + } + return false +} + +// nodeURL returns the single best forward URL for a node (the first candidate +// in deterministic, loopback-first order). It does no reachability probing — +// p.targetURL is the request-path entry point. Kept as a free function so the +// URL-construction unit test can exercise it without a Proxy. +func nodeURL(n Node) *url.URL { + candidates := nodeCandidates(n) + if len(candidates) == 0 { + return nil + } + return &url.URL{Scheme: "http", Host: candidates[0]} +} + +// targetURL picks the forward URL for a node, preferring an address we can +// actually reach. With a single candidate it's just that candidate; with +// several (a multi-homed peer) the shared reach.Chooser returns the confirmed +// last-good address, or the node's own top-ranked one while it confirms in the +// background. The confirmation is transport-neutral: a TCP accept proves the +// address is reachable, while the real pinned mTLS request still authenticates +// which peer answered there. +// +// reach.Prefer, not a blocking confirmation: this runs once per discovered node per +// request, including nodes this request will not be routed to, so a handshake here +// would charge every request for every node's connectivity. An address that is +// wrong is caught by the ErrorHandler below, which forgets it and fails over. +func (p *Proxy) targetURL(n Node) *url.URL { + candidates := nodeCandidates(n) + if len(candidates) == 0 { + return nil + } + host := p.targets.Prefer(n.ID, candidates) + return &url.URL{Scheme: "http", Host: host} +} + +// nodeCandidates returns the ordered, de-duplicated host:port targets for a node. +// +// Order comes from the node itself: netpick.Candidates keeps the node's published +// ranking, which it derived from evidence no observer has, and appends anything +// else it advertised. Re-sorting here by address class is what previously put a +// two-host direct-connect link ahead of a peer's real LAN address. +// +// Any local-interface address is rewritten to loopback (the engine binds loopback +// only) and floated to the front because it's unambiguously reachable. +func nodeCandidates(n Node) []string { + port := strconv.Itoa(n.Port) + sorted := netpick.Candidates(n.TXT, n.Addresses) + if len(sorted) == 0 { + // A non-IP entry (a .local hostname) that netpick cannot parse. + hosts := n.Addresses + if len(hosts) == 0 { + if n.Host == "" { + return nil + } + hosts = []string{n.Host} + } + sorted = append([]string(nil), hosts...) + } + + seen := make(map[string]bool, len(sorted)) + var loopback, rest []string + for _, h := range sorted { + // If the address belongs to a local interface, use loopback instead; + // connecting via the machine's own external IP would be refused. + if isLocalAddress(h) { + h = "127.0.0.1" + } + // net.JoinHostPort bracket-wraps IPv6 literals (fe80::1 -> [fe80::1]). + hp := net.JoinHostPort(h, port) + if seen[hp] { + continue + } + seen[hp] = true + if ip := net.ParseIP(h); ip != nil && ip.IsLoopback() { + loopback = append(loopback, hp) + } else { + rest = append(rest, hp) + } + } + return append(loopback, rest...) +} + +var ( + localAddrsMu sync.RWMutex + // localAddrs is the set of IPs currently bound to this host's interfaces. + // It's used to decide whether a discovered node is actually us, so we can + // dial loopback instead of our own external IP (the engine binds loopback + // only). The initial value is a one-shot enumeration; startLocalAddrWatch + // then keeps it in sync with live interface changes, so a late VPN/dock + // interface or a sleep/wake IP reassignment can't strand us dialing a + // stale address. + localAddrs = netmon.Enumerate().LocalIPs +) + +func setLocalAddrs(s map[string]bool) { + localAddrsMu.Lock() + localAddrs = s + localAddrsMu.Unlock() +} + +func isLocalAddress(addr string) bool { + localAddrsMu.RLock() + defer localAddrsMu.RUnlock() + return localAddrs[addr] +} + +// startLocalAddrWatch keeps localAddrs in sync with the host's live interface +// set for the lifetime of ctx. If the network monitor can't start, the set +// stays at its initial enumeration rather than failing the proxy. +func startLocalAddrWatch(ctx context.Context) { + mon, err := netmon.Watch(ctx) + if err != nil { + slog.Warn("proxy: network monitor unavailable; local address set is static", "err", err) + return + } + setLocalAddrs(mon.LocalIPs()) + ch := mon.Subscribe() + go func() { + for range ch { + setLocalAddrs(mon.LocalIPs()) + slog.Debug("proxy: refreshed local address set after network change") + } + }() +} + +func (p *Proxy) SelectedID() string { + p.selectedMu.RLock() + defer p.selectedMu.RUnlock() + return p.selectedID +} + +func (p *Proxy) SetSelected(id string) { + p.selectedMu.Lock() + p.selectedID = id + p.selectedMu.Unlock() +} + +// clearSelectionIfNotPresent resets the user-selected node (and notifies the +// client) when it's neither in the relay-fed set nor a manual node, so a stale +// selection can't pin routing to a departed target. +func (p *Proxy) clearSelectionIfNotPresent(present map[string]bool) { + p.selectedMu.Lock() + sel := p.selectedID + p.selectedMu.Unlock() + if sel == "" || present[sel] || p.discovery.IsManual(sel) { + return + } + p.selectedMu.Lock() + cleared := p.selectedID == sel + if cleared { + p.selectedID = "" + } + p.selectedMu.Unlock() + if cleared { + p.codec.Notify("node/selection-changed", SelectedResult{ID: ""}) + } +} + +// PriorityList returns a copy of the current scheduler-supplied priority order. +func (p *Proxy) PriorityList() []string { + p.priorityMu.RLock() + defer p.priorityMu.RUnlock() + return append([]string(nil), p.priority...) +} + +// SetPriority stores the auto-routing priority order (highest first) and returns +// the number of ids stored. The list is kept verbatim — unknown ids are retained +// (a node may appear in discovery later) and only consulted at request time. An +// empty list clears the scheduler's influence. +func (p *Proxy) SetPriority(nodes []string) int { + return p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: nodes}) +} + +// SetPrioritySnapshot replaces the scheduler baseline and clears optimistic +// reservations made against the previous snapshot. Nodes-only callers remain +// valid: a missing rank supplies zero pending and GPU-pressure baselines. +func (p *Proxy) SetPrioritySnapshot(priority schedulerwire.Priority) int { + cleaned := append([]string(nil), priority.Nodes...) + pending := make(map[string]int, len(priority.Ranks)) + gpuPressure := make(map[string]int, len(priority.Ranks)) + for _, rank := range priority.Ranks { + if rank.ID == "" { + continue + } + if rank.Pending < 0 { + rank.Pending = 0 + } + if rank.GPUPressure < 0 { + rank.GPUPressure = 0 + } else if rank.GPUPressure > schedulerwire.MaxGPUPressure { + rank.GPUPressure = schedulerwire.MaxGPUPressure + } + pending[rank.ID] = rank.Pending + gpuPressure[rank.ID] = rank.GPUPressure + } + + p.priorityMu.Lock() + p.priority = cleaned + p.priorityPending = pending + p.priorityGPUPressure = gpuPressure + p.priorityReservations = make(map[string]int) + p.priorityMu.Unlock() + return len(cleaned) +} + +// replaceSubscribed replaces the proxy's relay-fed routing overlay from a +// discovery:nodes snapshot: it projects every node advertising lc with a dialable +// IP into the overlay (dropping the rest) and clears a user selection pinned to a +// node that's no longer routable. The broker sends the full filtered set on every +// change, so this is a wholesale replace, not a per-node apply — a departed node +// is simply absent from the next snapshot. +func (p *Proxy) replaceSubscribed(params json.RawMessage) { + var res noderec.GetNodesResult + if err := json.Unmarshal(params, &res); err != nil { + slog.Warn("invalid discovery:nodes snapshot", "err", err) + return + } + nodes := make([]Node, 0, len(res.Nodes)) + present := make(map[string]bool, len(res.Nodes)) + for _, dn := range res.Nodes { + n, ok := subscribedToNode(dn) + if !ok { + continue + } + nodes = append(nodes, n) + present[n.ID] = true + } + discovered, updated, removed := p.discovery.SetSubscribed(nodes) + // Surface the relay-fed set to the client as node/* events — the signal a + // consumer (the UI) uses to show which peers run this engine — mirroring how + // manual nodes are announced. Without this the routing overlay updates + // silently and peers appear engine-less. A node dropping out is also the + // proxy's "this upstream is gone" signal, surfaced through the errors + // pipeline (the broker forwards these to nvpair-errors); a re-appearance clears + // it. NodeID/Timestamp are left unset so the broker stamps the authoritative + // values. + for _, n := range discovered { + p.codec.Notify("node/discovered", n.withPrimaryIP()) + if err := p.codec.Notify("errors:clear", errors.ClearParams{ID: upstreamUnreachableID(n.ID)}); err != nil { + slog.Debug("failed to send errors:clear", "node", n.ID, "err", err) + } + } + for _, n := range updated { + p.codec.Notify("node/updated", n.withPrimaryIP()) + } + for _, n := range removed { + p.codec.Notify("node/removed", n.withPrimaryIP()) + if err := p.codec.Notify("errors:report", errors.ServiceError{ + ID: upstreamUnreachableID(n.ID), + Message: fmt.Sprintf("Upstream node %q is no longer reachable (dropped from discovery)", n.Host), + Severity: "warning", + Action: "none", + }); err != nil { + slog.Debug("failed to send errors:report", "node", n.ID, "err", err) + } + } + p.clearSelectionIfNotPresent(present) +} + +// upstreamUnreachableID is the canonical ServiceError id for an upstream the +// proxy no longer sees in discovery. Kept as a single function so the report and +// clear can't drift (nvpair-errors matches by literal id). +func upstreamUnreachableID(nodeID string) string { + return "llamacpp-proxy:upstream-unreachable:" + nodeID +} + +// subscribedToNode projects a relay DirectoryNode onto the proxy's routable Node +// for the lc service, returning false when the node doesn't advertise lc or has +// no dialable address. The engine port comes from the lc service key (the real +// llama.cpp port the broker's engine poller registered, not the proxy's listen +// port). +func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { + svc, ok := n.Services[noderec.ServiceLlamaCpp] + if !ok || n.IP == "" { + return Node{}, false + } + // Key routing by the stable per-host UUID, not the hostname: candidate ids, + // scheduledOn, node selection, and the scheduler's priority list are all this + // value, so routing survives a PC rename and never conflates two same-named + // machines. Host stays the hostname (display / dial name). A relay + // DirectoryNode always carries a hostUuid (the scanner guarantees it at the + // browse boundary), so there is no name fallback here. + return Node{ + ID: n.HostUUID, + Host: n.Name, + Port: svc.Port, + // The node's whole ranked address list, not just its canonical one: a + // multi-homed peer's best address from its own vantage point may be a + // direct-connect link this host cannot reach, and routing needs somewhere + // to fail over to when that happens. + Addresses: n.CandidateIPs(), + TXT: n.AddressTXT(), + IP: n.IP, + ClusterUUID: n.ClusterUUID, + // Routing eligibility uses the loaded set, not the on-disk catalog, so a + // catalog-only id is never treated as a llama.cpp owner. Dual-engine + // nodes still project only this engine's loaded models, never the union. + Models: append([]string(nil), n.EngineLoadedModels("llamacpp")...), + }, true +} + +func (p *Proxy) readLoop(ctx context.Context) error { + for { + msg, err := p.codec.Read() + if err != nil { + if err == io.EOF || ctx.Err() != nil { + return nil + } + log.Printf("JSON-RPC read error: %v", err) + continue + } + p.handleMessage(msg) + } +} + +func (p *Proxy) handleMessage(msg *Message) { + if msg.Method == applog.SetLevelMethod { + resolved, err := applog.HandleSetLevelParams(msg.Params) + if msg.IsRequest() { + if err != nil { + p.codec.RespondError(msg.ID, -32602, err.Error()) + return + } + p.codec.Respond(msg.ID, map[string]string{"level": resolved}) + } + if err != nil { + slog.Warn("log/set-level rejected", "err", err) + } else { + slog.Info("log level changed", "level", resolved) + } + return + } + + switch msg.Method { + case noderec.NotifyNodes: + p.replaceSubscribed(msg.Params) + return + } + + if !msg.IsRequest() { + if msg.IsNotification() { + log.Printf("ignoring incoming notification: %s", msg.Method) + } + return + } + + switch msg.Method { + case "nodes/list": + nodes := p.discovery.Nodes() + if err := p.codec.Respond(msg.ID, NodesResult{Nodes: nodes}); err != nil { + log.Printf("failed to respond to nodes/list: %v", err) + } + + case "node/select": + var params SelectParams + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}") + return + } + if params.ID != "" { + found := false + for _, n := range p.discovery.Nodes() { + if n.ID == params.ID { + found = true + break + } + } + if !found { + p.codec.RespondError(msg.ID, -32602, fmt.Sprintf("node %q not found", params.ID)) + return + } + } + p.SetSelected(params.ID) + log.Printf("node selection changed to %q", params.ID) + if err := p.codec.Respond(msg.ID, SelectedResult{ID: params.ID}); err != nil { + log.Printf("failed to respond to node/select: %v", err) + } + p.codec.Notify("node/selection-changed", SelectedResult{ID: params.ID}) + + case "node/selected": + if err := p.codec.Respond(msg.ID, SelectedResult{ID: p.SelectedID()}); err != nil { + log.Printf("failed to respond to node/selected: %v", err) + } + + case "node/set-priority": + var params schedulerwire.Priority + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"nodes\": [\"id\", ...], \"ranks\": [...]}") + return + } + count := p.SetPrioritySnapshot(params) + log.Printf("priority snapshot set (%d nodes, %d ranks): %v", count, len(params.Ranks), params.Nodes) + if err := p.codec.Respond(msg.ID, map[string]int{"count": count}); err != nil { + log.Printf("failed to respond to node/set-priority: %v", err) + } + + case "set-port": + var params struct { + Port int `json:"port"` + } + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"port\": }") + return + } + if params.Port < 1 || params.Port > 65535 { + p.codec.RespondError(msg.ID, -32602, "port must be between 1 and 65535") + return + } + if err := p.setPort(params.Port); err != nil { + p.codec.RespondError(msg.ID, -32000, err.Error()) + return + } + if err := p.codec.Respond(msg.ID, ReadyParams{Version: Version, Port: params.Port}); err != nil { + log.Printf("failed to respond to set-port: %v", err) + } + + case "node/add-manual": + var node Node + if err := json.Unmarshal(msg.Params, &node); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\",\"host\",\"port\",\"addresses\"}") + return + } + if node.ID == "" || node.Port == 0 || len(node.Addresses) == 0 { + p.codec.RespondError(msg.ID, -32602, "id, port, and at least one address are required") + return + } + added := p.discovery.AddManual(node) + if err := p.codec.Respond(msg.ID, map[string]bool{"added": added}); err != nil { + log.Printf("failed to respond to node/add-manual: %v", err) + } + if added { + log.Printf("manual node added: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + p.codec.Notify("node/discovered", node.withPrimaryIP()) + } else { + log.Printf("manual node updated: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + p.codec.Notify("node/updated", node.withPrimaryIP()) + } + + case "node/remove-manual": + var params SelectParams + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}") + return + } + removed := p.discovery.RemoveManual(params.ID) + if err := p.codec.Respond(msg.ID, map[string]bool{"removed": removed}); err != nil { + log.Printf("failed to respond to node/remove-manual: %v", err) + } + if removed { + log.Printf("manual node removed: %s", params.ID) + p.selectedMu.Lock() + if p.selectedID == params.ID { + p.selectedID = "" + p.selectedMu.Unlock() + p.codec.Notify("node/selection-changed", SelectedResult{ID: ""}) + } else { + p.selectedMu.Unlock() + } + p.codec.Notify("node/removed", Node{ID: params.ID}) + } + + case "node/set-local-backend": + var b localBackend + if err := json.Unmarshal(msg.Params, &b); err != nil { + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"host\",\"port\",\"healthy\"}") + return + } + p.setLocalBackend(b) + slog.Info("local backend updated", "engine", b.Engine, "host", b.Host, "port", b.Port, "healthy", b.Healthy) + if err := p.codec.Respond(msg.ID, map[string]bool{"ok": true}); err != nil { + log.Printf("failed to respond to node/set-local-backend: %v", err) + } + + case "shutdown": + if err := p.codec.Respond(msg.ID, nil); err != nil { + log.Printf("failed to respond to shutdown: %v", err) + } + log.Println("shutdown requested via JSON-RPC") + p.cancel() + + default: + if err := p.codec.RespondError(msg.ID, -32601, fmt.Sprintf("method not found: %s", msg.Method)); err != nil { + log.Printf("failed to send error response: %v", err) + } + } +} diff --git a/services/llamacpp-proxy/proxy_test.go b/services/llamacpp-proxy/proxy_test.go new file mode 100644 index 00000000..d0a029c6 --- /dev/null +++ b/services/llamacpp-proxy/proxy_test.go @@ -0,0 +1,63 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import "testing" + +// TestNodeURL covers the URL-construction part of nodeURL — specifically +// that IPv6 literals are bracket-wrapped and IPv4/hostnames remain +// byte-identical to the pre-JoinHostPort implementation. The +// local-address shortcut is not exercised here because it depends on +// the host's network interfaces (see init() in proxy.go) and would be +// flaky across environments. +func TestNodeURL(t *testing.T) { + // Pick addresses that are unlikely to appear on any local interface. + tests := []struct { + name string + node Node + wantHost string + wantURL string + }{ + { + name: "ipv4", + node: Node{Addresses: []string{"192.0.2.10"}, Port: 11434}, + wantHost: "192.0.2.10:11434", + wantURL: "http://192.0.2.10:11434", + }, + { + name: "ipv6", + node: Node{Addresses: []string{"2001:db8::1"}, Port: 11434}, + wantHost: "[2001:db8::1]:11434", + wantURL: "http://[2001:db8::1]:11434", + }, + { + name: "hostname", + node: Node{Addresses: []string{"gpu-host.lan"}, Port: 11434}, + wantHost: "gpu-host.lan:11434", + wantURL: "http://gpu-host.lan:11434", + }, + { + // Empty Addresses slice — nodeURL should fall back to Host. + name: "fallback to Host", + node: Node{Host: "gpu-host.lan", Port: 11434}, + wantHost: "gpu-host.lan:11434", + wantURL: "http://gpu-host.lan:11434", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + u := nodeURL(tc.node) + if u == nil { + t.Fatalf("nodeURL returned nil") + } + if u.Host != tc.wantHost { + t.Errorf("Host = %q, want %q", u.Host, tc.wantHost) + } + if got := u.String(); got != tc.wantURL { + t.Errorf("String() = %q, want %q", got, tc.wantURL) + } + }) + } +} diff --git a/services/llamacpp-proxy/reservation_test.go b/services/llamacpp-proxy/reservation_test.go new file mode 100644 index 00000000..97049ebf --- /dev/null +++ b/services/llamacpp-proxy/reservation_test.go @@ -0,0 +1,235 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "sync" + "testing" + + "nvpair-shared/schedulerwire" +) + +func reservationCandidates(ids ...string) []candidate { + out := make([]candidate, 0, len(ids)) + for _, id := range ids { + out = append(out, candidate{id: id}) + } + return out +} + +func reservedID(p *Proxy, candidates []candidate) string { + candidates = append([]candidate(nil), candidates...) + return p.reserveCandidate(candidates)[0].id +} + +func TestReserveCandidate_ConcurrentEqualLoadHasAtMostOneSkew(t *testing.T) { + p := prProxy(t) + ids := []string{"a", "b", "c", "d"} + ranks := make([]schedulerwire.NodeRank, 0, len(ids)) + for i, id := range ids { + ranks = append(ranks, schedulerwire.NodeRank{ID: id, Rank: i}) + } + p.SetPrioritySnapshot(schedulerwire.Priority{Nodes: ids, Ranks: ranks}) + candidates := reservationCandidates(ids...) + + const requests = 100 + chosen := make(chan string, requests) + var wg sync.WaitGroup + for range requests { + wg.Add(1) + go func() { + defer wg.Done() + chosen <- reservedID(p, candidates) + }() + } + wg.Wait() + close(chosen) + + counts := make(map[string]int, len(ids)) + for id := range chosen { + counts[id]++ + } + min, max := requests, 0 + for _, id := range ids { + if counts[id] < min { + min = counts[id] + } + if counts[id] > max { + max = counts[id] + } + } + if max-min > 1 { + t.Fatalf("100 equal-load reservations are imbalanced: %v", counts) + } +} + +func TestReserveCandidate_ConvergesUnequalPendingDepths(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"a", "b", "c"}, + Ranks: []schedulerwire.NodeRank{ + {ID: "a", Pending: 0, Rank: 0}, + {ID: "b", Pending: 2, Rank: 1}, + {ID: "c", Pending: 4, Rank: 2}, + }, + }) + candidates := reservationCandidates("a", "b", "c") + assigned := map[string]int{} + for range 6 { + assigned[reservedID(p, candidates)]++ + } + + total := map[string]int{ + "a": assigned["a"], + "b": 2 + assigned["b"], + "c": 4 + assigned["c"], + } + if total["a"] != 4 || total["b"] != 4 || total["c"] != 4 { + t.Fatalf("unequal depths did not converge: assigned=%v total=%v", assigned, total) + } +} + +func TestReserveCandidate_CombinesPendingPressureAndReservations(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"a", "b", "c"}, + Ranks: []schedulerwire.NodeRank{ + {ID: "a", Pending: 0, GPUPressure: 3}, + {ID: "b", Pending: 1, GPUPressure: 0}, + {ID: "c", Pending: 0, GPUPressure: 2}, + }, + }) + candidates := reservationCandidates("a", "b", "c") + got := []string{ + reservedID(p, candidates), + reservedID(p, candidates), + reservedID(p, candidates), + } + want := []string{"b", "b", "c"} + for i := range want { + if got[i] != want[i] { + t.Fatalf("GPU-aware reservations = %v, want %v", got, want) + } + } +} + +func TestSetPrioritySnapshotClampsGPUPressure(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"low", "high"}, + Ranks: []schedulerwire.NodeRank{ + {ID: "low", GPUPressure: -1}, + {ID: "high", GPUPressure: schedulerwire.MaxGPUPressure + 1}, + }, + }) + p.priorityMu.RLock() + low := p.priorityGPUPressure["low"] + high := p.priorityGPUPressure["high"] + p.priorityMu.RUnlock() + if low != 0 || high != schedulerwire.MaxGPUPressure { + t.Fatalf("clamped GPU pressure = low:%d high:%d", low, high) + } +} + +func TestReserveCandidate_LegacyNodesOnlyUsesZeroBaseline(t *testing.T) { + p := prProxy(t) + p.SetPriority([]string{"a", "b", "c"}) + candidates := reservationCandidates("a", "b", "c") + counts := map[string]int{} + for range 5 { + counts[reservedID(p, candidates)]++ + } + want := map[string]int{"a": 2, "b": 2, "c": 1} + for id, n := range want { + if counts[id] != n { + t.Fatalf("legacy nodes-only assignments = %v, want %v", counts, want) + } + } +} + +func TestReserveCandidate_UsesEligibleCandidates(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"missing", "owner-a", "owner-b", "unknown"}, + Ranks: []schedulerwire.NodeRank{ + {ID: "missing", Pending: 0}, + {ID: "owner-a", Pending: 4}, + {ID: "owner-b", Pending: 5}, + {ID: "unknown", Pending: 0}, + }, + }) + candidates := reservationCandidates("owner-a", "owner-b") + + for range 8 { + got := reservedID(p, candidates) + if got != "owner-a" && got != "owner-b" { + t.Fatalf("reservation escaped eligible candidates to %q", got) + } + } +} + +func TestReserveCandidate_ManualPinBypassesReservations(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"a", "b"}, + Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}}, + }) + p.SetSelected("b") + candidates := reservationCandidates("b", "a") // resolveCandidates puts the pin first + if got := reservedID(p, candidates); got != "b" { + t.Fatalf("manual pin resolved to %q, want b", got) + } + p.priorityMu.RLock() + defer p.priorityMu.RUnlock() + if len(p.priorityReservations) != 0 { + t.Fatalf("manual pin created optimistic reservations: %v", p.priorityReservations) + } +} + +func TestReserveCandidate_IneligibleManualPinDoesNotBypassReservations(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"owner-b", "owner-a"}, + Ranks: []schedulerwire.NodeRank{{ID: "owner-b"}, {ID: "owner-a"}}, + }) + p.SetSelected("missing") + if got := reservedID(p, reservationCandidates("owner-a", "owner-b")); got != "owner-b" { + t.Fatalf("reservation with ineligible pin = %q, want owner-b", got) + } +} + +func TestReserveCandidate_PreservesFailoverAndSnapshotReset(t *testing.T) { + p := prProxy(t) + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"b", "a", "c"}, + Ranks: []schedulerwire.NodeRank{ + {ID: "b", Pending: 0}, + {ID: "a", Pending: 5}, + {ID: "c", Pending: 6}, + }, + }) + got := p.reserveCandidate(reservationCandidates("a", "b", "c")) + want := []string{"b", "a", "c"} + for i, id := range want { + if got[i].id != id { + t.Fatalf("reserved failover order = %v, want %v", candidateIDsFrom(got), want) + } + } + + p.SetPrioritySnapshot(schedulerwire.Priority{ + Nodes: []string{"a", "b", "c"}, + Ranks: []schedulerwire.NodeRank{{ID: "a"}, {ID: "b"}, {ID: "c"}}, + }) + if next := reservedID(p, reservationCandidates("a", "b", "c")); next != "a" { + t.Fatalf("new snapshot did not reset reservations: next = %q, want a", next) + } +} + +func candidateIDsFrom(candidates []candidate) []string { + out := make([]string, 0, len(candidates)) + for _, candidate := range candidates { + out = append(out, candidate.id) + } + return out +} diff --git a/services/llamacpp-proxy/server_timeout_test.go b/services/llamacpp-proxy/server_timeout_test.go new file mode 100644 index 00000000..39d9b881 --- /dev/null +++ b/services/llamacpp-proxy/server_timeout_test.go @@ -0,0 +1,43 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "net" + "testing" + + "nvpair-shared/clustertrust" +) + +func TestHTTPServersConfigureIdleTimeouts(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + p.mesh = clustertrust.Open(t.TempDir()) + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + p.serveHTTP(context.Background(), ln) + defer p.shutdown(context.Background()) + + if p.plainSrv == nil || p.tlsSrv == nil { + t.Fatal("servers not recorded") + } + for name, srv := range map[string]struct { + readHeader, idle interface{} + }{ + "plain": {p.plainSrv.ReadHeaderTimeout, p.plainSrv.IdleTimeout}, + "tls": {p.tlsSrv.ReadHeaderTimeout, p.tlsSrv.IdleTimeout}, + } { + if srv.readHeader != proxyReadHeaderTimeout { + t.Errorf("%s ReadHeaderTimeout = %v, want %v", name, srv.readHeader, proxyReadHeaderTimeout) + } + if srv.idle != proxyServerIdleTimeout { + t.Errorf("%s IdleTimeout = %v, want %v", name, srv.idle, proxyServerIdleTimeout) + } + } + if proxyServerIdleTimeout != proxyIdleConnTimeout { + t.Fatalf("server IdleTimeout %v != client IdleConnTimeout %v", proxyServerIdleTimeout, proxyIdleConnTimeout) + } +} diff --git a/services/llamacpp-proxy/subscribed_test.go b/services/llamacpp-proxy/subscribed_test.go new file mode 100644 index 00000000..65c6740d --- /dev/null +++ b/services/llamacpp-proxy/subscribed_test.go @@ -0,0 +1,103 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "testing" + + "nvpair-shared/noderec" +) + +// TestSubscribedToNode covers the DirectoryNode -> routable Node projection for +// the lc service, including per-engine model attribution: the proxy ranks on the +// node's llama.cpp models only, never the cross-engine union. +func TestSubscribedToNode(t *testing.T) { + withLC := noderec.DirectoryNode{ + HostUUID: "uuid-a", + Name: "host-a", + IP: "10.0.0.5", + Models: []string{"gguf"}, + LoadedByEngine: map[string][]string{ + "llamacpp": {"gguf"}, + }, + Services: map[noderec.ServiceKey]noderec.ServiceStatus{ + noderec.ServiceLlamaCpp: {Port: 1234}, + }, + } + got, ok := subscribedToNode(withLC) + if !ok { + t.Fatal("node with lc + IP should project") + } + if got.ID != "uuid-a" || got.Port != 1234 || got.IP != "10.0.0.5" || + len(got.Models) != 1 || got.Models[0] != "gguf" { + t.Fatalf("unexpected projection: %+v", got) + } + + noIP := withLC + noIP.IP = "" + if _, ok := subscribedToNode(noIP); ok { + t.Fatal("node without IP should not project") + } + + // A node advertising only a non-lc service must not be an lc routing target. + olOnly := noderec.DirectoryNode{ + Name: "host-b", + IP: "10.0.0.6", + Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceOllama: {Port: 11434}}, + } + if _, ok := subscribedToNode(olOnly); ok { + t.Fatal("node without lc should not project") + } + + // Per-engine attribution: a dual-engine node projects ONLY its loaded + // llama.cpp models, never the union — so an Ollama-only model isn't ranked + // as a llama.cpp owner. + dual := noderec.DirectoryNode{ + HostUUID: "uuid-d", + Name: "host-d", + IP: "10.0.0.7", + Models: []string{"ollama-model", "llamacpp-model"}, + ModelsByEngine: map[string][]string{ + "ollama": {"ollama-model"}, + "llamacpp": {"llamacpp-model"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {"llamacpp-model"}, + }, + Services: map[noderec.ServiceKey]noderec.ServiceStatus{ + noderec.ServiceLlamaCpp: {Port: 1234}, + }, + } + got, ok = subscribedToNode(dual) + if !ok { + t.Fatal("dual-engine node with lc should project") + } + if len(got.Models) != 1 || got.Models[0] != "llamacpp-model" { + t.Fatalf("dual-engine projection Models = %v, want [llamacpp-model] only", got.Models) + } +} + +// TestSubscribedToNodeKeysByHostUUID: the routable Node keys on the stable +// hostUuid, not the hostname, so routing/scheduledOn/selection survive a PC +// rename and never conflate same-named machines. Host stays the hostname for +// display. +func TestSubscribedToNodeKeysByHostUUID(t *testing.T) { + const uuid = "22222222-2222-2222-2222-222222222222" + n := noderec.DirectoryNode{ + HostUUID: uuid, + Name: "host-a", + IP: "10.0.0.5", + Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceLlamaCpp: {Port: 1234}}, + } + got, ok := subscribedToNode(n) + if !ok { + t.Fatal("node with lc + IP should project") + } + if got.ID != uuid { + t.Fatalf("ID = %q, want hostUuid %q", got.ID, uuid) + } + if got.Host != "host-a" { + t.Fatalf("Host = %q, want hostname for display", got.Host) + } +} diff --git a/services/llamacpp-proxy/transport.go b/services/llamacpp-proxy/transport.go new file mode 100644 index 00000000..b62d6278 --- /dev/null +++ b/services/llamacpp-proxy/transport.go @@ -0,0 +1,25 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "io" + "os" +) + +type stdioTransport struct { + io.Reader + io.Writer +} + +func newStdioTransport() io.ReadWriteCloser { + return &stdioTransport{ + Reader: os.Stdin, + Writer: os.Stdout, + } +} + +func (s *stdioTransport) Close() error { + return nil +} diff --git a/services/llamacpp-proxy/transport_pool_test.go b/services/llamacpp-proxy/transport_pool_test.go new file mode 100644 index 00000000..2dc50f0d --- /dev/null +++ b/services/llamacpp-proxy/transport_pool_test.go @@ -0,0 +1,76 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "path/filepath" + "testing" + + "nvpair-shared/clustertrust" + "nvpair-shared/clustertrusttest" +) + +func TestCandidateTransportReusesPlainTransport(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + a := p.candidateTransport(candidate{}) + b := p.candidateTransport(candidate{id: "manual"}) + if a != b { + t.Fatalf("plain candidates returned distinct Transports") + } + if a == nil { + t.Fatal("plain Transport is nil") + } +} + +func TestCandidateTransportReusesPeerTransport(t *testing.T) { + const peerUUID = "principal-peer" + clusterDir := filepath.Join(t.TempDir(), "cluster") + clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID) + + p := testProxy(NewDiscovery(), 1235) + p.mesh = clustertrust.Open(clusterDir) + + a := p.candidateTransport(candidate{peerUUID: peerUUID}) + b := p.candidateTransport(candidate{peerUUID: peerUUID}) + if a != b { + t.Fatalf("same peerUUID returned distinct Transports") + } + if a.TLSClientConfig == nil { + t.Fatal("peer Transport missing TLSClientConfig") + } + + other := p.candidateTransport(candidate{peerUUID: "principal-other"}) + if other == a { + t.Fatal("unpinned peer reused pinned peer Transport") + } +} + +func TestDropUnpinnedPeerTransportsRemovesEntry(t *testing.T) { + const peerUUID = "principal-peer" + clusterDir := filepath.Join(t.TempDir(), "cluster") + clustertrusttest.Join(t, clusterDir, "cluster-xyz", "principal-self", peerUUID) + + p := testProxy(NewDiscovery(), 1235) + p.mesh = clustertrust.Open(clusterDir) + + tr := p.candidateTransport(candidate{peerUUID: peerUUID}) + p.transportMu.Lock() + if _, ok := p.peerTransports[peerUUID]; !ok { + p.transportMu.Unlock() + t.Fatal("peer Transport was not cached") + } + p.transportMu.Unlock() + + clustertrusttest.RemovePeerPin(t, clusterDir, peerUUID) + p.mesh.Refresh() + p.dropUnpinnedPeerTransports() + + p.transportMu.Lock() + _, still := p.peerTransports[peerUUID] + p.transportMu.Unlock() + if still { + t.Fatal("peer Transport remained after pin removal") + } + _ = tr +} diff --git a/services/llamacpp-proxy/zombie_test.go b/services/llamacpp-proxy/zombie_test.go new file mode 100644 index 00000000..29fde4f0 --- /dev/null +++ b/services/llamacpp-proxy/zombie_test.go @@ -0,0 +1,494 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bufio" + "bytes" + "context" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" +) + +// count reports how many times s appears across the recorded codec frames. +// Method names appear once per emitted notification, so this counts emissions. +func (r *prRec) count(s string) int { + r.mu.Lock() + defer r.mu.Unlock() + hay := string(r.b) + n := 0 + for i := 0; i+len(s) <= len(hay); i++ { + if hay[i:i+len(s)] == s { + n++ + } + } + return n +} + +// clientGoneWriter is an http.ResponseWriter whose body Write fails after the +// status line is sent, standing in for a client that vanished mid-stream: the +// idle write deadline trips (statusCapture.Write) and the underlying +// connection write returns an error. It records the status and supports Flush +// so the reverse proxy streams through it. This is the shape of the zombie-job +// bug — the response has committed (200), so without the wroteErr check the +// terminal would be misreported as completed. +type clientGoneWriter struct { + header http.Header + status int + err error + wrote bool +} + +func (c *clientGoneWriter) Header() http.Header { + if c.header == nil { + c.header = make(http.Header) + } + return c.header +} + +func (c *clientGoneWriter) WriteHeader(code int) { c.status = code } + +func (c *clientGoneWriter) Write(b []byte) (int, error) { + c.wrote = true + return 0, c.err +} + +func (c *clientGoneWriter) Flush() {} + +// TestHandleHTTP_ClientWriteError_MarksFailed is the zombie-job regression: a +// streaming inference response that has committed (200 headers sent) but whose +// body write to the client fails — the signature of a killed / half-open client +// whose write deadline tripped — must terminate the workload as FAILED, not be +// silently reported completed. +func TestHandleHTTP_ClientWriteError_MarksFailed(t *testing.T) { + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + io.WriteString(w, `{"choices":[{"delta":{"content":"partial tokens"}}]}`) + })) + defer upstream.Close() + + rec := &prRec{} + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama")) + p := NewProxy(NewCodec(rec), disc, 1235) + + cw := &clientGoneWriter{err: errors.New("write tcp: connection reset by peer")} + + done := make(chan struct{}) + go func() { + defer close(done) + p.handleHTTP(cw, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`))) + }() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("handleHTTP did not return after the client write failed (zombie: handler blocked)") + } + + if !cw.wrote { + t.Fatal("reverse proxy never attempted a body write to the client; test did not exercise the streaming path") + } + if got := rec.count("workload:errored"); got != 1 { + t.Fatalf("workload:errored emitted %d times, want exactly 1", got) + } + if rec.has("workload:completed") { + t.Fatal("workload:completed emitted for a request whose client write failed (should be failed)") + } +} + +// TestHandleHTTP_ClientDisconnect_TerminalOnce covers the disconnect watcher: a +// request whose context is cancelled mid-flight must emit exactly one terminal +// (errored) — the watcher and post-handler path are guarded by terminalOnce — +// and handleHTTP must return promptly rather than hang. +func TestHandleHTTP_ClientDisconnect_TerminalOnce(t *testing.T) { + received := make(chan struct{}, 1) + release := make(chan struct{}) + var releaseOnce sync.Once + doRelease := func() { releaseOnce.Do(func() { close(release) }) } + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + if f, ok := w.(http.Flusher); ok { + io.WriteString(w, `{"choices":[{"delta":{"content":"first chunk"}}]}`+"\n") + f.Flush() + } + select { + case received <- struct{}{}: + default: + } + <-release + })) + defer upstream.Close() + defer doRelease() + + rec := &prRec{} + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama")) + p := NewProxy(NewCodec(rec), disc, 1235) + + ctx, cancel := context.WithCancel(context.Background()) + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"llama"}`)).WithContext(ctx) + + done := make(chan struct{}) + go func() { + defer close(done) + p.handleHTTP(httptest.NewRecorder(), req) + }() + + select { + case <-received: + case <-time.After(5 * time.Second): + t.Fatal("upstream never started streaming") + } + cancel() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("handleHTTP did not return after client disconnect (zombie: handler blocked)") + } + + if got := rec.count("workload:errored"); got != 1 { + t.Fatalf("workload:errored emitted %d times, want exactly 1 (terminalOnce guard)", got) + } + if rec.has("workload:completed") { + t.Fatal("workload:completed emitted for a cancelled request") + } +} + +// deadlineRW records SetWriteDeadline calls and can force a Write error, so the +// statusCapture write-deadline mechanics can be tested without a real socket. +type deadlineRW struct { + *httptest.ResponseRecorder + deadlines []time.Time + writeErr error + flushErr error + flushed int +} + +func (d *deadlineRW) SetWriteDeadline(t time.Time) error { + d.deadlines = append(d.deadlines, t) + return nil +} + +func (d *deadlineRW) Write(b []byte) (int, error) { + if d.writeErr != nil { + return 0, d.writeErr + } + return d.ResponseRecorder.Write(b) +} + +// FlushError lets a test drive statusCapture.FlushError against a controllable +// flush outcome (recorded so the deadline arm/clear can be asserted). +func (d *deadlineRW) FlushError() error { + d.flushed++ + return d.flushErr +} + +// TestStatusCapture_WriteDeadline verifies statusCapture arms a write deadline +// around each streamed write and clears it after a successful one, and that the +// first write error is retained for the caller to classify the workload failed. +func TestStatusCapture_WriteDeadline(t *testing.T) { + t.Run("armed then cleared on success", func(t *testing.T) { + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond} + if _, err := sc.Write([]byte("tokens")); err != nil { + t.Fatalf("Write returned error: %v", err) + } + if len(d.deadlines) != 2 { + t.Fatalf("SetWriteDeadline called %d times, want 2 (arm + clear)", len(d.deadlines)) + } + if d.deadlines[0].IsZero() { + t.Fatal("first SetWriteDeadline should arm a future deadline, got zero") + } + if !d.deadlines[1].IsZero() { + t.Fatal("second SetWriteDeadline should clear the deadline (zero time)") + } + if sc.wroteErr != nil { + t.Fatalf("wroteErr set after a successful write: %v", sc.wroteErr) + } + }) + + t.Run("write error retained, deadline not cleared", func(t *testing.T) { + boom := errors.New("i/o timeout") + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), writeErr: boom} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond} + if _, err := sc.Write([]byte("tokens")); !errors.Is(err, boom) { + t.Fatalf("Write err = %v, want %v", err, boom) + } + if !errors.Is(sc.wroteErr, boom) { + t.Fatalf("wroteErr = %v, want %v", sc.wroteErr, boom) + } + if len(d.deadlines) != 1 { + t.Fatalf("SetWriteDeadline called %d times, want 1 (arm only; not cleared on error)", len(d.deadlines)) + } + }) + + t.Run("no deadline when idle is zero", func(t *testing.T) { + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK} + if _, err := sc.Write([]byte("tokens")); err != nil { + t.Fatalf("Write returned error: %v", err) + } + if len(d.deadlines) != 0 { + t.Fatalf("SetWriteDeadline called %d times with idle=0, want 0", len(d.deadlines)) + } + }) +} + +// TestStatusCapture_FlushDeadline verifies the flush path is deadline-aware: +// statusCapture.FlushError arms the idle deadline around the underlying flush, +// clears it after a successful flush, and retains a real flush error (but not an +// unsupported-flush) so a stalled client's blocked flush is classified failed +// rather than hanging forever. +func TestStatusCapture_FlushDeadline(t *testing.T) { + t.Run("armed then cleared on success", func(t *testing.T) { + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder()} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond} + if err := sc.FlushError(); err != nil { + t.Fatalf("FlushError returned error: %v", err) + } + if d.flushed != 1 { + t.Fatalf("underlying flushed %d times, want 1", d.flushed) + } + if len(d.deadlines) != 2 { + t.Fatalf("SetWriteDeadline called %d times, want 2 (arm + clear)", len(d.deadlines)) + } + if d.deadlines[0].IsZero() { + t.Fatal("flush should arm a future deadline, got zero") + } + if !d.deadlines[1].IsZero() { + t.Fatal("flush should clear the deadline on success (zero time)") + } + if sc.wroteErr != nil { + t.Fatalf("wroteErr set after a successful flush: %v", sc.wroteErr) + } + }) + + t.Run("flush error retained, deadline not cleared", func(t *testing.T) { + boom := errors.New("i/o timeout") + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), flushErr: boom} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond} + if err := sc.FlushError(); !errors.Is(err, boom) { + t.Fatalf("FlushError = %v, want %v", err, boom) + } + if !errors.Is(sc.wroteErr, boom) { + t.Fatalf("wroteErr = %v, want %v", sc.wroteErr, boom) + } + if len(d.deadlines) != 1 { + t.Fatalf("SetWriteDeadline called %d times, want 1 (arm only; not cleared on error)", len(d.deadlines)) + } + }) + + t.Run("unsupported flush is not a client failure", func(t *testing.T) { + d := &deadlineRW{ResponseRecorder: httptest.NewRecorder(), flushErr: http.ErrNotSupported} + sc := &statusCapture{ResponseWriter: d, status: http.StatusOK, idle: 50 * time.Millisecond} + _ = sc.FlushError() + if sc.wroteErr != nil { + t.Fatalf("ErrNotSupported must not be retained as wroteErr, got %v", sc.wroteErr) + } + }) +} + +// TestHandleHTTP_RealSocketWriteDeadline is the end-to-end, OS-level proof of +// the zombie-job fix. It drives handleHTTP over a REAL TCP socket with a client +// that reads the response headers and then stops reading — the shape of a +// killed / half-open client whose receive window closes without a FIN/RST. The +// upstream streams without end, so the proxy's kernel send buffer to the client +// fills and its next write blocks. Before the fix that write blocks ~forever +// (no terminal event; the zombie job), so r.Context() never fires and the +// handler never returns. With the fix, statusCapture arms a real +// SetWriteDeadline that the Go runtime's netpoller enforces on every platform +// (IOCP on Windows, epoll/kqueue elsewhere) regardless of the peer's TCP state, +// so the stuck write fails and the workload terminates as failed. This is the +// piece the in-process tests stub out — here the deadline is genuinely enforced +// by the OS/runtime. +func TestHandleHTTP_RealSocketWriteDeadline(t *testing.T) { + // Shorten the idle write deadline so a stuck write trips quickly; restore + // the production default for any test that runs after this one. + orig := idleClientWriteTimeout + idleClientWriteTimeout = 300 * time.Millisecond + defer func() { idleClientWriteTimeout = orig }() + + // Upstream streams 64 KiB chunks endlessly. Once the proxy stops reading + // from it (because the proxy is itself blocked writing to the stalled + // client), the upstream's own writes block too — no busy loop — and it + // unwinds when the proxy tears the connection down (write error or context + // cancel). + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + flusher, _ := w.(http.Flusher) + chunk := bytes.Repeat([]byte("x"), 64*1024) + for { + if _, err := w.Write(chunk); err != nil { + return + } + if flusher != nil { + flusher.Flush() + } + select { + case <-r.Context().Done(): + return + default: + } + } + })) + defer upstream.Close() + + rec := &prRec{} + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama")) + p := NewProxy(NewCodec(rec), disc, 1235) + + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)} + go func() { _ = srv.Serve(ln) }() + defer srv.Close() + + conn, err := net.Dial("tcp", ln.Addr().String()) + if err != nil { + t.Fatalf("dial proxy: %v", err) + } + defer conn.Close() + + body := `{"model":"llama"}` + reqText := "POST /v1/chat/completions HTTP/1.1\r\n" + + "Host: localhost\r\n" + + "Content-Type: application/json\r\n" + + fmt.Sprintf("Content-Length: %d\r\n", len(body)) + + "\r\n" + body + if _, err := conn.Write([]byte(reqText)); err != nil { + t.Fatalf("write request: %v", err) + } + + // Read the status line only — enough to confirm the response committed and + // started streaming — then STOP reading so the proxy's send buffer backs + // up. A read deadline guards against a hang if the proxy never responds. + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + statusLine, err := bufio.NewReader(conn).ReadString('\n') + if err != nil { + t.Fatalf("read status line: %v", err) + } + if !strings.Contains(statusLine, "200") { + t.Fatalf("unexpected status line: %q", statusLine) + } + + // The stuck write must trip the deadline and terminate the workload as + // failed within a few multiples of the deadline — never completed. + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) && !rec.has("workload:errored") { + time.Sleep(10 * time.Millisecond) + } + if !rec.has("workload:errored") { + t.Fatal("workload never terminated after the client stopped reading (zombie: write deadline did not trip / no terminal emitted)") + } + if rec.has("workload:completed") { + t.Fatal("workload:completed emitted for a client that stopped reading (should be failed)") + } +} + +// TestHandleHTTP_RealSocketFlushDeadline is the flush-path counterpart to the +// write-deadline test. A streaming response is flushed after every chunk, so a +// small chunk buffers on a successful Write (no network I/O) and the actual +// network write happens in a separate Flush. If only Write is deadline-aware, a +// stalled client makes that Flush block unbounded and the handler never returns +// — a zombie the 64 KiB Write-blocking test does not catch. This drives real, +// paced small flushed chunks over a real socket and asserts the flush deadline +// terminates the workload as failed. +func TestHandleHTTP_RealSocketFlushDeadline(t *testing.T) { + orig := idleClientWriteTimeout + idleClientWriteTimeout = 300 * time.Millisecond + defer func() { idleClientWriteTimeout = orig }() + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/x-ndjson") + w.WriteHeader(http.StatusOK) + flusher, ok := w.(http.Flusher) + if !ok { + t.Errorf("upstream ResponseWriter is not a Flusher") + return + } + chunk := bytes.Repeat([]byte("x"), 1500) + for { + if _, err := w.Write(chunk); err != nil { + return + } + flusher.Flush() + // Pace so the reverse proxy's 32 KiB copy read returns one small + // chunk per iteration rather than coalescing many into a >2 KiB + // write (which would block inside Write, not Flush). + time.Sleep(2 * time.Millisecond) + select { + case <-r.Context().Done(): + return + default: + } + } + })) + defer upstream.Close() + + rec := &prRec{} + disc := NewDiscovery() + disc.AddManual(nodeForModel(t, "node-a", upstream.URL, "llama")) + p := NewProxy(NewCodec(rec), disc, 1235) + + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + srv := &http.Server{Handler: http.HandlerFunc(p.handleHTTP)} + go func() { _ = srv.Serve(ln) }() + defer srv.Close() + + conn, err := net.Dial("tcp", ln.Addr().String()) + if err != nil { + t.Fatalf("dial proxy: %v", err) + } + defer conn.Close() + + body := `{"model":"llama"}` + reqText := "POST /v1/chat/completions HTTP/1.1\r\n" + + "Host: localhost\r\n" + + "Content-Type: application/json\r\n" + + fmt.Sprintf("Content-Length: %d\r\n", len(body)) + + "\r\n" + body + if _, err := conn.Write([]byte(reqText)); err != nil { + t.Fatalf("write request: %v", err) + } + + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + statusLine, err := bufio.NewReader(conn).ReadString('\n') + if err != nil { + t.Fatalf("read status line: %v", err) + } + if !strings.Contains(statusLine, "200") { + t.Fatalf("unexpected status line: %q", statusLine) + } + + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) && !rec.has("workload:errored") { + time.Sleep(10 * time.Millisecond) + } + if !rec.has("workload:errored") { + t.Fatal("workload never terminated after the client stopped reading (flush path not deadline-aware)") + } + if rec.has("workload:completed") { + t.Fatal("workload:completed emitted for a stalled client (should be failed)") + } +} diff --git a/services/nvpair-engine-manager/adopt_test.go b/services/nvpair-engine-manager/adopt_test.go new file mode 100644 index 00000000..af15e741 --- /dev/null +++ b/services/nvpair-engine-manager/adopt_test.go @@ -0,0 +1,95 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "runtime" + "strconv" + "strings" + "testing" +) + +func TestAdoptModeStartsWithoutSpawning(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/models" { + http.NotFound(w, r) + return + } + // Identity probes send X-NVPAIR-Engine-Identity-Probe; still answer 200. + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "object": "list", + "data": []map[string]any{{"id": "m1", "status": map[string]string{"value": "unloaded"}}}, + }) + })) + defer srv.Close() + u, _ := url.Parse(srv.URL) + port, _ := strconv.Atoi(u.Port()) + + m := adoptManifest(port) + ex := newTestExecutor(t, m) + if err := ex.Start(context.Background(), "llamacpp"); err != nil { + t.Fatalf("adopt start: %v", err) + } + st, err := ex.Status("llamacpp") + if err != nil { + t.Fatal(err) + } + if !st.Running || !st.Healthy || st.Port != port { + t.Fatalf("status = %+v", st) + } + state, _ := ex.state("llamacpp") + state.mu.Lock() + proc := state.proc + state.mu.Unlock() + if proc != nil { + t.Fatal("adopt mode spawned a process") + } +} + +func TestAdoptModeDoesNotSpawnWhenDown(t *testing.T) { + m := adoptManifest(1) // nothing listens on :1 + ex := newTestExecutor(t, m) + err := ex.Start(context.Background(), "llamacpp") + if err == nil { + t.Fatal("expected start error when probe fails") + } + if !strings.Contains(err.Error(), "PAIR will not launch llama-server") { + t.Fatalf("start error = %v, want adopt-only refusal", err) + } + st, _ := ex.Status("llamacpp") + if st.Running { + t.Fatal("must not mark running") + } + state, _ := ex.state("llamacpp") + state.mu.Lock() + proc := state.proc + state.mu.Unlock() + if proc != nil { + t.Fatal("adopt mode spawned a process when the probe was down") + } +} + +func adoptManifest(port int) *Manifest { + key := runtime.GOOS + "/" + runtime.GOARCH + return &Manifest{ + Engine: "llamacpp", + DisplayName: "llama.cpp", + ManifestVersion: 1, + Platforms: map[string]Platform{ + key: { + Runtime: Runtime{ + Mode: "adopt", + Port: port, + Ready: &Probe{HTTP: "http://127.0.0.1:{port}/v1/models", Status: 200}, + }, + }, + }, + } +} diff --git a/services/nvpair-engine-manager/lifecycle.go b/services/nvpair-engine-manager/lifecycle.go index c10244b0..9e7d5517 100644 --- a/services/nvpair-engine-manager/lifecycle.go +++ b/services/nvpair-engine-manager/lifecycle.go @@ -75,8 +75,9 @@ func effectiveBind(manifestBind, override string) string { // Start launches the engine and waits for its readiness probe, then // begins the health loop. No-op if already running. It branches on the -// runtime mode: "process" (spawn + own the process) or "command" (run -// bring-up commands; liveness comes from the probe). +// runtime mode: "process" (spawn + own the process), "command" (run +// bring-up commands; liveness comes from the probe), or "adopt" +// (identify an already-running listener; never spawn). func (e *Executor) Start(ctx context.Context, engine string) error { return e.StartWith(ctx, engine, startOpts{}) } @@ -159,6 +160,9 @@ func (e *Executor) doStart(ctx context.Context, st *engineState, engine string, } return nil } + if rt.modeOrDefault() == "adopt" { + return fmt.Errorf("cannot start engine %q: nothing is serving on port %d (PAIR will not launch llama-server)", engine, port) + } if presence.Occupied && rt.modeOrDefault() == "process" { return fmt.Errorf("cannot start engine %q on port %d: the port is occupied by a service that did not identify as %s", engine, port, st.manifest.DisplayName) } diff --git a/services/nvpair-engine-manager/manifests/llamacpp.json b/services/nvpair-engine-manager/manifests/llamacpp.json new file mode 100644 index 00000000..304f3345 --- /dev/null +++ b/services/nvpair-engine-manager/manifests/llamacpp.json @@ -0,0 +1,45 @@ +{ + "engine": "llamacpp", + "display_name": "llama.cpp", + "manifest_version": 1, + "runtime": { + "mode": "adopt", + "bind": "127.0.0.1", + "port": 8082, + "ready": { "http": "http://127.0.0.1:{port}/v1/models", "status": 200, "timeout_s": 5 }, + "health": { "http": "http://127.0.0.1:{port}/v1/models", "status": 200, "interval_s": 5 } + }, + "platforms": { + "windows/amd64": { + "detect": [ + "%LOCALAPPDATA%\\Microsoft\\WinGet\\Packages\\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\\llama-server.exe", + "C:\\Users\\P-DLE\\Desktop\\AI Playground\\vendor\\llama.cpp-official\\bin\\llama-server.exe" + ] + }, + "windows/arm64": { + "detect": [ + "%LOCALAPPDATA%\\Microsoft\\WinGet\\Packages\\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\\llama-server.exe" + ] + }, + "darwin/arm64": { "detect": ["/opt/homebrew/bin/llama-server", "/usr/local/bin/llama-server"] }, + "darwin/amd64": { "detect": ["/usr/local/bin/llama-server"] }, + "linux/amd64": { "detect": ["/usr/local/bin/llama-server", "/usr/bin/llama-server"] }, + "linux/arm64": { "detect": ["/usr/local/bin/llama-server", "/usr/bin/llama-server"] } + }, + "actions": { + "list_models": { + "description": "List every model id advertised by llama-server.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "loaded_models": { + "description": "List model ids currently loaded in memory.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { + "array": "data", + "field": "id", + "match": { "field": "status.value", "in": ["loaded"] } + } + } + } +} diff --git a/services/nvpair-engine-manager/models.go b/services/nvpair-engine-manager/models.go index de89064e..68975a4e 100644 --- a/services/nvpair-engine-manager/models.go +++ b/services/nvpair-engine-manager/models.go @@ -8,6 +8,7 @@ import ( "context" "encoding/json" "log/slog" + "strings" "sync" "time" ) @@ -216,14 +217,37 @@ func extractStringsResult(raw json.RawMessage, spec *ActionResult) ([]string, bo return out, true } +// lookupField walks a dotted Field path (e.g. "status.value") through nested +// JSON objects. A missing segment or a non-object intermediate returns ok=false. +func lookupField(el map[string]json.RawMessage, field string) (json.RawMessage, bool) { + obj := el + parts := strings.Split(field, ".") + for i, part := range parts { + fv, ok := obj[part] + if !ok { + return nil, false + } + if i == len(parts)-1 { + return fv, true + } + next := map[string]json.RawMessage{} + if err := json.Unmarshal(fv, &next); err != nil { + return nil, false + } + obj = next + } + return nil, false +} + // matchRow reports whether an element passes an ActionResult row filter. -// With Match.In set, Match.Field must decode as a JSON string equal to one of -// In. With Match.Nonempty set, Match.Field must decode as a JSON array with +// Match.Field may be a dotted path into nested objects (e.g. "status.value"). +// With Match.In set, the resolved field must decode as a JSON string equal to +// one of In. With Match.Nonempty set, it must decode as a JSON array with // length > 0 (LM Studio /api/v1/models loaded_instances). A missing or // wrong-typed field fails the match, so a row we cannot classify is excluded // rather than counted as loaded. func matchRow(el map[string]json.RawMessage, m *ResultMatch) bool { - fv, ok := el[m.Field] + fv, ok := lookupField(el, m.Field) if !ok { return false } diff --git a/services/nvpair-engine-manager/models_test.go b/services/nvpair-engine-manager/models_test.go index a9794f80..c079ec7b 100644 --- a/services/nvpair-engine-manager/models_test.go +++ b/services/nvpair-engine-manager/models_test.go @@ -86,6 +86,18 @@ func TestExtractStrings(t *testing.T) { spec: &ActionResult{Array: "models", Field: "key", Match: &ResultMatch{Field: "loaded_instances", Nonempty: true}}, want: []string{"a"}, }, + { + name: "dotted status.value keeps only loaded llama.cpp rows", + raw: `{"data":[{"id":"a","status":{"value":"loaded"}},{"id":"b","status":{"value":"unloaded"}},{"id":"c"},{"id":"d","status":{"value":"loaded"}}]}`, + spec: &ActionResult{Array: "data", Field: "id", Match: &ResultMatch{Field: "status.value", In: []string{"loaded"}}}, + want: []string{"a", "d"}, + }, + { + name: "dotted match: missing status is unloaded", + raw: `{"data":[{"id":"a","status":{"value":"loaded"}},{"id":"b"}]}`, + spec: &ActionResult{Array: "data", Field: "id", Match: &ResultMatch{Field: "status.value", In: []string{"loaded"}}}, + want: []string{"a"}, + }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { diff --git a/services/nvpair-engine-manager/registry.go b/services/nvpair-engine-manager/registry.go index b44ec6e0..52229d05 100644 --- a/services/nvpair-engine-manager/registry.go +++ b/services/nvpair-engine-manager/registry.go @@ -112,6 +112,9 @@ type Fetch struct { // - "command": the engine is a daemon brought up/down by commands // (e.g. LM Studio's `lms`); liveness = the readiness/health probe, // and Stop.Cmd brings it down. +// - "adopt": the engine is an already-running service this process +// never launches; Start identifies it via the ready probe and +// fails if nothing is serving. Bin and start are forbidden. type Runtime struct { Mode string `json:"mode,omitempty"` Bin string `json:"bin,omitempty"` @@ -577,8 +580,18 @@ func (p *Platform) validate(key string) error { if len(p.Runtime.Start) == 0 { return fmt.Errorf("platform %q: runtime.start is required in command mode", key) } + case "adopt": + if strings.TrimSpace(p.Runtime.Bin) != "" { + return fmt.Errorf("platform %q: runtime.bin is forbidden in adopt mode", key) + } + if len(p.Runtime.Start) != 0 { + return fmt.Errorf("platform %q: runtime.start is forbidden in adopt mode", key) + } + if p.Runtime.Ready == nil { + return fmt.Errorf("platform %q: runtime.ready is required in adopt mode", key) + } default: - return fmt.Errorf("platform %q: runtime.mode %q invalid (want \"process\" or \"command\")", key, p.Runtime.Mode) + return fmt.Errorf("platform %q: runtime.mode %q invalid (want \"process\", \"command\", or \"adopt\")", key, p.Runtime.Mode) } if p.Install != nil { if len(p.Install.Script) > 0 && (p.Install.Fetch != nil || len(p.Install.Run) > 0) { diff --git a/services/nvpair-engine-manager/registry_test.go b/services/nvpair-engine-manager/registry_test.go index 2ae2c5c6..9a03c2bd 100644 --- a/services/nvpair-engine-manager/registry_test.go +++ b/services/nvpair-engine-manager/registry_test.go @@ -69,6 +69,18 @@ func TestValidateAcceptsCommandModeAndCmdAction(t *testing.T) { } } +func TestValidateAcceptsAdoptMode(t *testing.T) { + m := validManifest() + p := m.Platforms["linux/amd64"] + p.Runtime.Mode = "adopt" + p.Runtime.Bin = "" + p.Runtime.Start = nil + m.Platforms["linux/amd64"] = p + if err := m.Validate(); err != nil { + t.Fatalf("adopt-mode manifest with ready and empty bin/start rejected: %v", err) + } +} + func TestValidateAcceptsUnpinnedFetch(t *testing.T) { m := validManifest() p := m.Platforms["linux/amd64"] @@ -139,6 +151,26 @@ func TestValidateRejects(t *testing.T) { {"action missing method", func(m *Manifest) { m.Actions = map[string]Action{"x": {HTTP: &ActionHTTP{Path: "/p"}}} }, "http.method and http.path"}, + {"adopt with bin", func(m *Manifest) { + p := m.Platforms["linux/amd64"] + p.Runtime.Mode = "adopt" + m.Platforms["linux/amd64"] = p + }, "runtime.bin is forbidden"}, + {"adopt without ready", func(m *Manifest) { + p := m.Platforms["linux/amd64"] + p.Runtime.Mode = "adopt" + p.Runtime.Bin = "" + p.Runtime.Start = nil + p.Runtime.Ready = nil + m.Platforms["linux/amd64"] = p + }, "runtime.ready is required"}, + {"adopt with start", func(m *Manifest) { + p := m.Platforms["linux/amd64"] + p.Runtime.Mode = "adopt" + p.Runtime.Bin = "" + p.Runtime.Start = [][]string{{"lms", "server", "start"}} + m.Platforms["linux/amd64"] = p + }, "runtime.start is forbidden"}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -489,6 +521,85 @@ func TestLMStudioManifestUsesNativeSystemInventory(t *testing.T) { } } +// TestLlamaCppManifestIsAdoptOnly pins the bundled llama.cpp engine as +// adopt-only: LoadFS merges the shared runtime onto every platform, so +// inspecting Platforms without that merge would see empty Runtime and +// modeOrDefault() == "process". PAIR must never spawn llama-server. +func TestLlamaCppManifestIsAdoptOnly(t *testing.T) { + reg := NewRegistry() + if err := reg.LoadFS(bundledManifests, "manifests"); err != nil { + t.Fatal(err) + } + m, ok := reg.Get("llamacpp") + if !ok { + t.Fatal("llamacpp manifest not loaded") + } + if m.Engine != "llamacpp" || m.DisplayName != "llama.cpp" { + t.Fatalf("identity = %s %s", m.Engine, m.DisplayName) + } + for _, name := range []string{"pull_model", "load_model", "unload_model", "delete_model", "install", "uninstall"} { + if _, ok := m.Actions[name]; ok { + t.Errorf("%s must not exist", name) + } + } + wantPlatforms := []string{ + "windows/amd64", "windows/arm64", + "darwin/arm64", "darwin/amd64", + "linux/amd64", "linux/arm64", + } + for _, key := range wantPlatforms { + p, ok := m.Platforms[key] + if !ok { + t.Errorf("missing platform %s", key) + continue + } + if p.Runtime.modeOrDefault() != "adopt" { + t.Errorf("%s mode = %q, want adopt", key, p.Runtime.Mode) + } + if p.Runtime.Port != 8082 { + t.Errorf("%s port = %d, want 8082", key, p.Runtime.Port) + } + if p.Runtime.Bind != "127.0.0.1" { + t.Errorf("%s bind = %q, want 127.0.0.1", key, p.Runtime.Bind) + } + if p.Runtime.Ready == nil || !strings.Contains(p.Runtime.Ready.HTTP, "/v1/models") { + t.Errorf("%s ready probe must be /v1/models", key) + } + if p.Runtime.Health == nil || !strings.Contains(p.Runtime.Health.HTTP, "/v1/models") { + t.Errorf("%s health probe must be /v1/models", key) + } + if strings.TrimSpace(p.Runtime.Bin) != "" { + t.Errorf("%s bin is forbidden in adopt mode, got %q", key, p.Runtime.Bin) + } + if len(p.Runtime.Start) != 0 { + t.Errorf("%s start is forbidden in adopt mode, got %v", key, p.Runtime.Start) + } + if p.Install != nil { + t.Errorf("%s install must not exist", key) + } + if p.Uninstall != nil { + t.Errorf("%s uninstall must not exist", key) + } + } + list := m.Actions["list_models"] + if list.HTTP == nil || list.HTTP.Method != "GET" || list.HTTP.Path != "/v1/models" { + t.Errorf("list_models HTTP = %+v, want GET /v1/models", list.HTTP) + } + if list.Result == nil || list.Result.Array != "data" || list.Result.Field != "id" { + t.Errorf("list_models result = %+v, want data[].id", list.Result) + } + loaded := m.Actions["loaded_models"] + if loaded.HTTP == nil || loaded.HTTP.Method != "GET" || loaded.HTTP.Path != "/v1/models" { + t.Errorf("loaded_models HTTP = %+v, want GET /v1/models", loaded.HTTP) + } + if loaded.Result == nil || loaded.Result.Match == nil || loaded.Result.Match.Field != "status.value" { + t.Fatal("loaded_models must match status.value") + } + if !slices.Equal(loaded.Result.Match.In, []string{"loaded"}) { + t.Errorf("loaded_models match.in = %v, want [loaded]", loaded.Result.Match.In) + } +} + // TestLMStudioInstallBootstrapSafety verifies that bootstrap fetch failures are // visible and Windows executes a downloaded .ps1 // file rather than pipe remote content through Invoke-Expression. diff --git a/services/nvpair-engine-manager/setport.go b/services/nvpair-engine-manager/setport.go index 5a9554c2..259a96c9 100644 --- a/services/nvpair-engine-manager/setport.go +++ b/services/nvpair-engine-manager/setport.go @@ -23,6 +23,11 @@ func canMoveAdoptedEngine(rt Runtime) bool { // the port survives a restart with no separate override store. Held under the // engine's op lock so it can't interleave with another lifecycle op. // +// Adopt-mode engines only retarget the loopback probe: the persisted port +// changes and a later start/status identifies whatever is already listening +// there. PAIR never stops the foreign listener or binds a new one. A probe +// that finds nothing on the new port is not a SetPort failure. +// // A running, adopted process-mode engine is refused. An identified command-mode // engine may be moved only when its manifest provides an official stop command. func (e *Executor) SetPort(ctx context.Context, engine string, port int) (EngineStatus, error) { @@ -45,6 +50,24 @@ func (e *Executor) SetPort(ctx context.Context, engine string, port int) (Engine oldPort := st.port st.mu.Unlock() + if st.plat.Runtime.modeOrDefault() == "adopt" { + if err := e.persistPort(engine, port); err != nil { + return EngineStatus{}, err + } + st.mu.Lock() + st.port = port + if st.plat != nil { + st.plat.Runtime.Port = port + } + st.mu.Unlock() + if wasRunning { + _ = e.doStart(ctx, st, engine, startOpts{}) + } else { + e.emitState(engine) + } + return e.snapshot(engine, st), nil + } + // Adopted process-mode engines and command-mode engines without an official // stop command remain externally managed. Refuse rather than killing an // unknown process or spawning a duplicate listener on the new port. diff --git a/services/nvpair-engine-manager/setport_test.go b/services/nvpair-engine-manager/setport_test.go index ea95287c..4364a1df 100644 --- a/services/nvpair-engine-manager/setport_test.go +++ b/services/nvpair-engine-manager/setport_test.go @@ -6,6 +6,9 @@ package main import ( "context" "encoding/json" + "net/http" + "net/http/httptest" + "net/url" "os" "path/filepath" "strconv" @@ -246,6 +249,41 @@ func TestSetPortRejectsCommandEngineWithoutStopCommand(t *testing.T) { } } +func TestSetPortAdoptModeDoesNotStopListener(t *testing.T) { + hits := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits++ + w.WriteHeader(200) + _, _ = w.Write([]byte(`{"object":"list","data":[]}`)) + })) + defer srv.Close() + u, _ := url.Parse(srv.URL) + oldPort, _ := strconv.Atoi(u.Port()) + + ex := newTestExecutor(t, adoptManifest(oldPort)) + ex.overrideDir = t.TempDir() + if err := ex.Start(context.Background(), "llamacpp"); err != nil { + t.Fatal(err) + } + before := hits + _, err := ex.SetPort(context.Background(), "llamacpp", oldPort+1) + if err != nil { + t.Fatalf("set-port adopt: %v", err) + } + resp, err := http.Get(srv.URL + "/v1/models") + if err != nil { + t.Fatalf("foreign listener died: %v", err) + } + resp.Body.Close() + st, _ := ex.Status("llamacpp") + if st.Port != oldPort+1 { + t.Fatalf("port = %d, want %d", st.Port, oldPort+1) + } + if hits < before { + t.Fatal("listener should still be reachable on the old port") + } +} + func TestSetPortRestartsOldPortWhenPersistenceFails(t *testing.T) { ex, stopped, started := adoptedCommandEngineFixture(t, "lmstudio", 1234) blocked := filepath.Join(t.TempDir(), "not-a-directory") diff --git a/services/nvpair-engine-manager/status.go b/services/nvpair-engine-manager/status.go index 75419cb7..d48cae36 100644 --- a/services/nvpair-engine-manager/status.go +++ b/services/nvpair-engine-manager/status.go @@ -153,8 +153,11 @@ func (e *Executor) reconcilePresence(ctx context.Context, engine string, st *eng // A command-mode engine needs its control CLI. A compatible HTTP endpoint // alone (for example another OpenAI server on LM Studio's port) is not an - // installation and must not suppress the installer. - if !pathInstalled && st.plat.Runtime.modeOrDefault() != "process" { + // installation and must not suppress the installer. Adopt mode, like + // process mode, may identify a healthy ready probe with no detect-path + // binary (service-only adoption). + mode := st.plat.Runtime.modeOrDefault() + if !pathInstalled && mode != "process" && mode != "adopt" { return presenceResult{} } diff --git a/services/nvpair-job-scheduler/schedule.go b/services/nvpair-job-scheduler/schedule.go index 1afe2fe4..1cf9c47f 100644 --- a/services/nvpair-job-scheduler/schedule.go +++ b/services/nvpair-job-scheduler/schedule.go @@ -12,9 +12,9 @@ import ( "nvpair-shared/schedulerwire" ) -// schedulerEngines is the fixed set of engine-specific output contracts. Both +// schedulerEngines is the fixed set of engine-specific output contracts. All // receive the same node-wide ranking because their work shares node resources. -var schedulerEngines = []string{"ollama", "lmstudio"} +var schedulerEngines = []string{"ollama", "lmstudio", "llamacpp"} // NodeRank is retained as the scheduler's public status type while the wire // definition is shared with the broker and proxies. diff --git a/services/nvpair-job-scheduler/schedule_test.go b/services/nvpair-job-scheduler/schedule_test.go index 4bbfbabd..7c693c1b 100644 --- a/services/nvpair-job-scheduler/schedule_test.go +++ b/services/nvpair-job-scheduler/schedule_test.go @@ -675,3 +675,15 @@ func TestNewManager_Floor(t *testing.T) { t.Fatalf("interval = %v, want floor %v", m.interval, intervalFloor) } } + +func TestSchedulerEnginesIncludesLlamaCpp(t *testing.T) { + found := false + for _, e := range schedulerEngines { + if e == "llamacpp" { + found = true + } + } + if !found { + t.Fatal("schedulerEngines missing llamacpp") + } +} diff --git a/services/nvpair-manual-nodes/README.md b/services/nvpair-manual-nodes/README.md index 5fb648a6..9f8bef5a 100644 --- a/services/nvpair-manual-nodes/README.md +++ b/services/nvpair-manual-nodes/README.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0 # nvpair-manual-nodes -A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable. Accepts node addresses via JSON-RPC, probes each for Ollama, LM Studio, and node-info, and emits status events. +A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable. Accepts node addresses via JSON-RPC, probes each for Ollama, LM Studio, llama.cpp, and node-info, and emits status events. ## Communication @@ -50,6 +50,9 @@ Emitted when a manually added node has been probed and its initial status determ "lmstudio_up":true, "lmstudio_port":1234, "lmstudio_models":["qwen2.5-7b-instruct"], + "llamacpp_up":true, + "llamacpp_port":8082, + "llamacpp_models":["loaded-one"], "node_info_up":true, "node_info_port":14318, "gpus":[{"name":"NVIDIA GeForce RTX 3080","utilization_percent":37}], @@ -60,7 +63,7 @@ Emitted when a manually added node has been probed and its initial status determ } ``` -Each node is probed for both inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`) and LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` the same way it bridges Ollama into `ollama-proxy`. A node can run either engine, both, or neither. +Each node is probed for the three inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`), LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list), and llama.cpp on its default `:8082` (`GET /v1/models`, same liveness-plus-list shape). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` and `llamacpp_up` / `llamacpp_port` / `llamacpp_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` / `llamacpp-proxy` the same way it bridges Ollama into `ollama-proxy`. `llamacpp_models` is the loaded subset only (`status.value == "loaded"`; a missing status is not loaded); a 200 from `/v1/models` still sets `llamacpp_up` when that set is empty. A node can run any combination of engines, or none. ### `node/updated` @@ -136,11 +139,12 @@ Each manual node is probed every 10 seconds, with a 3-second timeout per leg, fo - **Ollama** on port 11434: health check (`GET /`) and model list (`GET /api/tags`) - **LM Studio** on port 1234: `GET /v1/models`, which doubles as the liveness check and the model list +- **llama.cpp** on port 8082: `GET /v1/models`, which doubles as the liveness check and the loaded-model list (`status.value == "loaded"`) - **Node Info** on port 14318, or `tls_port` over HTTPS: hardware inventory and identity (`GET /v1/node-info`) A node can have any combination of these, or none if the target is unreachable. Status changes trigger `node/updated` events. Because change detection compares CPU, memory, and GPU values, a node running node-info emits a `node/updated` on most probe cycles as utilization moves. -The three engine ports are compiled in: only the node-info leg's port can be moved, via `tls_port`. A remote engine on a non-default port is not discovered. +The engine ports are compiled in: only the node-info leg's port can be moved, via `tls_port`. A remote engine on a non-default port is not discovered. ## Shutdown diff --git a/services/nvpair-manual-nodes/manager.go b/services/nvpair-manual-nodes/manager.go index 55a4040a..510e2155 100644 --- a/services/nvpair-manual-nodes/manager.go +++ b/services/nvpair-manual-nodes/manager.go @@ -112,9 +112,17 @@ type ManualNodeStatus struct { // LM Studio is probed on its default OpenAI-API port the same way Ollama // is on 11434, so a manually-added node running LM Studio can be bridged // into lmstudio-proxy by a supervising broker. - LMStudioUp bool `json:"lmstudio_up"` - LMStudioPort int `json:"lmstudio_port"` - LMStudioModels []string `json:"lmstudio_models,omitempty"` + LMStudioUp bool `json:"lmstudio_up"` + LMStudioPort int `json:"lmstudio_port"` + LMStudioModels []string `json:"lmstudio_models,omitempty"` + // llama.cpp is probed on its default OpenAI-API port (8082) the same way + // LM Studio is on 1234. llamacpp_models is the loaded subset only + // (status.value == "loaded"); a 200 from GET /v1/models still counts as + // up when that set is empty so a supervising broker can bridge the node + // into llamacpp-proxy. + LlamaCppUp bool `json:"llamacpp_up"` + LlamaCppPort int `json:"llamacpp_port"` + LlamaCppModels []string `json:"llamacpp_models,omitempty"` NodeInfoUp bool `json:"node_info_up"` NodeInfoPort int `json:"node_info_port"` TLSEnabled bool `json:"tls_enabled,omitempty"` @@ -138,12 +146,11 @@ type trackedNode struct { entry ManualEntry status ManualNodeStatus - // consecutiveFails counts back-to-back probes where neither - // service answered (OllamaUp && NodeInfoUp both false). Reset - // to 0 on any probe where at least one service responded. - // Used to gate probe-failed errors:report emits at - // probeFailThreshold so a single transient failure doesn't - // generate UI noise. + // consecutiveFails counts back-to-back probes where no engine + // and no node-info answered (reachable is false). Reset to 0 + // on any probe where at least one service responded. Used to + // gate probe-failed errors:report emits at probeFailThreshold + // so a single transient failure doesn't generate UI noise. consecutiveFails int } @@ -253,6 +260,7 @@ func (m *Manager) probeNode(entry ManualEntry) { ollamaUp, ollamaModels := m.probeOllama(addr, 11434) lmStudioUp, lmStudioModels := m.probeLMStudio(addr, lmStudioPort) + llamaCppUp, llamaCppModels := m.probeLlamaCpp(addr, llamaCppPort) // Pick scheme + port + client based on the entry's TLS hint. // The operator decides which scheme this manual node uses; we @@ -290,6 +298,9 @@ func (m *Manager) probeNode(entry ManualEntry) { LMStudioUp: lmStudioUp, LMStudioPort: lmStudioPort, LMStudioModels: lmStudioModels, + LlamaCppUp: llamaCppUp, + LlamaCppPort: llamaCppPort, + LlamaCppModels: llamaCppModels, NodeInfoUp: nodeInfoUp, NodeInfoPort: nodeInfoPort, TLSEnabled: entry.TLSPort > 0, @@ -302,7 +313,7 @@ func (m *Manager) probeNode(entry ManualEntry) { HostUUID: info.HostUUID, } - reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.NodeInfoUp + reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.LlamaCppUp || newStatus.NodeInfoUp m.mu.Lock() tn, exists := m.nodes[id] @@ -331,10 +342,12 @@ func (m *Manager) probeNode(entry ManualEntry) { changed := prev.OllamaUp != newStatus.OllamaUp || prev.LMStudioUp != newStatus.LMStudioUp || + prev.LlamaCppUp != newStatus.LlamaCppUp || prev.NodeInfoUp != newStatus.NodeInfoUp || prev.HostUUID != newStatus.HostUUID || !sliceEqual(prev.OllamaModels, newStatus.OllamaModels) || !sliceEqual(prev.LMStudioModels, newStatus.LMStudioModels) || + !sliceEqual(prev.LlamaCppModels, newStatus.LlamaCppModels) || !gpusEqual(prev.GPUs, newStatus.GPUs) || !cpuEqual(prev.CPU, newStatus.CPU) || !memoryEqual(prev.Memory, newStatus.Memory) || @@ -446,6 +459,59 @@ func (m *Manager) probeLMStudio(addr string, port int) (bool, []string) { return true, models } +// llamaCppPort is llama.cpp's default OpenAI-API server port, probed the same +// way LM Studio is hardcoded to 1234. A manual node is remote, so (like the +// other engines) we assume the engine's default port rather than resolving it +// via the engine manager (which only governs the local engine). +const llamaCppPort = 8082 + +// probeLlamaCpp checks llama-server's OpenAI-compatible API on addr:port. A +// single GET /v1/models doubles as the liveness check and the model list. +// Only ids whose status.value is "loaded" are returned — a missing status is +// treated as not loaded — so the broker bridges a routing-eligible set into +// llamacpp-proxy. A 200 still reports the node up when that set is empty. +func (m *Manager) probeLlamaCpp(addr string, port int) (bool, []string) { + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/models" + start := time.Now() + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe llamacpp failed", + "addr", addr, "port", port, "duration_ms", time.Since(start).Milliseconds(), "err", err) + return false, nil + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + slog.Debug("manual probe llamacpp non-OK", + "addr", addr, "port", port, "status", resp.StatusCode, + "duration_ms", time.Since(start).Milliseconds()) + return false, nil + } + var result struct { + Data []struct { + ID string `json:"id"` + Status struct { + Value string `json:"value"` + } `json:"status"` + } `json:"data"` + } + if err := json.NewDecoder(resp.Body).Decode(&result); err != nil { + // Reachable, but the model list didn't parse — still report it up. + slog.Debug("manual probe llamacpp up (models parse failed)", + "addr", addr, "port", port, "err", err) + return true, nil + } + models := make([]string, 0, len(result.Data)) + for _, d := range result.Data { + if d.ID != "" && d.Status.Value == "loaded" { + models = append(models, d.ID) + } + } + slog.Debug("manual probe llamacpp up", + "addr", addr, "port", port, "models", len(models), + "duration_ms", time.Since(start).Milliseconds()) + return true, models +} + func (m *Manager) probeOllama(addr string, port int) (bool, []string) { url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/" start := time.Now() diff --git a/services/nvpair-manual-nodes/manager_test.go b/services/nvpair-manual-nodes/manager_test.go index e52d9d5d..77415ba8 100644 --- a/services/nvpair-manual-nodes/manager_test.go +++ b/services/nvpair-manual-nodes/manager_test.go @@ -154,6 +154,81 @@ func TestProbeLMStudioReportsModels(t *testing.T) { } } +// configureHealthyLlamaCpp registers a 200 GET /v1/models on addr:8082 with +// one loaded and one unloaded model, so probeLlamaCpp reports the node up +// with only the loaded id. +func configureHealthyLlamaCpp(rt *fakeRoundTripper, addr string) { + host := net.JoinHostPort(addr, "8082") + rt.set(http.MethodGet, host, "/v1/models", func(*http.Request) (*http.Response, error) { + return httpJSON(http.StatusOK, `{"object":"list","data":[{"id":"loaded-one","status":{"value":"loaded"}},{"id":"catalog-only","status":{"value":"unloaded"}}]}`) + }) +} + +// TestProbeLlamaCppReportsModels covers the llama.cpp probe: a reachable +// server reports up with only ids whose status.value is "loaded", a 200 with +// an empty loaded set is still up, and an absent one reports down. +func TestProbeLlamaCppReportsModels(t *testing.T) { + m, _, rt := newTestManager() + configureHealthyLlamaCpp(rt, "node.local") + + up, models := m.probeLlamaCpp("node.local", llamaCppPort) + if !up { + t.Fatal("expected llamacpp up") + } + if len(models) != 1 || models[0] != "loaded-one" { + t.Fatalf("models = %#v, want [loaded-one] (loaded only)", models) + } + + emptyHost := net.JoinHostPort("empty.local", "8082") + rt.set(http.MethodGet, emptyHost, "/v1/models", func(*http.Request) (*http.Response, error) { + return httpJSON(http.StatusOK, `{"object":"list","data":[{"id":"catalog-only","status":{"value":"unloaded"}},{"id":"no-status"}]}`) + }) + emptyUp, emptyModels := m.probeLlamaCpp("empty.local", llamaCppPort) + if !emptyUp { + t.Fatal("expected llamacpp up with empty loaded set") + } + if len(emptyModels) != 0 { + t.Fatalf("empty loaded set models = %#v, want empty", emptyModels) + } + + downUp, downModels := m.probeLlamaCpp("absent.local", llamaCppPort) + if downUp || downModels != nil { + t.Fatalf("expected absent llamacpp down, got up=%v models=%#v", downUp, downModels) + } +} + +func TestProbeNodeLlamaCppOnlyIsReachable(t *testing.T) { + m, rw, rt := newTestManager() + entry := ManualEntry{Name: "lab", Address: "node.local"} + m.nodes["lab"] = &trackedNode{entry: entry, status: ManualNodeStatus{ID: "lab", Address: "node.local"}} + configureHealthyLlamaCpp(rt, "node.local") + + m.probeNode(entry) + updated := decodeParams[ManualNodeStatus](t, readCaptureUntil(t, rw, methodIs("node/updated"))) + if !updated.LlamaCppUp { + t.Fatalf("expected llamacpp up: %+v", updated) + } + if updated.LlamaCppPort != llamaCppPort { + t.Fatalf("llamacpp_port = %d, want %d", updated.LlamaCppPort, llamaCppPort) + } + if len(updated.LlamaCppModels) != 1 || updated.LlamaCppModels[0] != "loaded-one" { + t.Fatalf("llamacpp_models = %#v", updated.LlamaCppModels) + } + if m.nodes["lab"].consecutiveFails != 0 { + t.Fatalf("llama.cpp-only node counted as unreachable: fails=%d", m.nodes["lab"].consecutiveFails) + } + + host := net.JoinHostPort("node.local", "8082") + rt.set(http.MethodGet, host, "/v1/models", func(*http.Request) (*http.Response, error) { + return httpJSON(http.StatusOK, `{"object":"list","data":[{"id":"loaded-two","status":{"value":"loaded"}}]}`) + }) + m.probeNode(entry) + second := decodeParams[ManualNodeStatus](t, readCaptureUntil(t, rw, methodIs("node/updated"))) + if len(second.LlamaCppModels) != 1 || second.LlamaCppModels[0] != "loaded-two" { + t.Fatalf("second llamacpp_models = %#v", second.LlamaCppModels) + } +} + func requestMessage(id int, method string, params any) *Message { idData, _ := json.Marshal(id) idRaw := json.RawMessage(idData) diff --git a/services/nvpair-tui/ui/health.go b/services/nvpair-tui/ui/health.go index c1863ea9..66631283 100644 --- a/services/nvpair-tui/ui/health.go +++ b/services/nvpair-tui/ui/health.go @@ -35,6 +35,7 @@ var healthWorkers = []string{ "node-info", "proxy", "lmstudio-proxy", + "llamacpp-proxy", "workload-manager", "engine-manager", "manual-nodes", diff --git a/services/nvpair-tui/ui/proxies.go b/services/nvpair-tui/ui/proxies.go index 072c1a8b..aec7419b 100644 --- a/services/nvpair-tui/ui/proxies.go +++ b/services/nvpair-tui/ui/proxies.go @@ -24,12 +24,12 @@ type proxyNode struct { Port int `json:"port"` } -// proxyEngine is one of the two reverse proxies the broker fronts. Both +// proxyEngine is one of the reverse proxies the broker fronts. All three // speak the same routing/failover contract; only the JSON-RPC prefix and // label differ. type proxyEngine struct { - label string // "Ollama" / "LM Studio" - prefix string // "proxy" / "lmstudio-proxy" + label string // "Ollama" / "LM Studio" / "llama.cpp" + prefix string // "proxy" / "lmstudio-proxy" / "llamacpp-proxy" ready bool port int selected string @@ -92,6 +92,7 @@ func newProxiesView(client *rpc.Client) *proxiesView { engines: []*proxyEngine{ {label: "Ollama", prefix: "proxy", table: newTable(nil)}, {label: "LM Studio", prefix: "lmstudio-proxy", table: newTable(nil)}, + {label: "llama.cpp", prefix: "llamacpp-proxy", table: newTable(nil)}, }, } return v @@ -215,6 +216,8 @@ func (v *proxiesView) handleNotification(msg *rpc.Message) tea.Cmd { switch { case strings.HasPrefix(msg.Method, "lmstudio-proxy:"): idx = 1 + case strings.HasPrefix(msg.Method, "llamacpp-proxy:"): + idx = 2 case strings.HasPrefix(msg.Method, "proxy:"): idx = 0 default: diff --git a/services/nvpair-tui/ui/proxies_test.go b/services/nvpair-tui/ui/proxies_test.go new file mode 100644 index 00000000..61a58d8d --- /dev/null +++ b/services/nvpair-tui/ui/proxies_test.go @@ -0,0 +1,41 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package ui + +import ( + "encoding/json" + "testing" + + "nvpair-tui/rpc" +) + +func TestProxiesViewIncludesLlamaCpp(t *testing.T) { + v := newProxiesView(nil) + if len(v.engines) != 3 { + t.Fatalf("engines = %d, want 3", len(v.engines)) + } + got := v.engines[2] + if got.label != "llama.cpp" || got.prefix != "llamacpp-proxy" { + t.Fatalf("engine[2] = {%q, %q}, want {llama.cpp, llamacpp-proxy}", got.label, got.prefix) + } +} + +func TestHandleNotificationLlamaCppProxy(t *testing.T) { + v := newProxiesView(nil) + params, err := json.Marshal(map[string]int{"port": 8084}) + if err != nil { + t.Fatal(err) + } + v.handleNotification(&rpc.Message{ + Method: "llamacpp-proxy:ready", + Params: params, + }) + e := v.engines[2] + if !e.ready || e.port != 8084 { + t.Fatalf("llamacpp engine ready=%v port=%d, want ready :8084", e.ready, e.port) + } + if v.engines[0].ready { + t.Fatal("ollama proxy must not consume llamacpp-proxy notifications") + } +} diff --git a/services/nvpair-ui-broker/README.md b/services/nvpair-ui-broker/README.md index d2b0e174..6893d4c5 100644 --- a/services/nvpair-ui-broker/README.md +++ b/services/nvpair-ui-broker/README.md @@ -12,7 +12,7 @@ the broker from the same installation directory. The broker is the parent process and canonical backend entry point: it supervises the worker subprocesses on the UI's behalf, speaking JSON-RPC over stdio. -The broker supervises **eleven** worker subprocesses, so a client gets the whole +The broker supervises **twelve** worker subprocesses, so a client gets the whole backend behind one endpoint. Each is spawned at startup and relayed under its own namespace: @@ -22,6 +22,7 @@ namespace: | `nvpair-node-info` | Local GPU / CPU / memory inventory over HTTP at `/v1/node-info` | — (HTTP only) | | `ollama-proxy` | Ollama-compatible inference proxy and router | `proxy:*` | | `lmstudio-proxy` | The LM Studio counterpart, supervised identically | `lmstudio-proxy:*` | +| `llamacpp-proxy` | The llama.cpp counterpart, supervised identically (no managed facade) | `llamacpp-proxy:*` | | `nvpair-engine-manager` | Local engine and model control plane; also serves `GET /v1/models` to peers | `engine:*` | | `nvpair-cluster-manager` | Node identity, trusted-node store, PIN pairing | `cluster:*`, `nodes:*` | | `nvpair-workload-manager` | Cluster workload relay between this node and peers | `workloads:*` | @@ -70,6 +71,7 @@ Bidirectional newline-delimited JSON-RPC 2.0 — same conventions as every other | `--node-info-path ` | `./nvpair-node-info[.exe]` in the CWD | Explicit path to the `nvpair-node-info` binary the broker should spawn. When omitted and no default sibling exists, the broker runs without the local inventory server (non-fatal); when set to an invalid path, the broker exits with an error | | `--proxy-path ` | `./ollama-proxy[.exe]` in the CWD | Explicit path to the `ollama-proxy` binary the broker spawns for the local Ollama reverse proxy. Same optional semantics as `--node-info-path`: an absent default sibling means no local proxy (non-fatal); an invalid explicit path exits with an error | | `--lmstudio-proxy-path ` | `./lmstudio-proxy[.exe]` in the CWD | Explicit path to the `lmstudio-proxy` binary the broker spawns for the local LM Studio reverse proxy. Same optional semantics as `--proxy-path` | +| `--llamacpp-proxy-path ` | `./llamacpp-proxy[.exe]` in the CWD | Explicit path to the `llamacpp-proxy` binary the broker spawns for the local llama.cpp reverse proxy. Same optional semantics as `--proxy-path` | | `--workload-manager-path ` | `./nvpair-workload-manager[.exe]` in the CWD | Explicit path to the `nvpair-workload-manager` binary the broker spawns for the cluster workload relay. Same optional semantics as `--node-info-path`: an absent default sibling means no workload relay (non-fatal); an invalid explicit path exits with an error | | `--errors-path ` | `./nvpair-errors[.exe]` in the CWD | Explicit path to the `nvpair-errors` binary the broker spawns (with `--peer-sync`) for the service-error pipeline. Same optional semantics as `--node-info-path`: an absent default sibling means the error pipeline is disabled — producers' errors are dropped (non-fatal); an invalid explicit path exits with an error | | `--engine-manager-path ` | `./nvpair-engine-manager[.exe]` in the CWD | Explicit path to the `nvpair-engine-manager` binary the broker spawns for engine management. Same optional semantics as `--node-info-path` | @@ -87,13 +89,13 @@ Logs go to **stderr** (shared `applog` format, same as every other NVPAIR binary On startup — **before** emitting `app:ready` — the broker spawns the scanner and (when available) node-info, ollama-proxy, the workload-manager, and the cluster-manager as child processes over stdio. The proxy is spawned up front but doesn't gate `app:ready` — it announces its listen port asynchronously (see below). None of the auxiliary workers gate `app:ready`. -**`nvpair-node-scanner`** (the consolidated discovery daemon) is spawned first. It pushes `discovery:node-discovered`, `discovery:node-updated`, and `discovery:node-removed` notifications into the broker, which maintains them in an in-memory map keyed by `id`. Clients query that map via `discovery:get-nodes` and — once they've opted in via `discovery:subscribe` — receive a `discovery:nodes-changed` notification on every store mutation. The raw `discovery:node-*` notifications are never forwarded as-is. The scanner polls healthy node-info endpoints on a staggered two-second cadence, backs consecutive remote failures off to a 30-second cap, and emits compact `discovery:node-telemetry` observations containing maximum GPU utilization, validity, and age; these remain internal to broker scheduling. The broker registers this node's local service ports (`ni`/`er`/`wl`/`cl`/`em`, plus `ol`/`lm` from the engine poller) with the daemon over the same link, so the daemon can advertise them all in one `_nvpair-node` record. +**`nvpair-node-scanner`** (the consolidated discovery daemon) is spawned first. It pushes `discovery:node-discovered`, `discovery:node-updated`, and `discovery:node-removed` notifications into the broker, which maintains them in an in-memory map keyed by `id`. Clients query that map via `discovery:get-nodes` and — once they've opted in via `discovery:subscribe` — receive a `discovery:nodes-changed` notification on every store mutation. The raw `discovery:node-*` notifications are never forwarded as-is. The scanner polls healthy node-info endpoints on a staggered two-second cadence, backs consecutive remote failures off to a 30-second cap, and emits compact `discovery:node-telemetry` observations containing maximum GPU utilization, validity, and age; these remain internal to broker scheduling. The broker registers this node's local service ports (`ni`/`er`/`wl`/`cl`/`em`, plus `ol`/`lm`/`lc` from the engine poller) with the daemon over the same link, so the daemon can advertise them all in one `_nvpair-node` record. **`nvpair-node-info`** is spawned next. It's a server, not an event source: it stands up the local `/v1/node-info` HTTP endpoint (GPU/CPU/memory inventory). It does not advertise itself — the broker registers its `ni` port with the scanner daemon, which carries it in the node record, and a peer's daemon fetches `/v1/node-info` over plain HTTP to enrich the node. The broker doesn't read anything back from node-info's stdout (drained and discarded). Spawning it is **optional**: if the binary can't be resolved (and no `--node-info-path` override was given) the broker logs a warning and continues serving discovery without it. -**Engine advertising.** The broker runs an internal 5 s poll loop against local Ollama at its configured backend port and LM Studio (`GET /v1/models`) and reconciles this node's engine registration with the scanner daemon: +**Engine advertising.** The broker runs an internal 5 s poll loop against local Ollama at its configured backend port, LM Studio, and llama.cpp (`GET /v1/models`) and reconciles this node's engine registration with the scanner daemon: -- engine **up** → register `ol` / `lm` at the engine's real port, never the proxy's own, to prevent a self-forward loop; +- engine **up** → register `ol` / `lm` / `lc` at the **proxy** port (never the engine) and hand the engine's loopback port to that proxy via `node/set-local-backend`; equal proxy/engine ports are refused so the proxy cannot self-forward; - engine **down** → unregister it. The daemon folds those registrations into this host's single `_nvpair-node` record, so a peer discovers the engine through the shared channel. The model list is not part of that registration — it's served over HTTP by `nvpair-engine-manager` (the `em` service, `GET /v1/models`) and enriched onto each node by the peer's daemon. There is no separate advertiser subprocess and no manual-advertise RPC. @@ -385,6 +387,10 @@ The same check runs whenever the proxy announces a (re)bound port (its restored The LM Studio counterpart of the `proxy:*` surface runs the supervised `lmstudio-proxy` on compatibility port `:1234` and tracks the managed LM Studio backend on `:1235`. With managed port ownership enabled (the default), the broker identifies and moves an existing LM Studio server through engine-manager before allowing the proxy to claim `1234`; unknown owners are left untouched and force a warned proxy fallback. Disabling managed ownership preserves explicit custom backend and proxy ports. `lmstudio-proxy:get-status` reports the actual bound port; `lmstudio-proxy:subscribe` / `lmstudio-proxy:unsubscribe` opt into / out of its `lmstudio-proxy:` stream; and any other `lmstudio-proxy:` is relayed verbatim with the prefix stripped (`nodes/list`, `node/select`, `node/add-manual`, `node/remove-manual`, ...). `lmstudio-proxy:shutdown` is refused because the broker owns lifecycle ordering. Workload and error events feed the shared streams exactly as Ollama's do. +#### `llamacpp-proxy:get-status` / `llamacpp-proxy:subscribe` / `llamacpp-proxy:unsubscribe` / `llamacpp-proxy:` (generic relay) + +The llama.cpp counterpart of the `proxy:*` surface runs the supervised `llamacpp-proxy` on `:8084` (or a persisted port). There is no managed facade and the broker never binds llama-server's stock `:8082`. `llamacpp-proxy:get-status` reports the actual bound port; `llamacpp-proxy:subscribe` / `llamacpp-proxy:unsubscribe` opt into / out of its `llamacpp-proxy:` stream; and any other `llamacpp-proxy:` is relayed verbatim with the prefix stripped. `llamacpp-proxy:shutdown` is refused because the broker owns lifecycle ordering. Workload and error events feed the shared streams exactly as Ollama's and LM Studio's do. `schedule:priority` for engine `llamacpp` is forwarded as `node/set-priority` on this proxy. + #### `proxy:subscribe` Opts the peer into the `proxy:` stream (off by default). On a fresh subscription the broker immediately replays the proxy's last `ready` payload as a baseline `proxy:ready` (if the proxy has come up), so a subscriber learns the port without a separate `proxy:get-status`. @@ -477,7 +483,7 @@ Any `settings/*` request is forwarded to `nvpair-node-settings` and its response Relayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, tls_port?, mtls? }`) registers a user-added node and probes it; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. A `nvpair-manual-nodes` restart loses the in-memory entries because neither that worker nor the broker persists an authoritative copy, so clients must re-add manual nodes after a restart. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised. -**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), and one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`) — a node running both is bridged into both. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before. +**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `proxy:node/select` / `lmstudio-proxy:node/select` / `llamacpp-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`), and one whose `llamacpp_*` status is up into `llamacpp-proxy` (from `llamacpp_port`) — a node running more than one is bridged into each. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before. #### `cluster:` / `nodes:` (generic relay) diff --git a/services/nvpair-ui-broker/advertiser.go b/services/nvpair-ui-broker/advertiser.go index c3f2e7d1..80d3b8ad 100644 --- a/services/nvpair-ui-broker/advertiser.go +++ b/services/nvpair-ui-broker/advertiser.go @@ -14,14 +14,16 @@ import ( ) const ( - // defaultOllamaPort / defaultLMStudioPort are the engines' stock ports, - // used ONLY as a fallback when engine-manager can't report the real one. - // Never hardcode the advertise/health port: the product-default proxy takes - // Ollama's :11434, so a fixed :11434 would advertise the proxy - // as Ollama and make the proxy (and peers) self-forward into a loop. The - // real port is resolved per poll via localEnginePort. - defaultOllamaPort = 11434 - defaultLMStudioPort = 1234 + // defaultOllamaPort / defaultLMStudioPort / defaultLlamaCppPort are the + // engines' stock ports, used ONLY as a fallback when engine-manager can't + // report the real one. Never hardcode the advertise/health port: the + // product-default proxy takes Ollama's :11434, so a fixed :11434 would + // advertise the proxy as Ollama and make the proxy (and peers) self-forward + // into a loop. The real port is resolved per poll via localEnginePort. + defaultOllamaPort = 11434 + defaultLMStudioPort = 1234 + defaultLlamaCppPort = 8082 + defaultLlamaCppProxyPort = 8084 // engineManagerHTTPPort is the fixed LAN port the broker tells // nvpair-engine-manager to serve its HTTP surface (/v1/models) on, and the port @@ -181,6 +183,51 @@ func (b *Broker) reconcileAdvertiseLMStudio(client *http.Client) { } } +// runAutoAdvertiseLlamaCpp is the llama.cpp sibling of runAutoAdvertiseLMStudio: +// it polls the local llama-server and reconciles this node's lc service +// registration against it. Kept parallel to the other engines rather than +// folded into them: the three are a deliberate temporary set, to be unified +// when the proxies are. +func (b *Broker) runAutoAdvertiseLlamaCpp(ctx context.Context) { + client := &http.Client{Timeout: 2 * time.Second} + ticker := time.NewTicker(autoAdvertiseInterval) + defer ticker.Stop() + + b.reconcileAdvertiseLlamaCpp(client) + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + b.reconcileAdvertiseLlamaCpp(client) + } + } +} + +// reconcileAdvertiseLlamaCpp brings this node's lc registration into line with +// the local llama.cpp server, mirroring reconcileAdvertiseLMStudio: it +// advertises the promoted proxy port (never the engine) and hands the engine's +// loopback port to llamacpp-proxy via node/set-local-backend. There is no +// managed facade, so equal proxy/engine ports are a hard refuse rather than a +// cached-backend recovery. +func (b *Broker) reconcileAdvertiseLlamaCpp(client *http.Client) { + enginePort, probe := b.localEnginePort("llamacpp", defaultLlamaCppPort) + proxyPort := b.llamaCppProxyListenPort() + if proxyPort != 0 && enginePort == proxyPort { + enginePort = 0 + probe = false + } + up := probe && proxyPort != 0 && enginePort != proxyPort && checkLlamaCppHealth(client, enginePort) + if up { + b.registerService(noderec.RegisterParams{Service: noderec.ServiceLlamaCpp, Port: proxyPort}) + b.setProxyLocalBackend(b.getLlamaCppProxy(), "llamacpp", enginePort, true) + } else { + b.unregisterService(noderec.ServiceLlamaCpp) + b.setProxyLocalBackend(b.getLlamaCppProxy(), "llamacpp", enginePort, false) + } +} + // proxyLocalBackend is the node/set-local-backend payload: the loopback engine // the proxy's cluster mTLS ingress forwards to, and the proxy's own self // candidate on the local routing path. @@ -261,6 +308,18 @@ func (b *Broker) lmstudioProxyListenPort() int { return 0 } +// llamaCppProxyListenPort is the llama.cpp sibling of proxyListenPort. Equal +// proxy/engine ports are refused so the broker never advertises lc at the +// proxy's own listener or hands that listener back as the local backend. +func (b *Broker) llamaCppProxyListenPort() int { + if p := b.getLlamaCppProxy(); p != nil { + if ready, port := p.Status(); ready { + return port + } + } + return 0 +} + // checkOllamaHealth reports whether a local ollama server is answering on the // given port. A plain GET of the root that returns 200 is ollama's liveness // convention. The port is resolved per poll (see @@ -287,3 +346,17 @@ func checkLMStudioHealth(client *http.Client, port int) bool { resp.Body.Close() return resp.StatusCode == http.StatusOK } + +// checkLlamaCppHealth reports whether a local llama-server is answering on the +// given port. llama.cpp serves the OpenAI-compatible API, so a 200 from +// /v1/models is its liveness signal. The port is resolved per poll (see +// localEnginePort), not hardcoded, so the proxy is never mistaken for the +// engine. +func checkLlamaCppHealth(client *http.Client, port int) bool { + resp, err := client.Get(fmt.Sprintf("http://localhost:%d/v1/models", port)) + if err != nil { + return false + } + resp.Body.Close() + return resp.StatusCode == http.StatusOK +} diff --git a/services/nvpair-ui-broker/advertiser_test.go b/services/nvpair-ui-broker/advertiser_test.go index e6e6a2bf..15f0677f 100644 --- a/services/nvpair-ui-broker/advertiser_test.go +++ b/services/nvpair-ui-broker/advertiser_test.go @@ -23,6 +23,9 @@ func TestLocalEnginePortFallback(t *testing.T) { if got, ok := b.localEnginePort("lmstudio", defaultLMStudioPort); !ok || got != defaultLMStudioPort { t.Errorf("no engine-manager: localEnginePort = (%d, %v), want (%d, true)", got, ok, defaultLMStudioPort) } + if got, ok := b.localEnginePort("llamacpp", defaultLlamaCppPort); !ok || got != defaultLlamaCppPort { + t.Errorf("no engine-manager: localEnginePort = (%d, %v), want (%d, true)", got, ok, defaultLlamaCppPort) + } } func TestRunningEnginePort(t *testing.T) { diff --git a/services/nvpair-ui-broker/broker.go b/services/nvpair-ui-broker/broker.go index 0d189578..af8b4814 100644 --- a/services/nvpair-ui-broker/broker.go +++ b/services/nvpair-ui-broker/broker.go @@ -153,6 +153,7 @@ type Broker struct { nodeInfoPath string proxyPath string lmstudioProxyPath string + llamaCppProxyPath string workloadMgrPath string errorsPath string engineMgrPath string @@ -189,6 +190,8 @@ type Broker struct { lmstudioPortReady chan struct{} lmstudioPortReadyOnce sync.Once lmstudioReadyMu sync.Mutex + llamaCppProxyStartupPort atomic.Int32 + llamaCppProxyGeneration atomic.Uint64 store *discoveryStore telemetry *telemetryCache // relayDir is the discovery directory, fed by the promoted daemon's @@ -213,6 +216,7 @@ type Broker struct { nodeInfo *nodeInfoProcess proxy *proxyProcess lmstudioProxy *proxyProcess + llamaCppProxy *proxyProcess workloadMgr *workloadManagerProcess errorsProc *errorsProcess engineMgr *rpcWorker @@ -228,6 +232,7 @@ type Broker struct { nodeInfoSup *supervisor proxySup *supervisor lmstudioProxySup *supervisor + llamaCppProxySup *supervisor workloadMgrSup *supervisor errorsSup *supervisor engineMgrSup *supervisor @@ -244,14 +249,16 @@ type Broker struct { subMu sync.Mutex subscribed bool - // proxyMu guards proxySubscribed and lmstudioProxySubscribed. The - // proxy: / lmstudio-proxy: streams are opt-in like - // discovery's: the forward*Notification hooks (on each proxy's reader - // goroutine) read the flags while the *:subscribe / *:unsubscribe - // handlers (on the read-loop goroutine) flip them. + // proxyMu guards proxySubscribed, lmstudioProxySubscribed, and + // llamaCppProxySubscribed. The proxy: / lmstudio-proxy: / + // llamacpp-proxy: streams are opt-in like discovery's: the + // forward*Notification hooks (on each proxy's reader goroutine) read the + // flags while the *:subscribe / *:unsubscribe handlers (on the read-loop + // goroutine) flip them. proxyMu sync.Mutex proxySubscribed bool lmstudioProxySubscribed bool + llamaCppProxySubscribed bool // workloadsMu guards workloadsSubscribed. The workloads:* stream is // opt-in too: emitWorkloadEvent (called on the proxy reader goroutine @@ -330,6 +337,7 @@ type workerPaths struct { nodeInfo string proxy string lmstudioProxy string + llamaCppProxy string workloadMgr string errors string engineMgr string @@ -369,6 +377,7 @@ func NewBroker(codec *Codec, paths workerPaths) *Broker { nodeInfoPath: paths.nodeInfo, proxyPath: paths.proxy, lmstudioProxyPath: paths.lmstudioProxy, + llamaCppProxyPath: paths.llamaCppProxy, workloadMgrPath: paths.workloadMgr, errorsPath: paths.errors, engineMgrPath: paths.engineMgr, @@ -509,12 +518,14 @@ func (b *Broker) runEngineAvailabilityAfterPortGates( ctx context.Context, runOllama func(context.Context), runLMStudio func(context.Context), + runLlamaCpp func(context.Context), ) bool { if !b.restoreEnabledEnginesAfterPortGate(ctx) { return false } go runOllama(ctx) - runLMStudio(ctx) + go runLMStudio(ctx) + runLlamaCpp(ctx) return true } @@ -1276,8 +1287,9 @@ func (b *Broker) upsertManualNode(s manualNodeStatus) { b.store.Upsert(en, sourceManual) b.ingestTelemetryAt(sourceManual, manualNodeTelemetry(s, key), receivedAt) // Bridge a reachable manual node into each engine's proxy (ollama-proxy / - // lmstudio-proxy) so inference can route to it; an unreachable engine is - // pulled back out. No-op for a proxy the broker doesn't supervise. + // lmstudio-proxy / llamacpp-proxy) so inference can route to it; an + // unreachable engine is pulled back out. No-op for a proxy the broker + // doesn't supervise. b.bridgeManualNode(s, key) if existed && oldKey != key { @@ -1464,6 +1476,8 @@ func (b *Broker) proxyForEngine(engine string) *proxyProcess { return b.getProxy() case "lmstudio": return b.getLMStudioProxy() + case "llamacpp": + return b.getLlamaCppProxy() default: return nil } @@ -1775,11 +1789,29 @@ func (b *Broker) Serve(ctx context.Context) error { b.finishLMStudioProxyTerminal() } - // Restore engines and begin both advertising loops only after both proxy - // startup attempts have established either readiness or a terminal outcome. - // This prevents a restored engine from taking a persisted proxy port before - // the broker can resolve ownership. - go b.runEngineAvailabilityAfterPortGates(ctx, b.runAutoAdvertise, b.runAutoAdvertiseLMStudio) + // llamacpp-proxy is the llama.cpp counterpart of lmstudio-proxy and is + // supervised identically (non-fatal, port learned via its "ready" + // notification, control plane relayed under llamacpp-proxy:). There is no + // managed facade: it binds :8084 (or a persisted port) and never claims + // llama-server's stock :8082. + if b.llamaCppProxyPath != "" { + b.llamaCppProxySup = newSupervisor("llamacpp-proxy", defaultRestartPolicy(), b.spawnLlamaCppProxy) + b.configureLlamaCppProxySupervisorCallbacks(b.llamaCppProxySup) + if err := b.llamaCppProxySup.Start(); err != nil { + slog.Warn("llamacpp-proxy failed to start; continuing without local llama.cpp proxy", "path", b.llamaCppProxyPath, "err", err) + b.llamaCppProxySup = nil + } else { + defer b.llamaCppProxySup.Stop() + } + } else { + slog.Info("llamacpp-proxy path not resolved; running without local llama.cpp proxy") + } + + // Restore engines and begin the advertising loops only after both managed + // proxy startup attempts have established either readiness or a terminal + // outcome. This prevents a restored engine from taking a persisted proxy + // port before the broker can resolve ownership. + go b.runEngineAvailabilityAfterPortGates(ctx, b.runAutoAdvertise, b.runAutoAdvertiseLMStudio, b.runAutoAdvertiseLlamaCpp) // nvpair-workload-manager is another auxiliary worker: it relays local // workload lifecycle events to peer nodes and surfaces peer events @@ -1856,6 +1888,10 @@ func (b *Broker) shutdownInferenceStack() { // Stop ingress first so no new inference can arrive while engine-manager is // draining engines. supervisor.Stop uses each proxy's stdin-close/join path; // it never adds a parent-side kill timeout. + if b.llamaCppProxySup != nil { + b.llamaCppProxySup.Stop() + b.setLlamaCppProxy(nil) + } if b.lmstudioProxySup != nil { b.lmstudioProxySup.Stop() b.setLMStudioProxy(nil) @@ -2241,6 +2277,11 @@ func (b *Broker) forwardLogLevel(level string) { slog.Warn("failed to forward log/set-level to lmstudio-proxy", "err", err) } } + if p := b.getLlamaCppProxy(); p != nil { + if err := p.SetLogLevel(level); err != nil { + slog.Warn("failed to forward log/set-level to llamacpp-proxy", "err", err) + } + } if wm := b.getWorkloadMgr(); wm != nil { if err := wm.SetLogLevel(level); err != nil { slog.Warn("failed to forward log/set-level to workload-manager", "err", err) @@ -2861,6 +2902,43 @@ func (b *Broker) handleMessage(msg *Message) { log.Printf("failed to respond to lmstudio-proxy:unsubscribe: %v", err) } + case "llamacpp-proxy:get-status": + var result ProxyStatusResult + if p := b.getLlamaCppProxy(); p != nil { + ready, port := p.Status() + result.Ready = ready + result.Port = port + } + if err := b.codec.Respond(msg.ID, result); err != nil { + log.Printf("failed to respond to llamacpp-proxy:get-status: %v", err) + } + + case "llamacpp-proxy:subscribe": + b.proxyMu.Lock() + wasSubscribed := b.llamaCppProxySubscribed + b.llamaCppProxySubscribed = true + b.proxyMu.Unlock() + if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: true}); err != nil { + log.Printf("failed to respond to llamacpp-proxy:subscribe: %v", err) + } + if !wasSubscribed { + if p := b.getLlamaCppProxy(); p != nil { + if rp := p.ReadyParams(); rp != nil { + if err := b.codec.Notify("llamacpp-proxy:ready", rp); err != nil { + slog.Warn("emit baseline llamacpp-proxy:ready failed", "err", err) + } + } + } + } + + case "llamacpp-proxy:unsubscribe": + b.proxyMu.Lock() + b.llamaCppProxySubscribed = false + b.proxyMu.Unlock() + if err := b.codec.Respond(msg.ID, SubscriptionResult{Subscribed: false}); err != nil { + log.Printf("failed to respond to llamacpp-proxy:unsubscribe: %v", err) + } + case "workloads:subscribe": b.workloadsMu.Lock() b.workloadsSubscribed = true @@ -2940,9 +3018,14 @@ func (b *Broker) handleMessage(msg *Message) { // proxy:get-status, and the subscription methods — are handled by // their own cases above). This makes the broker a thin pass-through // for the proxy's whole control plane without enumerating methods. - // lmstudio-proxy:* is checked before proxy:* — though the prefixes - // don't actually overlap (lmstudio-proxy: vs proxy:), keeping it - // first makes the LM Studio namespace explicit. + // llamacpp-proxy:* / lmstudio-proxy:* are checked before proxy:* — + // though the prefixes don't actually overlap (llamacpp-proxy: / + // lmstudio-proxy: vs proxy:), keeping them first makes the engine + // namespaces explicit. + if strings.HasPrefix(msg.Method, "llamacpp-proxy:") { + b.relayToLlamaCppProxy(msg) + return + } if strings.HasPrefix(msg.Method, "lmstudio-proxy:") { b.relayToLMStudioProxy(msg) return diff --git a/services/nvpair-ui-broker/broker_lifecycle_test.go b/services/nvpair-ui-broker/broker_lifecycle_test.go index 51cc0915..b18066ce 100644 --- a/services/nvpair-ui-broker/broker_lifecycle_test.go +++ b/services/nvpair-ui-broker/broker_lifecycle_test.go @@ -53,7 +53,7 @@ func TestEngineAvailabilityWaitsForBothProxyOutcomes(t *testing.T) { restore <- msg.Method } }() - advertised := make(chan string, 2) + advertised := make(chan string, 3) ctx, cancel := context.WithCancel(context.Background()) defer cancel() done := make(chan bool, 1) @@ -62,6 +62,7 @@ func TestEngineAvailabilityWaitsForBothProxyOutcomes(t *testing.T) { ctx, func(context.Context) { advertised <- "ollama" }, func(context.Context) { advertised <- "lmstudio" }, + func(context.Context) { advertised <- "llamacpp" }, ) }() @@ -91,12 +92,12 @@ func TestEngineAvailabilityWaitsForBothProxyOutcomes(t *testing.T) { t.Fatal("enabled-engine restore did not run after both proxy outcomes") } seen := map[string]bool{} - for len(seen) < 2 { + for len(seen) < 3 { select { case got := <-advertised: seen[got] = true case <-time.After(2 * time.Second): - t.Fatalf("advertising did not start for both engines: %v", seen) + t.Fatalf("advertising did not start for all engines: %v", seen) } } if !<-done { @@ -124,17 +125,22 @@ func (h *orderedLifecycleHandle) Stop() { } func TestInferenceShutdownStopsProxiesBeforeEngines(t *testing.T) { - order := make(chan string, 3) + order := make(chan string, 4) ollama := newOrderedLifecycleHandle("ollama-proxy", order) lmstudio := newOrderedLifecycleHandle("lmstudio-proxy", order) + llamacpp := newOrderedLifecycleHandle("llamacpp-proxy", order) proxySup := newSupervisor("proxy", noRestartPolicy(), func() (supervisedHandle, error) { return ollama, nil }) lmstudioSup := newSupervisor("lmstudio-proxy", noRestartPolicy(), func() (supervisedHandle, error) { return lmstudio, nil }) + llamaCppSup := newSupervisor("llamacpp-proxy", noRestartPolicy(), func() (supervisedHandle, error) { return llamacpp, nil }) if err := proxySup.Start(); err != nil { t.Fatal(err) } if err := lmstudioSup.Start(); err != nil { t.Fatal(err) } + if err := llamaCppSup.Start(); err != nil { + t.Fatal(err) + } engineClient, engineServer := net.Pipe() defer engineClient.Close() @@ -151,13 +157,13 @@ func TestInferenceShutdownStopsProxiesBeforeEngines(t *testing.T) { _ = codec.Respond(msg.ID, nil) }() - b := &Broker{proxySup: proxySup, lmstudioProxySup: lmstudioSup} + b := &Broker{proxySup: proxySup, lmstudioProxySup: lmstudioSup, llamaCppProxySup: llamaCppSup} b.setEngineMgr(engine) b.shutdownInferenceStack() - got := []string{<-order, <-order, <-order} - if got[0] != "lmstudio-proxy" || got[1] != "ollama-proxy" || got[2] != "engine-manager" { - t.Fatalf("shutdown order = %v, want [lmstudio-proxy ollama-proxy engine-manager]", got) + got := []string{<-order, <-order, <-order, <-order} + if got[0] != "llamacpp-proxy" || got[1] != "lmstudio-proxy" || got[2] != "ollama-proxy" || got[3] != "engine-manager" { + t.Fatalf("shutdown order = %v, want [llamacpp-proxy lmstudio-proxy ollama-proxy engine-manager]", got) } } diff --git a/services/nvpair-ui-broker/llamacpp_advertise_test.go b/services/nvpair-ui-broker/llamacpp_advertise_test.go new file mode 100644 index 00000000..ef0fcffa --- /dev/null +++ b/services/nvpair-ui-broker/llamacpp_advertise_test.go @@ -0,0 +1,161 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "testing" + "time" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +func TestReconcileAdvertiseLlamaCppRegistersProxyPort(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/models" { + http.NotFound(w, r) + return + } + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"data":[]}`)) + })) + t.Cleanup(engine.Close) + enginePort := engine.Listener.Addr().(*net.TCPAddr).Port + if enginePort == defaultLlamaCppProxyPort { + t.Fatal("httptest bound the llama.cpp proxy port; cannot distinguish engine from proxy") + } + + proxy, localBackend := llamaCppProxyPipe(t, defaultLlamaCppProxyPort) + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLlamaCppProxy(proxy) + b.setEngineMgr(serveEngineStatus(t, enginePort)) + + b.reconcileAdvertiseLlamaCpp(&http.Client{Timeout: 2 * time.Second}) + + got, ok := registrationFor(b, noderec.ServiceLlamaCpp) + if !ok || got.Port != defaultLlamaCppProxyPort { + t.Fatalf("lc registration = (%+v, %v), want port %d", got, ok, defaultLlamaCppProxyPort) + } + select { + case backend := <-localBackend: + if backend.Engine != "llamacpp" || backend.Port != enginePort || !backend.Healthy { + t.Fatalf("local backend = %+v, want healthy llamacpp:%d", backend, enginePort) + } + case <-time.After(2 * time.Second): + t.Fatal("llama.cpp proxy did not receive a healthy local backend") + } +} + +func TestLlamaCppFallbackNeverAdvertisesItsProxy(t *testing.T) { + proxy, localBackend := llamaCppProxyPipe(t, defaultLlamaCppPort) + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLlamaCppProxy(proxy) + + // A nil client is intentional: collision detection must short-circuit before + // any health request can mistake the proxy for llama.cpp. + b.reconcileAdvertiseLlamaCpp(nil) + if got := b.regCache.Snapshot(); len(got) != 0 { + t.Fatalf("llama.cpp proxy was advertised as an engine: %+v", got) + } + select { + case got := <-localBackend: + if got.Port != 0 || got.Healthy { + t.Fatalf("proxy listener was retained as the local backend: %+v", got) + } + case <-time.After(2 * time.Second): + t.Fatal("llama.cpp proxy did not receive a cleared local backend") + } +} + +func TestReconcileAdvertiseLlamaCppUnregistersWhenEngineDown(t *testing.T) { + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + deadPort := ln.Addr().(*net.TCPAddr).Port + _ = ln.Close() + + proxy, localBackend := llamaCppProxyPipe(t, defaultLlamaCppProxyPort) + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLlamaCppProxy(proxy) + b.setEngineMgr(serveEngineStatus(t, deadPort)) + b.registerService(noderec.RegisterParams{Service: noderec.ServiceLlamaCpp, Port: defaultLlamaCppProxyPort}) + + b.reconcileAdvertiseLlamaCpp(&http.Client{Timeout: 2 * time.Second}) + + if _, ok := registrationFor(b, noderec.ServiceLlamaCpp); ok { + t.Fatal("lc stayed registered while the engine was down") + } + select { + case got := <-localBackend: + if got.Healthy { + t.Fatalf("local backend stayed healthy while the engine was down: %+v", got) + } + case <-time.After(2 * time.Second): + t.Fatal("llama.cpp proxy did not receive an unhealthy local backend") + } +} + +func llamaCppProxyPipe(t *testing.T, listenPort int) (*proxyProcess, <-chan proxyLocalBackend) { + t.Helper() + proxyClient, proxyServer := net.Pipe() + t.Cleanup(func() { + _ = proxyClient.Close() + _ = proxyServer.Close() + }) + proxy := &proxyProcess{ + peer: NewPeer(NewCodec(proxyClient)), + ready: true, + port: listenPort, + } + go proxy.peer.Serve(nil, nil) + + localBackend := make(chan proxyLocalBackend, 1) + go func() { + codec := NewCodec(proxyServer) + msg, err := codec.Read() + if err != nil { + return + } + var got proxyLocalBackend + if json.Unmarshal(msg.Params, &got) == nil { + localBackend <- got + } + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + }() + return proxy, localBackend +} + +func serveEngineStatus(t *testing.T, port int) *rpcWorker { + t.Helper() + engineClient, engineServer := net.Pipe() + t.Cleanup(func() { + _ = engineClient.Close() + _ = engineServer.Close() + }) + engine := &rpcWorker{peer: NewPeer(NewCodec(engineClient))} + go engine.peer.Serve(nil, nil) + go func() { + codec := NewCodec(engineServer) + msg, err := codec.Read() + if err != nil { + return + } + _ = codec.Respond(msg.ID, map[string]any{"running": true, "port": port}) + }() + return engine +} + +func registrationFor(b *Broker, svc noderec.ServiceKey) (noderec.RegisterParams, bool) { + for _, p := range b.regCache.Snapshot() { + if p.Service == svc { + return p, true + } + } + return noderec.RegisterParams{}, false +} diff --git a/services/nvpair-ui-broker/llamacppproxy.go b/services/nvpair-ui-broker/llamacppproxy.go new file mode 100644 index 00000000..64630b8e --- /dev/null +++ b/services/nvpair-ui-broker/llamacppproxy.go @@ -0,0 +1,154 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "fmt" + "log" + "log/slog" + "strings" + + "nvpair-shared/applog" + "nvpair-shared/noderec" +) + +// llamacppproxy.go is the broker's llama.cpp counterpart to its lmstudio-proxy +// wiring. llamacpp-proxy speaks the same JSON-RPC control plane as the other +// engine proxies (a "ready" port notification, node/add-manual/remove-manual, +// nodes/list, node/select, the workload:* lifecycle stream), so it reuses the +// proxyProcess client type; only the namespace differs — the broker relays it +// under llamacpp-proxy:. There is no managed facade and PAIR never binds the +// engine's stock :8082; the proxy listens on :8084 (or a persisted port). + +func (b *Broker) setLlamaCppProxy(p *proxyProcess) { + b.workersMu.Lock() + b.llamaCppProxy = p + b.workersMu.Unlock() +} + +func (b *Broker) getLlamaCppProxy() *proxyProcess { + b.workersMu.Lock() + defer b.workersMu.Unlock() + return b.llamaCppProxy +} + +func (b *Broker) configureLlamaCppProxySupervisorCallbacks(sup *supervisor) { + sup.onCrash, sup.onRecovered = b.supervisedWorkerCallbacks("llamacpp-proxy", func() { b.setLlamaCppProxy(nil) }) +} + +func (b *Broker) llamaCppProxyArgs() []string { + var args []string + if port := int(b.llamaCppProxyStartupPort.Load()); port != 0 { + args = []string{"--port", fmt.Sprintf("%d", port), "--ignore-persisted-port"} + } + return append(args, b.clusterDirArgs()...) +} + +// spawnLlamaCppProxy is the llamacpp-proxy supervisor's spawn closure, +// mirroring spawnLMStudioProxy without the managed-facade port reconcile. +func (b *Broker) spawnLlamaCppProxy() (supervisedHandle, error) { + generation := b.llamaCppProxyGeneration.Add(1) + pp, err := startProxy( + "llamacpp-proxy", + b.llamaCppProxyPath, + applog.LevelString(), + b.relayDir, + func(method string, params json.RawMessage) { + b.forwardLlamaCppProxyNotificationForGeneration(generation, method, params) + }, + b.llamaCppProxyArgs()..., + ) + if err != nil { + return nil, err + } + b.setLlamaCppProxy(pp) + slog.Info("llamacpp-proxy started", "path", b.llamaCppProxyPath, "pid", pp.cmd.Process.Pid) + return pp, nil +} + +// forwardLlamaCppProxyNotification is the hook startProxy invokes on the +// llamacpp-proxy reader goroutine. It mirrors forwardLMStudioProxyNotification +// without the managed-facade bind-failed dance: errors:report / errors:clear +// go into the nvpair-errors pipeline; workload lifecycle events are stamped +// and forwarded to the workload-manager (llamacpp-proxy tags its workloads +// "llamacpp"); everything else is re-emitted to llamacpp-proxy:subscribe'd +// clients as llamacpp-proxy:. +func (b *Broker) forwardLlamaCppProxyNotification(method string, params json.RawMessage) { + b.forwardLlamaCppProxyNotificationForGeneration(b.llamaCppProxyGeneration.Load(), method, params) +} + +func (b *Broker) forwardLlamaCppProxyNotificationForGeneration(generation uint64, method string, params json.RawMessage) { + if b.llamaCppProxyGeneration.Load() != generation { + return + } + if b.dispatchErrorsNotif("llamacpp-proxy", method, params) { + return + } + if method == "error" { + var ep struct { + Code string `json:"code"` + Port int `json:"port"` + } + if json.Unmarshal(params, &ep) == nil && ep.Code == "bind-failed" { + slog.Warn("llama.cpp proxy bind failed", "port", ep.Port) + } + } + if proxyWorkloadMethods[method] { + b.routeProxyWorkload(method, params) + return + } + if method == noderec.NotifyNodeActivity { + b.routeNodeActivity(params) + return + } + b.proxyMu.Lock() + subscribed := b.llamaCppProxySubscribed + b.proxyMu.Unlock() + if !subscribed { + return + } + if err := b.codec.Notify("llamacpp-proxy:"+method, params); err != nil { + slog.Warn("forward llamacpp-proxy notification failed", "method", method, "err", err) + } +} + +// relayToLlamaCppProxy forwards an llamacpp-proxy: request to +// llamacpp-proxy as (prefix stripped) and maps its response straight +// back, mirroring relayToLMStudioProxy. llamacpp-proxy:shutdown is refused — +// the broker owns the proxy's lifecycle. +func (b *Broker) relayToLlamaCppProxy(msg *Message) { + method := strings.TrimPrefix(msg.Method, "llamacpp-proxy:") + if method == "shutdown" { + if err := b.codec.RespondError(msg.ID, -32601, "llamacpp-proxy:shutdown is not allowed; the broker owns the proxy lifecycle"); err != nil { + log.Printf("failed to respond to llamacpp-proxy:shutdown: %v", err) + } + return + } + + p := b.getLlamaCppProxy() + if p == nil { + if err := b.codec.RespondError(msg.ID, -32000, "llamacpp-proxy not available"); err != nil { + log.Printf("failed to respond to %s: %v", msg.Method, err) + } + return + } + + result, rpcErr, err := p.Call(context.Background(), method, msg.Params) + switch { + case err != nil: + if err := b.codec.RespondError(msg.ID, -32000, fmt.Sprintf("llamacpp-proxy call failed: %v", err)); err != nil { + log.Printf("failed to respond to %s: %v", msg.Method, err) + } + case rpcErr != nil: + if err := b.codec.RespondError(msg.ID, rpcErr.Code, rpcErr.Message); err != nil { + log.Printf("failed to relay llamacpp-proxy error for %s: %v", msg.Method, err) + } + default: + if err := b.codec.Respond(msg.ID, result); err != nil { + log.Printf("failed to relay llamacpp-proxy result for %s: %v", msg.Method, err) + } + } +} diff --git a/services/nvpair-ui-broker/main.go b/services/nvpair-ui-broker/main.go index ced0d783..d87dfcda 100644 --- a/services/nvpair-ui-broker/main.go +++ b/services/nvpair-ui-broker/main.go @@ -26,6 +26,7 @@ func main() { nodeInfoPath := flag.String("node-info-path", "", "path to nvpair-node-info binary (default: ./nvpair-node-info in the current working directory)") proxyPath := flag.String("proxy-path", "", "path to ollama-proxy binary (default: ./ollama-proxy in the current working directory)") lmstudioProxyPath := flag.String("lmstudio-proxy-path", "", "path to lmstudio-proxy binary (default: ./lmstudio-proxy in the current working directory)") + llamaCppProxyPath := flag.String("llamacpp-proxy-path", "", "path to llamacpp-proxy binary (default: ./llamacpp-proxy in the current working directory)") workloadMgrPath := flag.String("workload-manager-path", "", "path to nvpair-workload-manager binary (default: ./nvpair-workload-manager in the current working directory)") errorsPath := flag.String("errors-path", "", "path to nvpair-errors binary (default: ./nvpair-errors in the current working directory)") engineMgrPath := flag.String("engine-manager-path", "", "path to nvpair-engine-manager binary (default: ./nvpair-engine-manager in the current working directory)") @@ -136,6 +137,19 @@ func main() { resolvedLMStudioProxy = "" } + // llamacpp-proxy is auxiliary too, resolved with the same rules: an + // explicit --llamacpp-proxy-path that doesn't exist is a loud operator + // mistake (fatal), but an absent default sibling just means the broker + // runs without a local llama.cpp reverse proxy. + resolvedLlamaCppProxy, err := resolveLlamaCppProxyPath(*llamaCppProxyPath) + if err != nil { + if *llamaCppProxyPath != "" { + fatalf("llamacpp-proxy binary: %v", err) + } + slog.Warn("llamacpp-proxy binary not found; broker will run without local llama.cpp proxy", "err", err) + resolvedLlamaCppProxy = "" + } + // nvpair-workload-manager is auxiliary too, resolved with the same rules: // an explicit --workload-manager-path that doesn't exist is a loud // operator mistake (fatal), but an absent default sibling just means @@ -253,6 +267,7 @@ func main() { nodeInfo: resolvedNodeInfo, proxy: resolvedProxy, lmstudioProxy: resolvedLMStudioProxy, + llamaCppProxy: resolvedLlamaCppProxy, workloadMgr: resolvedWorkloadMgr, errors: resolvedErrors, engineMgr: resolvedEngineMgr, @@ -320,6 +335,14 @@ func resolveLMStudioProxyPath(override string) (string, error) { return resolveSiblingBinary(override, "lmstudio-proxy", "--lmstudio-proxy-path") } +// resolveLlamaCppProxyPath mirrors resolveLMStudioProxyPath for the +// llamacpp-proxy binary the broker supervises. Like the other engine proxies +// its result is optional at the call site: a not-found default sibling +// degrades to "no local llama.cpp proxy" rather than aborting the broker. +func resolveLlamaCppProxyPath(override string) (string, error) { + return resolveSiblingBinary(override, "llamacpp-proxy", "--llamacpp-proxy-path") +} + // resolveWorkloadManagerPath mirrors resolveProxyPath for the // nvpair-workload-manager binary the broker supervises. Like node-info and the // proxy its result is optional at the call site: a not-found default sibling diff --git a/services/nvpair-ui-broker/manualnodes.go b/services/nvpair-ui-broker/manualnodes.go index 47fe0ff1..d760420e 100644 --- a/services/nvpair-ui-broker/manualnodes.go +++ b/services/nvpair-ui-broker/manualnodes.go @@ -16,9 +16,9 @@ import ( // broker needs. Its JSON tags match the producer's so node/discovered| // updated|removed payloads unmarshal straight into it (the GPU/CPU/memory // sub-objects reuse the broker's discovery types, whose tags are identical). -// The ollama_* / lmstudio_* fields drive the per-engine manual→proxy bridge -// (bridgeManualNode); the rest project into the discovery store via -// manualToEnriched. +// The ollama_* / lmstudio_* / llamacpp_* fields drive the per-engine +// manual→proxy bridge (bridgeManualNode); the rest project into the discovery +// store via manualToEnriched. type manualNodeStatus struct { ID string `json:"id"` Address string `json:"address"` @@ -28,6 +28,9 @@ type manualNodeStatus struct { LMStudioUp bool `json:"lmstudio_up"` LMStudioPort int `json:"lmstudio_port"` LMStudioModels []string `json:"lmstudio_models,omitempty"` + LlamaCppUp bool `json:"llamacpp_up"` + LlamaCppPort int `json:"llamacpp_port"` + LlamaCppModels []string `json:"llamacpp_models,omitempty"` NodeInfoPort int `json:"node_info_port"` GPUs []GPUInfo `json:"gpus"` CPU *CPUInfo `json:"cpu"` @@ -87,7 +90,7 @@ func manualToEnriched(s manualNodeStatus) EnrichedNode { GPUs: s.GPUs, CPU: s.CPU, Memory: s.Memory, - Models: mergeModels(s.OllamaModels, s.LMStudioModels), + Models: mergeModels(s.OllamaModels, s.LMStudioModels, s.LlamaCppModels), ModelsByEngine: manualModelsByEngine(s), } if s.Address != "" { @@ -98,9 +101,9 @@ func manualToEnriched(s manualNodeStatus) EnrichedNode { // manualModelsByEngine builds the per-engine attribution for a manual node from // the per-engine lists the prober already collected, keyed by the same -// engine-manager engine names discovered nodes use ("ollama", "lmstudio") so the -// two discovery sources present ModelsByEngine identically. An engine with no -// models adds no key; returns nil when neither engine reports any. +// engine-manager engine names discovered nodes use ("ollama", "lmstudio", +// "llamacpp") so the two discovery sources present ModelsByEngine identically. +// An engine with no models adds no key; returns nil when no engine reports any. func manualModelsByEngine(s manualNodeStatus) map[string][]string { byEngine := map[string][]string{} if len(s.OllamaModels) > 0 { @@ -109,6 +112,9 @@ func manualModelsByEngine(s manualNodeStatus) map[string][]string { if len(s.LMStudioModels) > 0 { byEngine["lmstudio"] = s.LMStudioModels } + if len(s.LlamaCppModels) > 0 { + byEngine["llamacpp"] = s.LlamaCppModels + } if len(byEngine) == 0 { return nil } @@ -149,11 +155,11 @@ type proxyManualNode struct { // bridgeManualNode keeps every supervised proxy's manual-node set in step with // a manual node's per-engine reachability: a node whose Ollama is up is bridged -// into ollama-proxy and one whose LM Studio is up into lmstudio-proxy -// (idempotent — each proxy upserts on a repeat), while an engine that is not -// (or no longer) reachable is removed from its proxy. Each leg is a no-op when -// that proxy isn't supervised — the bridge only applies when the broker owns -// both ends. +// into ollama-proxy, one whose LM Studio is up into lmstudio-proxy, and one +// whose llama.cpp is up into llamacpp-proxy (idempotent — each proxy upserts on +// a repeat), while an engine that is not (or no longer) reachable is removed +// from its proxy. Each leg is a no-op when that proxy isn't supervised — the +// bridge only applies when the broker owns both ends. // // Manual nodes are, by definition, the nodes that never appear in the discovery // relay's snapshots — they advertise no _nvpair-node record for the scanner @@ -162,6 +168,7 @@ type proxyManualNode struct { func (b *Broker) bridgeManualNode(s manualNodeStatus, key string) { b.bridgeToProxy(b.getProxy(), "ollama", s, key, s.OllamaUp, s.OllamaPort, s.OllamaModels) b.bridgeToProxy(b.getLMStudioProxy(), "lmstudio", s, key, s.LMStudioUp, s.LMStudioPort, s.LMStudioModels) + b.bridgeToProxy(b.getLlamaCppProxy(), "llamacpp", s, key, s.LlamaCppUp, s.LlamaCppPort, s.LlamaCppModels) } // bridgeToProxy adds the node to p when its engine is reachable, or removes it @@ -196,6 +203,7 @@ func (b *Broker) bridgeToProxy(p *proxyProcess, engine string, s manualNodeStatu func (b *Broker) removeManualNodeFromProxies(id string) { b.callProxyManual(b.getProxy(), "ollama", "node/remove-manual", map[string]string{"id": id}, id) b.callProxyManual(b.getLMStudioProxy(), "lmstudio", "node/remove-manual", map[string]string{"id": id}, id) + b.callProxyManual(b.getLlamaCppProxy(), "llamacpp", "node/remove-manual", map[string]string{"id": id}, id) } // callProxyManual issues a best-effort node/add-manual|remove-manual to a diff --git a/services/nvpair-ui-broker/manualnodes_test.go b/services/nvpair-ui-broker/manualnodes_test.go new file mode 100644 index 00000000..73fc9ef1 --- /dev/null +++ b/services/nvpair-ui-broker/manualnodes_test.go @@ -0,0 +1,148 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "testing" + "time" +) + +func TestManualModelsByEngineIncludesLlamaCpp(t *testing.T) { + s := manualNodeStatus{ + ID: "lab", + OllamaModels: []string{"llama3"}, + LMStudioModels: []string{"qwen"}, + LlamaCppModels: []string{"loaded-one"}, + } + got := manualModelsByEngine(s) + if len(got["llamacpp"]) != 1 || got["llamacpp"][0] != "loaded-one" { + t.Fatalf("llamacpp = %#v, want [loaded-one]", got["llamacpp"]) + } + en := manualToEnriched(s) + found := false + for _, m := range en.Models { + if m == "loaded-one" { + found = true + break + } + } + if !found { + t.Fatalf("Models missing loaded-one: %#v", en.Models) + } +} + +func TestManualNodeBridgesLlamaCppIntoProxy(t *testing.T) { + proxy, calls := llamaCppManualProxyPipe(t) + b := newManualTestBroker() + b.setLlamaCppProxy(proxy) + + b.bridgeManualNode(manualNodeStatus{ + ID: "lab", + Address: "10.0.0.5", + LlamaCppUp: true, + LlamaCppPort: 8082, + LlamaCppModels: []string{"loaded-one"}, + }, "lab") + + call := readProxyManualCall(t, calls) + if call.method != "node/add-manual" { + t.Fatalf("method = %q, want node/add-manual", call.method) + } + var node proxyManualNode + if err := json.Unmarshal(call.params, &node); err != nil { + t.Fatalf("decode add-manual: %v", err) + } + if node.ID != "lab" || node.Host != "10.0.0.5" || node.Port != 8082 { + t.Fatalf("bridged node = %+v", node) + } + if len(node.Models) != 1 || node.Models[0] != "loaded-one" { + t.Fatalf("models = %#v, want [loaded-one]", node.Models) + } +} + +func TestManualNodeRemovesLlamaCppFromProxyWhenDown(t *testing.T) { + proxy, calls := llamaCppManualProxyPipe(t) + b := newManualTestBroker() + b.setLlamaCppProxy(proxy) + + b.bridgeManualNode(manualNodeStatus{ + ID: "lab", + Address: "10.0.0.5", + LlamaCppUp: false, + LlamaCppPort: 8082, + }, "lab") + + call := readProxyManualCall(t, calls) + if call.method != "node/remove-manual" { + t.Fatalf("method = %q, want node/remove-manual", call.method) + } + var params map[string]string + if err := json.Unmarshal(call.params, ¶ms); err != nil { + t.Fatalf("decode remove-manual: %v", err) + } + if params["id"] != "lab" { + t.Fatalf("remove id = %q, want lab", params["id"]) + } +} + +func TestRemoveManualNodeFromProxiesDropsLlamaCpp(t *testing.T) { + proxy, calls := llamaCppManualProxyPipe(t) + b := newManualTestBroker() + b.setLlamaCppProxy(proxy) + + b.removeManualNodeFromProxies("lab") + + call := readProxyManualCall(t, calls) + if call.method != "node/remove-manual" { + t.Fatalf("method = %q, want node/remove-manual", call.method) + } + var params map[string]string + if err := json.Unmarshal(call.params, ¶ms); err != nil { + t.Fatalf("decode remove-manual: %v", err) + } + if params["id"] != "lab" { + t.Fatalf("remove id = %q, want lab", params["id"]) + } +} + +type proxyManualCall struct { + method string + params json.RawMessage +} + +func llamaCppManualProxyPipe(t *testing.T) (*proxyProcess, <-chan proxyManualCall) { + t.Helper() + proxyClient, proxyServer := net.Pipe() + t.Cleanup(func() { + _ = proxyClient.Close() + _ = proxyServer.Close() + }) + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient))} + go proxy.peer.Serve(nil, nil) + + calls := make(chan proxyManualCall, 1) + go func() { + codec := NewCodec(proxyServer) + msg, err := codec.Read() + if err != nil { + return + } + calls <- proxyManualCall{method: msg.Method, params: msg.Params} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + }() + return proxy, calls +} + +func readProxyManualCall(t *testing.T, calls <-chan proxyManualCall) proxyManualCall { + t.Helper() + select { + case call := <-calls: + return call + case <-time.After(2 * time.Second): + t.Fatal("timed out waiting for llamacpp-proxy manual-node call") + return proxyManualCall{} + } +} diff --git a/services/readme.md b/services/readme.md index ee3523d5..3c18b32e 100644 --- a/services/readme.md +++ b/services/readme.md @@ -11,8 +11,9 @@ local network: each node advertises itself over mDNS as one consolidated node offers and where to reach them. What a discovered node can actually serve is a separate question, answered after -discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, both, -or neither, and its model inventory is fetched over HTTP from its engine-manager +discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, +llama.cpp, any combination, or none, and its model inventory is fetched over HTTP +from its engine-manager rather than crammed into mDNS TXT records, which are too small to carry it. Locally, each node exposes compatibility proxies — Ollama-compatible and @@ -35,13 +36,14 @@ see the [root README](../README.md#what-is-supported). ## Architecture -This tree builds thirteen Go binaries. `nvpair-ui-broker` is the parent service and supervises the eleven workers, all spawned at startup — only the scanner is required, and a missing binary for any other leaves the broker running without that capability. `nvpair-tui` is the thirteenth: a terminal client that launches and supervises its own broker rather than being supervised. Processes communicate via newline-delimited JSON-RPC 2.0 over stdio or, optionally, a Unix socket / Windows named pipe. +This tree builds fourteen Go binaries. `nvpair-ui-broker` is the parent service and supervises the twelve workers, all spawned at startup — only the scanner is required, and a missing binary for any other leaves the broker running without that capability. `nvpair-tui` is the fourteenth: a terminal client that launches and supervises its own broker rather than being supervised. Processes communicate via newline-delimited JSON-RPC 2.0 over stdio or, optionally, a Unix socket / Windows named pipe. | Binary | Role | | --- | --- | | `nvpair-ui-broker` | Parent service and JSON-RPC API surface used by the bundled UI and other clients. Supervises workers, relays consolidated discovery, and coordinates routing and scheduling. | | `ollama-proxy` | Ollama-compatible HTTP reverse proxy. Routes only to advertised model owners, with owner failover and scheduler priorities. | | `lmstudio-proxy` | LM Studio counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes with equivalent owner-only routing and failover behavior. | +| `llamacpp-proxy` | llama.cpp counterpart to `lmstudio-proxy`. OpenAI-compatible routes; eligibility is **loaded** models only. Default listen port `8084`; never binds the adopt probe (default `8082`). | | `nvpair-node-info` | Local HTTP service on `:14318` exposing GPU, CPU, and memory inventory at `/v1/node-info`. | | `nvpair-node-scanner` | Consolidated discovery daemon. Advertises and browses `_nvpair-node._tcp`, maintains the node directory, and enriches peers with hardware and model information over HTTP. | | `nvpair-manual-nodes` | Manages user-added nodes that don't appear via mDNS; probes them every 10 s. | @@ -59,7 +61,7 @@ The mDNS responder is our own rather than the host's, because Windows ships none The broker feeds every accepted local or peer workload transition plus compact GPU telemetry to the scheduler. Queued and running work is counted by destination -node across Ollama and LM Studio together. Fresh maximum-GPU utilization is +node across Ollama, LM Studio, and llama.cpp together. Fresh maximum-GPU utilization is smoothed into pressure 0–3; missing or stale telemetry is neutral. Rankings use `pending + gpuPressure`, and each proxy adds local reservations before choosing, so bursts spread without waiting for workload feedback. @@ -70,6 +72,7 @@ so bursts spread without waiting for workload feedback. nvpair-ui-broker/ Parent service / JSON-RPC API surface ollama-proxy/ Ollama-compatible routing proxy lmstudio-proxy/ OpenAI-compatible routing proxy for LM Studio +llamacpp-proxy/ OpenAI-compatible routing proxy for llama.cpp nvpair-node-info/ Local GPU-inventory HTTP service nvpair-node-scanner/ Consolidated _nvpair-node._tcp discovery daemon nvpair-manual-nodes/ Manual-node manager @@ -84,8 +87,8 @@ shared/ Shared Go module (nvpair-shared/…) eap-noob/ EAP-NOOB implementation used by cluster pairing tests/ Cross-process integration tests (separate go.mod) versions.json Single source of truth for every component version -build.bat Builds all thirteen binaries (Windows) -build.sh Builds all thirteen binaries (Linux) +build.bat Builds all fourteen binaries (Windows) +build.sh Builds all fourteen binaries (Linux) VERSIONING.md SemVer rules and version-bump workflow ``` @@ -115,7 +118,7 @@ On Linux and macOS: ./build.sh ``` -Both scripts read `versions.json`, build all thirteen Go binaries with `-X main.Version=…` ldflags, and stage them together in `services/build/bin/`. +Both scripts read `versions.json`, build all fourteen Go binaries with `-X main.Version=…` ldflags, and stage them together in `services/build/bin/`. Do **not** build individual components by hand without also copying their binaries into `build/bin/`: the broker will silently keep using the older binary there. @@ -190,7 +193,7 @@ cd shared go test ./... ``` -**Every one of the thirteen binaries has tests**, as do `shared/` and +**Every one of the fourteen binaries has tests**, as do `shared/` and `eap-noob/`. Depth varies with how much behaviour a component carries: `nvpair-engine-manager` and `nvpair-cluster-manager` have the largest suites, while a component with one test file may still hold twenty test functions in it. diff --git a/services/shared/noderec/noderec.go b/services/shared/noderec/noderec.go index 112e7fe3..d1e58189 100644 --- a/services/shared/noderec/noderec.go +++ b/services/shared/noderec/noderec.go @@ -87,6 +87,7 @@ const ( ServiceNodeInfo ServiceKey = "ni" ServiceOllama ServiceKey = "ol" ServiceLMStudio ServiceKey = "lm" + ServiceLlamaCpp ServiceKey = "lc" ServiceErrors ServiceKey = "er" ServiceWorkload ServiceKey = "wl" ServiceCluster ServiceKey = "cl" @@ -104,7 +105,7 @@ const ( // serviceKeyOrder is the deterministic emit order for service ports in TXT. var serviceKeyOrder = []ServiceKey{ - ServiceNodeInfo, ServiceOllama, ServiceLMStudio, + ServiceNodeInfo, ServiceOllama, ServiceLMStudio, ServiceLlamaCpp, ServiceErrors, ServiceWorkload, ServiceCluster, ServiceEngineManager, ServiceEngineControl, } @@ -577,6 +578,17 @@ func (n DirectoryNode) EngineModels(engine string) []string { return n.Models } +// EngineLoadedModels returns models currently resident in memory for one +// engine. It never falls back to Models or ModelsByEngine: a missing +// LoadedByEngine report means nothing is loaded, so a router cannot treat +// catalog ids as eligible. +func (n DirectoryNode) EngineLoadedModels(engine string) []string { + if n.LoadedByEngine == nil { + return nil + } + return n.LoadedByEngine[engine] +} + // SubscribeParams filters a subscription to nodes advertising any of the listed // services; an empty list subscribes to all nodes. type SubscribeParams struct { diff --git a/services/shared/noderec/noderec_test.go b/services/shared/noderec/noderec_test.go index 8b0757e5..a02aedf8 100644 --- a/services/shared/noderec/noderec_test.go +++ b/services/shared/noderec/noderec_test.go @@ -40,6 +40,48 @@ func TestEngineModels(t *testing.T) { } } +func TestEngineLoadedModels(t *testing.T) { + n := DirectoryNode{ + Models: []string{"catalog-a", "catalog-b"}, + ModelsByEngine: map[string][]string{ + "llamacpp": {"catalog-a", "catalog-b"}, + }, + LoadedByEngine: map[string][]string{ + "llamacpp": {"catalog-a"}, + }, + } + got := n.EngineLoadedModels("llamacpp") + if !reflect.DeepEqual(got, []string{"catalog-a"}) { + t.Fatalf("EngineLoadedModels(llamacpp) = %v, want [catalog-a]", got) + } + if got := n.EngineLoadedModels("ollama"); len(got) != 0 { + t.Fatalf("EngineLoadedModels(missing) = %v, want empty", got) + } + legacy := DirectoryNode{Models: []string{"catalog-a"}} + if got := legacy.EngineLoadedModels("llamacpp"); len(got) != 0 { + t.Fatalf("nil LoadedByEngine must not fall back to catalog, got %v", got) + } +} + +func TestServiceLlamaCppKey(t *testing.T) { + if ServiceLlamaCpp != "lc" { + t.Fatalf("ServiceLlamaCpp = %q, want lc", ServiceLlamaCpp) + } + found := false + for _, k := range serviceKeyOrder { + if k == ServiceLlamaCpp { + found = true + break + } + } + if !found { + t.Fatal("ServiceLlamaCpp missing from serviceKeyOrder") + } + if ServiceLlamaCpp.Transport() != TransportPlain { + t.Fatal("lc transport must be TransportPlain (same as ol/lm)") + } +} + func TestParseTXT(t *testing.T) { txt := []string{ "v=1", "uuid=host-abc", "cluster-uuid=clu-xyz", "ip=192.168.1.10", diff --git a/services/versions.json b/services/versions.json index 29d8c230..b22b4898 100644 --- a/services/versions.json +++ b/services/versions.json @@ -1,20 +1,21 @@ { "$comment": "Single source of truth for all version numbers. See VERSIONING.md for bump rules.", - "product": "0.91.7", - "installer": "0.91.7", + "product": "0.92.0", + "installer": "0.92.0", "components": { "ollama-proxy": "0.26.2", "lmstudio-proxy": "0.16.2", + "llamacpp-proxy": "0.1.0", "nvpair-node-info": "0.13.3", "nvpair-node-scanner": "0.20.3", - "nvpair-manual-nodes": "0.11.1", + "nvpair-manual-nodes": "0.12.0", "nvpair-workload-manager": "0.13.3", "nvpair-errors": "0.7.4", "nvpair-node-settings": "1.0.4", - "nvpair-ui-broker": "0.40.2", - "nvpair-engine-manager": "0.17.4", + "nvpair-ui-broker": "0.41.0", + "nvpair-engine-manager": "0.18.0", "nvpair-cluster-manager": "1.1.4", - "nvpair-job-scheduler": "0.4.1", - "nvpair-tui": "0.7.2" + "nvpair-job-scheduler": "0.5.0", + "nvpair-tui": "0.8.0" } }