diff --git a/README.md b/README.md index 0f0a7242..0939cb2c 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ one, and both report live GPU and memory use throughout. | **Architectures** | x64 and arm64 on all three. Windows on ARM is experimental. | | **Installers** | Windows `.exe`; Linux `.deb`; macOS `.dmg`. On other Linux distributions, [build from source](docs/building.mdx). | | **Mixing nodes** | Windows, Linux, and macOS nodes can all be paired with each other | -| **Inference engines** | Ollama and LM Studio | +| **Inference engines** | Ollama, LM Studio, and vLLM (Linux) | **PAIR running on a machine does not mean an engine will.** PAIR itself runs on any supported Windows, Linux, or macOS machine. Each engine sets its own requirements @@ -100,7 +100,8 @@ you want by its full filename instead. status there. - **Get an engine running.** On the node's card, open **Engine settings** and - select **Install** next to Ollama or LM Studio. PAIR downloads and sets the + select **Install** next to Ollama, LM Studio, or (on Linux) vLLM. PAIR + downloads and sets the engine up for you, so nothing needs to be in place beforehand. If PAIR already found an engine you installed yourself, start that one instead. @@ -149,7 +150,8 @@ The reply is ordinary OpenAI-shaped JSON, abbreviated here: } ``` -If you changed a port, or you are using LM Studio rather than Ollama, copy the +If you changed a port, or you are using LM Studio or vLLM rather than Ollama, +copy the URL from **Endpoints → API endpoints** instead of assuming the one above. That is a single machine working. To route across machines, pair a second one diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.md index 590e7264..5fcf1021 100644 --- a/desktop/docs/services-api.md +++ b/desktop/docs/services-api.md @@ -123,6 +123,7 @@ | `engine:remote-stop` | request (we call) | ✅ yes | | `engine:remote-unload-model` | request (we call) | ✅ yes | | `engine:restart` | request (we call) | ⚠️ not called | +| `engine:set-model` | request (we call) | ✅ yes | | `engine:set-port` | request (we call) | ✅ yes | | `engine:start` | request (we call) | ✅ yes | | `engine:status` | request (we call) | ✅ yes | diff --git a/desktop/docs/services-backend.md b/desktop/docs/services-backend.md index 0d54c4c9..28262a29 100644 --- a/desktop/docs/services-backend.md +++ b/desktop/docs/services-backend.md @@ -22,7 +22,7 @@ broker supervises every worker and relays its control plane. | ------------------------- | --------------------------------------------------------- | | `nvpair-ui-broker` | Worker supervision and relay | | `ollama-proxy` | Ollama-compatible routing proxy with cluster-mTLS ingress | -| `lmstudio-proxy` | LM Studio routing proxy with cluster-mTLS ingress | +| `lmstudio-proxy` | OpenAI-compatible routing proxy (LM Studio, vLLM) with cluster-mTLS ingress | | `nvpair-node-scanner` | Discovery and node announcement | | `nvpair-node-info` | Node metadata and telemetry | | `nvpair-manual-nodes` | User-managed node entries | @@ -90,8 +90,10 @@ engine, workload, cluster, and error relays. The bridge then emits renderer push events from backend notifications. Connector readiness follows the broker contract: `app:ready` establishes the -service connection, while Ollama and LM Studio proxy readiness remains an -asynchronous capability signal. Personal AI Router waits up to the canonical +service connection, while the Ollama and OpenAI-compatible proxy readiness +signals remain asynchronous capability signals. The OpenAI-compatible proxy +fronts LM Studio and vLLM together, so one `lmstudio-proxy:ready` records the +port for both engines. Personal AI Router waits up to the canonical startup deadline in `src/shared/constants/modular-runtime.ts` for `app:ready`; an outright failure or stalled broker startup is surfaced in Settings > Service with retry and log access. If a stalled broker reports ready @@ -266,7 +268,8 @@ LAN-reachable. Each node fronts its engine with its `ollama-proxy` / `lmstudio-proxy`, whose LAN ingress is gated by cluster mTLS: only a pinned cluster member can send it work. Discovery advertises the promoted **proxy** port (never the engine port), and the broker hands the private loopback engine to -the local proxy via `node/set-local-backend`. Every cluster-scoped worker derives +the local proxy via `node/set-local-backend`, which is keyed per engine so the +one OpenAI-compatible proxy can hold an LM Studio and a vLLM backend at once. Every cluster-scoped worker derives its own membership from the cluster directory continuously, so a proxy's mTLS ingress tracks a create, join, or leave with no worker restart. Loopback plaintext requests from local clients are unaffected. This is entirely diff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.md index 93d13290..b5cee7c6 100644 --- a/desktop/docs/services-parity.md +++ b/desktop/docs/services-parity.md @@ -24,6 +24,7 @@ history. | Manual nodes | Complete with local persistence | Broker owns probing and proxy registration; Electron persists entries for replay | | Ollama routing | Complete | Broker relay and backend scheduler drive proxy routing | | LM Studio routing | Complete | Parallel broker relay and scheduler path | +| vLLM routing | Complete on Linux | Shares the LM Studio relay, the OpenAI-compatible proxy, and the same scheduler path | | Local engine lifecycle | Complete | Install, start, stop, uninstall, update, and port configuration | | Remote engine lifecycle | Partial | Remote install, start, stop, status, and model pull are supported | | Engine models | Partial | Core list, pull, load, unload, and supported delete actions are wired | @@ -102,7 +103,9 @@ they survive worker restarts. Both text-engine proxies are broker-owned and cluster-aware: - `ollama-proxy` serves the Ollama-compatible surface; -- `lmstudio-proxy` serves the LM Studio/OpenAI-compatible surface. +- `lmstudio-proxy` serves the OpenAI-compatible surface for every + OpenAI-compatible engine — LM Studio and vLLM share the one listener, and a + node record can advertise `lm` and `vl` at the same port. Routing precedence is manual selection, scheduler priority, then deterministic proxy ordering. Personal AI Router leaves proxies in automatic mode. @@ -197,8 +200,16 @@ Personal AI Router uses: - Ollama `run_model`, `unload_model` (`keep_alive: 0`), and `delete_model`; - LM Studio `load_model`, `unload_model`, and `delete_model` (`remove_path`). -Both engines expose Load, Eject, and Delete in the model manager when the -backend action exists. Keep-alive / expiry controls remain unsupported. +Ollama and LM Studio expose Load, Eject, and Delete in the model manager when +the backend action exists. Keep-alive / expiry controls remain unsupported. + +vLLM has no model operations at all: it declares only `list_models`, +`loaded_models`, and `chat`, because a vLLM process serves exactly one model +chosen at launch and it neither downloads nor deletes weights on request. Its +capability entry therefore sets `hasDeleteModel: false`, `hasEject: false`, +`hasExpiry: false`, and `modelOpsWhenStopped: false`, and adds +`hasServedModel: true` — the model to serve is a *setting* (`engine:set-model`), +not a model action, and changing it restarts the engine. LM Studio's `delete_model` declares `restart_after`, so the engine manager restarts a running LM Studio once the files are removed — its `/v1/models` is diff --git a/desktop/src/electron/service-bridge/empty-handlers.ts b/desktop/src/electron/service-bridge/empty-handlers.ts index dc5c2dd8..8def7fd2 100644 --- a/desktop/src/electron/service-bridge/empty-handlers.ts +++ b/desktop/src/electron/service-bridge/empty-handlers.ts @@ -507,6 +507,24 @@ function routeEngineManagerCommand(payload: WsInvokeRequest<'engine:command'>): case 'toggle': void toggleLocalEngine(engine, payload.engineType) break + case 'setServedModel': + // vLLM serves one model per process, so the model is a start-time + // setting rather than a model operation. engine:set-model persists it + // as a manifest override — the same mechanism engine:set-port uses — + // and restarts a running engine onto it. Local only: the backend + // exposes no remote served-model control. + supervisor.sendProcess( + 'broker', + 'engine:set-model', + { engine, model: payload.model ?? '' }, + failAction('set the served model', { + nodeId: payload.nodeId, + engineType: payload.engineType, + modelName: payload.model + }), + true + ) + break case 'setPorts': // Both ports persist on this node: the engine HTTP server port via // engine:set-port (manifest override), the proxy port via the broker's diff --git a/desktop/src/electron/service-bridge/modular-state.ts b/desktop/src/electron/service-bridge/modular-state.ts index ab6c5fb5..cd15ef83 100644 --- a/desktop/src/electron/service-bridge/modular-state.ts +++ b/desktop/src/electron/service-bridge/modular-state.ts @@ -40,13 +40,23 @@ type BrokerNodeSource = ProxyNodeSource | 'broker' * Engines surfaced by the broker's proxy plane. Other engine-manager engines * are not currently routed across nodes. */ -export type ProxyEngine = Extract -export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] +export type ProxyEngine = Extract +export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio', 'vllm'] -/** Map a proxy node source onto the engine it describes. */ -const PROXY_SOURCE_ENGINE: Record = { - 'ollama-proxy': 'ollama', - 'lmstudio-proxy': 'lm-studio' +/** + * The engines each proxy fronts. `lmstudio-proxy` is the OpenAI-compatible + * router and serves every engine that speaks that API, so a node it reports is + * not necessarily LM Studio — which engine it actually runs comes from the + * node's own per-engine model attribution (see enginesOnProxyNode). + */ +const PROXY_SOURCE_ENGINES: Record = { + 'ollama-proxy': ['ollama'], + 'lmstudio-proxy': ['lm-studio', 'vllm'] +} + +/** The proxy that fronts an engine, and so the node source it reports under. */ +function proxySourceForEngine(engine: ProxyEngine): ProxyNodeSource { + return engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' } /** Per-engine presence on a node — each proxy reports its own engine. */ @@ -164,7 +174,7 @@ function emptyPresence(): EnginePresence { } function emptyEngines(): Record { - return { ollama: emptyPresence(), 'lm-studio': emptyPresence() } + return { ollama: emptyPresence(), 'lm-studio': emptyPresence(), vllm: emptyPresence() } } /** Immutably set one engine's presence, preserving the other. */ @@ -175,7 +185,8 @@ function setEngine( ): Record { return { ollama: engine === 'ollama' ? presence : engines.ollama, - 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'] + 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'], + vllm: engine === 'vllm' ? presence : engines.vllm } } @@ -390,7 +401,7 @@ export function parseWorkloadsInitial(value: JsonValue | undefined): Workload[] /** True for an engine fronted by a broker-supervised reverse proxy. */ export function isProxyEngine(engine: EngineType): engine is ProxyEngine { - return engine === 'ollama' || engine === 'lm-studio' + return engine === 'ollama' || engine === 'lm-studio' || engine === 'vllm' } const PENDING_OP_IDLE_TIMEOUT_MS = 90_000 @@ -709,26 +720,54 @@ function toEngineStatus( * the engine is determined by which relay namespace it arrived on. The presence * is stamped onto that engine only; the other engine stays empty. */ -function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): ModularNode | null { +/** + * Which of a proxy's engines a reported node actually runs. + * + * A proxy that fronts one engine answers for itself — the source names it. The + * OpenAI proxy fronts several, so a node it reports is not necessarily LM + * Studio: it stamps every routed node with per-engine model attribution keyed by + * engine-manager name, and that is what says whether the node runs LM Studio, + * vLLM, or both. A payload with no attribution names no engine, so callers that + * must clear presence use the proxy's whole engine set instead. + */ +function enginesOnProxyNode(source: ProxyNodeSource, params: JsonValue | undefined): ProxyEngine[] { + const fronted = PROXY_SOURCE_ENGINES[source] + if (fronted.length === 1) return [...fronted] + const byEngine = objectValue(objectValue(params)?.modelsByEngine) + if (!byEngine) return [] + // Presence of the key, not a non-empty list: an engine that is running with + // no models reports an empty (JSON null) list and is still present. + return fronted.filter(engine => Object.hasOwn(byEngine, proxyEngineToManagerName(engine))) +} + +function parseProxyNode( + params: JsonValue | undefined, + source: ProxyNodeSource, + engines: readonly ProxyEngine[] +): ModularNode | null { const obj = objectValue(params) if (!obj) return null // The proxy keys `Node.ID` by the stable per-host UUID (and rejects empty), // so this is already the canonical node key — no TXT parsing needed. const id = stringValue(obj.id) if (!id) return null - const engines = emptyEngines() - engines[engine] = { + const presence: EnginePresence = { up: true, // Under secure inference this is the peer's promoted inference proxy - // port, not the engine's own (loopback-private) port. + // port, not the engine's own (loopback-private) port. Every engine a + // proxy fronts shares that one port. port: numberValue(obj.port), // The proxy `Node` carries no version field; engine version comes from // nvpair-engine-manager, not discovery. version: null } + let nodeEngines = emptyEngines() + for (const engine of engines) { + nodeEngines = setEngine(nodeEngines, engine, presence) + } return { id, - sources: [engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'], + sources: [source], // `Node.Host` is the hostname; empty for the self-bridge manual node, // in which case the broker discovery entry supplies the display name on // merge (see mergeNode). Never fall back to the UUID id here. @@ -760,7 +799,7 @@ function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): Mod gpus: [], cpu: null, memory: null, - engines, + engines: nodeEngines, lastSeen: Date.now() } } @@ -900,7 +939,7 @@ class ModularBridgeState { // Per-engine bound proxy port reported by the broker. 0 = not reported yet; // we never fabricate a default — an unknown port surfaces as null, not a // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`. - private proxyPorts: Record = { ollama: 0, 'lm-studio': 0 } + private proxyPorts: Record = { ollama: 0, 'lm-studio': 0, vllm: 0 } private selfId: string | null = null /** * Authoritative local-engine facts from `nvpair-engine-manager`, keyed by @@ -910,7 +949,7 @@ class ModularBridgeState { */ private engineManagerFacts = new Map< EngineType, - { installed: boolean; running: boolean; port: number } + { installed: boolean; running: boolean; port: number; servedModel: string } >() /** * Local model lists pulled from `nvpair-engine-manager`'s `list_models` action by @@ -1375,7 +1414,10 @@ class ModularBridgeState { this.engineManagerFacts.set(engineType, { installed: booleanValue(obj.installed), running: booleanValue(obj.running), - port: numberValue(obj.port) + port: numberValue(obj.port), + // Empty for every engine that does not serve one model per process, + // and for one that does but has no model chosen yet. + servedModel: stringValue(obj.model) }) // A fresh authoritative state is the resolution of whatever op was in // flight (start/stop done, install `done`+installed, uninstall removed). @@ -2073,7 +2115,8 @@ class ModularBridgeState { nodeId, processStatus: pending, enginePort: facts && facts.running && facts.port > 0 ? facts.port : null, - proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null + proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null, + servedModel: facts?.servedModel } } @@ -2096,7 +2139,10 @@ class ModularBridgeState { // Each proxy-fronted engine has its own broker proxy // (`ollama-proxy` / `lmstudio-proxy`); report that engine's bound // proxy port. Loopback-only engines get null. - proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null + proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null, + // The engine's configured served model, for an engine that runs + // one model per process. Empty when none is chosen. + servedModel: facts.servedModel } } @@ -2280,11 +2326,11 @@ class ModularBridgeState { handleNotification(notification: JsonRpcNotification): void { if (notification.source === 'proxy') { - this.handleProxyNotification(notification, 'ollama') + this.handleProxyNotification(notification, 'ollama-proxy') return } if (notification.source === 'lmstudio-proxy') { - this.handleProxyNotification(notification, 'lm-studio') + this.handleProxyNotification(notification, 'lmstudio-proxy') return } if (notification.source === 'broker') { @@ -2292,28 +2338,36 @@ class ModularBridgeState { } } - private handleProxyNotification(notification: JsonRpcNotification, engine: ProxyEngine): void { + private handleProxyNotification( + notification: JsonRpcNotification, + source: ProxyNodeSource + ): void { + const fronted = PROXY_SOURCE_ENGINES[source] if (notification.method === 'ready') { const params = objectValue(notification.params) // Trust the broker-reported port only. If `ready` carries no port we // keep the last known value (0 = unknown) rather than guessing. const nextPort = numberValue(params?.port) if (nextPort <= 0) return - const changed = nextPort !== this.proxyPorts[engine] - this.proxyPorts[engine] = nextPort - // A runtime proxy:set-port (or a broker steer onto a free port) - // re-emits `ready` with the new port. Push a fresh status for that - // engine so the Edit Node proxy port reflects the actually-bound value - // immediately. Only on a real change so the startup baseline `ready` - // (and redundant re-readies) stay quiet. - if (changed) this.emitLocalEngineStatus(engine) + // Every engine this proxy fronts is reached through the one listener, + // so they all move together. + for (const engine of fronted) { + const changed = nextPort !== this.proxyPorts[engine] + this.proxyPorts[engine] = nextPort + // A runtime proxy:set-port (or a broker steer onto a free port) + // re-emits `ready` with the new port. Push a fresh status for that + // engine so the Edit Node proxy port reflects the actually-bound + // value immediately. Only on a real change so the startup baseline + // `ready` (and redundant re-readies) stay quiet. + if (changed) this.emitLocalEngineStatus(engine) + } return } if (notification.method === 'error') { const params = objectValue(notification.params) this.upsertError({ - id: `${engine}-proxy:${Date.now()}`, + id: `${source}:${Date.now()}`, message: stringValue(params?.message) || 'Modular proxy failed', timestamp: Date.now(), severity: 'error', @@ -2323,16 +2377,23 @@ class ModularBridgeState { } if (notification.method === 'node/removed') { - const node = parseProxyNode(notification.params, engine) + // A removal payload carries the node id alone, so it names no engine. + // The proxy only sends one once its last engine entry for that node is + // gone, so clearing every engine it fronts is the accurate reading. + const node = parseProxyNode(notification.params, source, fronted) if (!node) return - this.clearNodeEngine(node.id, engine) + for (const engine of fronted) { + this.clearNodeEngine(node.id, engine) + } return } if (notification.method === 'node/discovered' || notification.method === 'node/updated') { - const node = parseProxyNode(notification.params, engine) + const engines = enginesOnProxyNode(source, notification.params) + if (engines.length === 0) return + const node = parseProxyNode(notification.params, source, engines) if (!node) return - this.upsertNode(node, engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy') + this.upsertNode(node, source) } } @@ -2344,7 +2405,7 @@ class ModularBridgeState { private clearNodeEngine(nodeId: string, engine: ProxyEngine): void { const existing = this.nodes.get(nodeId) if (!existing) return - const source: BrokerNodeSource = engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' + const source: BrokerNodeSource = proxySourceForEngine(engine) const sources = removeSource(existing.sources, source) if (sources.length === 0 && !existing.nodeInfoUp) { this.removeNodeEntry(nodeId) @@ -2592,9 +2653,16 @@ class ModularBridgeState { } } - // A proxy source (ollama-proxy / lmstudio-proxy): refresh only that - // engine's presence; keep the other engine, telemetry, and node-info. - const engine = PROXY_SOURCE_ENGINE[source] + // A proxy source (ollama-proxy / lmstudio-proxy): refresh only the + // engines that proxy fronts; keep the others, telemetry, and node-info. + // The OpenAI proxy fronts more than one, and its event is authoritative + // for all of them — a node that dropped vLLM but kept LM Studio reports + // exactly that, so both presences come from the incoming node. + const fronted = PROXY_SOURCE_ENGINES[source] + const mergedEngines = fronted.reduce( + (acc, engine) => setEngine(acc, engine, next.engines[engine]), + existing.engines + ) return { ...next, sources: mergeSources(existing.sources, source), @@ -2623,7 +2691,7 @@ class ModularBridgeState { models: existing.models, modelsByEngine: existing.modelsByEngine, loadedByEngine: existing.loadedByEngine, - engines: setEngine(existing.engines, engine, next.engines[engine]), + engines: mergedEngines, lastSeen: Math.max(existing.lastSeen, next.lastSeen) } } diff --git a/desktop/src/electron/service-bridge/modular-supervisor.ts b/desktop/src/electron/service-bridge/modular-supervisor.ts index 943d0f84..00a7b37a 100644 --- a/desktop/src/electron/service-bridge/modular-supervisor.ts +++ b/desktop/src/electron/service-bridge/modular-supervisor.ts @@ -299,14 +299,31 @@ function engineManagerId(engine: ProxyEngine): string { function proxyEngineFromManagerId(id: string): ProxyEngine | null { if (id === 'ollama') return 'ollama' if (id === 'lmstudio') return 'lm-studio' + if (id === 'vllm') return 'vllm' return null } -/** The broker relay namespace fronting an engine's reverse proxy. */ +/** + * The broker relay namespace fronting an engine's reverse proxy. Every + * OpenAI-compatible engine shares one router, so they share one namespace; the + * `lmstudio-proxy` spelling is a wire contract and keeps its historical name. + */ function proxyRelayPrefix(engine: ProxyEngine): string { return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy' } +/** The bridge notification source for an engine's proxy. */ +function proxyNotificationSource(engine: ProxyEngine): 'proxy' | 'lmstudio-proxy' { + return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy' +} + +/** + * One engine per supervised proxy process, for the sweeps that are a property of + * the proxy rather than of an engine (status polling, node hydration). Iterating + * PROXY_ENGINES there would query the OpenAI proxy once per engine it fronts. + */ +const PROXY_REPRESENTATIVE_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] + /** * Spawns and supervises the modular backend. * @@ -891,7 +908,7 @@ class ModularSupervisor { await this.seedClusterPeerIds() await this.refreshAllRemoteEngineStatus() this.startRemoteStatusPolling() - for (const engine of PROXY_ENGINES) { + for (const engine of PROXY_REPRESENTATIVE_ENGINES) { await this.hydrateProxyNodes(engine) await this.pollProxyStatus(engine) } @@ -1079,7 +1096,7 @@ class ModularSupervisor { const obj = objectValue(result) if (obj && booleanValue(obj.ready)) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyNotificationSource(engine), method: 'ready', params: { port: numberValue(obj.port) } }) @@ -1100,7 +1117,7 @@ class ModularSupervisor { if (!obj || !Array.isArray(obj.nodes)) return for (const node of obj.nodes) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyNotificationSource(engine), method: 'node/discovered', params: node }) @@ -1266,19 +1283,23 @@ class ModularSupervisor { this.scheduleRemoteEngineStatusRefresh() } - const proxyEngine: ProxyEngine | null = + // Every engine the (re)bound proxy fronts loses its manual-node set at + // once, since they share the process. + const reboundEngines: readonly ProxyEngine[] = event.source === 'proxy' - ? 'ollama' + ? ['ollama'] : event.source === 'lmstudio-proxy' - ? 'lm-studio' - : null - if (proxyEngine && event.method === 'ready') { + ? PROXY_ENGINES.filter(engine => engine !== 'ollama') + : [] + if (reboundEngines.length > 0 && event.method === 'ready') { // A (re)bound proxy starts with an empty manual-node set, so forget // what we think we bridged and re-push the local node if applicable. - const bridge = this.getLocalBridge(proxyEngine) - bridge.bridgedId = '' - bridge.bridgedPort = 0 - void this.reconcileLocalNodeBridge(proxyEngine) + for (const proxyEngine of reboundEngines) { + const bridge = this.getLocalBridge(proxyEngine) + bridge.bridgedId = '' + bridge.bridgedPort = 0 + void this.reconcileLocalNodeBridge(proxyEngine) + } this.emitStateRefreshIfHydrated() } if (event.source === 'broker' && event.method === 'app:ready') { @@ -2188,6 +2209,9 @@ class ModularSupervisor { try { await this.callProxy(engine, 'node/add-manual', { id: selfId, + // The OpenAI proxy fronts several engines on different local + // ports, so a manual entry names the engine it belongs to. + engine: engineManagerId(engine), host: '127.0.0.1', port: bridge.port, addresses: ['127.0.0.1'] @@ -2208,7 +2232,10 @@ class ModularSupervisor { bridge.bridgedId = '' bridge.bridgedPort = 0 try { - await this.callProxy(engine, 'node/remove-manual', { id: previousId }) + await this.callProxy(engine, 'node/remove-manual', { + id: previousId, + engine: engineManagerId(engine) + }) } catch (err) { log.verbose({ sublevel: proxyRelayPrefix(engine), diff --git a/desktop/src/shared/constants/engines.ts b/desktop/src/shared/constants/engines.ts index 3a1472c8..23ac9d00 100644 --- a/desktop/src/shared/constants/engines.ts +++ b/desktop/src/shared/constants/engines.ts @@ -9,23 +9,28 @@ import { EngineType, ModelExpiry } from '@/shared/types/engines' // never-enabled placeholders; they were removed with the chat window, which was // their only in-app consumer. Adding an engine back means shipping its manifest // first -- an engine row without one renders commands that fail with `-32000`. -export const EngineTypes = ['ollama', 'lm-studio'] as const +export const EngineTypes = ['ollama', 'lm-studio', 'vllm'] as const // Kept as a distinct export so a future engine can ship behind it rather than // appearing the moment its type exists. -export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio'] as const +export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio', 'vllm'] as const export const EngineSources = ['bundled', 'detected', 'installed'] as const export const EngineDisplayNames: Record = { ollama: 'Ollama', - 'lm-studio': 'LM Studio' + 'lm-studio': 'LM Studio', + vllm: 'vLLM' } as const /** Default docs/install URLs for built-in backends. Single source of truth for UI and adapter buildInfo(). */ export const EngineDefaultLinks: Record = { ollama: { docsUrl: 'https://docs.ollama.com/', installUrl: 'https://ollama.com/download' }, - 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' } + 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' }, + vllm: { + docsUrl: 'https://docs.vllm.ai/', + installUrl: 'https://docs.vllm.ai/en/latest/getting_started/installation/' + } } as const export const ModelItemStatuses = ['idle', 'loading', 'loaded', 'ejecting', 'pulling'] as const diff --git a/desktop/src/shared/types/engine-api.ts b/desktop/src/shared/types/engine-api.ts index 351f5c5b..12ca5c21 100644 --- a/desktop/src/shared/types/engine-api.ts +++ b/desktop/src/shared/types/engine-api.ts @@ -50,12 +50,19 @@ export type EngineCommandType = | 'unloadModel' | 'deleteModel' | 'setModelExpiry' + | 'setServedModel' /** Payload for engine commands sent from the UI. */ export interface EngineCommandPayload { command: EngineCommandType engineType: EngineType nodeId: string + /** + * The model an operation acts on. For `setServedModel` it is the model the + * engine should serve from its next start; an empty string clears the + * choice, leaving an engine that requires one unable to start until it is + * set again. + */ model?: string /** * `setPorts` only. The engine HTTP server port to apply. Omitted when the diff --git a/desktop/src/shared/types/engines.ts b/desktop/src/shared/types/engines.ts index 54a228b4..d39088d0 100644 --- a/desktop/src/shared/types/engines.ts +++ b/desktop/src/shared/types/engines.ts @@ -40,6 +40,12 @@ export interface EngineStatusData { * reported version data or engines that are not installed. */ installedVersion?: string + /** + * The model this engine is configured to serve, for an engine that runs one + * model per process (vLLM). Empty string when none is chosen — the state in + * which the engine refuses to start. Undefined for every other engine. + */ + servedModel?: string } export type ModelItemStatus = (typeof ModelItemStatuses)[number] diff --git a/desktop/src/ui/api/engine-api.ts b/desktop/src/ui/api/engine-api.ts index 69fd71a2..2a6fc86a 100644 --- a/desktop/src/ui/api/engine-api.ts +++ b/desktop/src/ui/api/engine-api.ts @@ -47,6 +47,12 @@ export interface IEngineApi { deleteModel(engineType: EngineType, nodeId: string, model: string): void /** Set the model keep-alive expiry duration on a node. */ setModelExpiry(engineType: EngineType, nodeId: string, model: string, expiry: string): void + /** + * Choose the model an engine serves. For an engine that runs one model per + * process (vLLM), the choice is persisted and the engine restarts onto it; + * an empty model clears the choice. Local node only. + */ + setServedModel(engineType: EngineType, nodeId: string, model: string): void /** Search the model registry/hub for available models. */ searchHub(engineType: EngineType): Promise @@ -86,6 +92,8 @@ export function createEngineApi(transport: ServiceTransport): IEngineApi { enginePort: ports.enginePort, proxyPort: ports.proxyPort }), + setServedModel: (engineType, nodeId, model) => + fireCommand(transport, { command: 'setServedModel', engineType, nodeId, model }), pullModel: (engineType, nodeId, model) => fireCommand(transport, { command: 'pullModel', engineType, nodeId, model }), loadModel: (engineType, nodeId, model) => diff --git a/desktop/src/ui/components/BackendRow/BackendRow.tsx b/desktop/src/ui/components/BackendRow/BackendRow.tsx index a1b399df..58357702 100644 --- a/desktop/src/ui/components/BackendRow/BackendRow.tsx +++ b/desktop/src/ui/components/BackendRow/BackendRow.tsx @@ -71,7 +71,8 @@ export function BackendRow({ }) { const [edit, setEdit] = useState({ serverPort: String(backend.port ?? ''), - proxyPort: String(backend.proxyPort ?? '') + proxyPort: String(backend.proxyPort ?? ''), + servedModel: backend.servedModel ?? '' }) const addLocalError = useErrorsStore(state => state.addLocalError) @@ -84,6 +85,10 @@ export function BackendRow({ setEdit(prev => ({ ...prev, proxyPort: String(backend.proxyPort ?? '') })) }, [backend.proxyPort]) + useEffect(() => { + setEdit(prev => ({ ...prev, servedModel: backend.servedModel ?? '' })) + }, [backend.servedModel]) + const [expanded, setExpanded] = useState(false) const [confirmUninstall, setConfirmUninstall] = useState(false) const [confirmPorts, setConfirmPorts] = useState(false) @@ -102,6 +107,11 @@ export function BackendRow({ const portsChanged = serverPortChanged || proxyPortChanged + const servedModelChanged = useMemo( + () => caps.hasServedModel && edit.servedModel.trim() !== (backend.servedModel ?? ''), + [caps.hasServedModel, edit.servedModel, backend.servedModel] + ) + const installProgress = useEngineProgressStore(s => { const installKey = engineProgressKey({ nodeId, @@ -198,9 +208,18 @@ export function BackendRow({ const resetPortsToBackend = useCallback(() => { setEdit({ serverPort: String(backend.port ?? ''), - proxyPort: String(backend.proxyPort ?? '') + proxyPort: String(backend.proxyPort ?? ''), + servedModel: backend.servedModel ?? '' }) - }, [backend.port, backend.proxyPort]) + }, [backend.port, backend.proxyPort, backend.servedModel]) + + // Changing the served model restarts a running engine onto it, exactly as a + // port change does; the engine reports the applied value back, which resets + // the draft through the effect above. + const handleApplyServedModel = useCallback(() => { + if (!servedModelChanged) return + window.pairApi.engines.setServedModel(backend.type, nodeId, edit.servedModel.trim()) + }, [servedModelChanged, backend.type, nodeId, edit.servedModel]) const validateAndConfirmPorts = useCallback(() => { if (!serverPortChanged && !proxyPortChanged) return @@ -288,14 +307,17 @@ export function BackendRow({ )} - {canShowAccordions && (caps.hasEnginePort || edit.proxyPort) && ( + {canShowAccordions && (caps.hasEnginePort || caps.hasServedModel || edit.proxyPort) && ( setEdit(prev => ({ ...prev, servedModel: v }))} onServerChange={v => setEdit(prev => ({ ...prev, serverPort: v }))} onProxyChange={v => setEdit(prev => ({ ...prev, proxyPort: v }))} /> diff --git a/desktop/src/ui/components/BackendRow/PortsSection.tsx b/desktop/src/ui/components/BackendRow/PortsSection.tsx index 353ac45d..94e962c5 100644 --- a/desktop/src/ui/components/BackendRow/PortsSection.tsx +++ b/desktop/src/ui/components/BackendRow/PortsSection.tsx @@ -3,32 +3,43 @@ import { Text } from '@nvidia/foundations-react-core' import { BackendPorts } from './BackendPorts' +import { ServedModelRow } from './ServedModelRow' import { EditState } from '@/ui/types/engine-edit-state' import type { EngineCaps } from '@/ui/types/engine-manifest' export function PortsSection({ edit, portsChanged, + servedModelChanged, anyLoading, isLocalNode, caps, onApplyPorts, + onApplyServedModel, onServerChange, - onProxyChange + onProxyChange, + onServedModelChange }: { edit: EditState portsChanged: boolean + servedModelChanged: boolean anyLoading: boolean isLocalNode: boolean caps: EngineCaps onApplyPorts: () => void + onApplyServedModel: () => void onServerChange: (v: string) => void onProxyChange: (v: string) => void + onServedModelChange: (v: string) => void }) { + // The served model is a local-node control: the engine manager only writes + // manifest overrides for its own host, and there is no remote equivalent, so + // a remote row would be a field that can never hold a value. + const showServedModel = caps.hasServedModel && isLocalNode return (
- Ports + {showServedModel ? 'Ports and model' : 'Ports'}
+ {/* An engine that serves one model per process is told which model + before it starts, so the choice belongs with the other + start-time settings rather than in the model list. */} + {showServedModel && ( + + )}
) diff --git a/desktop/src/ui/components/BackendRow/ServedModelRow.tsx b/desktop/src/ui/components/BackendRow/ServedModelRow.tsx new file mode 100644 index 00000000..f9e47d71 --- /dev/null +++ b/desktop/src/ui/components/BackendRow/ServedModelRow.tsx @@ -0,0 +1,69 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { Button, Flex, Stack, Text, TextInput } from '@nvidia/foundations-react-core' +import { Check } from '@/ui/components/icons' + +/** + * The model an engine serves, for an engine that runs one model per process + * (vLLM). It is a start-time setting, not a model operation: the engine manager + * persists it and restarts the engine onto it, and the engine downloads the + * weights itself on that first start. The local node only — the engine manager + * writes manifest overrides for its own host and the backend exposes no remote + * served-model control, so `PortsSection` does not render this for a peer. + */ +export function ServedModelRow({ + model, + changed, + disabled, + onChange, + onApply +}: { + model: string + /** True when the draft differs from the engine-reported value. */ + changed: boolean + disabled: boolean + onChange: (value: string) => void + onApply: () => void +}) { + return ( + + + Model to serve + + A Hugging Face model id, for example Qwen/Qwen3-8B. The engine serves this one + model and downloads it on the next start, which can take several minutes. + + + + + + + + ) +} diff --git a/desktop/src/ui/components/EngineIcon.tsx b/desktop/src/ui/components/EngineIcon.tsx index 4ee81367..86ea9fe4 100644 --- a/desktop/src/ui/components/EngineIcon.tsx +++ b/desktop/src/ui/components/EngineIcon.tsx @@ -39,5 +39,35 @@ export default function EngineIcon({ type, size = 32 }: { type: EngineType; size ) } + // vLLM's own logo is not redistributable here, so its tile is drawn rather + // than shipped: a wordmark on the project's colours, in the same rounded + // square the other engines use so the row stays visually even. + if (type === 'vllm') { + return ( +
+ + + vL + + +
+ ) + } + return null } diff --git a/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx b/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx index 7581be68..e7e8f12d 100644 --- a/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx +++ b/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx @@ -98,6 +98,7 @@ export default function NodeEngineSettings({ nodeId }: { nodeId: string }) { docsUrl: EngineDefaultLinks[type]?.docsUrl, installUrl: EngineDefaultLinks[type]?.installUrl, installedVersion: status.installedVersion, + servedModel: status.servedModel, updateAvailable }, statusKnown diff --git a/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx b/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx index 3460c4c2..fb7c30dc 100644 --- a/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx +++ b/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx @@ -16,8 +16,10 @@ function wipeMessage() { engine installations under the app data folder. - Third-party model libraries (for example ~/.ollama and{' '} - ~/.lmstudio) are not deleted. + Third-party model libraries and engine installations outside the app data folder + (for example ~/.ollama, ~/.lmstudio, a vLLM environment at{' '} + ~/.vllm, and the Hugging Face cache at{' '} + ~/.cache/huggingface) are not deleted. ) diff --git a/desktop/src/ui/constants/engine-capabilities.ts b/desktop/src/ui/constants/engine-capabilities.ts index 77e6f2bf..c7092650 100644 --- a/desktop/src/ui/constants/engine-capabilities.ts +++ b/desktop/src/ui/constants/engine-capabilities.ts @@ -20,6 +20,7 @@ export const EngineCapabilities: Record = { hasModelSearchOnlyWhenRunning: true, modelOpsWhenStopped: false, hasDeleteModel: true, + hasServedModel: false, engineHub: { label: 'Ollama', url: 'https://ollama.com/library' } }, 'lm-studio': { @@ -42,6 +43,28 @@ export const EngineCapabilities: Record = { // exposes no rescan, so nvpair-engine-manager's delete_model restarts the // server. Deleting therefore interrupts inference and needs a warning. restartsOnModelDelete: true, + hasServedModel: false, engineHub: { label: 'LM Studio', url: 'https://lmstudio.ai/models' } + }, + vllm: { + hasExpiry: false, + // vLLM keeps its one served model resident for the life of the process. + // There is nothing to eject short of stopping the engine. + hasEject: false, + // vLLM publishes no Windows build and no macOS GPU build, so PAIR only + // offers to install it on Linux. Its manifest ships Linux platforms only, + // so the engine reports unavailable everywhere else. + hasInstall: ['linux'], + hasEnginePort: true, + hasInstallPath: false, + hasProxyWebUI: false, + hasPreferredNode: false, + hasCrashAlert: false, + hasModelSearchOnlyWhenRunning: true, + // vLLM has no model-management surface: weights are fetched by the engine + // itself when it starts, from the model id configured below. + modelOpsWhenStopped: false, + hasDeleteModel: false, + hasServedModel: true } } diff --git a/desktop/src/ui/constants/welcome.ts b/desktop/src/ui/constants/welcome.ts index 4f9fa04a..4bf18f0d 100644 --- a/desktop/src/ui/constants/welcome.ts +++ b/desktop/src/ui/constants/welcome.ts @@ -13,7 +13,10 @@ export const WELCOME_STEP_SUB_HEADINGS = ['', 'You can update later by clicking export const WELCOME_ENGINE_DEFAULT_SELECTED: Record = { ollama: true, - 'lm-studio': true + 'lm-studio': true, + // vLLM downloads a multi-gigabyte CUDA stack and then a model, and it runs + // on Linux only. Offer it, but never pre-select it during onboarding. + vllm: false } export function getWelcomeEngineCandidates(os: PlatformDisplayName): EngineType[] { diff --git a/desktop/src/ui/types/engine-edit-state.ts b/desktop/src/ui/types/engine-edit-state.ts index 67229c40..d96fd892 100644 --- a/desktop/src/ui/types/engine-edit-state.ts +++ b/desktop/src/ui/types/engine-edit-state.ts @@ -4,4 +4,9 @@ export type EditState = { serverPort: string proxyPort: string + /** + * Draft of the model an engine should serve, for an engine that runs one + * model per process. Empty clears the choice. + */ + servedModel: string } diff --git a/desktop/src/ui/types/engine-info.ts b/desktop/src/ui/types/engine-info.ts index 3c4cb780..6e4a5b02 100644 --- a/desktop/src/ui/types/engine-info.ts +++ b/desktop/src/ui/types/engine-info.ts @@ -95,6 +95,11 @@ export interface BackendInfo { proxyPort: number | null /** Installed engine binary version reported by the owning node */ installedVersion?: string + /** + * The model this engine is configured to serve, for an engine that runs one + * model per process (vLLM). Empty string when none is chosen. + */ + servedModel?: string /** Models on this backend with per-model status */ models: ModelItem[] /** System-level dependencies required before install/run (local node only) */ diff --git a/desktop/src/ui/types/engine-manifest.ts b/desktop/src/ui/types/engine-manifest.ts index 1cd998bf..5a9fcc56 100644 --- a/desktop/src/ui/types/engine-manifest.ts +++ b/desktop/src/ui/types/engine-manifest.ts @@ -36,6 +36,12 @@ export interface EngineCaps { modelOpsWhenStopped: boolean /** When true, show the Delete action in the model action menu. */ hasDeleteModel: boolean + /** + * When true, the engine serves exactly one model per process and the model is + * chosen before it starts, so Engine settings offers a "Model to serve" field. + * The engine manager persists the choice and restarts the engine onto it. + */ + hasServedModel: boolean /** * When true, the engine restarts as part of deleting a model, so Delete asks * for confirmation first. The engine manager owns the restart; this flag only diff --git a/desktop/src/ui/utils/format-model-display-name.ts b/desktop/src/ui/utils/format-model-display-name.ts index e16282ed..ace46540 100644 --- a/desktop/src/ui/utils/format-model-display-name.ts +++ b/desktop/src/ui/utils/format-model-display-name.ts @@ -45,6 +45,9 @@ export function formatModelDisplayName(name: string, engineType?: string | null) case 'ollama': return formatOllamaModelName(formatted) + // vLLM serves Hugging Face repo ids verbatim (Qwen/Qwen3-8B), which the + // shared Hugging Face formatter already renders; it needs no + // engine-specific rules of its own. default: if (formatted.includes('/')) { return formatHuggingFaceModelName(formatted) diff --git a/desktop/src/ui/utils/match-downloaded-model.ts b/desktop/src/ui/utils/match-downloaded-model.ts index 2edf8361..20ed5de2 100644 --- a/desktop/src/ui/utils/match-downloaded-model.ts +++ b/desktop/src/ui/utils/match-downloaded-model.ts @@ -39,6 +39,8 @@ const matchHfPullKeyOrName: DownloadedMatcher = (hubEntry, d) => { const MATCHERS: Partial> = { ollama: matchOllama, 'lm-studio': matchHfPullKeyOrName + // vLLM declares no engineHub, so no hub row is ever rendered for it and it + // needs no matcher. Adding one here without a hub would be dead code. } export function isHubEntryDownloaded( diff --git a/desktop/tests/modular/delete-model-restart.test.ts b/desktop/tests/modular/delete-model-restart.test.ts index 3563c5ee..3fea1fea 100644 --- a/desktop/tests/modular/delete-model-restart.test.ts +++ b/desktop/tests/modular/delete-model-restart.test.ts @@ -27,7 +27,8 @@ const MANIFEST_DIR = path.resolve(process.cwd(), '../services/nvpair-engine-mana /** Manifest engine ids differ from our `EngineType` for LM Studio only. */ const ENGINE_TYPE_BY_MANIFEST_ID: Record = { ollama: 'ollama', - lmstudio: 'lm-studio' + lmstudio: 'lm-studio', + vllm: 'vllm' } interface ManifestAction { diff --git a/desktop/tests/modular/lmstudio-stale-model.test.ts b/desktop/tests/modular/lmstudio-stale-model.test.ts index 8b37e862..b722abab 100644 --- a/desktop/tests/modular/lmstudio-stale-model.test.ts +++ b/desktop/tests/modular/lmstudio-stale-model.test.ts @@ -91,7 +91,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -202,7 +202,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -264,7 +264,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -323,7 +323,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -414,7 +414,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', diff --git a/desktop/tests/modular/node-address-rerank.test.ts b/desktop/tests/modular/node-address-rerank.test.ts index 454df5e1..1a042ca5 100644 --- a/desktop/tests/modular/node-address-rerank.test.ts +++ b/desktop/tests/modular/node-address-rerank.test.ts @@ -101,7 +101,8 @@ describe('broker address re-rank', () => { host: 'proxy-refresh-host', port: 1234, addresses: ['203.0.113.7'], - ip: '192.0.2.201' + ip: '192.0.2.201', + modelsByEngine: { lmstudio: [] } } }) diff --git a/desktop/tests/modular/openai-proxy-engines.test.ts b/desktop/tests/modular/openai-proxy-engines.test.ts new file mode 100644 index 00000000..06d27117 --- /dev/null +++ b/desktop/tests/modular/openai-proxy-engines.test.ts @@ -0,0 +1,110 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => ({ BrowserWindow: { getAllWindows: () => [] } })) +vi.mock('@/electron/window', () => ({ createOverviewWindow: vi.fn() })) + +import { getModularBridgeState } from '@/electron/service-bridge/modular-state' + +/** + * The OpenAI-compatible proxy fronts more than one engine, so a node it reports + * is no longer necessarily LM Studio. The node's own per-engine model + * attribution says which engine it runs; reading the proxy's name instead would + * label every vLLM node as LM Studio. + */ +function proxyNode(id: string, modelsByEngine: Record) { + return { + id, + host: id, + port: 1234, + addresses: ['192.0.2.60'], + ip: '192.0.2.60', + modelsByEngine + } +} + +describe('OpenAI proxy engine attribution', () => { + // The bridge state is a process singleton, so each case uses its own node + // ids rather than trying to reset it. + const state = getModularBridgeState() + + beforeEach(() => { + state.setSelfId('openai-proxy-engines-local') + }) + + it('labels a vLLM-only peer as vLLM, not LM Studio', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('vllm-peer', { vllm: ['Qwen/Qwen3-8B'] }) + }) + expect(state.isRemoteEngineRunning('vllm-peer', 'vllm')).toBe(true) + expect(state.isRemoteEngineRunning('vllm-peer', 'lm-studio')).toBe(false) + }) + + it('reports both engines for a peer running LM Studio and vLLM', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('dual-peer', { lmstudio: ['qwen2.5-7b'], vllm: ['Qwen/Qwen3-8B'] }) + }) + expect(state.isRemoteEngineRunning('dual-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('dual-peer', 'vllm')).toBe(true) + }) + + it('counts an engine that is running with no models as present', () => { + // The proxy reports a key with an empty (JSON null) list for an engine + // that is up but holds nothing, which is not the same as absent. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('empty-peer', { vllm: [] }) + }) + expect(state.isRemoteEngineRunning('empty-peer', 'vllm')).toBe(true) + }) + + it('drops an engine the node stopped advertising', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('shrinking-peer', { lmstudio: ['a'], vllm: ['b'] }) + }) + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/updated', + params: proxyNode('shrinking-peer', { lmstudio: ['a'] }) + }) + expect(state.isRemoteEngineRunning('shrinking-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('shrinking-peer', 'vllm')).toBe(false) + }) + + it('clears every engine the proxy fronts when the node is removed', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('leaving-peer', { lmstudio: ['a'], vllm: ['b'] }) + }) + // A removal payload carries the node id alone; the proxy sends it only + // once its last engine entry for that node is gone. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/removed', + params: { id: 'leaving-peer' } + }) + expect(state.isRemoteEngineRunning('leaving-peer', 'lm-studio')).toBe(false) + expect(state.isRemoteEngineRunning('leaving-peer', 'vllm')).toBe(false) + }) + + it('needs no attribution from a proxy that fronts one engine', () => { + // ollama-proxy routes Ollama and nothing else, so its source names the + // engine and its payload carries no per-engine map. + state.handleNotification({ + source: 'proxy', + method: 'node/discovered', + params: { id: 'ollama-peer', host: 'ollama-peer', port: 11434, ip: '192.0.2.61' } + }) + expect(state.isRemoteEngineRunning('ollama-peer', 'ollama')).toBe(true) + }) +}) diff --git a/desktop/tests/modular/remote-engine-install-state.test.ts b/desktop/tests/modular/remote-engine-install-state.test.ts index eed8aa89..634083ed 100644 --- a/desktop/tests/modular/remote-engine-install-state.test.ts +++ b/desktop/tests/modular/remote-engine-install-state.test.ts @@ -138,7 +138,8 @@ describe('remote engine install state', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.63'], - ip: '192.0.2.63' + ip: '192.0.2.63', + modelsByEngine: { lmstudio: [] } } }) @@ -179,7 +180,8 @@ describe('remote engine install state', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.65'], - ip: '192.0.2.65' + ip: '192.0.2.65', + modelsByEngine: { lmstudio: [] } } }) diff --git a/desktop/tests/modular/remote-engine-status.test.ts b/desktop/tests/modular/remote-engine-status.test.ts index 463e1ad6..d6500612 100644 --- a/desktop/tests/modular/remote-engine-status.test.ts +++ b/desktop/tests/modular/remote-engine-status.test.ts @@ -22,7 +22,8 @@ describe('remote engine status', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.190'], - ip: '192.0.2.190' + ip: '192.0.2.190', + modelsByEngine: { lmstudio: [] } } }) expect(state.isRemoteEngineRunning(remoteNodeId, 'lm-studio')).toBe(true) diff --git a/desktop/tests/modular/uuid-node-keying.test.ts b/desktop/tests/modular/uuid-node-keying.test.ts index 5632b53f..808164ea 100644 --- a/desktop/tests/modular/uuid-node-keying.test.ts +++ b/desktop/tests/modular/uuid-node-keying.test.ts @@ -71,7 +71,8 @@ describe('UUID node keying', () => { host: 'merge-host-2', port: 1234, addresses: ['192.0.2.12'], - ip: '192.0.2.12' + ip: '192.0.2.12', + modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ diff --git a/desktop/tests/modular/vllm-engine.test.ts b/desktop/tests/modular/vllm-engine.test.ts new file mode 100644 index 00000000..f2601b3b --- /dev/null +++ b/desktop/tests/modular/vllm-engine.test.ts @@ -0,0 +1,83 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'node:fs' +import path from 'node:path' +import { describe, expect, it } from 'vitest' +import { + EngineDefaultLinks, + EngineDisplayNames, + EngineTypes, + EnabledEngineTypes +} from '@/shared/constants/engines' +import { EngineCapabilities } from '@/ui/constants/engine-capabilities' +import { WELCOME_ENGINE_DEFAULT_SELECTED, getWelcomeEngineCandidates } from '@/ui/constants/welcome' +import { isEngineType } from '@/shared/utils/engines' +import { platformDisplayName } from '@/shared/utils/platform' +import { formatModelDisplayName } from '@/ui/utils/format-model-display-name' + +const MANIFEST_DIR = path.resolve(process.cwd(), '../services/nvpair-engine-manager/manifests') + +describe('vLLM engine registration', () => { + it('is a known, enabled engine type spelled the same as its manifest', () => { + expect(EngineTypes).toContain('vllm') + expect(EnabledEngineTypes).toContain('vllm') + expect(isEngineType('vllm')).toBe(true) + // The engine-manager id and our EngineType are the same string, so every + // name-translation helper is a pass-through. LM Studio is the only engine + // whose two spellings differ. + const manifest = JSON.parse( + fs.readFileSync(path.join(MANIFEST_DIR, 'vllm.json'), 'utf8') + ) as { engine: string; display_name: string } + expect(manifest.engine).toBe('vllm') + expect(EngineDisplayNames.vllm).toBe(manifest.display_name) + }) + + it('ships the vLLM documentation and install links', () => { + expect(EngineDefaultLinks.vllm.docsUrl).toBe('https://docs.vllm.ai/') + expect(EngineDefaultLinks.vllm.installUrl).toBe( + 'https://docs.vllm.ai/en/latest/getting_started/installation/' + ) + }) + + it('declares capabilities that match what vLLM can actually do', () => { + const caps = EngineCapabilities.vllm + // Linux only: vLLM publishes no Windows build and no macOS GPU build, and + // its manifest ships Linux platform blocks only. + expect(caps.hasInstall).toEqual(['linux']) + expect(caps.hasEnginePort).toBe(true) + // One model per process, chosen before start and resident for the life of + // the process: nothing to eject, delete, or expire. + expect(caps.hasEject).toBe(false) + expect(caps.hasDeleteModel).toBe(false) + expect(caps.hasExpiry).toBe(false) + expect(caps.modelOpsWhenStopped).toBe(false) + // The served model is a start-time setting, which is what gives the + // engine settings its "Model to serve" field. + expect(caps.hasServedModel).toBe(true) + // No public catalog is wired up, so no hub source selector is offered. + expect(caps.engineHub).toBeUndefined() + }) + + it('is the only engine that declares a served model', () => { + const declaring = EngineTypes.filter(type => EngineCapabilities[type].hasServedModel) + expect(declaring).toEqual(['vllm']) + }) + + it('is offered in onboarding on Linux only, and never pre-selected', () => { + expect(WELCOME_ENGINE_DEFAULT_SELECTED.vllm).toBe(false) + expect(getWelcomeEngineCandidates(platformDisplayName('linux'))).toContain('vllm') + expect(getWelcomeEngineCandidates(platformDisplayName('darwin'))).not.toContain('vllm') + expect(getWelcomeEngineCandidates(platformDisplayName('win32'))).not.toContain('vllm') + }) + + it('renders a Hugging Face repo id as a readable model name', () => { + // vLLM's model ids are Hugging Face repo ids, which the shared formatter + // already handles — the point of this case is that the engine falls into + // that path rather than an engine-specific one. + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'vllm')).toBe( + formatModelDisplayName('Qwen/Qwen3-8B', 'lm-studio') + ) + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'vllm')).not.toContain('/') + }) +}) diff --git a/docs/architecture.mdx b/docs/architecture.mdx index f7e7ffed..69fb3834 100644 --- a/docs/architecture.mdx +++ b/docs/architecture.mdx @@ -19,7 +19,7 @@ flowchart TB NodeA["Node
one machine running PAIR"] NodeB["Node"] EngineA1["Engine
Ollama"] - EngineA2["Engine
LM Studio"] + EngineA2["Engine
LM Studio / vLLM"] EngineB1["Engine
Ollama"] ModelA1["Models
present on this engine"] ModelA2["Models"] @@ -40,7 +40,8 @@ flowchart TB one cluster. - A **node** is one machine. Nodes are peers where each runs the same services, and each can both serve requests and route them elsewhere. -- An **engine** is an inference server on a node, Ollama or LM Studio. A node can +- An **engine** is an inference server on a node: Ollama, LM Studio, or vLLM. A + node can run both, one, or neither, and a node with no running engine is not eligible to serve. - **Models** belong to an engine on a specific node. Nothing is shared. The same @@ -314,7 +315,8 @@ ordering reaches a proxy in three steps: The ranking combines *pending work and GPU pressure*. A workload counts as pending while it is queued or running, and it is attributed to the node it was -placed on. Both engines count together, so Ollama load affects LM Studio ordering +placed on. Every engine counts together, so Ollama load affects LM Studio and +vLLM ordering and vice versa. GPU pressure is deliberately coarse. The scheduler smooths the busiest GPU's @@ -357,7 +359,7 @@ across the cluster as a whole. When an inference request contains a non-empty model, only nodes whose current inventory for that engine advertises the model enter the failover list. An empty inventory and an inventory that lists other models are both ineligible. Ollama's -implicit `:latest` tag is normalized; LM Studio model IDs match exactly. +implicit `:latest` tag is normalized; LM Studio and vLLM model IDs match exactly. If no advertised owner is routable, the proxy returns an actionable local `502` without sending the request to an engine. It does not broaden the candidate list @@ -428,7 +430,8 @@ one place lower in the order. to an engine's own port is absent from workload events. GPU-heavy external work can still raise pressure, but CPU-only work and queued demand remain invisible. -**Both engines are counted as one pool.** Ollama and LM Studio load is summed, +**Every engine is counted as one pool.** Ollama, LM Studio, and vLLM load is +summed, and maximum GPU pressure applies to the whole node. That is conservative on a typical single-GPU machine and can underuse a multi-GPU node where the engines occupy different devices. @@ -462,7 +465,8 @@ port it listens on. ### Finding an Engine PAIR does not assume it installed the engine. Detection checks the manifest's -known install locations for each engine, so an Ollama or LM Studio you installed +known install locations for each engine, so an Ollama, LM Studio, or vLLM you +installed yourself is found where it already is. "Installing" an engine that is already present downloads nothing and reports it as installed. @@ -521,7 +525,8 @@ A default installation listens on these ports: | Port | Listener | | --- | --- | | `11434` | Ollama-compatible proxy (Ollama itself moves to `11435`+) | -| `1234` | OpenAI-compatible proxy (LM Studio moves to `1235`+) | +| `1234` | OpenAI-compatible proxy for LM Studio and vLLM (LM Studio moves to `1235`+) | +| `8000` | vLLM engine (adopted or PAIR-started; the proxy never takes this port) | | `14318` | Node hardware and model inventory | | `14319` | Service-error synchronization between nodes | | `14320` | Workload propagation between nodes | diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index 81bf01d9..61f52f2d 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -6,8 +6,8 @@ SPDX-License-Identifier: Apache-2.0 # Managing Engines in NVIDIA Personal AI Router An **engine** is the local inference runtime Personal AI Router (PAIR) uses to -run models. Today that means Ollama or LM Studio on a given machine. PAIR can -install and run those engines for you, or work with a copy you already have. +run models. Today that means Ollama, LM Studio, or vLLM on a given machine. PAIR +can install and run those engines for you, or work with a copy you already have. This page explains what you can expect when you install, start, stop, update, or remove an engine. @@ -74,9 +74,9 @@ When you install an engine, consider the following: configures it, and owns it from then on. - A successful install **starts the engine** as part of the same flow, so you normally do not need a separate start step right after install. -- If Ollama or LM Studio is already running on the machine, PAIR can **adopt** - that install instead of downloading another copy. Adoption helps when the - usual engine port is already in use. +- If Ollama, LM Studio, or vLLM is already running on the machine, PAIR can + **adopt** that install instead of downloading another copy. Adoption helps when + the usual engine port is already in use. To download models: @@ -128,10 +128,50 @@ controlled: the engine in its own application first if you want PAIR to manage it fully. - **LM Studio** publishes an official stop command, so PAIR can stop an adopted instance that way and restart it on the port you chose. +- **vLLM** is managed as a process, so it behaves like Ollama: PAIR will not move + an adopted one. Refer to [Engines and Ports](architecture.mdx#engines-and-ports) in the architecture guide. +## vLLM + +vLLM works differently enough from the other engines to be worth reading before +you use it. + +**Linux only.** vLLM publishes no Windows build and no macOS GPU build, so PAIR +offers it on Linux and reports it as unavailable elsewhere. + +**Adopting an instance you already run is the normal path.** Most people run +vLLM themselves, either as `vllm serve ` or from the `vllm/vllm-openai` +container image. If something is already serving on port `8000`, turning vLLM on +in **Engine settings** adopts it rather than starting a second copy, and the +engine shows as running. PAIR does not stop an instance it did not start; stop +it the way you started it. + +**One model per process.** Unlike Ollama and LM Studio, a vLLM process serves +exactly one model, named when it starts. Set it in **Engine settings** under +**Model to serve**, using a Hugging Face model id such as `Qwen/Qwen3-8B`. PAIR +persists the choice and restarts the engine onto it. Starting vLLM with no model +chosen fails with a message saying so rather than starting something that cannot +answer. + +**The first start can take many minutes.** vLLM downloads the model's weights +and compiles CUDA graphs before it serves anything, so PAIR allows up to thirty +minutes for it to become ready. The engine shows as starting throughout; later +starts of the same model are much faster. + +**There are no model operations.** vLLM has no download, load, unload, or delete +surface — the engine fetches what it needs from the model id you configured, and +keeps it resident for the life of the process. The model list simply shows what +the running engine serves. + +**Installing it takes a while and uses a lot of disk.** Choosing install builds a +Python environment under PAIR's own engine directory and installs vLLM and its +CUDA dependencies into it, which is several gigabytes. Uninstalling removes that +environment; it does not remove downloaded weights, which live in the Hugging +Face cache at `~/.cache/huggingface`. + ## Managing Engines on Other Machines After systems are paired, you can view another machine's engines from diff --git a/docs/overview.mdx b/docs/overview.mdx index 95ea262c..01415aae 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -35,7 +35,8 @@ These terms have specific meanings in PAIR: is no server, controller, or primary node. - **Cluster** — the set of nodes you have paired together. A node belongs to at most one cluster, and it must leave before it can join another. -- **Engine** — the local inference server that runs models: Ollama or LM Studio. +- **Engine** — the local inference server that runs models: Ollama, LM Studio, + or vLLM. PAIR can install, start, stop, and update an engine, or adopt one you already run yourself. - **Model** — what you prepare on each node. Nodes do not share models, so a @@ -121,7 +122,8 @@ For the trust boundaries in detail, refer to - A local endpoint for compatible AI applications and development tools. - LAN discovery plus manually configured nodes. -- Ollama-compatible and LM Studio/OpenAI-compatible routing proxies. +- Ollama-compatible and OpenAI-compatible routing proxies. The + OpenAI-compatible one fronts both LM Studio and vLLM. - Pairing and cluster membership managed by the background services. - Model-aware, workload-informed routing of independent requests. - Encrypted routing between machines: a request sent to another node travels over diff --git a/services/lmstudio-proxy/README.md b/services/lmstudio-proxy/README.md index 71a8b70d..39953d7c 100644 --- a/services/lmstudio-proxy/README.md +++ b/services/lmstudio-proxy/README.md @@ -3,11 +3,15 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 --> -# LM Studio Proxy +# OpenAI-Compatible Proxy -A discovery-aware HTTP reverse proxy for LM Studio nodes on the local network. It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lm]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket). +A discovery-aware HTTP reverse proxy for every OpenAI-compatible engine on the local network — **LM Studio and vLLM** today. They speak the same HTTP surface, so they share one router rather than one binary each; the binary, its port file, and its `lmstudio-proxy:` relay namespace keep their historical spelling because those are wire contracts. -> **Clone of `ollama-proxy`.** This proxy is a deliberate clone of [`ollama-proxy`](../ollama-proxy/README.md) so the two share identical routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lm` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags workloads `lmstudio`, and persists its port to its own file. It has no `--alias-address`, so its self-forward guard covers only its own listener. +It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lm, vl]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket). + +**Which engine owns a model.** A peer advertises `lm` and `vl` at the same value — the port of its own OpenAI proxy — so a peer running both still resolves to one routing target. Which engine actually serves a given model comes from the node's per-engine model attribution (`modelsByEngine`, keyed by engine-manager name), not from the service key. That attribution is what tags each forwarded workload with the engine that ran it, and what the desktop reads to label a node. A model id advertised by more than one engine on a node resolves to the first in the proxy's engine order (LM Studio, then vLLM), so routing is deterministic. + +> **Sibling of `ollama-proxy`.** This proxy shares `ollama-proxy`'s routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lm` and `vl` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags each workload with the engine that owns its model, and persists its port to its own file. It has no `--alias-address`, so its self-forward guard covers only its own listener. ## Build @@ -34,14 +38,14 @@ lmstudio-proxy [flags] ### HTTP Reverse Proxy -The proxy listens on `--port` (default 1234) and forwards incoming HTTP requests to the currently active LM Studio node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your OpenAI-compatible client at `http://localhost:1234` and the proxy handles routing. +The proxy listens on `--port` (default 1234) and forwards incoming HTTP requests to the currently active node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your OpenAI-compatible client at `http://localhost:1234` and the proxy handles routing. -**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to the local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart. +**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to a local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. With more than one local OpenAI engine the ingress picks the one that actually serves the requested model, read from the engines themselves (short-TTL cached) rather than from discovery, falling back to the single healthy engine when there is only one and answering `503` when neither resolves. A peer's `GET /v1/models` is answered by merging every local engine's inventory, so nothing this node serves is hidden from the cluster. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart. **Persisted port.** A port chosen at runtime via the `set-port` request (see below) is saved as `lmstudio-proxy-port.json` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and **restored on startup**, taking precedence over `--port`/the default. One value is exempt: a stored `1235` is discarded and `--port` is used instead, so that port cannot be restored even when it was chosen deliberately via `set-port`. Any other stored port is honoured. The broker uses `--ignore-persisted-port` while reserving the managed `1234` facade. Node selection: -- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current LM Studio inventory advertises the exact requested model ID. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`. +- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current OpenAI-engine inventory advertises the exact requested model ID. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`. - **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order. - **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md). - **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model. @@ -88,10 +92,11 @@ Nodes are represented throughout the protocol with this shape: |-------|------|-------------| | `id` | string | Stable per-host UUID from the discovery record (the ID you supply, for a manual node) | | `host` | string | Hostname, for display — routing never keys on it | -| `port` | int | LM Studio port from the discovery record's `lm` service entry | +| `port` | int | The node's OpenAI proxy port, from whichever of the `lm` / `vl` service entries the record carries (they name the same port) | | `addresses` | string[] | Addresses to dial. A node fed by the discovery relay always carries exactly one canonical address; several only ever appear on a manual node | | `txt` | string[] | The discovery record's TXT pairs, carried verbatim | -| `models` | string[] | The node's LM Studio model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the exact requested model ID. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation | +| `modelsByEngine` | object | The node's inventory attributed per OpenAI engine (`lmstudio`, `vllm`), keyed by engine-manager engine name. A present key with an empty list means that engine is running and holds nothing; a missing key means the node does not run it. This is what says which engine a node runs and which one a model belongs to | +| `models` | string[] | The union of the above — the node's whole OpenAI-reachable model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the exact requested model ID. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation | | `ip` | string | The single canonical LAN address to dial or display, resolved from the node's `ip=` TXT if present and otherwise the best-scored advertised IPv4. Stamped onto outbound `node/*` notifications so consumers agree with the address the proxy routes to | --- @@ -349,7 +354,7 @@ Add a node manually (for networks where mDNS is blocked). If the node ID already **Request:** ```json -{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","host":"remote-server","port":1234,"addresses":["10.0.1.50"]}} +{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","engine":"vllm","host":"remote-server","port":8000,"addresses":["10.0.1.50"]}} ``` **Response:** @@ -365,7 +370,7 @@ Remove a previously added manual node. **Request:** ```json -{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server"}} +{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server","engine":"vllm"}} ``` **Response:** @@ -377,7 +382,7 @@ The proxy emits a `node/removed` notification and clears the active selection if #### `node/set-local-backend` -Tell the proxy which loopback engine this node's own traffic terminates on. The broker sends it once the local LM Studio address and health are known. It is the target the cluster mTLS ingress forwards to, and the substitute used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears it, and the ingress then answers `503`. +Tell the proxy which loopback engine this node's own traffic terminates on. Backends are held **per engine**, so one node may run LM Studio and vLLM at once and clearing one leaves the other routing. The broker sends one per engine once that engine's local address and health are known. They are the targets the cluster mTLS ingress forwards to, and the substitutes used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears that engine's backend, and its ingress then answers `503`. **Request:** ```json @@ -441,4 +446,4 @@ The proxy shuts down gracefully on any of: ## Discovery -The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lm` (LM Studio) nodes (`discovery:subscribe {services:[lm]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set. +The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lm` (LM Studio) and `vl` (vLLM) nodes (`discovery:subscribe {services:[lm, vl]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set. diff --git a/services/lmstudio-proxy/activity_test.go b/services/lmstudio-proxy/activity_test.go index 49a63541..b9692654 100644 --- a/services/lmstudio-proxy/activity_test.go +++ b/services/lmstudio-proxy/activity_test.go @@ -58,6 +58,7 @@ func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) { Addresses: []string{"127.0.0.1"}, Port: closedPortFor(t), Models: []string{"qwen"}, + Engine: "lmstudio", }) p := NewProxy(NewCodec(rec), disc, 1235) diff --git a/services/lmstudio-proxy/discovery.go b/services/lmstudio-proxy/discovery.go index 9b05581b..6ea1b735 100644 --- a/services/lmstudio-proxy/discovery.go +++ b/services/lmstudio-proxy/discovery.go @@ -5,10 +5,11 @@ package main // Discovery is the proxy's routing-target set. The proxy runs no mDNS of its // own: routing targets are pushed down from the broker's discovery relay -// (discovery:nodes snapshots for the lm service) into the subscribed overlay, -// merged with user-added manual nodes. The proxy is itself advertised — as an lm -// service — by the node-scanner daemon's single _nvpair-node record, keyed off -// the engine port the broker's poller registers. +// (discovery:nodes snapshots for the OpenAI-compatible engine services — lm and +// vl) into the subscribed overlay, merged with user-added manual nodes. The +// proxy is itself advertised — under one key per local OpenAI engine — by the +// node-scanner daemon's single _nvpair-node record, which the broker's poller +// registers against this proxy's own listen port. // // The routable Node projection (IP / withPrimaryIP) and the manual-node // overlay live here; request-path reachability (TCP-probe + failover) lives in @@ -36,10 +37,23 @@ type Node struct { Addresses []string `json:"addresses"` TXT []string `json:"txt"` // Models is the latest model inventory carried by the broker's discovery - // snapshot. Model-bearing inference is eligible only when this list - // advertises the requested model; an empty list stays in discovery but is - // not an inference candidate until a later inventory update. + // snapshot — the union across every OpenAI engine the node runs. Model-bearing + // inference is eligible only when this list advertises the requested model; an + // empty list stays in discovery but is not an inference candidate until a + // later inventory update. Models []string `json:"models,omitempty"` + // ModelsByEngine attributes each model to the engine on this node that serves + // it, so a request can be tagged with the engine that will actually run it and + // so a node running both LM Studio and vLLM is not conflated into one + // inventory. Keyed by engine-manager engine name ("lmstudio", "vllm"). + ModelsByEngine map[string][]string `json:"modelsByEngine,omitempty"` + // Engine names the single OpenAI engine a manual node was added for. A relay + // peer leaves it empty: one peer entry covers every engine that peer runs, + // because they all answer on that peer's one proxy port. A manual node is the + // exception — the user supplies an engine's own address, and LM Studio and + // vLLM sit on different ports — so one manual node per engine is added, each + // carrying the engine it represents. + Engine string `json:"engine,omitempty"` // IP is the single canonical LAN address a consumer should dial/display for // this node, resolved via the shared netpick ranker: the node's // own ip= TXT if present, else the best-scored advertised IPv4. It is @@ -64,14 +78,33 @@ func (n Node) withPrimaryIP() Node { return n } +// routeKey is the node's key for per-target routing caches (the reachability +// chooser). It is the node ID for a relay peer, and engine-qualified for a +// manual node, because two manual entries can share an ID while pointing at +// different engine ports on that host — caching them under one key would let +// one engine's confirmed address be used to dial the other's. +// +// It is deliberately NOT the attribution handle: scheduledOn, node/select and +// the scheduler's priority list stay keyed by the bare node ID, which names the +// machine rather than one engine on it. +func (n Node) routeKey() string { + if n.Engine == "" { + return n.ID + } + return n.Engine + "\x00" + n.ID +} + +// manualKey is the manual overlay's storage key: one entry per (engine, node). +func manualKey(engine, id string) string { return engine + "\x00" + id } + // Discovery holds the proxy's routing targets: the relay-fed subscribed overlay // and the user-added manual overlay. type Discovery struct { mu sync.RWMutex manualNodes map[string]Node // subscribedNodes are routing targets pushed down by the broker's discovery - // relay (discovery:nodes snapshots for the lm service), keyed by node ID (the - // directory instance name). + // relay (discovery:nodes snapshots for the OpenAI engine services), keyed by + // node ID (the directory instance name). subscribedNodes map[string]Node } @@ -93,8 +126,11 @@ func (d *Discovery) Nodes() []Node { out = append(out, n) seen[id] = struct{}{} } - for id, n := range d.manualNodes { - if _, exists := seen[id]; !exists { + // Manual entries are keyed per (engine, node), so a host added for both + // engines contributes one entry each; a node the relay already covers is + // skipped entirely, exactly as before. + for _, n := range d.manualNodes { + if _, exists := seen[n.ID]; !exists { out = append(out, n) } } @@ -135,31 +171,61 @@ func (d *Discovery) SetSubscribed(nodes []Node) (discovered, updated, removed [] // warrants a node/updated. func nodeEqual(a, b Node) bool { return a.ID == b.ID && a.Host == b.Host && a.Port == b.Port && a.IP == b.IP && + a.Engine == b.Engine && slices.Equal(a.Addresses, b.Addresses) && slices.Equal(a.TXT, b.TXT) && - slices.Equal(a.Models, b.Models) + slices.Equal(a.Models, b.Models) && engineModelsEqual(a.ModelsByEngine, b.ModelsByEngine) } +// engineModelsEqual compares two per-engine inventories. It is part of nodeEqual +// because a node that moves a model from one engine to the other keeps the same +// union — without this the change would not raise a node/updated and consumers +// would keep the stale attribution. +func engineModelsEqual(a, b map[string][]string) bool { + if len(a) != len(b) { + return false + } + for engine, models := range a { + other, ok := b[engine] + if !ok || !slices.Equal(models, other) { + return false + } + } + return true +} + +// AddManual upserts a manual node for one engine. The same node ID added for +// two engines is two entries, because each names that engine's own port. func (d *Discovery) AddManual(node Node) (added bool) { d.mu.Lock() defer d.mu.Unlock() - _, exists := d.manualNodes[node.ID] - d.manualNodes[node.ID] = node + key := manualKey(node.Engine, node.ID) + _, exists := d.manualNodes[key] + d.manualNodes[key] = node return !exists } -func (d *Discovery) RemoveManual(id string) (removed bool) { +// RemoveManual drops one engine's entry for a node. +func (d *Discovery) RemoveManual(engine, id string) (removed bool) { d.mu.Lock() defer d.mu.Unlock() - _, exists := d.manualNodes[id] + key := manualKey(engine, id) + _, exists := d.manualNodes[key] if exists { - delete(d.manualNodes, id) + delete(d.manualNodes, key) } return exists } +// IsManual reports whether any engine's manual entry exists for a node ID. It +// answers the routing question "did the user supply this address explicitly?", +// which is a property of the node, not of one engine on it. func (d *Discovery) IsManual(id string) bool { d.mu.RLock() defer d.mu.RUnlock() - _, exists := d.manualNodes[id] - return exists + for _, n := range d.manualNodes { + if n.ID == id { + return true + } + } + return false } diff --git a/services/lmstudio-proxy/e2e_test.go b/services/lmstudio-proxy/e2e_test.go index 5412ab7c..78b9bd6d 100644 --- a/services/lmstudio-proxy/e2e_test.go +++ b/services/lmstudio-proxy/e2e_test.go @@ -189,9 +189,9 @@ func TestE2EFailoverOverRealBinary(t *testing.T) { busyHost, busyPort := e2eSplitHostPort(t, busy.URL) goodHost, goodPort := e2eSplitHostPort(t, good.URL) - e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}}) + e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "engine": "lmstudio", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}}) e2eWaitResult(t, frames, "1", 5*time.Second) - e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}}) + e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "engine": "lmstudio", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}}) e2eWaitResult(t, frames, "2", 5*time.Second) // Select the busy node so the failover path is deterministic. e2eSend(t, stdin, 3, "node/select", map[string]any{"id": "busy"}) diff --git a/services/lmstudio-proxy/engines.go b/services/lmstudio-proxy/engines.go new file mode 100644 index 00000000..47d99d63 --- /dev/null +++ b/services/lmstudio-proxy/engines.go @@ -0,0 +1,238 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/url" + "slices" + "sort" + + "nvpair-shared/noderec" +) + +// openaiEngine binds one OpenAI-compatible engine's discovery service key to +// the engine-manager engine name that owns its models and tags its workloads. +// The two are separate contracts and are deliberately not derived from each +// other: the key is the mDNS TXT spelling, the name is what engine-manager's +// modelsByEngine map and the workload wire use. +type openaiEngine struct { + Service noderec.ServiceKey + Name string +} + +// openaiEngines is every engine this proxy fronts, in the order it resolves +// them. The order is the deterministic tie-break for a model id that more than +// one engine on the same node advertises — first listed wins — so routing and +// workload attribution can never depend on map iteration order. +// +// Every entry speaks the same OpenAI HTTP surface, which is what lets one proxy +// serve them all: the engines differ in how they are installed and managed, not +// in how a request is forwarded. +var openaiEngines = []openaiEngine{ + {Service: noderec.ServiceLMStudio, Name: "lmstudio"}, + {Service: noderec.ServiceVLLM, Name: "vllm"}, +} + +// subscribedServices is the discovery:subscribe service list: every engine key +// this proxy routes for. +func subscribedServices() []noderec.ServiceKey { + out := make([]noderec.ServiceKey, 0, len(openaiEngines)) + for _, e := range openaiEngines { + out = append(out, e.Service) + } + return out +} + +// engineNames returns the engine-manager names in resolution order. +func engineNames() []string { + out := make([]string, 0, len(openaiEngines)) + for _, e := range openaiEngines { + out = append(out, e.Name) + } + return out +} + +// isOpenAIEngine reports whether a name is one this proxy fronts. It gates the +// engine field on node/set-local-backend and node/add-manual so a typo is +// refused at the boundary rather than silently creating an unroutable entry. +func isOpenAIEngine(name string) bool { + return slices.Contains(engineNames(), name) +} + +// engineForModel returns the engine on this node that owns a model id, using +// openaiEngines order as the tie-break, and "" when none advertises it. A node +// with no per-engine attribution at all (a peer whose scanner has not enriched +// it yet) resolves to "" and is attributed by the caller. +func engineForModel(modelsByEngine map[string][]string, model string) string { + if model == "" { + return "" + } + for _, e := range openaiEngines { + if slices.Contains(modelsByEngine[e.Name], model) { + return e.Name + } + } + return "" +} + +// unionModels flattens a per-engine model map into one sorted, de-duplicated +// list — the node's whole OpenAI-reachable inventory, which is what eligibility +// filtering and the aggregated /v1/models answer both work from. Sorted so a +// node's Models list is stable across snapshots and cannot spuriously trip the +// node/updated diff. +func unionModels(modelsByEngine map[string][]string) []string { + seen := make(map[string]bool) + out := make([]string, 0) + for _, e := range openaiEngines { + for _, m := range modelsByEngine[e.Name] { + if m != "" && !seen[m] { + seen[m] = true + out = append(out, m) + } + } + } + sort.Strings(out) + return out +} + +// candidateEngine is the engine a forwarded request is attributed to. A +// candidate resolved from a model always names its owner; one resolved with no +// routing model (a control call) falls back to the first engine, matching the +// order routing itself resolves in. +func candidateEngine(c candidate) string { + if c.engine != "" { + return c.engine + } + return fallbackWorkloadEngine +} + +// appendCandidate adds a resolved candidate unless its backend host is already +// claimed or resolves back to this proxy. The self check is defensive: a local +// backend must never point at our own listener, which would loop. +func appendCandidate(out []candidate, seenHost map[string]bool, c candidate, selfPort int) []candidate { + if c.url == nil { + return out + } + if isSelfTarget(c.url, selfPort) { + slog.Debug("resolveCandidates: skipping self-target node", + "node_id", c.id, "engine", c.engine, "target", c.url.Host, "self_port", selfPort) + return out + } + if seenHost[c.url.Host] { + return out + } + seenHost[c.url.Host] = true + return append(out, c) +} + +// nodeEngineFor names the engine on a node that owns the routing model, or "" +// when there is no routing model or the node carries no attribution for it. A +// manual node has exactly one engine and answers with it directly. +func (p *Proxy) nodeEngineFor(n Node, model string) string { + if n.Engine != "" { + return n.Engine + } + return engineForModel(n.ModelsByEngine, model) +} + +// selfTargets are the local loopback engines a candidate for this node resolves +// to. With a routing model it is the one engine that serves it — read from the +// engines themselves, which is authoritative where the node's own discovery +// entry can lag. Without one (a control call or the aggregated model list) it is +// every healthy local engine, so nothing this node serves is hidden. +func (p *Proxy) selfTargets(n Node, model, discoveredEngine string) []localEngineTarget { + backends := p.localBackends() + if len(backends) == 0 { + slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID) + return nil + } + if model == "" { + return backends + } + engine := engineForModel(p.localModelsByEngine(context.Background(), backends), model) + if engine == "" { + engine = discoveredEngine + } + for _, b := range backends { + if b.Engine == engine { + return []localEngineTarget{b} + } + } + // The model is advertised for this node but no local engine claims it. Do not + // guess an engine that would answer 404: leave the node out and let another + // owner, or the no-owner rejection, decide. + slog.Debug("resolveCandidates: no local engine owns the requested model", "node_id", n.ID) + return nil +} + +// modelListTransportError marks a model-list failure that happened before any +// response arrived, so the caller can drop the confirmed address for that +// target. A protocol-level failure (bad status, unparseable body) proves the +// address is reachable and must not. +type modelListTransportError struct{ err error } + +func (e modelListTransportError) Error() string { return e.err.Error() } +func (e modelListTransportError) Unwrap() error { return e.err } + +func isTransportError(err error) bool { + var t modelListTransportError + return errors.As(err, &t) +} + +// fetchModelList reads one OpenAI /v1/models endpoint and returns its records +// with their ids. It is shared by the cluster-wide aggregation (over a peer's +// mTLS proxy) and the ingress aggregation (over this node's loopback engines), +// so both apply the same size cap and the same strict envelope validation. +func fetchModelList(ctx context.Context, client *http.Client, target *url.URL) ([]modelListItem, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, target.String(), nil) + if err != nil { + return nil, err + } + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + return nil, modelListTransportError{err} + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("upstream returned %s", resp.Status) + } + body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1)) + if err != nil { + return nil, err + } + if len(body) > maxModelListBytes { + return nil, fmt.Errorf("model list exceeds %d bytes", maxModelListBytes) + } + var envelope struct { + Data *[]json.RawMessage `json:"data"` + } + if err := json.Unmarshal(body, &envelope); err != nil { + return nil, err + } + if envelope.Data == nil { + return nil, fmt.Errorf("upstream response has no data array") + } + items := make([]modelListItem, 0, len(*envelope.Data)) + for _, raw := range *envelope.Data { + var identity struct { + ID string `json:"id"` + } + if err := json.Unmarshal(raw, &identity); err != nil { + return nil, fmt.Errorf("invalid model record: %w", err) + } + if identity.ID == "" { + return nil, fmt.Errorf("model record has no id") + } + items = append(items, modelListItem{key: identity.ID, raw: raw}) + } + return items, nil +} diff --git a/services/lmstudio-proxy/failover_test.go b/services/lmstudio-proxy/failover_test.go index 02e5361b..2ebf92fc 100644 --- a/services/lmstudio-proxy/failover_test.go +++ b/services/lmstudio-proxy/failover_test.go @@ -43,13 +43,16 @@ func nodeFor(t *testing.T, id, serverURL string) Node { if err != nil { t.Fatalf("port %q: %v", portStr, err) } - return Node{ID: id, Addresses: []string{host}, Port: port} + // Manual nodes always name the engine they were added for; the proxy keys its + // manual overlay by (engine, node) so one host can be added for both. + return Node{ID: id, Addresses: []string{host}, Port: port, Engine: "lmstudio"} } func nodeForModel(t *testing.T, id, serverURL, model string) Node { t.Helper() node := nodeFor(t, id, serverURL) node.Models = []string{model} + node.ModelsByEngine = map[string][]string{node.Engine: {model}} return node } diff --git a/services/lmstudio-proxy/ingress.go b/services/lmstudio-proxy/ingress.go index 2b70e697..d4138e31 100644 --- a/services/lmstudio-proxy/ingress.go +++ b/services/lmstudio-proxy/ingress.go @@ -4,25 +4,34 @@ package main import ( + "bytes" + "context" "encoding/json" + "io" "log/slog" "net" "net/http" "net/http/httputil" "net/url" "strconv" + "sync" + "time" "nvpair-shared/cors" ) const engineIdentityProbeHeader = "X-NVPAIR-Engine-Identity-Probe" -// localBackend is the explicit loopback engine the cluster mTLS ingress +// localBackend is one explicit loopback engine the cluster mTLS ingress // forwards to. It is supplied by the broker over node/set-local-backend and is // deliberately NOT sourced from the discovery overlay: a request that arrived // over the LAN mTLS ingress can only ever be dumped on this node's own local // engine, never re-routed to a peer, so the ingress path is strictly terminal // and cannot recurse or amplify. +// +// Engine names which OpenAI engine this is. One node may run LM Studio and vLLM +// at once, so the backends are held per engine and each is set and cleared +// independently — marking one unhealthy must not disturb the other. type localBackend struct { Engine string `json:"engine"` Host string `json:"host"` @@ -30,29 +39,68 @@ type localBackend struct { Healthy bool `json:"healthy"` } +// url is the loopback URL for this backend, and false when it is not set or not +// healthy. The host defaults to 127.0.0.1 and is always loopback. +func (b localBackend) url() (*url.URL, bool) { + if b.Port <= 0 || !b.Healthy { + return nil, false + } + host := b.Host + if host == "" { + host = "127.0.0.1" + } + return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true +} + // setLocalBackend records (or, with a zero port / unhealthy flag, effectively -// clears) the local engine the ingress serves. +// clears) one engine's local backend. Storing the unhealthy record rather than +// deleting the key keeps "known and down" distinguishable from "never +// advertised" in logs, while localBackendTarget treats both as unavailable. func (p *Proxy) setLocalBackend(b localBackend) { p.backendMu.Lock() - p.backend = b + if p.backends == nil { + p.backends = make(map[string]localBackend, len(openaiEngines)) + } + p.backends[b.Engine] = b p.backendMu.Unlock() } -// localBackendTarget returns the loopback URL of the current local engine, and -// false when none is set/healthy (the ingress then answers 503 rather than -// forwarding). The host defaults to 127.0.0.1 and is always loopback. -func (p *Proxy) localBackendTarget() (*url.URL, bool) { +// localBackendTarget returns the loopback URL of one engine's local backend, +// and false when that engine has none set or it is unhealthy (the ingress then +// answers 503 rather than forwarding). +func (p *Proxy) localBackendTarget(engine string) (*url.URL, bool) { p.backendMu.RLock() - b := p.backend + b, ok := p.backends[engine] p.backendMu.RUnlock() - if b.Port <= 0 || !b.Healthy { + if !ok { return nil, false } - host := b.Host - if host == "" { - host = "127.0.0.1" + return b.url() +} + +// localBackends returns every healthy local backend in openaiEngines order, +// paired with its engine name. It is the fan-out set for the ingress model list +// and the candidate set for this node's own entry in routing. +func (p *Proxy) localBackends() []localEngineTarget { + p.backendMu.RLock() + defer p.backendMu.RUnlock() + out := make([]localEngineTarget, 0, len(openaiEngines)) + for _, e := range openaiEngines { + b, ok := p.backends[e.Name] + if !ok { + continue + } + if u, ok := b.url(); ok { + out = append(out, localEngineTarget{Engine: e.Name, URL: u}) + } } - return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true + return out +} + +// localEngineTarget is one healthy local engine: its name and its loopback URL. +type localEngineTarget struct { + Engine string + URL *url.URL } // handlePlain is the plaintext personality: it accepts requests only from @@ -103,17 +151,66 @@ func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) { "client certificate is not a pinned member of this node's cluster") return } - target, ok := p.localBackendTarget() - if !ok { + backends := p.localBackends() + if len(backends) == 0 { writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend", "no local inference backend is available on this node") return } + // A peer aggregating the cluster's inventory must see everything this node + // serves. Forwarding to one engine would hide the other's models and the + // peer would then never route a request this node could have answered, so + // the list is merged locally instead. Still terminal: the fan-out is to this + // node's own loopback engines only. + if r.Method == http.MethodGet && r.URL.Path == "/v1/models" { + p.serveLocalModelList(w, r, backends) + return + } + target, engine, ok := p.ingressTarget(r, backends) + if !ok { + writeIngressError(w, http.StatusServiceUnavailable, "no-model-owner", + "no local inference engine serves the requested model") + return + } slog.Debug("cluster ingress forwarding to local backend", - "peer", peer, "method", r.Method, "path", r.URL.Path, "target", target.Host) + "peer", peer, "engine", engine, "method", r.Method, "path", r.URL.Path, "target", target.Host) p.reverseProxyToLocal(w, r, target) } +// ingressTarget picks which local engine answers a peer's request. Resolution +// order, top down: +// +// 1. the engine that actually serves the requested model, read live from the +// local engines themselves (short-TTL cached) rather than from discovery — +// this node's own entry in the discovery overlay is enriched by the scanner +// and can lag or, on a freshly started node, be absent entirely, which would +// misroute every ingress request; +// 2. the only healthy backend, when there is exactly one — the common case, and +// the answer regardless of what the body names; +// 3. no target, which the caller reports as 503. +// +// The body is buffered and restored, so the reverse proxy still forwards it. +func (p *Proxy) ingressTarget(r *http.Request, backends []localEngineTarget) (*url.URL, string, bool) { + if len(backends) == 1 { + return backends[0].URL, backends[0].Engine, true + } + body, model := bufferBodyAndModel(r) + if body != nil { + r.Body = io.NopCloser(bytes.NewReader(body)) + r.ContentLength = int64(len(body)) + } + if model != "" { + if engine := engineForModel(p.localModelsByEngine(r.Context(), backends), model); engine != "" { + for _, b := range backends { + if b.Engine == engine { + return b.URL, b.Engine, true + } + } + } + } + return nil, "", false +} + // reverseProxyToLocal streams the request to the local engine, preserving // cancellation (the request context is the proxy's root context, so a client // disconnect or shutdown tears down the upstream call and stops generation). @@ -163,3 +260,109 @@ func writeIngressError(w http.ResponseWriter, status int, code, msg string) { } _, _ = w.Write(body) } + +// localModelsTTL bounds how stale the ingress engine attribution may be. Long +// enough that a burst of peer requests costs one loopback query per engine, +// short enough that a model swap is picked up within a few seconds. +const localModelsTTL = 5 * time.Second + +// localModelsCache memoizes the per-engine model ids read from this node's own +// engines, so ingress routing does not pay a loopback round trip per request. +type localModelsCache struct { + mu sync.Mutex + fetched time.Time + byEngine map[string][]string +} + +// localModelsByEngine returns which local engine serves which models, asking the +// engines themselves rather than the discovery overlay. Authoritative for this +// node and cheap: loopback, bounded, and cached for localModelsTTL. An engine +// that fails to answer contributes nothing, so a request for its model falls +// through to the caller's next resolution step. +func (p *Proxy) localModelsByEngine(ctx context.Context, backends []localEngineTarget) map[string][]string { + p.localModels.mu.Lock() + if time.Since(p.localModels.fetched) < localModelsTTL && p.localModels.byEngine != nil { + cached := p.localModels.byEngine + p.localModels.mu.Unlock() + return cached + } + p.localModels.mu.Unlock() + + fresh := make(map[string][]string, len(backends)) + for _, b := range backends { + ids, err := p.fetchModelIDs(ctx, b.URL) + if err != nil { + slog.Debug("local model list unavailable", "engine", b.Engine, "target", b.URL.Host, "err", err) + continue + } + fresh[b.Engine] = ids + } + + p.localModels.mu.Lock() + p.localModels.fetched = time.Now() + p.localModels.byEngine = fresh + p.localModels.mu.Unlock() + return fresh +} + +// fetchModelIDs reads one loopback engine's /v1/models and returns its ids. +func (p *Proxy) fetchModelIDs(ctx context.Context, target *url.URL) ([]string, error) { + records, err := p.fetchModelRecords(ctx, target) + if err != nil { + return nil, err + } + ids := make([]string, 0, len(records)) + for _, rec := range records { + ids = append(ids, rec.key) + } + return ids, nil +} + +// serveLocalModelList answers a peer's GET /v1/models by merging this node's +// own engines' inventories, first engine's metadata winning on a duplicate id. +func (p *Proxy) serveLocalModelList(w http.ResponseWriter, r *http.Request, backends []localEngineTarget) { + seen := make(map[string]bool) + models := make([]json.RawMessage, 0) + ok := false + for _, b := range backends { + records, err := p.fetchModelRecords(r.Context(), b.URL) + if err != nil { + slog.Debug("ingress model list candidate unavailable", "engine", b.Engine, "target", b.URL.Host, "err", err) + continue + } + ok = true + for _, rec := range records { + if !seen[rec.key] { + seen[rec.key] = true + models = append(models, rec.raw) + } + } + } + if !ok { + writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend", + "no local inference backend answered its model list") + return + } + body, err := json.Marshal(struct { + Object string `json:"object"` + Data []json.RawMessage `json:"data"` + }{Object: "list", Data: models}) + if err != nil { + writeIngressError(w, http.StatusInternalServerError, "backend-error", "failed to encode model inventory") + return + } + cors.Apply(w.Header()) + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-Content-Type-Options", "nosniff") + w.WriteHeader(http.StatusOK) + _, _ = w.Write(body) +} + +// fetchModelRecords reads one loopback engine's /v1/models over the shared plain +// transport. Loopback only: the caller supplies a target from localBackends. +func (p *Proxy) fetchModelRecords(ctx context.Context, target *url.URL) ([]modelListItem, error) { + list := *target + list.Path = "/v1/models" + client := &http.Client{Timeout: modelListClient.Timeout, Transport: p.plainHTTPTransport()} + return fetchModelList(ctx, client, &list) +} diff --git a/services/lmstudio-proxy/multiengine_test.go b/services/lmstudio-proxy/multiengine_test.go new file mode 100644 index 00000000..5d46f577 --- /dev/null +++ b/services/lmstudio-proxy/multiengine_test.go @@ -0,0 +1,358 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "strings" + "testing" + "time" + + "nvpair-shared/noderec" +) + +// dirNode builds a relay DirectoryNode advertising the given engine services on +// one shared proxy port, with per-engine model attribution. +func dirNode(id string, port int, byEngine map[string][]string, services ...noderec.ServiceKey) noderec.DirectoryNode { + svc := make(map[noderec.ServiceKey]noderec.ServiceStatus, len(services)) + for _, s := range services { + svc[s] = noderec.ServiceStatus{Port: port} + } + return noderec.DirectoryNode{ + Name: id, + HostUUID: id, + IP: "192.0.2.10", + IPs: []string{"192.0.2.10"}, + Services: svc, + ModelsByEngine: byEngine, + } +} + +// TestSubscribedNodeCarriesEveryOpenAIEngine proves one peer advertising both +// lm and vl projects to a single routable node — they name the same proxy port +// — whose inventory is the union and whose attribution is kept per engine. +func TestSubscribedNodeCarriesEveryOpenAIEngine(t *testing.T) { + n, ok := subscribedToNode(dirNode("peer", 1234, map[string][]string{ + "lmstudio": {"qwen2.5-7b"}, + "vllm": {"Qwen/Qwen3-8B"}, + "ollama": {"llama3.2:1b"}, + }, noderec.ServiceLMStudio, noderec.ServiceVLLM)) + if !ok { + t.Fatal("peer advertising lm and vl was dropped") + } + if n.Port != 1234 { + t.Errorf("port = %d, want the shared proxy port 1234", n.Port) + } + want := []string{"Qwen/Qwen3-8B", "qwen2.5-7b"} + if strings.Join(n.Models, ",") != strings.Join(want, ",") { + t.Errorf("models = %v, want the sorted OpenAI union %v (Ollama-only models are not ours)", n.Models, want) + } + if got := engineForModel(n.ModelsByEngine, "Qwen/Qwen3-8B"); got != "vllm" { + t.Errorf("owner of Qwen/Qwen3-8B = %q, want vllm", got) + } + if got := engineForModel(n.ModelsByEngine, "qwen2.5-7b"); got != "lmstudio" { + t.Errorf("owner of qwen2.5-7b = %q, want lmstudio", got) + } + if got := engineForModel(n.ModelsByEngine, "llama3.2:1b"); got != "" { + t.Errorf("an Ollama-only model must not resolve to an OpenAI engine, got %q", got) + } +} + +// TestSubscribedNodeVLLMOnly proves a node that runs only vLLM is routable +// through this proxy, which is the whole point of the second service key. +func TestSubscribedNodeVLLMOnly(t *testing.T) { + n, ok := subscribedToNode(dirNode("vpeer", 1234, map[string][]string{ + "vllm": {"Qwen/Qwen3-8B"}, + }, noderec.ServiceVLLM)) + if !ok { + t.Fatal("vLLM-only peer was dropped") + } + if len(n.Models) != 1 || n.Models[0] != "Qwen/Qwen3-8B" { + t.Errorf("models = %v", n.Models) + } +} + +// TestSubscribedNodeWithNoOpenAIEngineIsDropped proves a node running only +// Ollama is not a candidate here, however many models it advertises. +func TestSubscribedNodeWithNoOpenAIEngineIsDropped(t *testing.T) { + if _, ok := subscribedToNode(dirNode("opeer", 11434, map[string][]string{ + "ollama": {"llama3.2:1b"}, + }, noderec.ServiceOllama)); ok { + t.Fatal("an Ollama-only node must not be an OpenAI routing candidate") + } +} + +// TestEngineForModelTieBreakIsDeterministic proves a model id served by both +// local engines resolves the same way every time, following openaiEngines order +// rather than map iteration order. +func TestEngineForModelTieBreakIsDeterministic(t *testing.T) { + byEngine := map[string][]string{ + "lmstudio": {"Qwen/Qwen3-8B"}, + "vllm": {"Qwen/Qwen3-8B"}, + } + for range 50 { + if got := engineForModel(byEngine, "Qwen/Qwen3-8B"); got != "lmstudio" { + t.Fatalf("tie-break = %q, want the first listed engine lmstudio", got) + } + } +} + +// TestMixedEngineCandidatesRouteToTheModelOwner proves eligibility and workload +// attribution both follow the per-engine inventory: a request for a vLLM-only +// model reaches the vLLM node and is tagged vllm, and vice versa. +func TestMixedEngineCandidatesRouteToTheModelOwner(t *testing.T) { + lmNode := nodeForModel(t, "lm-node", "http://192.0.2.11:1234", "qwen2.5-7b") + vlNode := nodeForModel(t, "vl-node", "http://192.0.2.12:8000", "Qwen/Qwen3-8B") + vlNode.Engine = "vllm" + vlNode.ModelsByEngine = map[string][]string{"vllm": {"Qwen/Qwen3-8B"}} + + disc := NewDiscovery() + disc.AddManual(lmNode) + disc.AddManual(vlNode) + p := testProxy(disc, 1235) + + for _, tc := range []struct{ model, wantID, wantEngine string }{ + {"qwen2.5-7b", "lm-node", "lmstudio"}, + {"Qwen/Qwen3-8B", "vl-node", "vllm"}, + } { + cands := p.resolveCandidates(tc.model) + if len(cands) != 1 { + t.Fatalf("%s: %d candidates, want only the owner", tc.model, len(cands)) + } + if cands[0].id != tc.wantID { + t.Errorf("%s routed to %q, want %q", tc.model, cands[0].id, tc.wantID) + } + if got := candidateEngine(cands[0]); got != tc.wantEngine { + t.Errorf("%s tagged %q, want %q", tc.model, got, tc.wantEngine) + } + } +} + +// TestManualNodeIsKeyedPerEngine proves one host added for both engines keeps +// two entries — LM Studio and vLLM sit on different ports, so collapsing them +// would lose one — and that removing one leaves the other routable. +func TestManualNodeIsKeyedPerEngine(t *testing.T) { + lm := nodeForModel(t, "host", "http://192.0.2.20:1234", "qwen2.5-7b") + vl := nodeForModel(t, "host", "http://192.0.2.20:8000", "Qwen/Qwen3-8B") + vl.Engine = "vllm" + vl.ModelsByEngine = map[string][]string{"vllm": {"Qwen/Qwen3-8B"}} + + disc := NewDiscovery() + if !disc.AddManual(lm) || !disc.AddManual(vl) { + t.Fatal("both engine entries should be new") + } + if got := len(disc.Nodes()); got != 2 { + t.Fatalf("nodes = %d, want one entry per engine", got) + } + if !disc.RemoveManual("lmstudio", "host") { + t.Fatal("removing the LM Studio entry should report removed") + } + if !disc.IsManual("host") { + t.Error("the node is still manual while its vLLM entry remains") + } + nodes := disc.Nodes() + if len(nodes) != 1 || nodes[0].Port != 8000 { + t.Fatalf("surviving node = %+v, want the vLLM entry on 8000", nodes) + } + if !disc.RemoveManual("vllm", "host") { + t.Fatal("removing the vLLM entry should report removed") + } + if disc.IsManual("host") { + t.Error("the node should be gone once its last engine entry is removed") + } +} + +// TestRouteKeyIsEngineQualifiedForManualNodes proves the reachability cache +// cannot carry one engine's confirmed address over to the other engine on the +// same host, while attribution stays keyed by the bare node id. +func TestRouteKeyIsEngineQualifiedForManualNodes(t *testing.T) { + lm := Node{ID: "host", Engine: "lmstudio"} + vl := Node{ID: "host", Engine: "vllm"} + if lm.routeKey() == vl.routeKey() { + t.Error("two engines on one host must not share a reachability key") + } + peer := Node{ID: "host"} + if peer.routeKey() != "host" { + t.Errorf("a relay peer's route key = %q, want the bare node id", peer.routeKey()) + } +} + +// TestIngressForwardsToTheEngineThatOwnsTheModel proves a peer's inference +// request lands on whichever local OpenAI engine actually serves the model, +// read from the engines themselves rather than from discovery. +func TestIngressForwardsToTheEngineThatOwnsTheModel(t *testing.T) { + var lmHits, vlHits int + lm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"qwen2.5-7b"}]}`)) + return + } + lmHits++ + _, _ = w.Write([]byte(`{"ok":"lm"}`)) + })) + defer lm.Close() + vl := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"Qwen/Qwen3-8B"}]}`)) + return + } + vlHits++ + _, _ = w.Write([]byte(`{"ok":"vl"}`)) + })) + defer vl.Close() + + p := testProxy(NewDiscovery(), 1235) + setBackendFromURL(t, p, "lmstudio", lm.URL) + setBackendFromURL(t, p, "vllm", vl.URL) + + backends := p.localBackends() + if len(backends) != 2 { + t.Fatalf("localBackends = %d, want both engines", len(backends)) + } + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"Qwen/Qwen3-8B"}`)) + target, engine, ok := p.ingressTarget(req, backends) + if !ok || engine != "vllm" { + t.Fatalf("ingress engine = %q ok=%v, want vllm", engine, ok) + } + p.reverseProxyToLocal(httptest.NewRecorder(), req, target) + if vlHits != 1 || lmHits != 0 { + t.Errorf("hits lm=%d vl=%d, want the request on vLLM only", lmHits, vlHits) + } + + // The body must survive the peek that found the model. + req2 := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen2.5-7b"}`)) + target2, engine2, ok := p.ingressTarget(req2, backends) + if !ok || engine2 != "lmstudio" { + t.Fatalf("ingress engine = %q ok=%v, want lmstudio", engine2, ok) + } + var seen string + echo := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + buf := make([]byte, 64) + n, _ := r.Body.Read(buf) + seen = string(buf[:n]) + })) + defer echo.Close() + _ = target2 + setBackendFromURL(t, p, "lmstudio", echo.URL) + p.reverseProxyToLocal(httptest.NewRecorder(), req2, mustURL(t, echo.URL)) + if !strings.Contains(seen, "qwen2.5-7b") { + t.Errorf("forwarded body = %q, want the original request body", seen) + } +} + +// TestIngressModelListMergesEveryLocalEngine proves a peer aggregating this +// node's inventory sees both engines. Forwarding to one would hide the other's +// models and the peer would never route work this node could have run. +func TestIngressModelListMergesEveryLocalEngine(t *testing.T) { + lm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"qwen2.5-7b"}]}`)) + })) + defer lm.Close() + vl := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"Qwen/Qwen3-8B"}]}`)) + })) + defer vl.Close() + + p := testProxy(NewDiscovery(), 1235) + setBackendFromURL(t, p, "lmstudio", lm.URL) + setBackendFromURL(t, p, "vllm", vl.URL) + + rec := httptest.NewRecorder() + p.serveLocalModelList(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil), p.localBackends()) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d", rec.Code) + } + var envelope struct { + Data []struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &envelope); err != nil { + t.Fatal(err) + } + got := map[string]bool{} + for _, d := range envelope.Data { + got[d.ID] = true + } + if !got["qwen2.5-7b"] || !got["Qwen/Qwen3-8B"] { + t.Errorf("aggregated ids = %v, want both engines' models", got) + } +} + +// TestUnhealthyBackendDoesNotDisturbTheOther is the regression guard for the +// backends map: clearing one engine must leave the other routable. +func TestUnhealthyBackendDoesNotDisturbTheOther(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + p.setLocalBackend(localBackend{Engine: "lmstudio", Port: 1234, Healthy: true}) + p.setLocalBackend(localBackend{Engine: "vllm", Port: 8000, Healthy: true}) + + p.setLocalBackend(localBackend{Engine: "vllm", Port: 8000, Healthy: false}) + if _, ok := p.localBackendTarget("vllm"); ok { + t.Error("an unhealthy vLLM backend must not be a target") + } + if _, ok := p.localBackendTarget("lmstudio"); !ok { + t.Error("clearing vLLM must not disturb LM Studio") + } + if got := len(p.localBackends()); got != 1 { + t.Errorf("healthy backends = %d, want 1", got) + } + if _, ok := p.localBackendTarget("ollama"); ok { + t.Error("an engine this proxy never fronts must have no target") + } +} + +// TestOpenAIEnginesCoverEveryServiceKey guards the two halves of the engine +// table against drifting apart: every engine has a discovery key and a name, +// and the subscription list is exactly those keys. +func TestOpenAIEnginesCoverEveryServiceKey(t *testing.T) { + if len(subscribedServices()) != len(openaiEngines) { + t.Fatalf("subscribed services = %v, want one per engine", subscribedServices()) + } + for _, e := range openaiEngines { + if e.Service == "" || e.Name == "" { + t.Errorf("incomplete engine binding: %+v", e) + } + if !isOpenAIEngine(e.Name) { + t.Errorf("%q is in the table but not recognized", e.Name) + } + } + if isOpenAIEngine("ollama") { + t.Error("ollama is fronted by its own proxy, not this one") + } +} + +// setBackendFromURL registers an httptest server as one engine's local backend. +func setBackendFromURL(t *testing.T, p *Proxy, engine, serverURL string) { + t.Helper() + u := mustURL(t, serverURL) + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split %q: %v", u.Host, err) + } + port, err := strconv.Atoi(portStr) + if err != nil { + t.Fatalf("port %q: %v", portStr, err) + } + p.setLocalBackend(localBackend{Engine: engine, Host: host, Port: port, Healthy: true}) + // Each registration changes what the engines serve, so drop the memoized + // attribution rather than waiting out its TTL. + p.localModels.mu.Lock() + p.localModels.byEngine = nil + p.localModels.fetched = time.Time{} + p.localModels.mu.Unlock() +} + +func mustURL(t *testing.T, raw string) *url.URL { + t.Helper() + u, err := url.Parse(raw) + if err != nil { + t.Fatalf("parse %q: %v", raw, err) + } + return u +} diff --git a/services/lmstudio-proxy/priority_test.go b/services/lmstudio-proxy/priority_test.go index 6661b30b..13d9c8f7 100644 --- a/services/lmstudio-proxy/priority_test.go +++ b/services/lmstudio-proxy/priority_test.go @@ -47,7 +47,7 @@ func contains(hay, needle string) bool { // probe) and it never trips the loopback rewrite or self-forward guard. The // octet is derived from the id's first byte so each id gets a distinct valid IP. func prNode(id string) Node { - return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234} + return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234, Engine: "lmstudio"} } // prProxy returns a proxy whose discovery holds the given node ids. diff --git a/services/lmstudio-proxy/proxy.go b/services/lmstudio-proxy/proxy.go index 6e619e77..49553a2d 100644 --- a/services/lmstudio-proxy/proxy.go +++ b/services/lmstudio-proxy/proxy.go @@ -21,6 +21,7 @@ import ( "net/url" "sort" "strconv" + "strings" "sync" "sync/atomic" "time" @@ -138,16 +139,19 @@ const ( workloadCompletedMethod = "workload:completed" workloadErroredMethod = "workload:errored" - // workloadEngine is the opaque engine identifier carried in every - // workload this proxy produces. This proxy only ever fronts LM Studio. - workloadEngine = "lmstudio" + // fallbackWorkloadEngine is the opaque engine identifier used when a + // forwarded request could not be attributed to one of this proxy's engines — + // a peer whose per-engine model attribution has not arrived yet. It is the + // first engine in openaiEngines, which is also the tie-break order routing + // uses, so the tag matches what the request was routed as. + fallbackWorkloadEngine = "lmstudio" ) // inferenceEndpoints is the set of request paths that count as cluster // workloads. Health checks, model listings (/v1/models), and other control // traffic are deliberately excluded so we don't flood the cluster with -// non-inference noise. LM Studio serves the OpenAI-compatible API, so these -// are the OpenAI inference routes. +// non-inference noise. Every engine behind this proxy serves the +// OpenAI-compatible API, so these are the OpenAI inference routes. var inferenceEndpoints = map[string]bool{ "/v1/chat/completions": true, "/v1/completions": true, @@ -335,12 +339,19 @@ type Proxy struct { // only loopback-plaintext local routing. Read-only after startup. mesh *clustertrust.Mesh - // backendMu guards backend, the explicit loopback engine the cluster mTLS - // ingress forwards to. The broker sets/clears it via node/set-local-backend; - // it is never sourced from discovery, so an ingress request can only ever - // reach this node's own local engine and can never be re-routed to a peer. + // backendMu guards backends, this node's own loopback engines keyed by + // engine-manager name. The broker sets/clears each via + // node/set-local-backend; they are never sourced from discovery, so an + // ingress request can only ever reach this node's own local engines and can + // never be re-routed to a peer. One node may run several at once, so each is + // set and cleared independently. backendMu sync.RWMutex - backend localBackend + backends map[string]localBackend + + // localModels caches which of this node's own engines serves which models, + // read from the engines themselves. It is what the mTLS ingress uses to pick + // the engine for a peer's request. See localModelsByEngine. + localModels localModelsCache selectedMu sync.RWMutex selectedID string @@ -475,12 +486,14 @@ func (p *Proxy) Run(ctx context.Context) error { return fmt.Errorf("failed to send ready notification: %w", err) } - // Routing targets come from the broker's discovery relay. Subscribe - // for lm nodes; they arrive as discovery:nodes snapshots (handled in - // handleMessage), each replacing the subscribed overlay. Non-fatal: if the - // parent isn't a relay-aware broker the proxy still routes to manual nodes. - slog.Debug("subscribing to discovery relay for routing targets", "service", string(noderec.ServiceLMStudio)) - if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: []noderec.ServiceKey{noderec.ServiceLMStudio}}); err != nil { + // Routing targets come from the broker's discovery relay. Subscribe for every + // OpenAI-compatible engine service (lm, vl); they arrive as discovery:nodes + // snapshots (handled in handleMessage), each replacing the subscribed + // overlay. Non-fatal: if the parent isn't a relay-aware broker the proxy + // still routes to manual nodes. + services := subscribedServices() + slog.Debug("subscribing to discovery relay for routing targets", "services", services) + if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: services}); err != nil { slog.Warn("failed to subscribe to discovery relay", "err", err) } @@ -697,8 +710,16 @@ func (p *Proxy) emitWorkload(method string, w Workload) { // pinned to that peer's exact server cert. Empty peerUUID means a plain-HTTP // dial — the local backend (self) or an explicit manual node. type candidate struct { - id string - url *url.URL + id string + url *url.URL + // routeKey keys the reachability chooser. It equals id for a relay peer and + // is engine-qualified for a manual node, so two manual entries for one host's + // two engines never share a confirmed address. See Node.routeKey. + routeKey string + // engine is the OpenAI engine on that node this request is routed to. It is + // what the emitted workload is tagged with, so a dual-engine node's work is + // attributed to the engine that actually ran it. + engine string peerUUID string } @@ -828,12 +849,6 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidate target.Path = r.URL.Path target.RawPath = r.URL.RawPath target.RawQuery = r.URL.RawQuery - upstream, err := http.NewRequestWithContext(r.Context(), http.MethodGet, target.String(), nil) - if err != nil { - results[i].err = err - continue - } - upstream.Header.Set("Accept", "application/json") // A cluster-peer candidate is queried over mTLS to its promoted proxy; // self/manual candidates use the shared plain client. @@ -843,57 +858,18 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidate } wg.Add(1) - go func(i int, cand candidate, req *http.Request, client *http.Client) { + go func(i int, cand candidate, target url.URL, client *http.Client) { defer wg.Done() - resp, err := client.Do(req) + items, err := fetchModelList(r.Context(), client, &target) if err != nil { - p.targets.Forget(cand.id) - results[i].err = err - return - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - results[i].err = fmt.Errorf("upstream returned %s", resp.Status) - return - } - body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1)) - if err != nil { - results[i].err = err - return - } - if len(body) > maxModelListBytes { - results[i].err = fmt.Errorf("model list exceeds %d bytes", maxModelListBytes) - return - } - var envelope struct { - Data *[]json.RawMessage `json:"data"` - } - if err := json.Unmarshal(body, &envelope); err != nil { + if isTransportError(err) { + p.targets.Forget(cand.routeKey) + } results[i].err = err return } - if envelope.Data == nil { - results[i].err = fmt.Errorf("upstream response has no data array") - return - } - models := *envelope.Data - items := make([]modelListItem, 0, len(models)) - for _, raw := range models { - var identity struct { - ID string `json:"id"` - } - if err := json.Unmarshal(raw, &identity); err != nil { - results[i].err = fmt.Errorf("invalid model record: %w", err) - return - } - if identity.ID == "" { - results[i].err = fmt.Errorf("model record has no id") - return - } - items = append(items, modelListItem{key: identity.ID, raw: raw}) - } results[i] = modelListResult{items: items, ok: true} - }(i, cand, upstream, client) + }(i, cand, target, client) } wg.Wait() @@ -1042,7 +1018,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) { wl = &Workload{ ID: reqID, Model: model, - Engine: workloadEngine, + Engine: candidateEngine(candidates[0]), RunID: p.runID, State: "running", ScheduledOn: candidates[0].id, @@ -1213,7 +1189,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) { // can fail over to another of its published addresses // (multi-homed peer). The in-request failover below moves on to // the next node. - p.targets.Forget(cand.id) + p.targets.Forget(cand.routeKey) if !last { // Transport/dial error with candidates left: fail over. retry = true @@ -1369,7 +1345,9 @@ func (p *Proxy) resolveCandidates(model string) []candidate { out := make([]candidate, 0, len(nodes)) // Dedup by resolved backend host: the same physical node can appear under two // IDs (e.g. a manually-added entry and its relay-discovered record), and - // routing to the same engine twice is wasteful. + // routing to the same engine twice is wasteful. It is also what collapses a + // peer that advertises several engine keys back into one candidate: every + // key on a peer names that peer's single proxy port. seenHost := make(map[string]bool, len(nodes)) // placed tracks node IDs already considered so the priority-ordered and // fallback passes don't reconsider one (scheduler ordering). @@ -1383,19 +1361,23 @@ func (p *Proxy) resolveCandidates(model string) []candidate { if u == nil { return } + engine := p.nodeEngineFor(n, model) peerUUID := "" switch { case isSelfTarget(u, selfPort): - // Our own advertised endpoint (lm now points at this proxy). Serve - // it from the explicit local backend — the loopback engine — rather - // than dialing our own mTLS ingress, which would recurse. Ranking - // still used this node's real (discovered) model list above. - lb, ok := p.localBackendTarget() - if !ok { - slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID) - return + // Our own advertised endpoint (the engine keys now point at this + // proxy). Serve it from the explicit local backends — the loopback + // engines — rather than dialing our own mTLS ingress, which would + // recurse. Ranking still used this node's real (discovered) model + // list above. With no routing model there is nothing to attribute, so + // every healthy local engine becomes its own candidate and the + // aggregated model list covers them all. + for _, local := range p.selfTargets(n, model, engine) { + out = appendCandidate(out, seenHost, candidate{ + id: n.ID, routeKey: n.routeKey(), engine: local.Engine, url: local.URL, + }, selfPort) } - u = lb + return case p.mesh.HasPin(n.ClusterUUID): // A pinned cluster peer: reach it only over mTLS to its promoted // proxy (the lm port now advertises the proxy, not the engine). @@ -1419,21 +1401,13 @@ func (p *Proxy) resolveCandidates(model string) []candidate { "node_id", n.ID, "cluster_uuid", n.ClusterUUID) return } - // Defensive: the local backend must never resolve back to this proxy. - if isSelfTarget(u, selfPort) { - slog.Debug("resolveCandidates: skipping self-target node", - "node_id", n.ID, "target", u.Host, "self_port", selfPort) - return - } - if seenHost[u.Host] { - return - } - seenHost[u.Host] = true - out = append(out, candidate{ + out = appendCandidate(out, seenHost, candidate{ id: n.ID, + routeKey: n.routeKey(), + engine: engine, url: u, peerUUID: peerUUID, - }) + }, selfPort) } // Capability has already been enforced. An eligible explicit selection wins, @@ -1585,7 +1559,7 @@ func (p *Proxy) targetURL(n Node) *url.URL { if len(candidates) == 0 { return nil } - host := p.targets.Prefer(n.ID, candidates) + host := p.targets.Prefer(n.routeKey(), candidates) return &url.URL{Scheme: "http", Host: host} } @@ -1826,8 +1800,26 @@ func upstreamUnreachableID(nodeID string) string { // LM Studio port the broker's engine poller registered, not the proxy's listen // port). func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { - svc, ok := n.Services[noderec.ServiceLMStudio] - if !ok || n.IP == "" { + if n.IP == "" { + return Node{}, false + } + // A node may advertise several OpenAI engine keys at once. They all carry the + // same value — the port of that node's one OpenAI-compatible proxy — so the + // node projects to a single routable entry whose per-engine model attribution + // records which engine actually serves what. + port := 0 + byEngine := make(map[string][]string, len(openaiEngines)) + for _, e := range openaiEngines { + svc, ok := n.Services[e.Service] + if !ok { + continue + } + if port == 0 { + port = svc.Port + } + byEngine[e.Name] = append([]string(nil), n.EngineModels(e.Name)...) + } + if port == 0 { return Node{}, false } // Key routing by the stable per-host UUID, not the hostname: candidate ids, @@ -1839,7 +1831,7 @@ func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { return Node{ ID: n.HostUUID, Host: n.Name, - Port: svc.Port, + Port: port, // The node's whole ranked address list, not just its canonical one: a // multi-homed peer's best address from its own vantage point may be a // direct-connect link this host cannot reach, and routing needs somewhere @@ -1848,11 +1840,12 @@ func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { TXT: n.AddressTXT(), IP: n.IP, ClusterUUID: n.ClusterUUID, - // Filter on this node's LM Studio models only, not the cross-engine union, - // so a model a dual-engine node serves solely via Ollama isn't accepted as - // an LM Studio owner here (falls back to the union for a peer that sends - // no attribution — see DirectoryNode.EngineModels). - Models: append([]string(nil), n.EngineModels("lmstudio")...), + // Filter on this node's OpenAI-engine models only, not the cross-engine + // union, so a model a multi-engine node serves solely via Ollama isn't + // accepted as an owner here (each key falls back to the union for a peer + // that sends no attribution — see DirectoryNode.EngineModels). + Models: unionModels(byEngine), + ModelsByEngine: byEngine, }, true } @@ -1981,29 +1974,48 @@ func (p *Proxy) handleMessage(msg *Message) { p.codec.RespondError(msg.ID, -32602, "id, port, and at least one address are required") return } + // engine names which OpenAI engine on that host this address serves. + // LM Studio and vLLM listen on different ports, so a node running both is + // added once per engine and the entries must not collide. + if !isOpenAIEngine(node.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) + return + } + // A manual node's whole inventory belongs to the engine it was added for. + node.ModelsByEngine = map[string][]string{node.Engine: append([]string(nil), node.Models...)} added := p.discovery.AddManual(node) if err := p.codec.Respond(msg.ID, map[string]bool{"added": added}); err != nil { log.Printf("failed to respond to node/add-manual: %v", err) } if added { - log.Printf("manual node added: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + log.Printf("manual node added: %s/%s (%s:%d)", node.Engine, node.ID, node.Addresses[0], node.Port) p.codec.Notify("node/discovered", node.withPrimaryIP()) } else { - log.Printf("manual node updated: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + log.Printf("manual node updated: %s/%s (%s:%d)", node.Engine, node.ID, node.Addresses[0], node.Port) p.codec.Notify("node/updated", node.withPrimaryIP()) } case "node/remove-manual": - var params SelectParams + var params struct { + ID string `json:"id"` + Engine string `json:"engine"` + } if err := json.Unmarshal(msg.Params, ¶ms); err != nil { - p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}") + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\", \"engine\": \"...\"}") + return + } + if !isOpenAIEngine(params.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) return } - removed := p.discovery.RemoveManual(params.ID) + removed := p.discovery.RemoveManual(params.Engine, params.ID) if err := p.codec.Respond(msg.ID, map[string]bool{"removed": removed}); err != nil { log.Printf("failed to respond to node/remove-manual: %v", err) } - if removed { + // The node itself is only gone once its last engine's entry is dropped; + // until then it is still a routing target for the other engine, so the + // selection and the node/removed event must not fire early. + if removed && !p.discovery.IsManual(params.ID) { log.Printf("manual node removed: %s", params.ID) p.selectedMu.Lock() if p.selectedID == params.ID { @@ -2022,6 +2034,10 @@ func (p *Proxy) handleMessage(msg *Message) { p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"host\",\"port\",\"healthy\"}") return } + if !isOpenAIEngine(b.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) + return + } p.setLocalBackend(b) slog.Info("local backend updated", "engine", b.Engine, "host", b.Host, "port", b.Port, "healthy", b.Healthy) if err := p.codec.Respond(msg.ID, map[string]bool{"ok": true}); err != nil { diff --git a/services/nvpair-engine-manager/README.md b/services/nvpair-engine-manager/README.md index 8c3dfa3c..92dc9ca9 100644 --- a/services/nvpair-engine-manager/README.md +++ b/services/nvpair-engine-manager/README.md @@ -5,8 +5,8 @@ SPDX-License-Identifier: Apache-2.0 # nvpair-engine-manager -A config-driven control plane for local inference engines (Ollama today; -Intel/others via a dropped-in manifest). It manages everything about an +A config-driven control plane for local inference engines (Ollama, LM Studio and +vLLM today; others via a dropped-in manifest). It manages everything about an engine **except serving inference**: detect, user-mode install, start/stop/restart, health, and config-declared actions. Adding an engine is a JSON manifest, not code. @@ -35,10 +35,11 @@ Requests (caller → service): | `engine:status` | `{ engine }` | `EngineStatus` | | `engine:install` | `{ engine, start?, port?, bind? }` | `EngineStatus` (after install; also starts it if `start:true`) | | `engine:uninstall` | `{ engine }` | `EngineStatus` (after removal) | -| `engine:start` | `{ engine, port?, bind? }` | `EngineStatus` (after readiness) | +| `engine:start` | `{ engine, port?, bind?, model? }` | `EngineStatus` (after readiness). `model` is a one-shot served-model override for this launch, mirroring `port` | | `engine:stop` | `{ engine }` | `EngineStatus` | | `engine:restart` | `{ engine }` | `EngineStatus` | | `engine:set-port` | `{ engine, port }` | `EngineStatus` (after rebind) | +| `engine:set-model` | `{ engine, model }` | `EngineStatus` (after restart). The persistent served-model setter for an engine that runs one model per process; an empty `model` removes the override | | `engine:action` | `{ engine, action, params }` | the engine's raw response. `action:"pull_model"` is streamed: it emits live `engine:pull-progress` notifications and returns the pull's terminal result (see below). An action whose manifest declares `restart_after` (LM Studio's `delete_model`) restarts a running engine before replying, so the response also means the engine is back and healthy | | `engine:logs` | `{ engine }` | `{ lines: [LogLine] }` | | `engine:errors` | — | `{ errors: [ServiceError] }` | @@ -51,7 +52,9 @@ Requests (caller → service): | `shutdown` | — | `null` | | `log/set-level` | `{ level }` | `{ level }` | -`EngineStatus` = `{ engine, display_name, installed, running, healthy, port }`. +`EngineStatus` = `{ engine, display_name, installed, running, healthy, port, model? }`. +`model` is the engine's configured served model, present only for an engine that +serves one model per process (vLLM) and only once one has been chosen. Notifications (service → caller): `engine:ready{version}`, `engine:state-changed{EngineStatus}`, @@ -85,6 +88,19 @@ pinned cluster peer. See "Remote engine management" below. service side, so the read loop never blocks and their responses arrive when the op finishes. +`engine:set-model` is the persistent served-model setter, for an engine whose +launch command names the model it serves (vLLM's `vllm serve `). It +writes a `{ engine, runtime: { model } }` delta into the same per-user override +file `engine:set-port` uses — both setters read-modify-write it, so neither +clobbers the other, and the file is removed only once no override remains — and +applies the choice by restarting a running engine onto it. An empty model +removes the override, which leaves an engine that requires one unable to start +until a model is chosen again. A running, **adopted** engine is refused for the +same reason `engine:set-port` refuses one. Starting an engine whose launch +template substitutes `{model}` with none configured fails with an actionable +error rather than spawning a process that cannot serve anything. The one-shot +counterpart is `engine:start {model}`, which reverts on the next restart. + `engine:set-port` is the **persistent** port setter (distinct from the one-shot `engine:start {port}` override, which reverts on the next restart). It validates `1-65535`, persists the choice as a manifest override (a @@ -113,7 +129,9 @@ when one is pinned (an unpinned fetch runs with a loud warning). Start waits for the readiness probe, then runs a periodic health probe; an unexpected exit is reported. The bundled Ollama manifest allows up to ten minutes for startup because GPU discovery can exceed the previous 30-second -allowance on supported Windows systems. The deadline remains finite: if Ollama +allowance on supported Windows systems. The bundled vLLM manifest allows thirty: +its first start downloads the model's weights and captures CUDA graphs before it +serves anything. The deadline remains finite: if Ollama never serves its readiness endpoint, engine-manager stops the owned process and reports the failed start. Stop sends one stop signal and waits for the engine to exit, with no timeout: SIGTERM to the process group on Unix (graceful, no diff --git a/services/nvpair-engine-manager/actions.go b/services/nvpair-engine-manager/actions.go index 372ee104..286dbbc0 100644 --- a/services/nvpair-engine-manager/actions.go +++ b/services/nvpair-engine-manager/actions.go @@ -162,7 +162,7 @@ func (e *Executor) runRemovePathAction(ctx context.Context, st *engineState, act var pm map[string]any if err := json.Unmarshal(params, &pm); err == nil { for k, v := range pm { - if allowedPlaceholders[k] { + if reservedActionPlaceholders[k] { continue } vars[k] = fmt.Sprint(v) @@ -224,7 +224,7 @@ func (e *Executor) runCmdAction(ctx context.Context, st *engineState, act Action var pm map[string]any if err := json.Unmarshal(params, &pm); err == nil { for k, v := range pm { - if allowedPlaceholders[k] { + if reservedActionPlaceholders[k] { continue } vars[k] = fmt.Sprint(v) diff --git a/services/nvpair-engine-manager/executor.go b/services/nvpair-engine-manager/executor.go index 85ca5536..c81dec5f 100644 --- a/services/nvpair-engine-manager/executor.go +++ b/services/nvpair-engine-manager/executor.go @@ -33,6 +33,11 @@ type EngineStatus struct { Running bool `json:"running"` Healthy bool `json:"healthy"` Port int `json:"port,omitempty"` + // Model is the engine's configured served model, for an engine that runs one + // model per process (vLLM). Empty for every other engine, and for one that + // needs a model but has none chosen yet — the state in which a start is + // refused. Set by engine:set-model; see runtime.model. + Model string `json:"model,omitempty"` } // engineState is the per-engine runtime state. diff --git a/services/nvpair-engine-manager/lifecycle.go b/services/nvpair-engine-manager/lifecycle.go index c10244b0..9b9d030f 100644 --- a/services/nvpair-engine-manager/lifecycle.go +++ b/services/nvpair-engine-manager/lifecycle.go @@ -57,6 +57,9 @@ func (e *Executor) waitDetect(engine string, want bool, timeout time.Duration) b type startOpts struct { Port int Bind string + // Model overrides runtime.model for this launch only, for an engine that + // serves one model per process. "" => the manifest's configured model. + Model string } // effectiveBind picks the listen address substituted as {host}: a per-call @@ -165,6 +168,16 @@ func (e *Executor) doStart(ctx context.Context, st *engineState, engine string, if !pathInstalled { return fmt.Errorf("engine %q is not installed", engine) } + model := rt.Model + if opts.Model != "" { + model = opts.Model + } + // An engine whose launch template substitutes {model} serves exactly one + // model per process and has nothing to serve until one is chosen. Fail with + // guidance rather than spawning a process that cannot answer. + if rt.referencesModel() && strings.TrimSpace(model) == "" { + return fmt.Errorf("cannot start %s: it serves one model per process, and no model is configured — choose the model to serve in Engine settings", st.manifest.DisplayName) + } vars := map[string]string{ "host": effectiveBind(rt.Bind, opts.Bind), "port": strconv.Itoa(port), @@ -173,6 +186,9 @@ func (e *Executor) doStart(ctx context.Context, st *engineState, engine string, if rt.CLI != "" { vars["cli"] = expandPath(rt.CLI) } + if model != "" { + vars["model"] = model + } st.mu.Lock() st.port = port @@ -227,7 +243,7 @@ func (e *Executor) bringUpProcess(ctx context.Context, st *engineState, engine s } vars["bin"] = binPath - args, err := resolveArgs(rt.Args, vars) + args, err := resolveArgs(rt.launchArgs(), vars) if err != nil { return err } diff --git a/services/nvpair-engine-manager/manager.go b/services/nvpair-engine-manager/manager.go index 80d572b4..01ea68f5 100644 --- a/services/nvpair-engine-manager/manager.go +++ b/services/nvpair-engine-manager/manager.go @@ -40,6 +40,10 @@ type opParam struct { Port int `json:"port,omitempty"` Bind string `json:"bind,omitempty"` Start bool `json:"start,omitempty"` + // Model is the one-shot served-model override for engine:start, mirroring + // Port: it applies to this launch only and is never persisted. Use + // engine:set-model for the persistent choice. + Model string `json:"model,omitempty"` } type actionParam struct { @@ -56,6 +60,16 @@ type setPortParam struct { Port int `json:"port"` } +// setModelParam is the engine:set-model input: the engine whose served model to +// change and the model id. It is the persistent counterpart of the one-shot +// engine:start {model} override — the choice is written as a manifest override +// exactly the way engine:set-port writes the port, and an empty model removes +// the override (back to the bundled default). +type setModelParam struct { + Engine string `json:"engine"` + Model string `json:"model"` +} + // Manager is the engine-manager's JSON-RPC front end. It dispatches the // engine:* surface to the Executor. Long-running operations (install, // start, stop, restart, action) run in their own goroutine so the read @@ -248,6 +262,9 @@ func (m *Manager) handleMessage(ctx context.Context, msg *Message) { case "engine:set-port": go m.runSetPort(ctx, msg) + case "engine:set-model": + go m.runSetModel(ctx, msg) + case "internal:set-reserved-port": var p struct { Port int `json:"port"` @@ -294,7 +311,7 @@ func (m *Manager) runOp(ctx context.Context, msg *Message) { return } start := func() error { - return m.exec.StartWith(ctx, p.Engine, startOpts{Port: p.Port, Bind: p.Bind}) + return m.exec.StartWith(ctx, p.Engine, startOpts{Port: p.Port, Bind: p.Bind, Model: p.Model}) } var err error switch msg.Method { @@ -338,6 +355,21 @@ func (m *Manager) runSetPort(ctx context.Context, msg *Message) { m.respondOrErr(msg, st, err) } +// runSetModel persists the model an engine serves (as a manifest override) and +// applies it, responding with the engine's resulting status. +func (m *Manager) runSetModel(ctx context.Context, msg *Message) { + var p setModelParam + if !m.parse(msg, &p) { + return + } + if p.Engine == "" { + m.codec.RespondError(msg.ID, -32602, "engine is required") + return + } + st, err := m.exec.SetModel(ctx, p.Engine, p.Model) + m.respondOrErr(msg, st, err) +} + // runModels answers engine:models with the union of running engines' model // lists plus per-engine attribution (modelsByEngine). It runs async (like // engine:action) because it makes one loopback HTTP call per running engine and diff --git a/services/nvpair-engine-manager/manifests/vllm.json b/services/nvpair-engine-manager/manifests/vllm.json new file mode 100644 index 00000000..14fa7c99 --- /dev/null +++ b/services/nvpair-engine-manager/manifests/vllm.json @@ -0,0 +1,51 @@ +{ + "engine": "vllm", + "display_name": "vLLM", + "manifest_version": 1, + "detect": ["{install_dir}/venv/bin/vllm", "~/.vllm/bin/vllm"], + "install": { + "fetch": { "url": "https://astral.sh/uv/install.sh" }, + "run": [ + "sh", + "-c", + "set -e; mkdir -p \"$1\"; UV_INSTALL_DIR=\"$1/uv\" UV_NO_MODIFY_PATH=1 sh \"$2\"; \"$1/uv/uv\" venv \"$1/venv\"; \"$1/uv/uv\" pip install --python \"$1/venv/bin/python\" vllm", + "sh", + "{install_dir}", + "{download}" + ], + "mode": "user" + }, + "uninstall": { + "run": ["rm", "-rf", "{install_dir}"] + }, + "runtime": { + "mode": "process", + "bin": "{install_dir}/venv/bin/vllm", + "args": ["serve", "{model}", "--port", "{port}", "--host", "{host}"], + "extra_args": [], + "port": 8000, + "bind": "127.0.0.1", + "ready": { "http": "http://127.0.0.1:{port}/health", "status": 200, "timeout_s": 1800 }, + "health": { "http": "http://127.0.0.1:{port}/health", "status": 200, "interval_s": 10 } + }, + "platforms": { + "linux/amd64": {}, + "linux/arm64": {} + }, + "actions": { + "list_models": { + "description": "List the model this vLLM process serves (OpenAI-compatible).", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "loaded_models": { + "description": "List models resident in memory. vLLM serves exactly one model per process and keeps it resident, so this is the same list as list_models.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "chat": { + "description": "OpenAI-compatible chat completion (params: {\"model\": \"\", \"messages\": [...]}).", + "http": { "method": "POST", "path": "/v1/chat/completions", "body_schema": { "model": "string", "messages": "array" } } + } + } +} diff --git a/services/nvpair-engine-manager/registry.go b/services/nvpair-engine-manager/registry.go index b44ec6e0..09b978e6 100644 --- a/services/nvpair-engine-manager/registry.go +++ b/services/nvpair-engine-manager/registry.go @@ -25,11 +25,11 @@ import ( // schema growth stays backward compatible. const ManifestSchemaVersion = 1 -// allowedPlaceholders is the set of `{token}`s the runner can resolve -// at execution time. Validation rejects any other token so a typo in -// a manifest fails at load with a clear message rather than at run -// time with a mangled command line. -var allowedPlaceholders = map[string]bool{ +// reservedActionPlaceholders is the set of `{token}`s the runner always +// resolves itself from trusted state (the manifest, the chosen port, the +// install dir). An engine:action caller's params are refused these names, so a +// hostile param can never hijack one — notably {cli}, which is argv[0]. +var reservedActionPlaceholders = map[string]bool{ "bin": true, "cli": true, "host": true, @@ -39,6 +39,27 @@ var allowedPlaceholders = map[string]bool{ "models_dir": true, } +// allowedPlaceholders is the set of `{token}`s a manifest may template. +// Validation rejects any other token so a typo fails at load with a clear +// message rather than at run time with a mangled command line. It is the +// reserved set plus {model}: unlike the reserved names, model is a value the +// caller legitimately supplies per call on a model action (`lms get {model}`) +// as well as one the runner resolves from runtime.model at launch, so it is +// resolvable without being caller-proof. +var allowedPlaceholders = buildAllowedPlaceholders() + +func buildAllowedPlaceholders() map[string]bool { + out := make(map[string]bool, len(reservedActionPlaceholders)+1) + for k := range reservedActionPlaceholders { + out[k] = true + } + // model is the served model of an engine that runs one model per process + // (vLLM's `vllm serve `): read from runtime.model at start, set + // persistently by engine:set-model, overridable per call by engine:start. + out["model"] = true + return out +} + var placeholderRe = regexp.MustCompile(`\{([a-zA-Z_][a-zA-Z0-9_]*)\}`) // engineNameRe restricts engine names to a safe charset — the name is @@ -113,17 +134,28 @@ type Fetch struct { // (e.g. LM Studio's `lms`); liveness = the readiness/health probe, // and Stop.Cmd brings it down. type Runtime struct { - Mode string `json:"mode,omitempty"` - Bin string `json:"bin,omitempty"` - Args []string `json:"args,omitempty"` - Env map[string]string `json:"env,omitempty"` + Mode string `json:"mode,omitempty"` + Bin string `json:"bin,omitempty"` + Args []string `json:"args,omitempty"` + // ExtraArgs are appended after Args on every launch. They exist so a + // per-user manifest override can add engine flags (vLLM's + // --gpu-memory-utilization, --max-model-len, …) without restating the + // bundled Args array, which a deep merge would replace wholesale. + ExtraArgs []string `json:"extra_args,omitempty"` + Env map[string]string `json:"env,omitempty"` Port int `json:"port"` // 0 => auto-assign a free loopback port Bind string `json:"bind,omitempty"` // listen addr, substituted as {host}; "" => 127.0.0.1 Start [][]string `json:"start,omitempty"` // command mode: ordered bring-up commands // CLI is the engine's control-CLI path for this platform, referenced // elsewhere as {cli}. It lets the manifest's global actions resolve // to the correct per-OS binary (e.g. lms.exe vs lms). - CLI string `json:"cli,omitempty"` + CLI string `json:"cli,omitempty"` + // Model is the model this engine serves, for an engine that serves exactly + // one model per process (vLLM). It is substituted as {model} in Args / + // ExtraArgs / Start. Empty means "not configured": starting an engine whose + // launch template references {model} then fails with an actionable error + // instead of spawning a process that cannot serve anything. + Model string `json:"model,omitempty"` Ready *Probe `json:"ready,omitempty"` Stop *StopSpec `json:"stop,omitempty"` Health *Probe `json:"health,omitempty"` @@ -394,29 +426,92 @@ func (r *Registry) LoadOverrideDir(dir string) error { return nil } -// bundledDefaultPort returns the host-platform runtime.port from the bundled -// (un-overridden) manifest for an engine, used to decide whether a chosen -// port is back at the default (so its override file can be dropped). -func (r *Registry) bundledDefaultPort(engine string) (int, bool) { +// bundledHostRuntime returns the host-platform runtime block from the bundled +// (un-overridden) manifest for an engine. It is how the persistent setters +// decide whether a chosen value is back at the bundled default, so that +// field's override can be dropped instead of pinned forever. +func (r *Registry) bundledHostRuntime(engine string) (Runtime, bool) { raw, ok := r.bundledRaw[engine] if !ok { - return 0, false + return Runtime{}, false } merged, err := applyPlatformDefaults(raw) if err != nil { - return 0, false + return Runtime{}, false } var m Manifest if err := json.Unmarshal(merged, &m); err != nil { - return 0, false + return Runtime{}, false } p, ok := m.HostPlatform() + if !ok { + return Runtime{}, false + } + return p.Runtime, true +} + +// bundledDefaultPort returns the host-platform runtime.port from the bundled +// (un-overridden) manifest for an engine, used to decide whether a chosen +// port is back at the default (so its override can be dropped). +func (r *Registry) bundledDefaultPort(engine string) (int, bool) { + rt, ok := r.bundledHostRuntime(engine) if !ok { return 0, false } - return p.Runtime.Port, true + return rt.Port, true } +// bundledDefaultModel returns the host-platform runtime.model from the bundled +// manifest — the counterpart of bundledDefaultPort for engine:set-model. +func (r *Registry) bundledDefaultModel(engine string) (string, bool) { + rt, ok := r.bundledHostRuntime(engine) + if !ok { + return "", false + } + return rt.Model, true +} + +// launchArgs is the full argv tail for a process-mode launch: the manifest's +// runtime.args followed by runtime.extra_args, before placeholder resolution. +func (r *Runtime) launchArgs() []string { + if len(r.ExtraArgs) == 0 { + return r.Args + } + out := make([]string, 0, len(r.Args)+len(r.ExtraArgs)) + out = append(out, r.Args...) + out = append(out, r.ExtraArgs...) + return out +} + +// referencesModel reports whether this runtime's launch template substitutes +// {model} anywhere — the signal that the engine serves one model per process +// and cannot start until a model has been chosen. +func (r *Runtime) referencesModel() bool { + if referencesModelToken(r.Bin) { + return true + } + for _, s := range r.launchArgs() { + if referencesModelToken(s) { + return true + } + } + for _, cmd := range r.Start { + for _, s := range cmd { + if referencesModelToken(s) { + return true + } + } + } + for _, v := range r.Env { + if referencesModelToken(v) { + return true + } + } + return false +} + +func referencesModelToken(s string) bool { return strings.Contains(s, "{model}") } + // mergeOntoBundled deep-merges an override manifest's raw JSON onto the // bundled manifest of the same engine (override wins). When the override // declares no engine, or no bundled base exists for it, the override is @@ -719,6 +814,7 @@ func (m *Manifest) templatedStrings() []string { } out = append(out, p.Runtime.Bin) out = append(out, p.Runtime.Args...) + out = append(out, p.Runtime.ExtraArgs...) for _, cmd := range p.Runtime.Start { out = append(out, cmd...) } diff --git a/services/nvpair-engine-manager/setmodel_test.go b/services/nvpair-engine-manager/setmodel_test.go new file mode 100644 index 00000000..228165cc --- /dev/null +++ b/services/nvpair-engine-manager/setmodel_test.go @@ -0,0 +1,289 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +// hostModel returns the effective host-platform runtime.model for an engine. +func hostModel(t *testing.T, reg *Registry, engine string) string { + t.Helper() + m, ok := reg.Get(engine) + if !ok { + t.Fatalf("engine %q not in registry", engine) + } + p, ok := m.HostPlatform() + if !ok { + t.Fatalf("engine %q has no host platform block", engine) + } + return p.Runtime.Model +} + +// vllmManifest is the bundled vLLM manifest re-keyed onto the running host so +// the Linux-only engine's lifecycle can be exercised on any developer machine. +// Only the platform key changes; every field under test is the bundled one. +func vllmManifest(t *testing.T) *Manifest { + t.Helper() + reg := NewRegistry() + if err := reg.LoadFS(bundledManifests, "manifests"); err != nil { + t.Fatalf("LoadFS bundled: %v", err) + } + m, ok := reg.Get("vllm") + if !ok { + t.Fatal("vllm manifest not bundled") + } + linux, ok := m.PlatformFor("linux", "amd64") + if !ok { + t.Fatal("vllm manifest has no linux/amd64 block") + } + clone := *m + clone.Platforms = map[string]Platform{runtime.GOOS + "/" + runtime.GOARCH: *linux} + return &clone +} + +// TestVLLMManifestLoadsAndValidates proves the bundled vLLM manifest passes +// schema validation, including the {model} placeholder in its launch args, and +// that it is offered on Linux only. +func TestVLLMManifestLoadsAndValidates(t *testing.T) { + reg := NewRegistry() + if err := reg.LoadFS(bundledManifests, "manifests"); err != nil { + t.Fatalf("LoadFS bundled: %v", err) + } + m, ok := reg.Get("vllm") + if !ok { + t.Fatal("vllm manifest not bundled") + } + if m.DisplayName != "vLLM" { + t.Errorf("display_name = %q, want %q", m.DisplayName, "vLLM") + } + want := map[string]bool{"linux/amd64": true, "linux/arm64": true} + for key := range m.Platforms { + if !want[key] { + t.Errorf("unexpected platform %q: vLLM ships on Linux only", key) + } + delete(want, key) + } + for key := range want { + t.Errorf("missing platform %q", key) + } + + p, ok := m.PlatformFor("linux", "amd64") + if !ok { + t.Fatal("no linux/amd64 block") + } + if p.Runtime.modeOrDefault() != "process" { + t.Errorf("runtime.mode = %q, want process (vllm serve is a foreground server, not a bring-up command)", p.Runtime.modeOrDefault()) + } + if p.Runtime.Port != 8000 { + t.Errorf("runtime.port = %d, want 8000", p.Runtime.Port) + } + if !p.Runtime.referencesModel() { + t.Error("vLLM's launch template must substitute {model}") + } + if p.Runtime.Ready == nil || p.Runtime.Ready.TimeoutS != 1800 { + t.Errorf("runtime.ready timeout = %+v, want 1800s (first start downloads weights)", p.Runtime.Ready) + } + if p.Runtime.Health == nil || !strings.HasSuffix(p.Runtime.Health.HTTP, "/health") { + t.Errorf("runtime.health = %+v, want the /health endpoint", p.Runtime.Health) + } + for _, name := range []string{"list_models", "loaded_models", "chat"} { + if _, ok := m.Actions[name]; !ok { + t.Errorf("missing action %q", name) + } + } + for _, name := range []string{"pull_model", "load_model", "unload_model", "delete_model"} { + if _, ok := m.Actions[name]; ok { + t.Errorf("action %q must not be declared: vLLM has no model management surface", name) + } + } +} + +// TestVLLMHasNoHostPlatformOffHostLinux proves an engine with no block for the +// running platform is simply not offered there: it still lists (so the UI can +// say "unavailable on this system") but every lifecycle op refuses. +func TestVLLMHasNoHostPlatformOffHostLinux(t *testing.T) { + if runtime.GOOS == "linux" { + t.Skip("vLLM has a platform block on Linux; this covers the other hosts") + } + ex := newBundledExecutor(t, t.TempDir()) + var found *EngineStatus + for _, st := range ex.GetInstalled() { + if st.Engine == "vllm" { + s := st + found = &s + } + } + if found == nil { + t.Fatal("vllm missing from engine:get-installed") + } + if found.Installed || found.Running { + t.Errorf("vllm reported present on %s: %+v", runtime.GOOS, *found) + } + if _, err := ex.state("vllm"); err == nil { + t.Error("expected vllm to have no host platform block on this OS") + } + if err := ex.Start(context.Background(), "vllm"); err == nil { + t.Error("expected Start to refuse an engine with no host platform block") + } +} + +// TestStartWithoutConfiguredModelIsActionable proves an engine whose launch +// template needs {model} refuses to spawn until a model is chosen, with a +// message that tells the user what to do rather than leaking a placeholder. +func TestStartWithoutConfiguredModelIsActionable(t *testing.T) { + m := vllmManifest(t) + // Move off 8000 so an unrelated listener on this host cannot pre-empt the + // model check with a port-occupied error. + key := runtime.GOOS + "/" + runtime.GOARCH + plat := m.Platforms[key] + free, err := freePort() + if err != nil { + t.Fatal(err) + } + plat.Runtime.Port = free + // Point detect at a real file so the refusal is about the missing model, + // not about the engine being uninstalled. + plat.Detect = []string{fakeEngineBin} + m.Platforms[key] = plat + + ex := newTestExecutor(t, m) + err = ex.Start(context.Background(), "vllm") + if err == nil { + t.Fatal("expected Start to refuse with no model configured") + } + for _, want := range []string{"vLLM", "one model per process", "Engine settings"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q does not mention %q", err.Error(), want) + } + } + if strings.Contains(err.Error(), "{model}") { + t.Errorf("error leaks the raw placeholder: %q", err.Error()) + } +} + +// TestSetModelPersistsAsManifestOverride proves engine:set-model survives a +// "restart" through the same override-dir merge engine:set-port uses, and that +// clearing it removes the override again. It drives ollama because the writer is +// engine-agnostic and ollama has a platform block on every host; vLLM's own +// Linux-only manifest is covered by the manifest test above. +func TestSetModelPersistsAsManifestOverride(t *testing.T) { + dir := t.TempDir() + ex := newBundledExecutor(t, dir) + if err := ex.persistModel("ollama", "Qwen/Qwen3-8B"); err != nil { + t.Fatalf("persistModel: %v", err) + } + if got := hostModel(t, loadWithOverrides(t, dir), "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model after reload = %q, want Qwen/Qwen3-8B", got) + } + + if err := ex.persistModel("ollama", ""); err != nil { + t.Fatalf("persistModel clear: %v", err) + } + if got := hostModel(t, loadWithOverrides(t, dir), "ollama"); got != "" { + t.Errorf("model after clear = %q, want empty", got) + } + if _, err := os.Stat(filepath.Join(dir, "ollama.json")); !os.IsNotExist(err) { + t.Errorf("override file should be gone once no override remains: %v", err) + } +} + +// TestSetModelAndSetPortDoNotClobberEachOther is the regression guard for the +// two persistent setters sharing one override file: each writes its own +// runtime field and must leave the other's alone, in either order. +func TestSetModelAndSetPortDoNotClobberEachOther(t *testing.T) { + dir := t.TempDir() + ex := newBundledExecutor(t, dir) + bundledPort := hostPort(t, ex.reg, "ollama") + + if err := ex.persistModel("ollama", "Qwen/Qwen3-8B"); err != nil { + t.Fatalf("persistModel: %v", err) + } + if err := ex.persistPort("ollama", 9001); err != nil { + t.Fatalf("persistPort: %v", err) + } + reg := loadWithOverrides(t, dir) + if got := hostModel(t, reg, "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model lost when the port was set: got %q", got) + } + if got := hostPort(t, reg, "ollama"); got != 9001 { + t.Errorf("port = %d, want 9001", got) + } + + // Back to the bundled default port: the port override drops, the model stays. + if err := ex.persistPort("ollama", bundledPort); err != nil { + t.Fatalf("persistPort default: %v", err) + } + reg = loadWithOverrides(t, dir) + if got := hostModel(t, reg, "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model lost when the port returned to default: got %q", got) + } + if got := hostPort(t, reg, "ollama"); got != bundledPort { + t.Errorf("port = %d, want the bundled %d", got, bundledPort) + } + + raw, err := os.ReadFile(filepath.Join(dir, "ollama.json")) + if err != nil { + t.Fatalf("read override: %v", err) + } + var doc struct { + Runtime map[string]any `json:"runtime"` + } + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + if _, ok := doc.Runtime["port"]; ok { + t.Errorf("default port should not stay pinned in the override: %s", raw) + } +} + +// TestSetModelAppliesToTheLaunchArgs proves the configured model reaches the +// spawned process's argv, and that runtime.extra_args is appended after it. +func TestSetModelAppliesToTheLaunchArgs(t *testing.T) { + m := vllmManifest(t) + key := runtime.GOOS + "/" + runtime.GOARCH + plat := m.Platforms[key] + // Drive the fake engine instead of a real vLLM: it takes FAKE_ADDR, so the + // vLLM argv is carried purely to prove substitution, and extra_args adds a + // flag the fake ignores. + plat.Detect = []string{fakeEngineBin} + plat.Runtime.Bin = fakeEngineBin + plat.Runtime.Model = "Qwen/Qwen3-8B" + plat.Runtime.Args = []string{"echo", "{model}", "--port", "{port}"} + plat.Runtime.ExtraArgs = []string{"--gpu-memory-utilization", "0.9"} + plat.Runtime.Ready = nil + plat.Runtime.Health = nil + m.Platforms[key] = plat + + ex := newTestExecutor(t, m) + st, err := ex.state("vllm") + if err != nil { + t.Fatal(err) + } + if err := ex.Start(context.Background(), "vllm"); err != nil { + t.Fatalf("start: %v", err) + } + t.Cleanup(func() { _ = ex.Stop("vllm") }) + + // `echo` prints its args and exits, so the captured stdout is the resolved argv. + captured := func() string { + var line string + for _, l := range st.logs.snapshot() { + line += l.Text + "\n" + } + return line + } + waitFor(t, 5*time.Second, func() bool { return strings.Contains(captured(), "Qwen/Qwen3-8B") }) + line := captured() + if !strings.Contains(line, "--gpu-memory-utilization 0.9") { + t.Errorf("extra_args not appended to the launch argv: %q", line) + } +} diff --git a/services/nvpair-engine-manager/setport.go b/services/nvpair-engine-manager/setport.go index 5a9554c2..98fca571 100644 --- a/services/nvpair-engine-manager/setport.go +++ b/services/nvpair-engine-manager/setport.go @@ -10,6 +10,7 @@ import ( "fmt" "os" "path/filepath" + "strings" ) func canMoveAdoptedEngine(rt Runtime) bool { @@ -93,50 +94,133 @@ func (e *Executor) SetPort(ctx context.Context, engine string, port int) (Engine return e.snapshot(engine, st), nil } -// persistPort writes (or removes) the per-engine manifest override that pins -// runtime.port so the chosen port survives a restart. For a bundled engine it -// writes only the {engine, runtime:{port}} delta (which deep-merges onto the -// bundled manifest at load), and removes the override entirely when the port -// is back at the bundled default — keeping the override set minimal. For a -// non-bundled engine (a full manifest that lives only in the override dir) it -// merges the port into the existing file rather than clobbering it. Atomic -// (tmp + rename). +// persistPort pins runtime.port in the per-engine manifest override so the +// chosen port survives a restart. Back at the bundled default, the port key is +// dropped again rather than pinned forever. See persistRuntimeField. func (e *Executor) persistPort(engine string, port int) error { + def, bundled := e.reg.bundledDefaultPort(engine) + return e.persistRuntimeField(engine, "port", port, bundled && def == port) +} + +// persistModel pins runtime.model in the same override file persistPort writes. +func (e *Executor) persistModel(engine, model string) error { + def, bundled := e.reg.bundledDefaultModel(engine) + return e.persistRuntimeField(engine, "model", model, bundled && def == model) +} + +// persistRuntimeField writes one runtime. value into the per-engine +// manifest override that deep-merges onto the bundled manifest at load, so the +// manifest stays the single source of truth for the setting and it is restored +// on the next start with no separate store. +// +// The file is always read-modify-written rather than replaced with a one-key +// delta: port and model are set independently, and a wholesale write would +// silently drop whichever the caller wasn't changing. atDefault removes just +// that field; the file itself is unlinked only once no override remains, which +// keeps the override set minimal without discarding a sibling setting. Atomic +// (tmp + rename). +func (e *Executor) persistRuntimeField(engine, field string, value any, atDefault bool) error { if e.overrideDir == "" { - return fmt.Errorf("no config directory available to persist the port") + return fmt.Errorf("no config directory available to persist the %s", field) } if err := os.MkdirAll(e.overrideDir, 0o755); err != nil { return fmt.Errorf("create override dir: %w", err) } path := filepath.Join(e.overrideDir, engine+".json") - delta := map[string]any{"engine": engine, "runtime": map[string]any{"port": port}} - - if def, ok := e.reg.bundledDefaultPort(engine); ok { - // Bundled engine: back to default ⇒ drop the override; else persist - // just the delta so bundled upgrades to everything else still apply. - if def == port { - if err := os.Remove(path); err != nil && !os.IsNotExist(err) { - return fmt.Errorf("remove override: %w", err) - } - return nil + + doc := map[string]any{} + existing, err := os.ReadFile(path) + switch { + case err == nil: + if err := json.Unmarshal(existing, &doc); err != nil { + return fmt.Errorf("parse override %s: %w", path, err) } - return writeJSONAtomic(path, delta) + case !os.IsNotExist(err): + return fmt.Errorf("read override: %w", err) } - // Non-bundled engine: the full manifest lives only here, so merge the - // port into it rather than overwriting the file with a partial. - existing, err := os.ReadFile(path) + rt, _ := doc["runtime"].(map[string]any) + if rt == nil { + rt = map[string]any{} + } + if atDefault { + delete(rt, field) + } else { + rt[field] = value + } + doc["engine"] = engine + doc["runtime"] = rt + + // A bundled engine's override exists only to carry deltas, so an override + // with none left is removed. A non-bundled engine's full manifest lives + // only here and is never removed. + if _, bundled := e.reg.bundledDefaultPort(engine); bundled && len(rt) == 0 { + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove override: %w", err) + } + return nil + } + return writeJSONAtomic(path, doc) +} + +// SetModel persists the model an engine serves as a manifest override and +// applies it: a running engine is restarted onto the new model, since an engine +// that templates {model} into its launch command can only serve the model it +// was started with. Empty removes the override (back to the bundled default), +// which leaves an engine that requires a model unable to start until one is +// chosen again. Held under the engine's op lock, like SetPort. +// +// A running, adopted engine is refused for the same reason SetPort refuses one: +// NVPAIR cannot restart a process it did not start, and the externally-managed +// instance would keep serving its own model while the manifest claimed another. +func (e *Executor) SetModel(ctx context.Context, engine, model string) (EngineStatus, error) { + model = strings.TrimSpace(model) + st, err := e.state(engine) if err != nil { - if os.IsNotExist(err) { - return writeJSONAtomic(path, delta) + return EngineStatus{}, err + } + st.opMu.Lock() + defer st.opMu.Unlock() + + st.mu.Lock() + wasRunning := st.running + adopted := st.adopted + st.mu.Unlock() + + if wasRunning && adopted && !canMoveAdoptedEngine(st.plat.Runtime) { + return EngineStatus{}, fmt.Errorf("cannot change the model %s serves: it is running under external management (NVPAIR adopted it rather than starting it), so NVPAIR cannot restart it — stop it in its own app first, then set the model", engine) + } + + oldModel := st.plat.Runtime.Model + if wasRunning { + if err := e.doStop(st, engine); err != nil { + return EngineStatus{}, err } - return fmt.Errorf("read override: %w", err) } - var m map[string]any - if err := json.Unmarshal(existing, &m); err != nil { - return fmt.Errorf("parse override %s: %w", path, err) + + if err := e.persistModel(engine, model); err != nil { + if !wasRunning { + return EngineStatus{}, err + } + st.mu.Lock() + st.plat.Runtime.Model = oldModel + st.mu.Unlock() + restartErr := e.doStart(ctx, st, engine, startOpts{}) + return EngineStatus{}, errors.Join(err, restartErr) + } + + st.mu.Lock() + st.plat.Runtime.Model = model + st.mu.Unlock() + + if wasRunning { + if err := e.doStart(ctx, st, engine, startOpts{}); err != nil { + return EngineStatus{}, err + } + } else { + e.emitState(engine) } - return writeJSONAtomic(path, deepMerge(m, delta)) + return e.snapshot(engine, st), nil } // writeJSONAtomic marshals v and writes it to path via a tmp file + rename so diff --git a/services/nvpair-engine-manager/status.go b/services/nvpair-engine-manager/status.go index 75419cb7..674f9c46 100644 --- a/services/nvpair-engine-manager/status.go +++ b/services/nvpair-engine-manager/status.go @@ -124,7 +124,7 @@ func (e *Executor) Errors() []serviceError { func (e *Executor) snapshot(engine string, st *engineState) EngineStatus { st.mu.Lock() defer st.mu.Unlock() - return EngineStatus{ + status := EngineStatus{ Engine: engine, DisplayName: st.manifest.DisplayName, Installed: st.installed, @@ -132,6 +132,10 @@ func (e *Executor) snapshot(engine string, st *engineState) EngineStatus { Healthy: st.healthy, Port: st.port, } + if st.plat != nil { + status.Model = st.plat.Runtime.Model + } + return status } // reconcilePresence reconciles filesystem detection with a fixed-port engine diff --git a/services/nvpair-engine-manager/testdata/fakeengine/main.go b/services/nvpair-engine-manager/testdata/fakeengine/main.go index 76626c20..e1ca9fdb 100644 --- a/services/nvpair-engine-manager/testdata/fakeengine/main.go +++ b/services/nvpair-engine-manager/testdata/fakeengine/main.go @@ -146,6 +146,15 @@ func main() { log.Fatalf("write FAKE_PID_FILE %q: %v", path, err) } } + // vLLM mode: a vLLM process serves exactly one model, named on its command + // line, and keeps it resident. FAKE_ENGINE_MODEL reseeds the registry with + // just that model so the OpenAI list/loaded endpoints report vLLM's shape. + if m := os.Getenv("FAKE_ENGINE_MODEL"); m != "" { + modelsMu.Lock() + models = map[string]bool{m: true} + loaded = map[string]bool{m: true} + modelsMu.Unlock() + } if raw := os.Getenv("FAKE_START_DELAY"); raw != "" { delay, err := time.ParseDuration(raw) if err != nil { @@ -291,6 +300,17 @@ func main() { _ = json.NewEncoder(w).Encode(map[string]any{"model": bstr(b, "model"), "object": "chat.completion", "choices": []map[string]any{{"index": 0, "message": map[string]string{"role": "assistant", "content": "ok"}, "finish_reason": "stop"}}}) }) + // vLLM-shape readiness/health: a bare 200 on /health, and a /version that + // answers JSON with a version field. /version is what distinguishes vLLM + // from any other OpenAI-compatible server on a probe (LM Studio has no + // such route), so nvpair-manual-nodes uses it as its disambiguator. + mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + }) + mux.HandleFunc("/version", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{"version": "0.0.0-fake"}) + }) mux.HandleFunc("/api/error", func(w http.ResponseWriter, r *http.Request) { http.Error(w, "boom", http.StatusInternalServerError) }) diff --git a/services/nvpair-job-scheduler/README.md b/services/nvpair-job-scheduler/README.md index 8e810467..543188c8 100644 --- a/services/nvpair-job-scheduler/README.md +++ b/services/nvpair-job-scheduler/README.md @@ -57,8 +57,8 @@ rank thrash. Invalid, missing, or older-than-10-second telemetry contributes a neutral pressure of 1. Nodes are sorted by `pending + gpuPressure`, then lower GPU pressure, then -stable node ID. Pending counts include **both** engines together, so Ollama load -affects the LM Studio ordering and vice versa. +stable node ID. Pending counts include **every** engine together, so Ollama load +affects the LM Studio and vLLM ordering and vice versa. Rankings are recomputed when the node set, catalog, or effective pressure changes, and reconciled on the interval timer. A ranking is only emitted when @@ -66,7 +66,7 @@ the order, pending counts, or pressure actually changed. ## Output -One `schedule:priority` notification per engine (`ollama`, `lmstudio`): +One `schedule:priority` notification per engine (`ollama`, `lmstudio`, `vllm`): ```json { @@ -83,9 +83,10 @@ One `schedule:priority` notification per engine (`ollama`, `lmstudio`): } ``` -The broker relays each snapshot to the matching proxy as `node/set-priority`. -Both engines currently receive the same node-wide ordering; the per-engine -envelope exists so the routing contract can diverge later without a wire change. +The broker relays each snapshot to the matching proxy as `node/set-priority`; +`lmstudio` and `vllm` both resolve to the one OpenAI-compatible proxy. Every +engine currently receives the same node-wide ordering; the per-engine envelope +exists so the routing contract can diverge later without a wire change. Each proxy then adds its own reservations for in-flight requests whose workload feedback has not arrived yet, so a burst of concurrent requests does not all diff --git a/services/nvpair-job-scheduler/schedule.go b/services/nvpair-job-scheduler/schedule.go index 1afe2fe4..27c56a7d 100644 --- a/services/nvpair-job-scheduler/schedule.go +++ b/services/nvpair-job-scheduler/schedule.go @@ -12,9 +12,10 @@ import ( "nvpair-shared/schedulerwire" ) -// schedulerEngines is the fixed set of engine-specific output contracts. Both -// receive the same node-wide ranking because their work shares node resources. -var schedulerEngines = []string{"ollama", "lmstudio"} +// schedulerEngines is the fixed set of engine-specific output contracts. They +// all receive the same node-wide ranking because their work shares node +// resources — a node's GPU is one queue however many engines front it. +var schedulerEngines = []string{"ollama", "lmstudio", "vllm"} // NodeRank is retained as the scheduler's public status type while the wire // definition is shared with the broker and proxies. diff --git a/services/nvpair-job-scheduler/vllmengine_test.go b/services/nvpair-job-scheduler/vllmengine_test.go new file mode 100644 index 00000000..c9e406d5 --- /dev/null +++ b/services/nvpair-job-scheduler/vllmengine_test.go @@ -0,0 +1,51 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "slices" + "testing" +) + +// TestSchedulerEnginesCoversVLLM proves vLLM gets its own priority contract. +// A proxy only applies the snapshot published for the engine it is routing, so +// an engine missing from this list would route with no scheduler ordering at +// all and every request would fall back to the stable-ID pass. +func TestSchedulerEnginesCoversVLLM(t *testing.T) { + for _, want := range []string{"ollama", "lmstudio", "vllm"} { + if !slices.Contains(schedulerEngines, want) { + t.Errorf("schedulerEngines is missing %q: %v", want, schedulerEngines) + } + } +} + +// TestVLLMWorkloadsCountTowardTheNodeWideRanking proves vLLM work shares the +// node's queue depth with the other engines rather than being ranked +// separately: one GPU serves them all, so a node busy with vLLM must rank below +// an idle one for every engine, including Ollama. +func TestVLLMWorkloadsCountTowardTheNodeWideRanking(t *testing.T) { + rec := &capRW{} + m := mgrWith(rec, []string{"busy", "idle"}, + workload{ID: "1", Engine: "vllm", RunID: "v1", State: "running", OriginatedFrom: "x", ScheduledOn: "busy"}, + workload{ID: "2", Engine: "vllm", RunID: "v2", State: "queued", OriginatedFrom: "x", ScheduledOn: "busy"}, + ) + order, ranks := m.rank() + if len(order) == 0 || order[len(order)-1] != "busy" { + t.Fatalf("order = %v, want the vLLM-busy node last", order) + } + if pendingOf(ranks, "busy") != 2 { + t.Fatalf("vLLM work was not counted node-wide: %+v", ranks) + } + + m.recomputeAll(false) + for _, engine := range schedulerEngines { + got := rec.orders(engine) + if len(got) != 1 { + t.Fatalf("%s emissions = %d, want 1", engine, len(got)) + } + if got[0][len(got[0])-1] != "busy" { + t.Errorf("%s order = %v, want the busy node last", engine, got[0]) + } + } +} diff --git a/services/nvpair-manual-nodes/README.md b/services/nvpair-manual-nodes/README.md index 5fb648a6..be9ff40c 100644 --- a/services/nvpair-manual-nodes/README.md +++ b/services/nvpair-manual-nodes/README.md @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0 # nvpair-manual-nodes -A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable. Accepts node addresses via JSON-RPC, probes each for Ollama, LM Studio, and node-info, and emits status events. +A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable. Accepts node addresses via JSON-RPC, probes each for Ollama, LM Studio, vLLM, and node-info, and emits status events. ## Communication @@ -50,6 +50,9 @@ Emitted when a manually added node has been probed and its initial status determ "lmstudio_up":true, "lmstudio_port":1234, "lmstudio_models":["qwen2.5-7b-instruct"], + "vllm_up":true, + "vllm_port":8000, + "vllm_models":["Qwen/Qwen3-8B"], "node_info_up":true, "node_info_port":14318, "gpus":[{"name":"NVIDIA GeForce RTX 3080","utilization_percent":37}], @@ -60,7 +63,9 @@ Emitted when a manually added node has been probed and its initial status determ } ``` -Each node is probed for both inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`) and LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` the same way it bridges Ollama into `ollama-proxy`. A node can run either engine, both, or neither. +Each node is probed for every inference engine: Ollama on its default `:11434` (`GET /` + `/api/tags`), LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list), and vLLM on its default `:8000`. The `lmstudio_*` and `vllm_*` fields mirror the `ollama_*` ones and let a supervising broker bridge the node into the right proxy the same way it bridges Ollama into `ollama-proxy`. A node can run any combination of them, or none. + +vLLM is OpenAI-compatible, so `/v1/models` alone cannot tell it from LM Studio. The vLLM probe therefore also requires `GET /version` to answer 200 with a JSON `version` field, which LM Studio does not serve — both must pass before a node is reported as running vLLM. ### `node/updated` @@ -136,11 +141,12 @@ Each manual node is probed every 10 seconds, with a 3-second timeout per leg, fo - **Ollama** on port 11434: health check (`GET /`) and model list (`GET /api/tags`) - **LM Studio** on port 1234: `GET /v1/models`, which doubles as the liveness check and the model list +- **vLLM** on port 8000: `GET /version` (the disambiguator from any other OpenAI-compatible server) followed by `GET /v1/models` for the model list - **Node Info** on port 14318, or `tls_port` over HTTPS: hardware inventory and identity (`GET /v1/node-info`) A node can have any combination of these, or none if the target is unreachable. Status changes trigger `node/updated` events. Because change detection compares CPU, memory, and GPU values, a node running node-info emits a `node/updated` on most probe cycles as utilization moves. -The three engine ports are compiled in: only the node-info leg's port can be moved, via `tls_port`. A remote engine on a non-default port is not discovered. +The engine ports are compiled in: only the node-info leg's port can be moved, via `tls_port`. A remote engine on a non-default port is not discovered. ## Shutdown diff --git a/services/nvpair-manual-nodes/manager.go b/services/nvpair-manual-nodes/manager.go index 55a4040a..be3e25f0 100644 --- a/services/nvpair-manual-nodes/manager.go +++ b/services/nvpair-manual-nodes/manager.go @@ -112,9 +112,15 @@ type ManualNodeStatus struct { // LM Studio is probed on its default OpenAI-API port the same way Ollama // is on 11434, so a manually-added node running LM Studio can be bridged // into lmstudio-proxy by a supervising broker. - LMStudioUp bool `json:"lmstudio_up"` - LMStudioPort int `json:"lmstudio_port"` - LMStudioModels []string `json:"lmstudio_models,omitempty"` + LMStudioUp bool `json:"lmstudio_up"` + LMStudioPort int `json:"lmstudio_port"` + LMStudioModels []string `json:"lmstudio_models,omitempty"` + // vLLM is probed on its default OpenAI-API port alongside LM Studio. Both + // speak the same API, so /v1/models alone cannot tell them apart; the probe + // additionally requires vLLM's own /version, which LM Studio does not serve. + VLLMUp bool `json:"vllm_up"` + VLLMPort int `json:"vllm_port"` + VLLMModels []string `json:"vllm_models,omitempty"` NodeInfoUp bool `json:"node_info_up"` NodeInfoPort int `json:"node_info_port"` TLSEnabled bool `json:"tls_enabled,omitempty"` @@ -138,9 +144,9 @@ type trackedNode struct { entry ManualEntry status ManualNodeStatus - // consecutiveFails counts back-to-back probes where neither - // service answered (OllamaUp && NodeInfoUp both false). Reset - // to 0 on any probe where at least one service responded. + // consecutiveFails counts back-to-back probes where no service + // answered — no engine and no node-info. Reset to 0 on any probe + // where at least one service responded. // Used to gate probe-failed errors:report emits at // probeFailThreshold so a single transient failure doesn't // generate UI noise. @@ -253,6 +259,7 @@ func (m *Manager) probeNode(entry ManualEntry) { ollamaUp, ollamaModels := m.probeOllama(addr, 11434) lmStudioUp, lmStudioModels := m.probeLMStudio(addr, lmStudioPort) + vllmUp, vllmModels := m.probeVLLM(addr, vllmPort) // Pick scheme + port + client based on the entry's TLS hint. // The operator decides which scheme this manual node uses; we @@ -290,6 +297,9 @@ func (m *Manager) probeNode(entry ManualEntry) { LMStudioUp: lmStudioUp, LMStudioPort: lmStudioPort, LMStudioModels: lmStudioModels, + VLLMUp: vllmUp, + VLLMPort: vllmPort, + VLLMModels: vllmModels, NodeInfoUp: nodeInfoUp, NodeInfoPort: nodeInfoPort, TLSEnabled: entry.TLSPort > 0, @@ -302,7 +312,7 @@ func (m *Manager) probeNode(entry ManualEntry) { HostUUID: info.HostUUID, } - reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.NodeInfoUp + reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.VLLMUp || newStatus.NodeInfoUp m.mu.Lock() tn, exists := m.nodes[id] @@ -331,10 +341,12 @@ func (m *Manager) probeNode(entry ManualEntry) { changed := prev.OllamaUp != newStatus.OllamaUp || prev.LMStudioUp != newStatus.LMStudioUp || + prev.VLLMUp != newStatus.VLLMUp || prev.NodeInfoUp != newStatus.NodeInfoUp || prev.HostUUID != newStatus.HostUUID || !sliceEqual(prev.OllamaModels, newStatus.OllamaModels) || !sliceEqual(prev.LMStudioModels, newStatus.LMStudioModels) || + !sliceEqual(prev.VLLMModels, newStatus.VLLMModels) || !gpusEqual(prev.GPUs, newStatus.GPUs) || !cpuEqual(prev.CPU, newStatus.CPU) || !memoryEqual(prev.Memory, newStatus.Memory) || @@ -403,6 +415,12 @@ func probeFailedID(nodeID string) string { // manager (which only governs the local engine). const lmStudioPort = 1234 +// vllmPort is vLLM's default OpenAI-API server port, hardcoded on the same terms +// as Ollama's 11434 and LM Studio's 1234: a manual node is remote, so we assume +// the engine's default rather than resolving it through the engine manager, +// which only governs the local engine. +const vllmPort = 8000 + // probeLMStudio checks LM Studio's OpenAI-compatible server on addr:port. A // single GET /v1/models doubles as the liveness check and the model list (the // response is {"data":[{"id":"..."}],...}). Returns whether it is up and the @@ -446,6 +464,76 @@ func (m *Manager) probeLMStudio(addr string, port int) (bool, []string) { return true, models } +// probeVLLM checks vLLM's OpenAI-compatible server on addr:port. LM Studio +// serves the same /v1/models, so a model list alone would let one engine be +// reported as the other on a host running both. vLLM additionally serves +// GET /version returning {"version": "..."} and LM Studio does not, so that +// route is the disambiguator: both must answer before the node is reported as +// running vLLM. Returns whether it is up and the model ids it serves. +func (m *Manager) probeVLLM(addr string, port int) (bool, []string) { + if !m.probeVLLMVersion(addr, port) { + return false, nil + } + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/models" + start := time.Now() + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe vllm failed", + "addr", addr, "port", port, "duration_ms", time.Since(start).Milliseconds(), "err", err) + return false, nil + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + slog.Debug("manual probe vllm non-OK", + "addr", addr, "port", port, "status", resp.StatusCode, + "duration_ms", time.Since(start).Milliseconds()) + return false, nil + } + var result struct { + Data *[]struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.NewDecoder(resp.Body).Decode(&result); err != nil || result.Data == nil { + // /version identified it as vLLM, but its model list is not the OpenAI + // shape. Nothing can be routed to it, so it is not usable here. + slog.Debug("manual probe vllm model list unusable", "addr", addr, "port", port, "err", err) + return false, nil + } + models := make([]string, 0, len(*result.Data)) + for _, d := range *result.Data { + if d.ID != "" { + models = append(models, d.ID) + } + } + slog.Debug("manual probe vllm up", + "addr", addr, "port", port, "models", len(models), + "duration_ms", time.Since(start).Milliseconds()) + return true, models +} + +// probeVLLMVersion reports whether addr:port answers vLLM's GET /version with a +// JSON body carrying a version field. +func (m *Manager) probeVLLMVersion(addr string, port int) bool { + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/version" + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe vllm version failed", "addr", addr, "port", port, "err", err) + return false + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return false + } + var body struct { + Version string `json:"version"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return false + } + return body.Version != "" +} + func (m *Manager) probeOllama(addr string, port int) (bool, []string) { url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/" start := time.Now() @@ -540,6 +628,8 @@ func (m *Manager) addNode(entry ManualEntry) ManualNodeStatus { Name: entry.Name, Address: entry.Address, OllamaPort: 11434, + LMStudioPort: lmStudioPort, + VLLMPort: vllmPort, NodeInfoPort: nodeInfoPort, TLSEnabled: entry.TLSPort > 0, MTLSRequired: entry.TLSPort > 0 && entry.MTLS, diff --git a/services/nvpair-manual-nodes/vllmprobe_test.go b/services/nvpair-manual-nodes/vllmprobe_test.go new file mode 100644 index 00000000..55051637 --- /dev/null +++ b/services/nvpair-manual-nodes/vllmprobe_test.go @@ -0,0 +1,113 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +// vllmStub serves the two routes probeVLLM requires. version == "" omits the +// version field, standing in for an OpenAI-compatible server that is not vLLM. +func vllmStub(t *testing.T, version string, models []string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/version": + if version == "" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"version": version}) + case "/v1/models": + data := make([]map[string]string, 0, len(models)) + for _, id := range models { + data = append(data, map[string]string{"id": id, "object": "model"}) + } + _ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data}) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func stubHostPort(t *testing.T, srv *httptest.Server) (string, int) { + t.Helper() + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatal(err) + } + port, err := strconv.Atoi(u.Port()) + if err != nil { + t.Fatalf("port %q: %v", u.Port(), err) + } + return u.Hostname(), port +} + +// TestProbeVLLMReportsModels covers the vLLM probe against a real HTTP server: +// a node answering both /version and /v1/models reports up with its model ids. +func TestProbeVLLMReportsModels(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := vllmStub(t, "0.11.0", []string{"Qwen/Qwen3-8B"}) + host, port := stubHostPort(t, srv) + + up, models := m.probeVLLM(host, port) + if !up { + t.Fatal("expected vllm up") + } + if len(models) != 1 || models[0] != "Qwen/Qwen3-8B" { + t.Fatalf("models = %#v", models) + } +} + +// TestProbeVLLMRejectsAnOpenAIServerThatIsNotVLLM is the disambiguator guard: +// LM Studio serves the same /v1/models but has no /version, so an OpenAI server +// without one must never be reported as vLLM. +func TestProbeVLLMRejectsAnOpenAIServerThatIsNotVLLM(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := vllmStub(t, "", []string{"qwen2.5-7b"}) + host, port := stubHostPort(t, srv) + + if up, models := m.probeVLLM(host, port); up || models != nil { + t.Fatalf("an OpenAI server without /version was reported as vLLM: up=%v models=%#v", up, models) + } +} + +// TestProbeVLLMAbsentNodeIsDown covers the unreachable case. +func TestProbeVLLMAbsentNodeIsDown(t *testing.T) { + m, _, _ := newTestManager() + if up, models := m.probeVLLM("absent.local", vllmPort); up || models != nil { + t.Fatalf("expected absent vllm down, got up=%v models=%#v", up, models) + } +} + +// TestVLLMCountsAsReachable proves a node that runs only vLLM is not treated as +// unreachable, which would otherwise raise a probe-failed error for a healthy +// node and drop it from routing. +func TestVLLMCountsAsReachable(t *testing.T) { + for _, tc := range []struct { + name string + status ManualNodeStatus + want bool + }{ + {name: "vllm only", status: ManualNodeStatus{VLLMUp: true}, want: true}, + {name: "lmstudio only", status: ManualNodeStatus{LMStudioUp: true}, want: true}, + {name: "nothing", status: ManualNodeStatus{}, want: false}, + } { + t.Run(tc.name, func(t *testing.T) { + got := tc.status.OllamaUp || tc.status.LMStudioUp || tc.status.VLLMUp || tc.status.NodeInfoUp + if got != tc.want { + t.Errorf("reachable = %v, want %v", got, tc.want) + } + }) + } +} diff --git a/services/nvpair-node-scanner/README.md b/services/nvpair-node-scanner/README.md index ae2376e0..7b6b640d 100644 --- a/services/nvpair-node-scanner/README.md +++ b/services/nvpair-node-scanner/README.md @@ -9,7 +9,7 @@ SPDX-License-Identifier: Apache-2.0 Concretely, the daemon: -- **Advertises** this node's one `_nvpair-node._tcp` record, built from a registry of the local services its parent (the broker) registers — `ni`/`ol`/`lm`/`er`/`wl`/`cl`/`em`/`ec` ports plus the node identity (`uuid=`, `cluster-uuid=` when clustered, `ip=`). One record per node covers every service. +- **Advertises** this node's one `_nvpair-node._tcp` record, built from a registry of the local services its parent (the broker) registers — `ni`/`ol`/`lm`/`vl`/`er`/`wl`/`cl`/`em`/`ec` ports plus the node identity (`uuid=`, `cluster-uuid=` when clustered, `ip=`). One record per node covers every service. - **Browses** `_nvpair-node._tcp` for every node on the LAN (including itself) and maintains a queryable directory keyed by host UUID. - **Enriches** each discovered node: GPU/CPU/memory inventory from its `ni` (node-info) port over plain HTTP, and the model list from its `em` (engine-manager) port (`GET /v1/models`) over cluster mTLS — a peer's model inventory is cluster data, so it is fetched only when this node holds a pin for the principal that peer advertises, while this node's own list is read over loopback in plaintext — the flat union, the per-engine breakdown (`modelsByEngine`), and the per-engine set of models loaded in memory (`loadedByEngine`), all carried onto the node's directory entry so remote cards can show loaded state. Each enrichment has a last-good cache so a transient fetch miss doesn't blank the node's card. Node-info enrichment can optionally be moved onto HTTPS (see the TLS flags below) — off by default and gated only on operator flags, never inferred per-node. - **Samples scheduling telemetry** from healthy nodes every two seconds with diff --git a/services/nvpair-tui/README.md b/services/nvpair-tui/README.md index e5b92ca7..eba0932f 100644 --- a/services/nvpair-tui/README.md +++ b/services/nvpair-tui/README.md @@ -30,7 +30,7 @@ Tabs: | **Overview** | Broker liveness/version/uptime (`ping`) and a per-worker health table derived from the broker's `supervisor:subprocess-crashed:*` errors. | | **Errors** | The service-error datastore (`errors:get-initial` + live `errors:update`); `c` clears the selected entry. | | **Nodes** | mDNS-discovered Ollama nodes (`discovery:subscribe` / `discovery:nodes-changed`). | -| **Proxies** | Ollama and LM Studio reverse proxies: status, discovered upstreams, select a node (`enter`/`a`), set the listen port (`p`). | +| **Proxies** | The Ollama and OpenAI-compatible (LM Studio, vLLM) reverse proxies: status, discovered upstreams, select a node (`enter`/`a`), set the listen port (`p`). | | **Workloads** | Live cluster workloads (`workloads:subscribe` / `workloads:upsert` / `workloads:remove`). | | **Engines** | Local inference engines: install (`i`), start (`s`), stop (`x`), restart (`r`), uninstall (`u`). | | **Cluster** | Pairing + membership: invite by address (`i`, shows the six-digit PIN — the first invite auto-founds a cluster of one), accept (`a`) / decline (`d`) an inbound invite, remove a member (`r`), leave (`L`). | diff --git a/services/nvpair-tui/ui/proxies.go b/services/nvpair-tui/ui/proxies.go index 072c1a8b..05d5aa74 100644 --- a/services/nvpair-tui/ui/proxies.go +++ b/services/nvpair-tui/ui/proxies.go @@ -24,11 +24,12 @@ type proxyNode struct { Port int `json:"port"` } -// proxyEngine is one of the two reverse proxies the broker fronts. Both +// proxyEngine is one of the reverse proxies the broker fronts. They all // speak the same routing/failover contract; only the JSON-RPC prefix and -// label differ. +// label differ. One proxy may front several engines — the OpenAI-compatible +// one serves LM Studio and vLLM — so the label names the engines, not a binary. type proxyEngine struct { - label string // "Ollama" / "LM Studio" + label string // "Ollama" / "LM Studio & vLLM" prefix string // "proxy" / "lmstudio-proxy" ready bool port int @@ -91,7 +92,7 @@ func newProxiesView(client *rpc.Client) *proxiesView { portInput: ti, engines: []*proxyEngine{ {label: "Ollama", prefix: "proxy", table: newTable(nil)}, - {label: "LM Studio", prefix: "lmstudio-proxy", table: newTable(nil)}, + {label: "LM Studio & vLLM", prefix: "lmstudio-proxy", table: newTable(nil)}, }, } return v diff --git a/services/nvpair-ui-broker/README.md b/services/nvpair-ui-broker/README.md index d2b0e174..55de96b9 100644 --- a/services/nvpair-ui-broker/README.md +++ b/services/nvpair-ui-broker/README.md @@ -37,9 +37,13 @@ lifecycle, and relay rules. Two responsibilities live in the broker itself rather than in a worker: -- **Engine advertising.** The broker polls local Ollama and LM Studio every 5 s - and registers each running engine's port (`ol` / `lm`) with the discovery - daemon, so both are carried in this host's single `_nvpair-node` record. The +- **Engine advertising.** The broker polls local Ollama, LM Studio, and vLLM + every 5 s and registers each running engine's port (`ol` / `lm` / `vl`) with + the discovery daemon, so all of them are carried in this host's single + `_nvpair-node` record. `lm` and `vl` both carry the port of the one + OpenAI-compatible proxy, so a host serving from both engines advertises the + same port twice; which engine owns a given model comes from engine-manager + model attribution, not from the key. The model list is not part of that record — it is served over HTTP by `nvpair-engine-manager` on the `em` service and fetched by a peer's daemon during discovery enrichment. @@ -115,7 +119,7 @@ Two classes of proxy notification are **not** re-emitted under the `proxy:` name - **Inbound (peers -> manager -> broker).** The manager translates peer-origin lifecycle events into `workloads:upsert` and peer-origin removals into `workloads:remove` on stdout. The broker applies each accepted transition to the same store, fans it to the scheduler, and relays it to clients subscribed via `workloads:subscribe`. - **Local echo.** Local-origin proxy workloads are also emitted to the same `workloads:*` client stream (lifecycle translated to `workloads:upsert`), so a subscribed client sees a coherent cluster-wide view — its own workloads alongside peers'. -**`nvpair-job-scheduler`** consumes the accepted workload stream, compact GPU telemetry, and discovery snapshot. It smooths fresh utilization into pressure 0–3, uses neutral pressure 1 for invalid/missing/older-than-10-second samples, and orders by `pending + gpuPressure`, then pressure, then stable UUID. Load is node-wide across Ollama and LM Studio because both normally contend for the same resources. Each engine-specific `schedule:priority` carries `{engine,nodes,ranks}` and refreshes when order, pending counts, or pressure changes. The broker caches, generation-orders, and replays the full `{nodes,ranks}` snapshot to the matching proxy, where a newly delivered snapshot resets optimistic reservation deltas. On scheduler spawn/restart the broker replays active workloads and telemetry before discovery, then resumes all three live feeds. +**`nvpair-job-scheduler`** consumes the accepted workload stream, compact GPU telemetry, and discovery snapshot. It smooths fresh utilization into pressure 0–3, uses neutral pressure 1 for invalid/missing/older-than-10-second samples, and orders by `pending + gpuPressure`, then pressure, then stable UUID. Load is node-wide across Ollama, LM Studio, and vLLM because they normally contend for the same resources. Each engine-specific `schedule:priority` carries `{engine,nodes,ranks}` and refreshes when order, pending counts, or pressure changes. The broker caches, generation-orders, and replays the full `{nodes,ranks}` snapshot to the matching proxy, where a newly delivered snapshot resets optimistic reservation deltas. On scheduler spawn/restart the broker replays active workloads and telemetry before discovery, then resumes all three live feeds. `schedule:priority` and `node/set-priority` are internal worker contracts: the broker does not expose either notification to its connected client. diff --git a/services/nvpair-ui-broker/advertiser.go b/services/nvpair-ui-broker/advertiser.go index c3f2e7d1..dfc59e0a 100644 --- a/services/nvpair-ui-broker/advertiser.go +++ b/services/nvpair-ui-broker/advertiser.go @@ -22,6 +22,9 @@ const ( // real port is resolved per poll via localEnginePort. defaultOllamaPort = 11434 defaultLMStudioPort = 1234 + // defaultVLLMPort is vLLM's stock OpenAI-API port, used on the same + // fallback-only terms as the two above. + defaultVLLMPort = 8000 // engineManagerHTTPPort is the fixed LAN port the broker tells // nvpair-engine-manager to serve its HTTP surface (/v1/models) on, and the port @@ -181,6 +184,56 @@ func (b *Broker) reconcileAdvertiseLMStudio(client *http.Client) { } } +// runAutoAdvertiseVLLM is the vLLM sibling of runAutoAdvertiseLMStudio. vLLM +// speaks the same OpenAI API, so it is fronted by the same proxy; only the +// discovery service key (vl) and the engine name differ. There is no managed +// facade for vLLM — nothing serves :8000 on the proxy's behalf — so this loop +// has none of the facade/backend-cache handling the LM Studio one carries. +func (b *Broker) runAutoAdvertiseVLLM(ctx context.Context) { + client := &http.Client{Timeout: 2 * time.Second} + ticker := time.NewTicker(autoAdvertiseInterval) + defer ticker.Stop() + + b.reconcileAdvertiseVLLM(client) + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + b.reconcileAdvertiseVLLM(client) + } + } +} + +// reconcileAdvertiseVLLM brings this node's vl registration into line with the +// local vLLM server: it advertises the OpenAI proxy's port (never the engine's) +// and hands the engine's loopback port to that same proxy via +// node/set-local-backend, which keys its backends by engine so this never +// disturbs LM Studio's. +func (b *Broker) reconcileAdvertiseVLLM(client *http.Client) { + enginePort, probe := b.localEnginePort("vllm", defaultVLLMPort) + b.reconcileAdvertiseVLLMAt(client, enginePort, probe) +} + +// reconcileAdvertiseVLLMAt is reconcileAdvertiseVLLM with the engine port +// already resolved, so the registration and backend decisions can be exercised +// without an engine-manager. The health probe is the last term of the guard, so +// a collision or an authoritative "not running" short-circuits before any +// request is made — which is what lets a caller pass a nil client to assert +// exactly that. +func (b *Broker) reconcileAdvertiseVLLMAt(client *http.Client, enginePort int, probe bool) { + proxyPort := b.lmstudioProxyListenPort() + up := probe && proxyPort != 0 && enginePort != proxyPort && checkVLLMHealth(client, enginePort) + if up { + b.registerService(noderec.RegisterParams{Service: noderec.ServiceVLLM, Port: proxyPort}) + b.setProxyLocalBackend(b.getOpenAIProxy(), "vllm", enginePort, true) + } else { + b.unregisterService(noderec.ServiceVLLM) + b.setProxyLocalBackend(b.getOpenAIProxy(), "vllm", enginePort, false) + } +} + // proxyLocalBackend is the node/set-local-backend payload: the loopback engine // the proxy's cluster mTLS ingress forwards to, and the proxy's own self // candidate on the local routing path. @@ -249,6 +302,13 @@ func (b *Broker) proxyListenPort() int { return 0 } +// getOpenAIProxy returns the proxy that fronts every OpenAI-compatible engine. +// One process serves LM Studio and vLLM: they speak the same HTTP surface, so +// they need one router, and node/set-local-backend keys its backends by engine +// so both are held at once. Named for the role rather than the binary, which +// keeps its historical lmstudio-proxy spelling as a wire contract. +func (b *Broker) getOpenAIProxy() *proxyProcess { return b.getLMStudioProxy() } + // lmstudioProxyListenPort is the LM Studio sibling of proxyListenPort. It // prevents the compatibility fallback from mistaking a proxy moved onto :1234 // for the actual engine. @@ -287,3 +347,17 @@ func checkLMStudioHealth(client *http.Client, port int) bool { resp.Body.Close() return resp.StatusCode == http.StatusOK } + +// checkVLLMHealth reports whether a local vLLM server is answering on the given +// port. vLLM exposes a dedicated /health, which is a cheaper and more precise +// liveness signal than its model list — and, unlike /v1/models, one the OpenAI +// proxy's own facade does not answer, so the proxy can never be mistaken for the +// engine. The port is resolved per poll (see localEnginePort), not hardcoded. +func checkVLLMHealth(client *http.Client, port int) bool { + resp, err := client.Get(fmt.Sprintf("http://localhost:%d/health", port)) + if err != nil { + return false + } + resp.Body.Close() + return resp.StatusCode == http.StatusOK +} diff --git a/services/nvpair-ui-broker/broker.go b/services/nvpair-ui-broker/broker.go index 0d189578..e997323c 100644 --- a/services/nvpair-ui-broker/broker.go +++ b/services/nvpair-ui-broker/broker.go @@ -505,16 +505,21 @@ func (b *Broker) restoreEnabledEnginesAfterPortGate(ctx context.Context) bool { return true } -func (b *Broker) runEngineAvailabilityAfterPortGates( - ctx context.Context, - runOllama func(context.Context), - runLMStudio func(context.Context), -) bool { +// runEngineAvailabilityAfterPortGates starts every engine-advertise loop once +// the managed port gates have settled, so no loop can advertise a port that is +// still being moved. The last loop runs on this goroutine; the rest get their +// own. Variadic because the set grows with each engine PAIR fronts. +func (b *Broker) runEngineAvailabilityAfterPortGates(ctx context.Context, loops ...func(context.Context)) bool { if !b.restoreEnabledEnginesAfterPortGate(ctx) { return false } - go runOllama(ctx) - runLMStudio(ctx) + if len(loops) == 0 { + return true + } + for _, run := range loops[:len(loops)-1] { + go run(ctx) + } + loops[len(loops)-1](ctx) return true } @@ -1462,8 +1467,9 @@ func (b *Broker) proxyForEngine(engine string) *proxyProcess { switch engine { case "ollama": return b.getProxy() - case "lmstudio": - return b.getLMStudioProxy() + case "lmstudio", "vllm": + // Both OpenAI-compatible engines are fronted by the one OpenAI proxy. + return b.getOpenAIProxy() default: return nil } @@ -1779,7 +1785,8 @@ func (b *Broker) Serve(ctx context.Context) error { // startup attempts have established either readiness or a terminal outcome. // This prevents a restored engine from taking a persisted proxy port before // the broker can resolve ownership. - go b.runEngineAvailabilityAfterPortGates(ctx, b.runAutoAdvertise, b.runAutoAdvertiseLMStudio) + go b.runEngineAvailabilityAfterPortGates(ctx, + b.runAutoAdvertise, b.runAutoAdvertiseLMStudio, b.runAutoAdvertiseVLLM) // nvpair-workload-manager is another auxiliary worker: it relays local // workload lifecycle events to peer nodes and surfaces peer events diff --git a/services/nvpair-ui-broker/manualnodes.go b/services/nvpair-ui-broker/manualnodes.go index 47fe0ff1..61840785 100644 --- a/services/nvpair-ui-broker/manualnodes.go +++ b/services/nvpair-ui-broker/manualnodes.go @@ -28,6 +28,9 @@ type manualNodeStatus struct { LMStudioUp bool `json:"lmstudio_up"` LMStudioPort int `json:"lmstudio_port"` LMStudioModels []string `json:"lmstudio_models,omitempty"` + VLLMUp bool `json:"vllm_up"` + VLLMPort int `json:"vllm_port"` + VLLMModels []string `json:"vllm_models,omitempty"` NodeInfoPort int `json:"node_info_port"` GPUs []GPUInfo `json:"gpus"` CPU *CPUInfo `json:"cpu"` @@ -109,6 +112,9 @@ func manualModelsByEngine(s manualNodeStatus) map[string][]string { if len(s.LMStudioModels) > 0 { byEngine["lmstudio"] = s.LMStudioModels } + if len(s.VLLMModels) > 0 { + byEngine["vllm"] = s.VLLMModels + } if len(byEngine) == 0 { return nil } @@ -139,7 +145,12 @@ func mergeModels(lists ...[]string) []string { // It mirrors the proxy's Node wire shape (id/host/port/addresses[/txt]); // the proxy requires a non-empty address list and a port to forward to. type proxyManualNode struct { - ID string `json:"id"` + ID string `json:"id"` + // Engine names which engine on that host this address serves. The OpenAI + // proxy fronts more than one engine on different ports, so it keys its manual + // overlay by (engine, node) and would otherwise collapse a host running both + // into a single entry. ollama-proxy fronts one engine and ignores the field. + Engine string `json:"engine"` Host string `json:"host"` Port int `json:"port"` Addresses []string `json:"addresses"` @@ -147,6 +158,12 @@ type proxyManualNode struct { Models []string `json:"models,omitempty"` } +// proxyManualRef identifies a manual node to remove from one proxy engine. +type proxyManualRef struct { + ID string `json:"id"` + Engine string `json:"engine"` +} + // bridgeManualNode keeps every supervised proxy's manual-node set in step with // a manual node's per-engine reachability: a node whose Ollama is up is bridged // into ollama-proxy and one whose LM Studio is up into lmstudio-proxy @@ -161,7 +178,8 @@ type proxyManualNode struct { // inference to them even though both workers are broker-owned. func (b *Broker) bridgeManualNode(s manualNodeStatus, key string) { b.bridgeToProxy(b.getProxy(), "ollama", s, key, s.OllamaUp, s.OllamaPort, s.OllamaModels) - b.bridgeToProxy(b.getLMStudioProxy(), "lmstudio", s, key, s.LMStudioUp, s.LMStudioPort, s.LMStudioModels) + b.bridgeToProxy(b.getOpenAIProxy(), "lmstudio", s, key, s.LMStudioUp, s.LMStudioPort, s.LMStudioModels) + b.bridgeToProxy(b.getOpenAIProxy(), "vllm", s, key, s.VLLMUp, s.VLLMPort, s.VLLMModels) } // bridgeToProxy adds the node to p when its engine is reachable, or removes it @@ -177,6 +195,7 @@ func (b *Broker) bridgeToProxy(p *proxyProcess, engine string, s manualNodeStatu if up && s.Address != "" && port > 0 { node := proxyManualNode{ ID: key, + Engine: engine, Host: s.Address, Port: port, Addresses: []string{s.Address}, @@ -187,15 +206,17 @@ func (b *Broker) bridgeToProxy(p *proxyProcess, engine string, s manualNodeStatu } // Engine unreachable (down, or this node doesn't run it): make sure the // proxy isn't left holding a stale manual entry it would try to route to. - b.callProxyManual(p, engine, "node/remove-manual", map[string]string{"id": key}, key) + b.callProxyManual(p, engine, "node/remove-manual", proxyManualRef{ID: key, Engine: engine}, key) } // removeManualNodeFromProxies drops a manual node from every supervised proxy. // Idempotent: a no-op for a proxy where the node was never bridged or that // isn't supervised (the proxy's RemoveManual just reports removed=false). func (b *Broker) removeManualNodeFromProxies(id string) { - b.callProxyManual(b.getProxy(), "ollama", "node/remove-manual", map[string]string{"id": id}, id) - b.callProxyManual(b.getLMStudioProxy(), "lmstudio", "node/remove-manual", map[string]string{"id": id}, id) + b.callProxyManual(b.getProxy(), "ollama", "node/remove-manual", proxyManualRef{ID: id, Engine: "ollama"}, id) + for _, engine := range []string{"lmstudio", "vllm"} { + b.callProxyManual(b.getOpenAIProxy(), engine, "node/remove-manual", proxyManualRef{ID: id, Engine: engine}, id) + } } // callProxyManual issues a best-effort node/add-manual|remove-manual to a diff --git a/services/nvpair-ui-broker/vllmadvertise_test.go b/services/nvpair-ui-broker/vllmadvertise_test.go new file mode 100644 index 00000000..edd37c68 --- /dev/null +++ b/services/nvpair-ui-broker/vllmadvertise_test.go @@ -0,0 +1,307 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "strconv" + "testing" + "time" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +// openAIProxyFixture returns a broker with a fake OpenAI proxy on the given +// listen port, plus a channel receiving every node/set-local-backend it is sent. +func openAIProxyFixture(t *testing.T, listenPort int) (*Broker, <-chan proxyLocalBackend) { + t.Helper() + proxyClient, proxyServer := net.Pipe() + t.Cleanup(func() { proxyClient.Close(); proxyServer.Close() }) + proxy := &proxyProcess{ + peer: NewPeer(NewCodec(proxyClient)), + ready: true, + port: listenPort, + } + go proxy.peer.Serve(nil, nil) + + backends := make(chan proxyLocalBackend, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var got proxyLocalBackend + if json.Unmarshal(msg.Params, &got) == nil { + backends <- got + } + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + return b, backends +} + +func waitBackend(t *testing.T, ch <-chan proxyLocalBackend, engine string) proxyLocalBackend { + t.Helper() + deadline := time.After(2 * time.Second) + for { + select { + case got := <-ch: + if got.Engine == engine { + return got + } + case <-deadline: + t.Fatalf("no node/set-local-backend for %q", engine) + } + } +} + +// TestVLLMAdvertisesTheProxyPortNotTheEnginePort proves a healthy local vLLM +// registers the vl service at the OpenAI proxy's listen port — peers must reach +// this node through the proxy, never the engine — while the engine's own +// loopback port is handed to that proxy as the vllm backend. +func TestVLLMAdvertisesTheProxyPortNotTheEnginePort(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/health" { + http.Error(w, "not vllm", http.StatusNotFound) + return + } + w.WriteHeader(http.StatusOK) + })) + defer engine.Close() + _, portStr, err := net.SplitHostPort(engine.Listener.Addr().String()) + if err != nil { + t.Fatal(err) + } + enginePort, err := strconv.Atoi(portStr) + if err != nil { + t.Fatal(err) + } + + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + // No engine-manager, so localEnginePort answers with the fallback; point the + // fallback at the stub by probing it directly. + client := &http.Client{Timeout: 2 * time.Second} + if !checkVLLMHealth(client, enginePort) { + t.Fatal("stub did not answer /health") + } + + b.reconcileAdvertiseVLLMAt(client, enginePort, true) + + snapshot := b.regCache.Snapshot() + reg, ok := registrationFor(snapshot, noderec.ServiceVLLM) + if !ok { + t.Fatalf("vl was not registered: %+v", snapshot) + } + if reg.Port != proxyPort { + t.Errorf("vl advertised port = %d, want the proxy's %d", reg.Port, proxyPort) + } + got := waitBackend(t, backends, "vllm") + if got.Port != enginePort || !got.Healthy { + t.Errorf("vllm local backend = %+v, want the healthy engine port %d", got, enginePort) + } +} + +// TestVLLMUnregistersWhenTheEngineIsDown proves the vl key is withdrawn and the +// proxy's vllm backend cleared when the engine stops answering, without any +// health request reaching a listener that is not vLLM. +func TestVLLMUnregistersWhenTheEngineIsDown(t *testing.T) { + b, backends := openAIProxyFixture(t, 1234) + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceVLLM, Port: 1234}) + + b.reconcileAdvertiseVLLMAt(nil, defaultVLLMPort, false) + + if _, ok := registrationFor(b.regCache.Snapshot(), noderec.ServiceVLLM); ok { + t.Error("vl stayed registered with the engine down") + } + got := waitBackend(t, backends, "vllm") + if got.Healthy { + t.Errorf("vllm backend = %+v, want cleared", got) + } +} + +// TestVLLMNeverAdvertisesItsOwnProxy is the self-forward guard: when the +// resolved engine port equals the proxy's own listener, the node must not +// advertise vl and must not be handed its own listener as a backend. +func TestVLLMNeverAdvertisesItsOwnProxy(t *testing.T) { + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + + // A nil client is intentional: the collision check must short-circuit before + // any health request could mistake the proxy for the engine. + b.reconcileAdvertiseVLLMAt(nil, proxyPort, true) + + if got := b.regCache.Snapshot(); len(got) != 0 { + t.Fatalf("the OpenAI proxy was advertised as vLLM: %+v", got) + } + if got := waitBackend(t, backends, "vllm"); got.Healthy { + t.Errorf("proxy listener was retained as the vLLM backend: %+v", got) + } +} + +// TestOpenAIProxyServesBothEngines proves both OpenAI engines resolve to the one +// proxy process, which is what lets a node advertise lm and vl at the same port. +func TestOpenAIProxyServesBothEngines(t *testing.T) { + b, _ := openAIProxyFixture(t, 1234) + lm := b.proxyForEngine("lmstudio") + vl := b.proxyForEngine("vllm") + if lm == nil || vl == nil { + t.Fatalf("proxyForEngine: lmstudio=%v vllm=%v", lm, vl) + } + if lm != vl { + t.Error("both OpenAI engines must be fronted by the same proxy process") + } + if b.proxyForEngine("ollama") == vl { + t.Error("Ollama must keep its own proxy") + } +} + +// TestManualVLLMNodeIsBridgedWithItsEngine proves a manual node whose vllm_up is +// set is bridged into the OpenAI proxy tagged vllm, and that an LM Studio entry +// on the same host is a separate entry rather than overwriting it. +func TestManualVLLMNodeIsBridgedWithItsEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + node proxyManualNode + } + calls := make(chan call, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var n proxyManualNode + _ = json.Unmarshal(msg.Params, &n) + calls <- call{method: msg.Method, node: n} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", + Address: "192.0.2.7", + LMStudioUp: true, + LMStudioPort: 1234, + LMStudioModels: []string{"qwen2.5-7b"}, + VLLMUp: true, + VLLMPort: 8000, + VLLMModels: []string{"Qwen/Qwen3-8B"}, + }, "host-uuid") + + byEngine := map[string]proxyManualNode{} + deadline := time.After(2 * time.Second) + for len(byEngine) < 2 { + select { + case c := <-calls: + if c.method == "node/add-manual" { + byEngine[c.node.Engine] = c.node + } + case <-deadline: + t.Fatalf("only bridged %v", byEngine) + } + } + if got := byEngine["vllm"]; got.Port != 8000 || got.ID != "host-uuid" { + t.Errorf("vllm bridge = %+v, want the vLLM port on the node's operational key", got) + } + if got := byEngine["lmstudio"]; got.Port != 1234 { + t.Errorf("lmstudio bridge = %+v, want its own port preserved", got) + } +} + +// TestManualNodeWithoutVLLMIsRemovedFromThatEngine proves an unreachable engine +// leg clears only its own entry, so the other engine on that host keeps routing. +func TestManualNodeWithoutVLLMIsRemovedFromThatEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + ref proxyManualRef + } + calls := make(chan call, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var ref proxyManualRef + _ = json.Unmarshal(msg.Params, &ref) + calls <- call{method: msg.Method, ref: ref} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", Address: "192.0.2.7", + LMStudioUp: true, LMStudioPort: 1234, + }, "host-uuid") + + deadline := time.After(2 * time.Second) + for { + select { + case c := <-calls: + if c.method == "node/remove-manual" && c.ref.Engine == "vllm" { + if c.ref.ID != "host-uuid" { + t.Fatalf("remove ref = %+v", c.ref) + } + return + } + if c.method == "node/remove-manual" && c.ref.Engine == "lmstudio" { + t.Fatal("a reachable LM Studio must not be removed") + } + case <-deadline: + t.Fatal("vLLM entry was never cleared") + } + } +} + +// TestManualModelsByEngineCarriesVLLM proves a manual node's vLLM inventory is +// attributed under the same engine name discovered nodes use. +func TestManualModelsByEngineCarriesVLLM(t *testing.T) { + got := manualModelsByEngine(manualNodeStatus{ + OllamaModels: []string{"llama3.2:1b"}, + VLLMModels: []string{"Qwen/Qwen3-8B"}, + }) + if len(got["vllm"]) != 1 || got["vllm"][0] != "Qwen/Qwen3-8B" { + t.Errorf("modelsByEngine = %+v, want a vllm key", got) + } + if _, ok := got["lmstudio"]; ok { + t.Errorf("an engine with no models must add no key: %+v", got) + } +} + +func registrationFor(regs []noderec.RegisterParams, svc noderec.ServiceKey) (noderec.RegisterParams, bool) { + for _, r := range regs { + if r.Service == svc { + return r, true + } + } + return noderec.RegisterParams{}, false +} diff --git a/services/readme.md b/services/readme.md index ee3523d5..52d3e812 100644 --- a/services/readme.md +++ b/services/readme.md @@ -11,9 +11,10 @@ local network: each node advertises itself over mDNS as one consolidated node offers and where to reach them. What a discovered node can actually serve is a separate question, answered after -discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, both, -or neither, and its model inventory is fetched over HTTP from its engine-manager -rather than crammed into mDNS TXT records, which are too small to carry it. +discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, +[vLLM](https://docs.vllm.ai/), any combination of them, or none, and its model +inventory is fetched over HTTP from its engine-manager rather than crammed into +mDNS TXT records, which are too small to carry it. Locally, each node exposes compatibility proxies — Ollama-compatible and OpenAI-compatible — so an unmodified client on that machine can reach any capable @@ -41,7 +42,7 @@ This tree builds thirteen Go binaries. `nvpair-ui-broker` is the parent service | --- | --- | | `nvpair-ui-broker` | Parent service and JSON-RPC API surface used by the bundled UI and other clients. Supervises workers, relays consolidated discovery, and coordinates routing and scheduling. | | `ollama-proxy` | Ollama-compatible HTTP reverse proxy. Routes only to advertised model owners, with owner failover and scheduler priorities. | -| `lmstudio-proxy` | LM Studio counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes with equivalent owner-only routing and failover behavior. | +| `lmstudio-proxy` | OpenAI-compatible counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes for **every** such engine — LM Studio and vLLM — with equivalent owner-only routing and failover behavior. One process fronts them all; the binary name is a wire contract and keeps its historical spelling. | | `nvpair-node-info` | Local HTTP service on `:14318` exposing GPU, CPU, and memory inventory at `/v1/node-info`. | | `nvpair-node-scanner` | Consolidated discovery daemon. Advertises and browses `_nvpair-node._tcp`, maintains the node directory, and enriches peers with hardware and model information over HTTP. | | `nvpair-manual-nodes` | Manages user-added nodes that don't appear via mDNS; probes them every 10 s. | @@ -59,7 +60,7 @@ The mDNS responder is our own rather than the host's, because Windows ships none The broker feeds every accepted local or peer workload transition plus compact GPU telemetry to the scheduler. Queued and running work is counted by destination -node across Ollama and LM Studio together. Fresh maximum-GPU utilization is +node across every engine together. Fresh maximum-GPU utilization is smoothed into pressure 0–3; missing or stale telemetry is neutral. Rankings use `pending + gpuPressure`, and each proxy adds local reservations before choosing, so bursts spread without waiting for workload feedback. @@ -69,7 +70,7 @@ so bursts spread without waiting for workload feedback. ``` nvpair-ui-broker/ Parent service / JSON-RPC API surface ollama-proxy/ Ollama-compatible routing proxy -lmstudio-proxy/ OpenAI-compatible routing proxy for LM Studio +lmstudio-proxy/ OpenAI-compatible routing proxy (LM Studio, vLLM) nvpair-node-info/ Local GPU-inventory HTTP service nvpair-node-scanner/ Consolidated _nvpair-node._tcp discovery daemon nvpair-manual-nodes/ Manual-node manager diff --git a/services/shared/noderec/noderec.go b/services/shared/noderec/noderec.go index 112e7fe3..5d2c6f14 100644 --- a/services/shared/noderec/noderec.go +++ b/services/shared/noderec/noderec.go @@ -10,7 +10,7 @@ // whose TXT map carries a schema version, the node's identity, its LAN address, // and one compact key per local service port, e.g.: // -// v=1;uuid=;cluster-uuid=;ip=192.168.1.10;ni=14318;ol=11434;lm=1234;er=14319;wl=14320;cl=14321;em=14322 +// v=1;uuid=;cluster-uuid=;ip=192.168.1.10;ni=14318;ol=11434;lm=1234;vl=8000;er=14319;wl=14320;cl=14321;em=14322 // // Design decisions this package encodes: // - SRV port is a fixed, NON-authoritative constant; consumers ignore it and @@ -87,7 +87,12 @@ const ( ServiceNodeInfo ServiceKey = "ni" ServiceOllama ServiceKey = "ol" ServiceLMStudio ServiceKey = "lm" - ServiceErrors ServiceKey = "er" + // ServiceVLLM is a node's vLLM engine, reached through the same + // OpenAI-compatible proxy that fronts LM Studio. A node may advertise lm and + // vl at once (both point at that one proxy port); which engine owns a given + // model comes from the engine-manager model attribution, not from the key. + ServiceVLLM ServiceKey = "vl" + ServiceErrors ServiceKey = "er" ServiceWorkload ServiceKey = "wl" ServiceCluster ServiceKey = "cl" // ServiceEngineManager is nvpair-engine-manager's LAN HTTP endpoint (the model @@ -104,7 +109,7 @@ const ( // serviceKeyOrder is the deterministic emit order for service ports in TXT. var serviceKeyOrder = []ServiceKey{ - ServiceNodeInfo, ServiceOllama, ServiceLMStudio, + ServiceNodeInfo, ServiceOllama, ServiceLMStudio, ServiceVLLM, ServiceErrors, ServiceWorkload, ServiceCluster, ServiceEngineManager, ServiceEngineControl, } diff --git a/services/tests/model_routing_interop_test.go b/services/tests/model_routing_interop_test.go index b6c13dd7..7e36c4f8 100644 --- a/services/tests/model_routing_interop_test.go +++ b/services/tests/model_routing_interop_test.go @@ -64,12 +64,16 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) { type proxyCase struct { name string rpcPrefix string + // engine names which engine the proxy's manual entry represents. The + // OpenAI proxy fronts more than one, so its manual overlay is keyed by + // (engine, node) and the field is required. + engine string path string port int } cases := []proxyCase{ - {name: "ollama", rpcPrefix: "proxy", path: "/api/chat", port: ollamaPort}, - {name: "lmstudio", rpcPrefix: "lmstudio-proxy", path: "/v1/chat/completions", port: lmstudioPort}, + {name: "ollama", rpcPrefix: "proxy", engine: "ollama", path: "/api/chat", port: ollamaPort}, + {name: "lmstudio", rpcPrefix: "lmstudio-proxy", engine: "lmstudio", path: "/v1/chat/completions", port: lmstudioPort}, } client := &http.Client{Timeout: 5 * time.Second} t.Cleanup(client.CloseIdleConnections) @@ -99,6 +103,7 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) { for _, node := range nodes { params := map[string]any{ "id": node.id, + "engine": tc.engine, "host": "127.0.0.1", "port": node.port, "addresses": []string{"127.0.0.1"}, diff --git a/services/tests/scheduler_interop_test.go b/services/tests/scheduler_interop_test.go index 102f44cf..fb829663 100644 --- a/services/tests/scheduler_interop_test.go +++ b/services/tests/scheduler_interop_test.go @@ -583,7 +583,7 @@ func TestLMStudioProxyIgnoresPriorityNodesAbsentFromDiscovery(t *testing.T) { proxyPort := waitLMStudioProxyReady(t, stdin, msgs, 15*time.Second) writeRawFrame(t, stdin, fmt.Sprintf( - `{"jsonrpc":"2.0","id":90,"method":"lmstudio-proxy:node/add-manual","params":{"id":"real-lm","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["chat-model"]}}`, + `{"jsonrpc":"2.0","id":90,"method":"lmstudio-proxy:node/add-manual","params":{"id":"real-lm","engine":"lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["chat-model"]}}`, realPort, )) if resp := waitForResponse(t, msgs, 5*time.Second); resp.Error != nil { diff --git a/services/tests/workload_identity_interop_test.go b/services/tests/workload_identity_interop_test.go index f3879a74..a0f904e1 100644 --- a/services/tests/workload_identity_interop_test.go +++ b/services/tests/workload_identity_interop_test.go @@ -93,7 +93,7 @@ func TestWorkloadCrossEngineIdentityDistinct(t *testing.T) { waitForResponse(t, msgs, 5*time.Second) writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`) waitForResponse(t, msgs, 5*time.Second) - writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":52,"method":"lmstudio-proxy:node/add-manual","params":{"id":"fake-lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, lmstudioPort)) + writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":52,"method":"lmstudio-proxy:node/add-manual","params":{"id":"fake-lmstudio","engine":"lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, lmstudioPort)) waitForResponse(t, msgs, 5*time.Second) writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":53,"method":"lmstudio-proxy:node/select","params":{"id":"fake-lmstudio"}}`) waitForResponse(t, msgs, 5*time.Second) diff --git a/services/versions.json b/services/versions.json index 29d8c230..7b73ed4e 100644 --- a/services/versions.json +++ b/services/versions.json @@ -1,20 +1,20 @@ { "$comment": "Single source of truth for all version numbers. See VERSIONING.md for bump rules.", - "product": "0.91.7", - "installer": "0.91.7", + "product": "0.92.0", + "installer": "0.92.0", "components": { "ollama-proxy": "0.26.2", - "lmstudio-proxy": "0.16.2", + "lmstudio-proxy": "1.0.0", "nvpair-node-info": "0.13.3", - "nvpair-node-scanner": "0.20.3", - "nvpair-manual-nodes": "0.11.1", - "nvpair-workload-manager": "0.13.3", - "nvpair-errors": "0.7.4", + "nvpair-node-scanner": "0.20.4", + "nvpair-manual-nodes": "0.12.0", + "nvpair-workload-manager": "0.13.4", + "nvpair-errors": "0.7.5", "nvpair-node-settings": "1.0.4", - "nvpair-ui-broker": "0.40.2", - "nvpair-engine-manager": "0.17.4", + "nvpair-ui-broker": "0.41.0", + "nvpair-engine-manager": "0.18.0", "nvpair-cluster-manager": "1.1.4", - "nvpair-job-scheduler": "0.4.1", - "nvpair-tui": "0.7.2" + "nvpair-job-scheduler": "0.5.0", + "nvpair-tui": "0.7.3" } }