diff --git a/README.md b/README.md index 0f0a7242..06ce2e6b 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,12 @@ requests can be routed to eligible nodes according to engine availability, model availability, and current workload. PAIR is useful for concurrent local workloads such as multi-agent applications. -Prompts and responses are intended to remain on the local network when every -configured client, model source, engine, and node is local. +Prompts and responses are intended to stay on the network you route them over — +your local network, or an encrypted overlay such as a Tailscale tailnet that you +configured yourself — when every configured client, model source, engine, and +node is one of yours. Refer to +[Running PAIR across a Tailscale tailnet](docs/remote-networks.mdx) for nodes +that are not on the same local link. > PAIR routes each independent request to one node. It does **not** pool GPU > memory, combine GPUs into a larger logical GPU, shard one model across @@ -38,7 +42,7 @@ one, and both report live GPU and memory use throughout. | **Architectures** | x64 and arm64 on all three. Windows on ARM is experimental. | | **Installers** | Windows `.exe`; Linux `.deb`; macOS `.dmg`. On other Linux distributions, [build from source](docs/building.mdx). | | **Mixing nodes** | Windows, Linux, and macOS nodes can all be paired with each other | -| **Inference engines** | Ollama and LM Studio | +| **Inference engines** | Ollama, LM Studio, and — on Linux — vLLM and SGLang | **PAIR running on a machine does not mean an engine will.** PAIR itself runs on any supported Windows, Linux, or macOS machine. Each engine sets its own requirements @@ -100,9 +104,10 @@ you want by its full filename instead. status there. - **Get an engine running.** On the node's card, open **Engine settings** and - select **Install** next to Ollama or LM Studio. PAIR downloads and sets the - engine up for you, so nothing needs to be in place beforehand. If PAIR already - found an engine you installed yourself, start that one instead. + select **Install** next to Ollama, LM Studio, or (on Linux) vLLM or SGLang. + PAIR downloads and sets the engine up for you, so nothing needs to be in place + beforehand. If PAIR already found an engine you installed yourself, start that + one instead. ![The Install engines dialog with Ollama downloading, reporting progress as it installs.](docs/assets/onboarding/engine-lifecycle/01-engine-installing.png) @@ -149,8 +154,9 @@ The reply is ordinary OpenAI-shaped JSON, abbreviated here: } ``` -If you changed a port, or you are using LM Studio rather than Ollama, copy the -URL from **Endpoints → API endpoints** instead of assuming the one above. +If you changed a port, or you are using LM Studio, vLLM, or SGLang rather than +Ollama, copy the URL from **Endpoints → API endpoints** instead of assuming the +one above. That is a single machine working. To route across machines, pair a second one from **Settings → Cluster** and repeat the engine and model steps there. The @@ -224,19 +230,22 @@ Each entry assumes the ones before it. 4. **[Terminal interface](docs/terminal-interface.mdx)** — the same tasks from a terminal, for a machine with no desktop environment. Skip it if every machine you run has a desktop. -5. **[Troubleshooting](docs/troubleshooting.mdx)** — worth skimming once before +5. **[Remote networks](docs/remote-networks.mdx)** — read this if your machines + are joined by a VPN or overlay network rather than a local link: discovery + does not cross one, so you add the peer by name instead. +6. **[Troubleshooting](docs/troubleshooting.mdx)** — worth skimming once before you need it, so you know where the diagnostics live. Alongside it, **[Known issues](docs/known-issues.mdx)** lists the significant limitations we are already aware of, and **[Collecting and sanitizing logs](docs/log-collection.mdx)** covers preparing a log you can share. -6. **[Architecture](docs/architecture.mdx)** — the process model, how a request is +7. **[Architecture](docs/architecture.mdx)** — the process model, how a request is routed, and where the trust boundaries are. Read this before changing anything, or if you want to know why PAIR behaves the way it does. -7. **[Building and running](docs/building.mdx)** — prerequisites, building from +8. **[Building and running](docs/building.mdx)** — prerequisites, building from source, running the services without the desktop application, and writing your own client against the JSON-RPC API. -8. **[Developer guide](docs/developing.mdx)** — read this before contributing: +9. **[Developer guide](docs/developing.mdx)** — read this before contributing: where the code lives, how a change travels through the layers, and the conventions the project enforces. diff --git a/SECURITY.md b/SECURITY.md index 957f9da4..8b3639fe 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -48,7 +48,10 @@ PAIR is a LAN-first, multi-process application: JSON-RPC 2.0 over the broker's standard input and output. - The broker supervises Go workers and relays their control-plane methods and notifications. -- Discovery and selected metadata endpoints operate on the local network. +- Discovery and selected metadata endpoints operate on the local network, or on + an encrypted overlay network the operator configured, such as a Tailscale + tailnet. Discovery itself is multicast and does not cross one; a node on the + far side is added by address. - Ollama-compatible and OpenAI-compatible local HTTP proxies carry inference traffic and may route a request to another paired node. - The cluster manager uses a six-digit PIN to bootstrap trust. Cluster-scoped @@ -76,7 +79,7 @@ open relay for inference to anything that can route to it. Run an application on a node and use that node's local endpoint. Exposing an engine to the network directly is outside PAIR and is the operator's decision and risk. -### Local Network Is a Trust-Relevant Boundary +### The Network You Route Over Is a Trust-Relevant Boundary PAIR discovers nodes and exposes service metadata on the LAN. Some discovery enrichment and node-information traffic can use plain HTTP. Treat an untrusted @@ -84,9 +87,18 @@ Wi-Fi, shared office network, compromised router, and hostile local process as potentially adversarial. Network segmentation and host firewall rules remain the operator's responsibility. +Nodes joined by an encrypted overlay network such as a Tailscale tailnet, rather +than by a local link, are the same boundary reached a different way. The overlay +carries its own encryption and its own admission policy, and PAIR's inter-node +traffic is certificate-pinned mutual TLS in either case; what changes is that +everyone admitted to the overlay is on this boundary, so who may join it is the +operator's decision. Refer to +[Running PAIR across a Tailscale tailnet](docs/remote-networks.mdx). + “Local-first” describes the intended topology. It does not prove that no data -leaves the machine or LAN. Inference engines, model catalogs, update systems, -applications, and user configuration may contact external services. +leaves the machine, the LAN, or an overlay you configured. Inference engines, +model catalogs, update systems, applications, and user configuration may contact +external services. ### Pairing PIN Is a Bootstrap Convenience diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.md index 590e7264..5fcf1021 100644 --- a/desktop/docs/services-api.md +++ b/desktop/docs/services-api.md @@ -123,6 +123,7 @@ | `engine:remote-stop` | request (we call) | ✅ yes | | `engine:remote-unload-model` | request (we call) | ✅ yes | | `engine:restart` | request (we call) | ⚠️ not called | +| `engine:set-model` | request (we call) | ✅ yes | | `engine:set-port` | request (we call) | ✅ yes | | `engine:start` | request (we call) | ✅ yes | | `engine:status` | request (we call) | ✅ yes | diff --git a/desktop/docs/services-backend.md b/desktop/docs/services-backend.md index 0d54c4c9..626c9eb9 100644 --- a/desktop/docs/services-backend.md +++ b/desktop/docs/services-backend.md @@ -22,7 +22,7 @@ broker supervises every worker and relays its control plane. | ------------------------- | --------------------------------------------------------- | | `nvpair-ui-broker` | Worker supervision and relay | | `ollama-proxy` | Ollama-compatible routing proxy with cluster-mTLS ingress | -| `lmstudio-proxy` | LM Studio routing proxy with cluster-mTLS ingress | +| `lmstudio-proxy` | OpenAI-compatible routing proxy (LM Studio, vLLM, SGLang) with cluster-mTLS ingress | | `nvpair-node-scanner` | Discovery and node announcement | | `nvpair-node-info` | Node metadata and telemetry | | `nvpair-manual-nodes` | User-managed node entries | @@ -90,8 +90,10 @@ engine, workload, cluster, and error relays. The bridge then emits renderer push events from backend notifications. Connector readiness follows the broker contract: `app:ready` establishes the -service connection, while Ollama and LM Studio proxy readiness remains an -asynchronous capability signal. Personal AI Router waits up to the canonical +service connection, while the Ollama and OpenAI-compatible proxy readiness +signals remain asynchronous capability signals. The OpenAI-compatible proxy +fronts LM Studio, vLLM and SGLang together, so one `lmstudio-proxy:ready` records +the port for all of them. Personal AI Router waits up to the canonical startup deadline in `src/shared/constants/modular-runtime.ts` for `app:ready`; an outright failure or stalled broker startup is surfaced in Settings > Service with retry and log access. If a stalled broker reports ready @@ -266,9 +268,11 @@ LAN-reachable. Each node fronts its engine with its `ollama-proxy` / `lmstudio-proxy`, whose LAN ingress is gated by cluster mTLS: only a pinned cluster member can send it work. Discovery advertises the promoted **proxy** port (never the engine port), and the broker hands the private loopback engine to -the local proxy via `node/set-local-backend`. Every cluster-scoped worker derives -its own membership from the cluster directory continuously, so a proxy's mTLS -ingress tracks a create, join, or leave with no worker restart. Loopback +the local proxy via `node/set-local-backend`, which is keyed per engine so the +one OpenAI-compatible proxy can hold an LM Studio, a vLLM and an SGLang backend +at once. Every cluster-scoped worker derives its own membership from the cluster +directory continuously, so a proxy's mTLS ingress tracks a create, join, or leave +with no worker restart. Loopback plaintext requests from local clients are unaffected. This is entirely backend-owned; Personal AI Router only reflects the advertised proxy port and does not implement any of the transport security. diff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.md index 93d13290..8f3e7edb 100644 --- a/desktop/docs/services-parity.md +++ b/desktop/docs/services-parity.md @@ -24,6 +24,8 @@ history. | Manual nodes | Complete with local persistence | Broker owns probing and proxy registration; Electron persists entries for replay | | Ollama routing | Complete | Broker relay and backend scheduler drive proxy routing | | LM Studio routing | Complete | Parallel broker relay and scheduler path | +| vLLM routing | Complete on Linux | Shares the LM Studio relay, the OpenAI-compatible proxy, and the same scheduler path | +| SGLang routing | Complete on Linux | Shares the LM Studio relay, the OpenAI-compatible proxy, and the same scheduler path | | Local engine lifecycle | Complete | Install, start, stop, uninstall, update, and port configuration | | Remote engine lifecycle | Partial | Remote install, start, stop, status, and model pull are supported | | Engine models | Partial | Core list, pull, load, unload, and supported delete actions are wired | @@ -102,7 +104,9 @@ they survive worker restarts. Both text-engine proxies are broker-owned and cluster-aware: - `ollama-proxy` serves the Ollama-compatible surface; -- `lmstudio-proxy` serves the LM Studio/OpenAI-compatible surface. +- `lmstudio-proxy` serves the OpenAI-compatible surface for every + OpenAI-compatible engine — LM Studio, vLLM and SGLang share the one listener, + and a node record can advertise `lm`, `vl` and `sg` at the same port. Routing precedence is manual selection, scheduler priority, then deterministic proxy ordering. Personal AI Router leaves proxies in automatic mode. @@ -197,8 +201,19 @@ Personal AI Router uses: - Ollama `run_model`, `unload_model` (`keep_alive: 0`), and `delete_model`; - LM Studio `load_model`, `unload_model`, and `delete_model` (`remove_path`). -Both engines expose Load, Eject, and Delete in the model manager when the -backend action exists. Keep-alive / expiry controls remain unsupported. +Ollama and LM Studio expose Load, Eject, and Delete in the model manager when +the backend action exists. Keep-alive / expiry controls remain unsupported. + +vLLM and SGLang have no model operations at all: each declares only +`list_models`, `loaded_models`, and `chat`, because such a process serves exactly +one model chosen at launch and it neither downloads nor deletes weights on +request. Both capability entries therefore set `hasDeleteModel: false`, +`hasEject: false`, `hasExpiry: false`, and `modelOpsWhenStopped: false`, and add +`hasServedModel: true` — the model to serve is a *setting* (`engine:set-model`), +not a model action, and changing it restarts the engine. That setting takes +either a Hugging Face model id or the path of a local model directory: vLLM's +`--model` and SGLang's `--model-path` both accept both, and `/v1/models` reports +back whichever was given, verbatim. LM Studio's `delete_model` declares `restart_after`, so the engine manager restarts a running LM Studio once the files are removed — its `/v1/models` is diff --git a/desktop/src/electron/service-bridge/empty-handlers.ts b/desktop/src/electron/service-bridge/empty-handlers.ts index dc5c2dd8..a69a002e 100644 --- a/desktop/src/electron/service-bridge/empty-handlers.ts +++ b/desktop/src/electron/service-bridge/empty-handlers.ts @@ -11,6 +11,7 @@ import { MODULAR_ENGINE_LIFECYCLE_CALL_TIMEOUT_MS } from '@/shared/constants/modular-runtime' import getErrorString from '@/shared/utils/get-error-string' +import { createStructuredLogger } from '@/shared/utils/log' import { getEngineHubModels } from '@/electron/model-hub' import { getModularSupervisor } from './modular-supervisor' import { @@ -23,7 +24,12 @@ import { import type { ProxyEngine } from './modular-state' import type { JsonObject, JsonValue } from './json-rpc-subprocess' import { emptyInvite, parseClusterNodes, parseInvite, parseNodeIdentity } from './cluster-json' -import { removeManualNodeEntry, resolveManualNodeKey } from './manual-nodes-store' +import { + addManualNodeEntry, + manualPortsToWire, + removeManualNodeEntry, + resolveManualNodeKey +} from './manual-nodes-store' type BridgeHandler = ( payload?: WsInvokeRequest @@ -34,6 +40,8 @@ type BridgeHandlerMap = { } // The broker allows pairing exchanges up to 30 seconds; keep the UI bridge outside that deadline. +const log = createStructuredLogger('service-bridge') + const CLUSTER_PAIRING_CALL_TIMEOUT_MS = 35_000 function wait(ms: number): Promise { @@ -507,6 +515,26 @@ function routeEngineManagerCommand(payload: WsInvokeRequest<'engine:command'>): case 'toggle': void toggleLocalEngine(engine, payload.engineType) break + case 'setServedModel': + // vLLM and SGLang serve one model per process, so the model is a + // start-time setting rather than a model operation. engine:set-model + // persists it as a manifest override — the same mechanism + // engine:set-port uses — and restarts a running engine onto it. It + // is engine-agnostic: the engine id travels with the call, so no + // branch here needs to know which engine declared a served model. + // Local only: the backend exposes no remote served-model control. + supervisor.sendProcess( + 'broker', + 'engine:set-model', + { engine, model: payload.model ?? '' }, + failAction('set the served model', { + nodeId: payload.nodeId, + engineType: payload.engineType, + modelName: payload.model + }), + true + ) + break case 'setPorts': // Both ports persist on this node: the engine HTTP server port via // engine:set-port (manifest override), the proxy port via the broker's @@ -798,6 +826,32 @@ async function handleClusterInviteNode( ): Promise { if (!payload) return emptyInvite() + // Add the address as a manual node BEFORE inviting it. + // + // On a network that carries no multicast — a Tailscale tailnet, a routed + // subnet — nothing is ever discovered, so a peer that is only paired stays + // invisible: no record arrives for it, ever. Adding it first also gives the + // operator the one piece of feedback that matters when an address was typed + // by hand: the node appears with its hardware as soon as it answers, whether + // or not the pairing that follows succeeds. + const supervisor = getModularSupervisor() + if (supervisor.hasProcess('broker')) { + const entry = addManualNodeEntry(payload.ipAddress, payload.ports) + const params: JsonObject = { address: entry.address, name: entry.name } + const wirePorts = manualPortsToWire(entry.ports) + if (wirePorts) params.ports = wirePorts + try { + await supervisor.callProcess('broker', 'node/add', params) + } catch (err) { + // Non-fatal: the entry is persisted and replayed on the next start, + // and the invite below is what the operator asked for. + log.warn({ + sublevel: 'manual-nodes', + message: `Failed to add manual node ${entry.address}: ${getErrorString(err)}` + }) + } + } + // cluster-manager auto-founds a solo cluster on the first invite while // unclustered (under inviteMu, with invite-created provenance). Do not // pre-call cluster:create here: parallel Invites used to race concurrent @@ -813,7 +867,7 @@ async function handleClusterInviteNode( 'cluster:invite-node', { address: payload.ipAddress, - port: MODULAR_CLUSTER_MANAGER_PORT + port: payload.ports?.cluster ?? MODULAR_CLUSTER_MANAGER_PORT }, CLUSTER_PAIRING_CALL_TIMEOUT_MS ) diff --git a/desktop/src/electron/service-bridge/manual-nodes-store.ts b/desktop/src/electron/service-bridge/manual-nodes-store.ts index d79ccea9..ddd01f7a 100644 --- a/desktop/src/electron/service-bridge/manual-nodes-store.ts +++ b/desktop/src/electron/service-bridge/manual-nodes-store.ts @@ -4,12 +4,19 @@ import fs from 'fs' import path from 'path' import { getPaths } from '@/electron/globals' +import type { ManualServicePorts } from '@/shared/types/manual-node' import type { JsonObject, JsonValue } from './json-rpc-subprocess' interface ManualNodeEntry { id: string address: string name: string + /** + * Per-service port overrides, persisted because the replay after a restart + * is the only thing that re-creates the entry: without them a node reachable + * on non-default ports comes back unreachable. + */ + ports?: ManualServicePorts } function configFilePath(): string { @@ -25,6 +32,60 @@ function stringValue(value: JsonValue | undefined): string { return typeof value === 'string' ? value : '' } +/** A usable TCP port, or undefined for anything else. Narrowing, never a cast. */ +function portValue(value: JsonValue | undefined): number | undefined { + if (typeof value !== 'number' || !Number.isInteger(value)) return undefined + if (value < 1 || value > 65535) return undefined + return value +} + +function portsValue(value: JsonValue | undefined): ManualServicePorts | undefined { + const obj = objectValue(value) + if (!obj) return undefined + const ports: ManualServicePorts = { + nodeInfo: portValue(obj.nodeInfo), + cluster: portValue(obj.cluster), + ollama: portValue(obj.ollama), + lmstudio: portValue(obj.lmstudio), + vllm: portValue(obj.vllm), + sglang: portValue(obj.sglang) + } + return definedPorts(ports) +} + +/** + * Drops every unset field, and the object itself when nothing is set. An empty + * overrides object is not the same thing as none: it would persist and replay as + * a value the operator never chose. + */ +function definedPorts(ports: ManualServicePorts): ManualServicePorts | undefined { + const kept: ManualServicePorts = {} + if (ports.nodeInfo !== undefined) kept.nodeInfo = ports.nodeInfo + if (ports.cluster !== undefined) kept.cluster = ports.cluster + if (ports.ollama !== undefined) kept.ollama = ports.ollama + if (ports.lmstudio !== undefined) kept.lmstudio = ports.lmstudio + if (ports.vllm !== undefined) kept.vllm = ports.vllm + if (ports.sglang !== undefined) kept.sglang = ports.sglang + return Object.keys(kept).length > 0 ? kept : undefined +} + +/** + * Projects the overrides onto the snake_case field names `node/add` reads + * (`node_info`, `cluster`, `ollama`, `lmstudio`, `vllm`, `sglang`). The two + * spellings meet here and nowhere else. + */ +export function manualPortsToWire(ports: ManualServicePorts | undefined): JsonObject | undefined { + if (!ports) return undefined + const wire: JsonObject = {} + if (ports.nodeInfo !== undefined) wire.node_info = ports.nodeInfo + if (ports.cluster !== undefined) wire.cluster = ports.cluster + if (ports.ollama !== undefined) wire.ollama = ports.ollama + if (ports.lmstudio !== undefined) wire.lmstudio = ports.lmstudio + if (ports.vllm !== undefined) wire.vllm = ports.vllm + if (ports.sglang !== undefined) wire.sglang = ports.sglang + return Object.keys(wire).length > 0 ? wire : undefined +} + function entryValue(value: JsonValue | undefined): ManualNodeEntry | null { const obj = objectValue(value) if (!obj) return null @@ -34,7 +95,30 @@ function entryValue(value: JsonValue | undefined): ManualNodeEntry | null { const name = stringValue(obj.name) || address const id = stringValue(obj.id) || name - return { id, address, name } + const ports = portsValue(obj.ports) + return ports ? { id, address, name, ports } : { id, address, name } +} + +/** + * Record a manually added node so it survives a restart, and hand back the entry + * the broker should be told about. + * + * `nvpair-manual-nodes` keeps no durable state of its own — the list belongs to + * the application — and the backend keys the node by `name`, which is the + * address here so {@link resolveManualNodeKey} and the `node/remove` relay agree + * on it. Re-adding the same address replaces its entry rather than duplicating + * it, so changing a node's ports is just adding it again. + */ +export function addManualNodeEntry(address: string, ports?: ManualServicePorts): ManualNodeEntry { + const trimmed = address.trim() + const kept = ports ? definedPorts(ports) : undefined + const entry: ManualNodeEntry = kept + ? { id: trimmed, address: trimmed, name: trimmed, ports: kept } + : { id: trimmed, address: trimmed, name: trimmed } + const entries = listManualNodeEntries().filter(existing => existing.address !== trimmed) + entries.push(entry) + saveManualNodeEntries(entries) + return entry } export function listManualNodeEntries(): ManualNodeEntry[] { diff --git a/desktop/src/electron/service-bridge/modular-state.ts b/desktop/src/electron/service-bridge/modular-state.ts index ab6c5fb5..6f7da6d9 100644 --- a/desktop/src/electron/service-bridge/modular-state.ts +++ b/desktop/src/electron/service-bridge/modular-state.ts @@ -40,13 +40,23 @@ type BrokerNodeSource = ProxyNodeSource | 'broker' * Engines surfaced by the broker's proxy plane. Other engine-manager engines * are not currently routed across nodes. */ -export type ProxyEngine = Extract -export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] +export type ProxyEngine = Extract +export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio', 'vllm', 'sglang'] -/** Map a proxy node source onto the engine it describes. */ -const PROXY_SOURCE_ENGINE: Record = { - 'ollama-proxy': 'ollama', - 'lmstudio-proxy': 'lm-studio' +/** + * The engines each proxy fronts. `lmstudio-proxy` is the OpenAI-compatible + * router and serves every engine that speaks that API, so a node it reports is + * not necessarily LM Studio — which engine it actually runs comes from the + * node's own per-engine model attribution (see enginesOnProxyNode). + */ +const PROXY_SOURCE_ENGINES: Record = { + 'ollama-proxy': ['ollama'], + 'lmstudio-proxy': ['lm-studio', 'vllm', 'sglang'] +} + +/** The proxy that fronts an engine, and so the node source it reports under. */ +function proxySourceForEngine(engine: ProxyEngine): ProxyNodeSource { + return engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' } /** Per-engine presence on a node — each proxy reports its own engine. */ @@ -164,10 +174,15 @@ function emptyPresence(): EnginePresence { } function emptyEngines(): Record { - return { ollama: emptyPresence(), 'lm-studio': emptyPresence() } + return { + ollama: emptyPresence(), + 'lm-studio': emptyPresence(), + vllm: emptyPresence(), + sglang: emptyPresence() + } } -/** Immutably set one engine's presence, preserving the other. */ +/** Immutably set one engine's presence, preserving the rest. */ function setEngine( engines: Record, engine: ProxyEngine, @@ -175,7 +190,9 @@ function setEngine( ): Record { return { ollama: engine === 'ollama' ? presence : engines.ollama, - 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'] + 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'], + vllm: engine === 'vllm' ? presence : engines.vllm, + sglang: engine === 'sglang' ? presence : engines.sglang } } @@ -390,7 +407,7 @@ export function parseWorkloadsInitial(value: JsonValue | undefined): Workload[] /** True for an engine fronted by a broker-supervised reverse proxy. */ export function isProxyEngine(engine: EngineType): engine is ProxyEngine { - return engine === 'ollama' || engine === 'lm-studio' + return engine === 'ollama' || engine === 'lm-studio' || engine === 'vllm' || engine === 'sglang' } const PENDING_OP_IDLE_TIMEOUT_MS = 90_000 @@ -709,26 +726,55 @@ function toEngineStatus( * the engine is determined by which relay namespace it arrived on. The presence * is stamped onto that engine only; the other engine stays empty. */ -function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): ModularNode | null { +/** + * Which of a proxy's engines a reported node actually runs. + * + * A proxy that fronts one engine answers for itself — the source names it. The + * OpenAI proxy fronts several, so a node it reports is not necessarily LM + * Studio: it stamps every routed node with per-engine model attribution keyed by + * engine-manager name, and that is what says which of LM Studio, vLLM and SGLang + * the node actually runs — any of them, or several at once. A payload with no + * attribution names no engine, so callers that must clear presence use the + * proxy's whole engine set instead. + */ +function enginesOnProxyNode(source: ProxyNodeSource, params: JsonValue | undefined): ProxyEngine[] { + const fronted = PROXY_SOURCE_ENGINES[source] + if (fronted.length === 1) return [...fronted] + const byEngine = objectValue(objectValue(params)?.modelsByEngine) + if (!byEngine) return [] + // Presence of the key, not a non-empty list: an engine that is running with + // no models reports an empty (JSON null) list and is still present. + return fronted.filter(engine => Object.hasOwn(byEngine, proxyEngineToManagerName(engine))) +} + +function parseProxyNode( + params: JsonValue | undefined, + source: ProxyNodeSource, + engines: readonly ProxyEngine[] +): ModularNode | null { const obj = objectValue(params) if (!obj) return null // The proxy keys `Node.ID` by the stable per-host UUID (and rejects empty), // so this is already the canonical node key — no TXT parsing needed. const id = stringValue(obj.id) if (!id) return null - const engines = emptyEngines() - engines[engine] = { + const presence: EnginePresence = { up: true, // Under secure inference this is the peer's promoted inference proxy - // port, not the engine's own (loopback-private) port. + // port, not the engine's own (loopback-private) port. Every engine a + // proxy fronts shares that one port. port: numberValue(obj.port), // The proxy `Node` carries no version field; engine version comes from // nvpair-engine-manager, not discovery. version: null } + let nodeEngines = emptyEngines() + for (const engine of engines) { + nodeEngines = setEngine(nodeEngines, engine, presence) + } return { id, - sources: [engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'], + sources: [source], // `Node.Host` is the hostname; empty for the self-bridge manual node, // in which case the broker discovery entry supplies the display name on // merge (see mergeNode). Never fall back to the UUID id here. @@ -760,7 +806,7 @@ function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): Mod gpus: [], cpu: null, memory: null, - engines, + engines: nodeEngines, lastSeen: Date.now() } } @@ -899,8 +945,14 @@ class ModularBridgeState { private logs: LogEntry[] = [] // Per-engine bound proxy port reported by the broker. 0 = not reported yet; // we never fabricate a default — an unknown port surfaces as null, not a - // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`. - private proxyPorts: Record = { ollama: 0, 'lm-studio': 0 } + // guess. `ollama` is the `ollama-proxy`; `lm-studio`, `vllm` and `sglang` + // are all the one `lmstudio-proxy` and therefore always hold the same port. + private proxyPorts: Record = { + ollama: 0, + 'lm-studio': 0, + vllm: 0, + sglang: 0 + } private selfId: string | null = null /** * Authoritative local-engine facts from `nvpair-engine-manager`, keyed by @@ -910,7 +962,7 @@ class ModularBridgeState { */ private engineManagerFacts = new Map< EngineType, - { installed: boolean; running: boolean; port: number } + { installed: boolean; running: boolean; port: number; servedModel: string } >() /** * Local model lists pulled from `nvpair-engine-manager`'s `list_models` action by @@ -1375,7 +1427,10 @@ class ModularBridgeState { this.engineManagerFacts.set(engineType, { installed: booleanValue(obj.installed), running: booleanValue(obj.running), - port: numberValue(obj.port) + port: numberValue(obj.port), + // Empty for every engine that does not serve one model per process, + // and for one that does but has no model chosen yet. + servedModel: stringValue(obj.model) }) // A fresh authoritative state is the resolution of whatever op was in // flight (start/stop done, install `done`+installed, uninstall removed). @@ -2073,7 +2128,8 @@ class ModularBridgeState { nodeId, processStatus: pending, enginePort: facts && facts.running && facts.port > 0 ? facts.port : null, - proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null + proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null, + servedModel: facts?.servedModel } } @@ -2096,7 +2152,10 @@ class ModularBridgeState { // Each proxy-fronted engine has its own broker proxy // (`ollama-proxy` / `lmstudio-proxy`); report that engine's bound // proxy port. Loopback-only engines get null. - proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null + proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null, + // The engine's configured served model, for an engine that runs + // one model per process. Empty when none is chosen. + servedModel: facts.servedModel } } @@ -2280,11 +2339,11 @@ class ModularBridgeState { handleNotification(notification: JsonRpcNotification): void { if (notification.source === 'proxy') { - this.handleProxyNotification(notification, 'ollama') + this.handleProxyNotification(notification, 'ollama-proxy') return } if (notification.source === 'lmstudio-proxy') { - this.handleProxyNotification(notification, 'lm-studio') + this.handleProxyNotification(notification, 'lmstudio-proxy') return } if (notification.source === 'broker') { @@ -2292,28 +2351,36 @@ class ModularBridgeState { } } - private handleProxyNotification(notification: JsonRpcNotification, engine: ProxyEngine): void { + private handleProxyNotification( + notification: JsonRpcNotification, + source: ProxyNodeSource + ): void { + const fronted = PROXY_SOURCE_ENGINES[source] if (notification.method === 'ready') { const params = objectValue(notification.params) // Trust the broker-reported port only. If `ready` carries no port we // keep the last known value (0 = unknown) rather than guessing. const nextPort = numberValue(params?.port) if (nextPort <= 0) return - const changed = nextPort !== this.proxyPorts[engine] - this.proxyPorts[engine] = nextPort - // A runtime proxy:set-port (or a broker steer onto a free port) - // re-emits `ready` with the new port. Push a fresh status for that - // engine so the Edit Node proxy port reflects the actually-bound value - // immediately. Only on a real change so the startup baseline `ready` - // (and redundant re-readies) stay quiet. - if (changed) this.emitLocalEngineStatus(engine) + // Every engine this proxy fronts is reached through the one listener, + // so they all move together. + for (const engine of fronted) { + const changed = nextPort !== this.proxyPorts[engine] + this.proxyPorts[engine] = nextPort + // A runtime proxy:set-port (or a broker steer onto a free port) + // re-emits `ready` with the new port. Push a fresh status for that + // engine so the Edit Node proxy port reflects the actually-bound + // value immediately. Only on a real change so the startup baseline + // `ready` (and redundant re-readies) stay quiet. + if (changed) this.emitLocalEngineStatus(engine) + } return } if (notification.method === 'error') { const params = objectValue(notification.params) this.upsertError({ - id: `${engine}-proxy:${Date.now()}`, + id: `${source}:${Date.now()}`, message: stringValue(params?.message) || 'Modular proxy failed', timestamp: Date.now(), severity: 'error', @@ -2323,16 +2390,23 @@ class ModularBridgeState { } if (notification.method === 'node/removed') { - const node = parseProxyNode(notification.params, engine) + // A removal payload carries the node id alone, so it names no engine. + // The proxy only sends one once its last engine entry for that node is + // gone, so clearing every engine it fronts is the accurate reading. + const node = parseProxyNode(notification.params, source, fronted) if (!node) return - this.clearNodeEngine(node.id, engine) + for (const engine of fronted) { + this.clearNodeEngine(node.id, engine) + } return } if (notification.method === 'node/discovered' || notification.method === 'node/updated') { - const node = parseProxyNode(notification.params, engine) + const engines = enginesOnProxyNode(source, notification.params) + if (engines.length === 0) return + const node = parseProxyNode(notification.params, source, engines) if (!node) return - this.upsertNode(node, engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy') + this.upsertNode(node, source) } } @@ -2344,7 +2418,7 @@ class ModularBridgeState { private clearNodeEngine(nodeId: string, engine: ProxyEngine): void { const existing = this.nodes.get(nodeId) if (!existing) return - const source: BrokerNodeSource = engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' + const source: BrokerNodeSource = proxySourceForEngine(engine) const sources = removeSource(existing.sources, source) if (sources.length === 0 && !existing.nodeInfoUp) { this.removeNodeEntry(nodeId) @@ -2592,9 +2666,16 @@ class ModularBridgeState { } } - // A proxy source (ollama-proxy / lmstudio-proxy): refresh only that - // engine's presence; keep the other engine, telemetry, and node-info. - const engine = PROXY_SOURCE_ENGINE[source] + // A proxy source (ollama-proxy / lmstudio-proxy): refresh only the + // engines that proxy fronts; keep the others, telemetry, and node-info. + // The OpenAI proxy fronts more than one, and its event is authoritative + // for all of them — a node that dropped vLLM but kept LM Studio reports + // exactly that, so both presences come from the incoming node. + const fronted = PROXY_SOURCE_ENGINES[source] + const mergedEngines = fronted.reduce( + (acc, engine) => setEngine(acc, engine, next.engines[engine]), + existing.engines + ) return { ...next, sources: mergeSources(existing.sources, source), @@ -2623,7 +2704,7 @@ class ModularBridgeState { models: existing.models, modelsByEngine: existing.modelsByEngine, loadedByEngine: existing.loadedByEngine, - engines: setEngine(existing.engines, engine, next.engines[engine]), + engines: mergedEngines, lastSeen: Math.max(existing.lastSeen, next.lastSeen) } } diff --git a/desktop/src/electron/service-bridge/modular-supervisor.ts b/desktop/src/electron/service-bridge/modular-supervisor.ts index 943d0f84..05514fcf 100644 --- a/desktop/src/electron/service-bridge/modular-supervisor.ts +++ b/desktop/src/electron/service-bridge/modular-supervisor.ts @@ -38,7 +38,7 @@ import { isModularLogLevel, type ModularLogLevel } from '@/shared/constants/modular-runtime' -import { listManualNodeEntries } from './manual-nodes-store' +import { listManualNodeEntries, manualPortsToWire } from './manual-nodes-store' import { MODULAR_RUNTIME_BINARIES, modularBinaryFileName @@ -299,14 +299,32 @@ function engineManagerId(engine: ProxyEngine): string { function proxyEngineFromManagerId(id: string): ProxyEngine | null { if (id === 'ollama') return 'ollama' if (id === 'lmstudio') return 'lm-studio' + if (id === 'vllm') return 'vllm' + if (id === 'sglang') return 'sglang' return null } -/** The broker relay namespace fronting an engine's reverse proxy. */ +/** + * The broker relay namespace fronting an engine's reverse proxy. Every + * OpenAI-compatible engine shares one router, so they share one namespace; the + * `lmstudio-proxy` spelling is a wire contract and keeps its historical name. + */ function proxyRelayPrefix(engine: ProxyEngine): string { return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy' } +/** The bridge notification source for an engine's proxy. */ +function proxyNotificationSource(engine: ProxyEngine): 'proxy' | 'lmstudio-proxy' { + return engine === 'ollama' ? 'proxy' : 'lmstudio-proxy' +} + +/** + * One engine per supervised proxy process, for the sweeps that are a property of + * the proxy rather than of an engine (status polling, node hydration). Iterating + * PROXY_ENGINES there would query the OpenAI proxy once per engine it fronts. + */ +const PROXY_REPRESENTATIVE_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] + /** * Spawns and supervises the modular backend. * @@ -851,10 +869,10 @@ class ModularSupervisor { const entries = listManualNodeEntries() for (const entry of entries) { try { - await this.callProcess('broker', 'node/add', { - address: entry.address, - name: entry.name - }) + const params: JsonObject = { address: entry.address, name: entry.name } + const wirePorts = manualPortsToWire(entry.ports) + if (wirePorts) params.ports = wirePorts + await this.callProcess('broker', 'node/add', params) } catch (err) { log.warn({ sublevel: 'manual-nodes', @@ -891,7 +909,7 @@ class ModularSupervisor { await this.seedClusterPeerIds() await this.refreshAllRemoteEngineStatus() this.startRemoteStatusPolling() - for (const engine of PROXY_ENGINES) { + for (const engine of PROXY_REPRESENTATIVE_ENGINES) { await this.hydrateProxyNodes(engine) await this.pollProxyStatus(engine) } @@ -1079,7 +1097,7 @@ class ModularSupervisor { const obj = objectValue(result) if (obj && booleanValue(obj.ready)) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyNotificationSource(engine), method: 'ready', params: { port: numberValue(obj.port) } }) @@ -1100,7 +1118,7 @@ class ModularSupervisor { if (!obj || !Array.isArray(obj.nodes)) return for (const node of obj.nodes) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'proxy' : 'lmstudio-proxy', + source: proxyNotificationSource(engine), method: 'node/discovered', params: node }) @@ -1266,19 +1284,23 @@ class ModularSupervisor { this.scheduleRemoteEngineStatusRefresh() } - const proxyEngine: ProxyEngine | null = + // Every engine the (re)bound proxy fronts loses its manual-node set at + // once, since they share the process. + const reboundEngines: readonly ProxyEngine[] = event.source === 'proxy' - ? 'ollama' + ? ['ollama'] : event.source === 'lmstudio-proxy' - ? 'lm-studio' - : null - if (proxyEngine && event.method === 'ready') { + ? PROXY_ENGINES.filter(engine => engine !== 'ollama') + : [] + if (reboundEngines.length > 0 && event.method === 'ready') { // A (re)bound proxy starts with an empty manual-node set, so forget // what we think we bridged and re-push the local node if applicable. - const bridge = this.getLocalBridge(proxyEngine) - bridge.bridgedId = '' - bridge.bridgedPort = 0 - void this.reconcileLocalNodeBridge(proxyEngine) + for (const proxyEngine of reboundEngines) { + const bridge = this.getLocalBridge(proxyEngine) + bridge.bridgedId = '' + bridge.bridgedPort = 0 + void this.reconcileLocalNodeBridge(proxyEngine) + } this.emitStateRefreshIfHydrated() } if (event.source === 'broker' && event.method === 'app:ready') { @@ -2188,6 +2210,9 @@ class ModularSupervisor { try { await this.callProxy(engine, 'node/add-manual', { id: selfId, + // The OpenAI proxy fronts several engines on different local + // ports, so a manual entry names the engine it belongs to. + engine: engineManagerId(engine), host: '127.0.0.1', port: bridge.port, addresses: ['127.0.0.1'] @@ -2208,7 +2233,10 @@ class ModularSupervisor { bridge.bridgedId = '' bridge.bridgedPort = 0 try { - await this.callProxy(engine, 'node/remove-manual', { id: previousId }) + await this.callProxy(engine, 'node/remove-manual', { + id: previousId, + engine: engineManagerId(engine) + }) } catch (err) { log.verbose({ sublevel: proxyRelayPrefix(engine), diff --git a/desktop/src/shared/constants/engines.ts b/desktop/src/shared/constants/engines.ts index 3a1472c8..fede948d 100644 --- a/desktop/src/shared/constants/engines.ts +++ b/desktop/src/shared/constants/engines.ts @@ -9,23 +9,33 @@ import { EngineType, ModelExpiry } from '@/shared/types/engines' // never-enabled placeholders; they were removed with the chat window, which was // their only in-app consumer. Adding an engine back means shipping its manifest // first -- an engine row without one renders commands that fail with `-32000`. -export const EngineTypes = ['ollama', 'lm-studio'] as const +export const EngineTypes = ['ollama', 'lm-studio', 'vllm', 'sglang'] as const // Kept as a distinct export so a future engine can ship behind it rather than // appearing the moment its type exists. -export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio'] as const +export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio', 'vllm', 'sglang'] as const export const EngineSources = ['bundled', 'detected', 'installed'] as const export const EngineDisplayNames: Record = { ollama: 'Ollama', - 'lm-studio': 'LM Studio' + 'lm-studio': 'LM Studio', + vllm: 'vLLM', + sglang: 'SGLang' } as const /** Default docs/install URLs for built-in backends. Single source of truth for UI and adapter buildInfo(). */ export const EngineDefaultLinks: Record = { ollama: { docsUrl: 'https://docs.ollama.com/', installUrl: 'https://ollama.com/download' }, - 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' } + 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' }, + vllm: { + docsUrl: 'https://docs.vllm.ai/', + installUrl: 'https://docs.vllm.ai/en/latest/getting_started/installation/' + }, + sglang: { + docsUrl: 'https://docs.sglang.ai/', + installUrl: 'https://docs.sglang.ai/get_started/install.html' + } } as const export const ModelItemStatuses = ['idle', 'loading', 'loaded', 'ejecting', 'pulling'] as const diff --git a/desktop/src/shared/types/engine-api.ts b/desktop/src/shared/types/engine-api.ts index 351f5c5b..12ca5c21 100644 --- a/desktop/src/shared/types/engine-api.ts +++ b/desktop/src/shared/types/engine-api.ts @@ -50,12 +50,19 @@ export type EngineCommandType = | 'unloadModel' | 'deleteModel' | 'setModelExpiry' + | 'setServedModel' /** Payload for engine commands sent from the UI. */ export interface EngineCommandPayload { command: EngineCommandType engineType: EngineType nodeId: string + /** + * The model an operation acts on. For `setServedModel` it is the model the + * engine should serve from its next start; an empty string clears the + * choice, leaving an engine that requires one unable to start until it is + * set again. + */ model?: string /** * `setPorts` only. The engine HTTP server port to apply. Omitted when the diff --git a/desktop/src/shared/types/engines.ts b/desktop/src/shared/types/engines.ts index 54a228b4..39b6b7f4 100644 --- a/desktop/src/shared/types/engines.ts +++ b/desktop/src/shared/types/engines.ts @@ -40,6 +40,13 @@ export interface EngineStatusData { * reported version data or engines that are not installed. */ installedVersion?: string + /** + * The model this engine is configured to serve, for an engine that runs one + * model per process (vLLM, SGLang). Empty string when none is chosen — the + * state in which the engine refuses to start. Undefined for every other + * engine. + */ + servedModel?: string } export type ModelItemStatus = (typeof ModelItemStatuses)[number] diff --git a/desktop/src/shared/types/manual-node.ts b/desktop/src/shared/types/manual-node.ts new file mode 100644 index 00000000..af169caf --- /dev/null +++ b/desktop/src/shared/types/manual-node.ts @@ -0,0 +1,25 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +/** + * Per-service port overrides for a manually added node. + * + * The defaults `nvpair-manual-nodes` assumes describe a machine it cannot + * introspect: a peer may run an engine on a second port, a whole node may be + * reachable only through a forwarded range, and a test may put two nodes on one + * loopback. An unset field keeps that service's default, so an entry overrides + * only what the operator meant to. + * + * `vllm` and `sglang` name the OpenAI-compatible engines' own server ports + * (stock 8000 and 30000). They are carried and persisted here so a manually + * added node keeps whatever the operator set for them, whether or not the + * backend probes that port yet. + */ +export interface ManualServicePorts { + nodeInfo?: number + cluster?: number + ollama?: number + lmstudio?: number + vllm?: number + sglang?: number +} diff --git a/desktop/src/shared/types/ws-channels.ts b/desktop/src/shared/types/ws-channels.ts index 07398eab..841a96e5 100644 --- a/desktop/src/shared/types/ws-channels.ts +++ b/desktop/src/shared/types/ws-channels.ts @@ -23,6 +23,7 @@ * - `export` is reserved for the handful of symbols consumed by the * service bridge generics — nothing else. */ +import type { ManualServicePorts } from '@/shared/types/manual-node' import type { EngineCommandPayload, EngineHubSearchResponse, @@ -75,7 +76,17 @@ export interface WsInvokeChannelMap { // Cluster — PIN-pairing handshake (nvpair-cluster-manager) 'cluster:get-initial': { request: void; response: ClusterInitialSnapshot } - 'cluster:invite-node': { request: { ipAddress: string }; response: Invite } + // Add a node by address and start pairing with it, in that order. The + // address is added as a manual node first so it appears with its hardware + // as soon as it answers -- on a network without multicast that is the only + // feedback the operator gets about whether the address is right, and it is + // also what keeps a successfully paired peer visible afterwards, since no + // discovery record will ever arrive for it. `ports` overrides the assumed + // port of any single service on that host. + 'cluster:invite-node': { + request: { ipAddress: string; ports?: ManualServicePorts } + response: Invite + } 'cluster:invite-status': { request: { inviteId: string }; response: Invite } 'cluster:respond-to-invite': { request: { inviteId: string; accept: boolean; pin?: string } diff --git a/desktop/src/ui/api/engine-api.ts b/desktop/src/ui/api/engine-api.ts index 69fd71a2..1f59a928 100644 --- a/desktop/src/ui/api/engine-api.ts +++ b/desktop/src/ui/api/engine-api.ts @@ -47,6 +47,12 @@ export interface IEngineApi { deleteModel(engineType: EngineType, nodeId: string, model: string): void /** Set the model keep-alive expiry duration on a node. */ setModelExpiry(engineType: EngineType, nodeId: string, model: string, expiry: string): void + /** + * Choose the model an engine serves. For an engine that runs one model per + * process (vLLM, SGLang), the choice is persisted and the engine restarts + * onto it; an empty model clears the choice. Local node only. + */ + setServedModel(engineType: EngineType, nodeId: string, model: string): void /** Search the model registry/hub for available models. */ searchHub(engineType: EngineType): Promise @@ -86,6 +92,8 @@ export function createEngineApi(transport: ServiceTransport): IEngineApi { enginePort: ports.enginePort, proxyPort: ports.proxyPort }), + setServedModel: (engineType, nodeId, model) => + fireCommand(transport, { command: 'setServedModel', engineType, nodeId, model }), pullModel: (engineType, nodeId, model) => fireCommand(transport, { command: 'pullModel', engineType, nodeId, model }), loadModel: (engineType, nodeId, model) => diff --git a/desktop/src/ui/api/pair-api.ts b/desktop/src/ui/api/pair-api.ts index a0a34155..980e7c30 100644 --- a/desktop/src/ui/api/pair-api.ts +++ b/desktop/src/ui/api/pair-api.ts @@ -9,6 +9,7 @@ import type { ClusterNode, Invite } from '@/shared/types/cluster' +import type { ManualServicePorts } from '@/shared/types/manual-node' import type { NodeItem } from '@/shared/types/nodes' import type { ServiceError } from '@/shared/types/errors' import type { NodeItemMetrics } from '@/shared/types/metrics' @@ -51,7 +52,7 @@ export interface IClusterApi { /** Fetch cluster bootstrap state: identity, settings, and membership. */ getInitial(): Promise /** Start PIN pairing with a remote node; the returned invite carries the PIN to display. */ - inviteNode(ipAddress: string): Promise + inviteNode(ipAddress: string, ports?: ManualServicePorts): Promise /** Poll the state of an outbound pairing session. */ inviteStatus(inviteId: string): Promise /** Respond to an inbound invite: accept with the PIN from the inviter, or decline. */ @@ -152,7 +153,8 @@ export function createPairApi(transport: ServiceTransport): IPairApi { }, cluster: { getInitial: () => transport.invoke('cluster:get-initial'), - inviteNode: ipAddress => transport.invoke('cluster:invite-node', { ipAddress }), + inviteNode: (ipAddress, ports) => + transport.invoke('cluster:invite-node', { ipAddress, ports }), inviteStatus: inviteId => transport.invoke('cluster:invite-status', { inviteId }), respondToInvite: (inviteId, accept, pin) => transport.invoke('cluster:respond-to-invite', { inviteId, accept, pin }), diff --git a/desktop/src/ui/components/AddNodeModal.tsx b/desktop/src/ui/components/AddNodeModal.tsx index d54abde9..dc0f6ba5 100644 --- a/desktop/src/ui/components/AddNodeModal.tsx +++ b/desktop/src/ui/components/AddNodeModal.tsx @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { useCallback, useState } from 'react' +import { useCallback, useMemo, useState } from 'react' import { Button, Divider, @@ -19,6 +19,42 @@ import { InvitePairingPanel } from './InvitePairingPanel' import { useBlurOnOpen } from '@/ui/hooks/useBlurOnOpen' import { useInvitePairing } from '@/ui/hooks/useInvitePairing' import { useInvitablePeers } from '@/ui/hooks/useInvitablePeers' +import type { ManualServicePorts } from '@/shared/types/manual-node' + +/** + * The advanced port fields, in the order they are shown. A node on its default + * ports needs none of them; a node behind a forwarded range, or one running an + * engine somewhere else, needs exactly the one it moved. + */ +const PORT_FIELDS = [ + { key: 'nodeInfo', label: 'Node info', placeholder: '14318' }, + { key: 'cluster', label: 'Pairing', placeholder: '14321' }, + { key: 'ollama', label: 'Ollama', placeholder: '11434' }, + { key: 'lmstudio', label: 'LM Studio', placeholder: '1234' } +] as const + +type PortField = (typeof PORT_FIELDS)[number]['key'] + +type PortDrafts = Partial> + +/** + * Reads the typed ports, dropping anything that is not a usable TCP port so a + * half-typed field never travels as an override. Returns undefined when nothing + * was set: an empty overrides object is not the same as none. + */ +function draftPorts(drafts: PortDrafts): ManualServicePorts | undefined { + const ports: ManualServicePorts = {} + let any = false + for (const field of PORT_FIELDS) { + const raw = drafts[field.key]?.trim() + if (!raw) continue + const port = Number(raw) + if (!Number.isInteger(port) || port < 1 || port > 65535) continue + ports[field.key] = port + any = true + } + return any ? ports : undefined +} interface AddNodeModalProps { open: boolean @@ -28,12 +64,18 @@ interface AddNodeModalProps { export function AddNodeModal({ open, onOpenChange }: AddNodeModalProps) { useBlurOnOpen(open) const [manualIp, setManualIp] = useState('') + const [showPorts, setShowPorts] = useState(false) + const [portDrafts, setPortDrafts] = useState({}) const pairing = useInvitePairing() const nodesThatCanBeAdded = useInvitablePeers() + const ports = useMemo(() => draftPorts(portDrafts), [portDrafts]) + const handleOpenChange = useCallback( (next: boolean) => { setManualIp('') + setShowPorts(false) + setPortDrafts({}) pairing.reset() onOpenChange(next) }, @@ -41,10 +83,14 @@ export function AddNodeModal({ open, onOpenChange }: AddNodeModalProps) { ) const handleManualInvite = useCallback(() => { - const ip = manualIp.trim() - if (!ip) return - void pairing.start(ip) - }, [manualIp, pairing]) + const address = manualIp.trim() + if (!address) return + void pairing.start(address, ports) + }, [manualIp, ports, pairing]) + + const setPortDraft = useCallback((field: PortField, value: string) => { + setPortDrafts(previous => ({ ...previous, [field]: value })) + }, []) const showPairing = pairing.invite !== null || pairing.error !== null const inviteInFlight = pairing.submitting || pairing.invite?.state === 'pending' @@ -73,7 +119,7 @@ export function AddNodeModal({ open, onOpenChange }: AddNodeModalProps) { ) : ( <> - + + + On a VPN or overlay network such as Tailscale, use the host name + — for example gpu-box.tail1234.ts.net. A name is re-resolved on + every check, so the node keeps working after it changes address. + + + + + {showPorts && ( + + + Leave a field empty to use this service's + default port. Enter the address on its own above — + ports belong here, not after a colon. + + + {PORT_FIELDS.map(field => ( + + + setPortDraft(field.key, value) + } + placeholder={field.placeholder} + inputMode="numeric" + disabled={inviteInFlight} + /> + + ))} + + + )} + {nodesThatCanBeAdded.length > 0 && ( diff --git a/desktop/src/ui/components/BackendRow/BackendRow.tsx b/desktop/src/ui/components/BackendRow/BackendRow.tsx index a1b399df..58357702 100644 --- a/desktop/src/ui/components/BackendRow/BackendRow.tsx +++ b/desktop/src/ui/components/BackendRow/BackendRow.tsx @@ -71,7 +71,8 @@ export function BackendRow({ }) { const [edit, setEdit] = useState({ serverPort: String(backend.port ?? ''), - proxyPort: String(backend.proxyPort ?? '') + proxyPort: String(backend.proxyPort ?? ''), + servedModel: backend.servedModel ?? '' }) const addLocalError = useErrorsStore(state => state.addLocalError) @@ -84,6 +85,10 @@ export function BackendRow({ setEdit(prev => ({ ...prev, proxyPort: String(backend.proxyPort ?? '') })) }, [backend.proxyPort]) + useEffect(() => { + setEdit(prev => ({ ...prev, servedModel: backend.servedModel ?? '' })) + }, [backend.servedModel]) + const [expanded, setExpanded] = useState(false) const [confirmUninstall, setConfirmUninstall] = useState(false) const [confirmPorts, setConfirmPorts] = useState(false) @@ -102,6 +107,11 @@ export function BackendRow({ const portsChanged = serverPortChanged || proxyPortChanged + const servedModelChanged = useMemo( + () => caps.hasServedModel && edit.servedModel.trim() !== (backend.servedModel ?? ''), + [caps.hasServedModel, edit.servedModel, backend.servedModel] + ) + const installProgress = useEngineProgressStore(s => { const installKey = engineProgressKey({ nodeId, @@ -198,9 +208,18 @@ export function BackendRow({ const resetPortsToBackend = useCallback(() => { setEdit({ serverPort: String(backend.port ?? ''), - proxyPort: String(backend.proxyPort ?? '') + proxyPort: String(backend.proxyPort ?? ''), + servedModel: backend.servedModel ?? '' }) - }, [backend.port, backend.proxyPort]) + }, [backend.port, backend.proxyPort, backend.servedModel]) + + // Changing the served model restarts a running engine onto it, exactly as a + // port change does; the engine reports the applied value back, which resets + // the draft through the effect above. + const handleApplyServedModel = useCallback(() => { + if (!servedModelChanged) return + window.pairApi.engines.setServedModel(backend.type, nodeId, edit.servedModel.trim()) + }, [servedModelChanged, backend.type, nodeId, edit.servedModel]) const validateAndConfirmPorts = useCallback(() => { if (!serverPortChanged && !proxyPortChanged) return @@ -288,14 +307,17 @@ export function BackendRow({ )} - {canShowAccordions && (caps.hasEnginePort || edit.proxyPort) && ( + {canShowAccordions && (caps.hasEnginePort || caps.hasServedModel || edit.proxyPort) && ( setEdit(prev => ({ ...prev, servedModel: v }))} onServerChange={v => setEdit(prev => ({ ...prev, serverPort: v }))} onProxyChange={v => setEdit(prev => ({ ...prev, proxyPort: v }))} /> diff --git a/desktop/src/ui/components/BackendRow/PortsSection.tsx b/desktop/src/ui/components/BackendRow/PortsSection.tsx index 353ac45d..94e962c5 100644 --- a/desktop/src/ui/components/BackendRow/PortsSection.tsx +++ b/desktop/src/ui/components/BackendRow/PortsSection.tsx @@ -3,32 +3,43 @@ import { Text } from '@nvidia/foundations-react-core' import { BackendPorts } from './BackendPorts' +import { ServedModelRow } from './ServedModelRow' import { EditState } from '@/ui/types/engine-edit-state' import type { EngineCaps } from '@/ui/types/engine-manifest' export function PortsSection({ edit, portsChanged, + servedModelChanged, anyLoading, isLocalNode, caps, onApplyPorts, + onApplyServedModel, onServerChange, - onProxyChange + onProxyChange, + onServedModelChange }: { edit: EditState portsChanged: boolean + servedModelChanged: boolean anyLoading: boolean isLocalNode: boolean caps: EngineCaps onApplyPorts: () => void + onApplyServedModel: () => void onServerChange: (v: string) => void onProxyChange: (v: string) => void + onServedModelChange: (v: string) => void }) { + // The served model is a local-node control: the engine manager only writes + // manifest overrides for its own host, and there is no remote equivalent, so + // a remote row would be a field that can never hold a value. + const showServedModel = caps.hasServedModel && isLocalNode return (
- Ports + {showServedModel ? 'Ports and model' : 'Ports'}
+ {/* An engine that serves one model per process is told which model + before it starts, so the choice belongs with the other + start-time settings rather than in the model list. */} + {showServedModel && ( + + )}
) diff --git a/desktop/src/ui/components/BackendRow/ServedModelRow.tsx b/desktop/src/ui/components/BackendRow/ServedModelRow.tsx new file mode 100644 index 00000000..2b7872bd --- /dev/null +++ b/desktop/src/ui/components/BackendRow/ServedModelRow.tsx @@ -0,0 +1,71 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { Button, Flex, Stack, Text, TextInput } from '@nvidia/foundations-react-core' +import { Check } from '@/ui/components/icons' + +/** + * The model an engine serves, for an engine that runs one model per process + * (vLLM, SGLang). It is a start-time setting, not a model operation: the engine + * manager persists it and restarts the engine onto it, and the engine downloads + * the weights itself on that first start — a Hugging Face id is fetched, a local + * model directory is read in place. The local node only — the engine manager + * writes manifest overrides for its own host and the backend exposes no remote + * served-model control, so `PortsSection` does not render this for a peer. + */ +export function ServedModelRow({ + model, + changed, + disabled, + onChange, + onApply +}: { + model: string + /** True when the draft differs from the engine-reported value. */ + changed: boolean + disabled: boolean + onChange: (value: string) => void + onApply: () => void +}) { + return ( + + + Model to serve + + A Hugging Face model id, for example Qwen/Qwen3-8B, or the path of a local model + directory. The engine serves this one model and, for a Hugging Face id, + downloads it on the next start, which can take several minutes. + + + + + + + + ) +} diff --git a/desktop/src/ui/components/EngineIcon.tsx b/desktop/src/ui/components/EngineIcon.tsx index 4ee81367..0ef4cfc8 100644 --- a/desktop/src/ui/components/EngineIcon.tsx +++ b/desktop/src/ui/components/EngineIcon.tsx @@ -39,5 +39,69 @@ export default function EngineIcon({ type, size = 32 }: { type: EngineType; size ) } + // vLLM's own logo is not redistributable here, so its tile is drawn rather + // than shipped: a wordmark on the project's colours, in the same rounded + // square the other engines use so the row stays visually even. + if (type === 'vllm') { + return ( +
+ + + vL + + +
+ ) + } + + // SGLang's own logo is not redistributable here either, so its tile is drawn + // the same way vLLM's is. The colour is deliberately across the wheel from + // vLLM's blue: these two engines sit next to each other in the backend list + // and the tile is the only thing telling them apart at a glance, and + // orange-vs-blue survives the common colour-vision deficiencies that would + // collapse a violet-vs-blue pair. #c2410c carries white text at 5.2:1, past + // the 4.5:1 contrast floor. + if (type === 'sglang') { + return ( +
+ + + SG + + +
+ ) + } + return null } diff --git a/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx b/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx index 7581be68..e7e8f12d 100644 --- a/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx +++ b/desktop/src/ui/components/NodeList/NodeEngineSettings.tsx @@ -98,6 +98,7 @@ export default function NodeEngineSettings({ nodeId }: { nodeId: string }) { docsUrl: EngineDefaultLinks[type]?.docsUrl, installUrl: EngineDefaultLinks[type]?.installUrl, installedVersion: status.installedVersion, + servedModel: status.servedModel, updateAvailable }, statusKnown diff --git a/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx b/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx index 3460c4c2..69eea719 100644 --- a/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx +++ b/desktop/src/ui/components/ServiceSettings/WipeAppDataCard.tsx @@ -16,8 +16,11 @@ function wipeMessage() { engine installations under the app data folder. - Third-party model libraries (for example ~/.ollama and{' '} - ~/.lmstudio) are not deleted. + Third-party model libraries and engine installations outside the app data folder + (for example ~/.ollama, ~/.lmstudio, a vLLM environment at{' '} + ~/.vllm, an SGLang environment at ~/.sglang, and the + Hugging Face cache at ~/.cache/huggingface) are not{' '} + deleted. ) diff --git a/desktop/src/ui/constants/engine-capabilities.ts b/desktop/src/ui/constants/engine-capabilities.ts index 77e6f2bf..d971a547 100644 --- a/desktop/src/ui/constants/engine-capabilities.ts +++ b/desktop/src/ui/constants/engine-capabilities.ts @@ -20,6 +20,7 @@ export const EngineCapabilities: Record = { hasModelSearchOnlyWhenRunning: true, modelOpsWhenStopped: false, hasDeleteModel: true, + hasServedModel: false, engineHub: { label: 'Ollama', url: 'https://ollama.com/library' } }, 'lm-studio': { @@ -42,6 +43,50 @@ export const EngineCapabilities: Record = { // exposes no rescan, so nvpair-engine-manager's delete_model restarts the // server. Deleting therefore interrupts inference and needs a warning. restartsOnModelDelete: true, + hasServedModel: false, engineHub: { label: 'LM Studio', url: 'https://lmstudio.ai/models' } + }, + vllm: { + hasExpiry: false, + // vLLM keeps its one served model resident for the life of the process. + // There is nothing to eject short of stopping the engine. + hasEject: false, + // vLLM publishes no Windows build and no macOS GPU build, so PAIR only + // offers to install it on Linux. Its manifest ships Linux platforms only, + // so the engine reports unavailable everywhere else. + hasInstall: ['linux'], + hasEnginePort: true, + hasInstallPath: false, + hasProxyWebUI: false, + hasPreferredNode: false, + hasCrashAlert: false, + hasModelSearchOnlyWhenRunning: true, + // vLLM has no model-management surface: weights are fetched by the engine + // itself when it starts, from the model id configured below. + modelOpsWhenStopped: false, + hasDeleteModel: false, + hasServedModel: true + }, + sglang: { + hasExpiry: false, + // SGLang, like vLLM, keeps its one served model resident for the life of + // the process. There is nothing to eject short of stopping the engine. + hasEject: false, + // SGLang targets Linux with CUDA/ROCm and publishes no Windows or macOS + // GPU build, so PAIR only offers to install it on Linux. Its manifest + // ships Linux platforms only, so the engine reports unavailable + // everywhere else. + hasInstall: ['linux'], + hasEnginePort: true, + hasInstallPath: false, + hasProxyWebUI: false, + hasPreferredNode: false, + hasCrashAlert: false, + hasModelSearchOnlyWhenRunning: true, + // SGLang has no model-management surface: weights are fetched by the + // engine itself when it starts, from the model path configured below. + modelOpsWhenStopped: false, + hasDeleteModel: false, + hasServedModel: true } } diff --git a/desktop/src/ui/constants/welcome.ts b/desktop/src/ui/constants/welcome.ts index 4f9fa04a..5f241490 100644 --- a/desktop/src/ui/constants/welcome.ts +++ b/desktop/src/ui/constants/welcome.ts @@ -13,7 +13,13 @@ export const WELCOME_STEP_SUB_HEADINGS = ['', 'You can update later by clicking export const WELCOME_ENGINE_DEFAULT_SELECTED: Record = { ollama: true, - 'lm-studio': true + 'lm-studio': true, + // vLLM downloads a multi-gigabyte CUDA stack and then a model, and it runs + // on Linux only. Offer it, but never pre-select it during onboarding. + vllm: false, + // SGLang installs the same kind of multi-gigabyte GPU stack and then a + // model, and it too runs on Linux only. Same treatment as vLLM. + sglang: false } export function getWelcomeEngineCandidates(os: PlatformDisplayName): EngineType[] { diff --git a/desktop/src/ui/hooks/useInvitePairing.ts b/desktop/src/ui/hooks/useInvitePairing.ts index d7b0e6bb..cb562261 100644 --- a/desktop/src/ui/hooks/useInvitePairing.ts +++ b/desktop/src/ui/hooks/useInvitePairing.ts @@ -3,6 +3,7 @@ import { useCallback, useEffect, useRef, useState } from 'react' import type { Invite } from '@/shared/types/cluster' +import type { ManualServicePorts } from '@/shared/types/manual-node' import { MODULAR_INVITE_STATUS_POLL_INTERVAL_MS } from '@/shared/constants/modular-runtime' import getErrorString from '@/shared/utils/get-error-string' import { formatClusterInviteError } from '@/ui/utils/cluster-invite-error' @@ -13,8 +14,12 @@ interface InvitePairing { /** True while the initial `cluster:invite-node` request is in flight. */ submitting: boolean error: string | null - /** Begin PIN pairing with a node, then poll its status until it resolves. */ - start: (ipAddress: string) => Promise + /** + * Add a node by address and begin PIN pairing with it, then poll its status + * until it resolves. `ports` overrides the assumed port of any single service + * on that host. + */ + start: (address: string, ports?: ManualServicePorts) => Promise /** * Cancel a still-pending outbound invite: tell the backend to tear down the * pairing session (invalidating the PIN so a remote user can no longer @@ -73,12 +78,12 @@ export function useInvitePairing(): InvitePairing { }, [reset, stopPolling]) const start = useCallback( - async (ipAddress: string) => { + async (address: string, ports?: ManualServicePorts) => { setSubmitting(true) setError(null) stopPolling() try { - const result = await window.pairApi.cluster.inviteNode(ipAddress) + const result = await window.pairApi.cluster.inviteNode(address, ports) setInvite(result) if (result.state === 'pending' && result.inviteId) { inviteIdRef.current = result.inviteId diff --git a/desktop/src/ui/types/engine-edit-state.ts b/desktop/src/ui/types/engine-edit-state.ts index 67229c40..d96fd892 100644 --- a/desktop/src/ui/types/engine-edit-state.ts +++ b/desktop/src/ui/types/engine-edit-state.ts @@ -4,4 +4,9 @@ export type EditState = { serverPort: string proxyPort: string + /** + * Draft of the model an engine should serve, for an engine that runs one + * model per process. Empty clears the choice. + */ + servedModel: string } diff --git a/desktop/src/ui/types/engine-info.ts b/desktop/src/ui/types/engine-info.ts index 3c4cb780..8fce3b41 100644 --- a/desktop/src/ui/types/engine-info.ts +++ b/desktop/src/ui/types/engine-info.ts @@ -95,6 +95,11 @@ export interface BackendInfo { proxyPort: number | null /** Installed engine binary version reported by the owning node */ installedVersion?: string + /** + * The model this engine is configured to serve, for an engine that runs one + * model per process (vLLM, SGLang). Empty string when none is chosen. + */ + servedModel?: string /** Models on this backend with per-model status */ models: ModelItem[] /** System-level dependencies required before install/run (local node only) */ diff --git a/desktop/src/ui/types/engine-manifest.ts b/desktop/src/ui/types/engine-manifest.ts index 1cd998bf..5a9fcc56 100644 --- a/desktop/src/ui/types/engine-manifest.ts +++ b/desktop/src/ui/types/engine-manifest.ts @@ -36,6 +36,12 @@ export interface EngineCaps { modelOpsWhenStopped: boolean /** When true, show the Delete action in the model action menu. */ hasDeleteModel: boolean + /** + * When true, the engine serves exactly one model per process and the model is + * chosen before it starts, so Engine settings offers a "Model to serve" field. + * The engine manager persists the choice and restarts the engine onto it. + */ + hasServedModel: boolean /** * When true, the engine restarts as part of deleting a model, so Delete asks * for confirmation first. The engine manager owns the restart; this flag only diff --git a/desktop/src/ui/utils/format-model-display-name.ts b/desktop/src/ui/utils/format-model-display-name.ts index e16282ed..f83236da 100644 --- a/desktop/src/ui/utils/format-model-display-name.ts +++ b/desktop/src/ui/utils/format-model-display-name.ts @@ -23,7 +23,8 @@ export function formatModelDisplayName(name: string, engineType?: string | null) return name } - if (name.length > MAX_FORMAT_INPUT_CHARS) { + const capped = name.length > MAX_FORMAT_INPUT_CHARS + if (capped) { name = name.slice(0, MAX_FORMAT_INPUT_CHARS) } @@ -45,7 +46,21 @@ export function formatModelDisplayName(name: string, engineType?: string | null) case 'ollama': return formatOllamaModelName(formatted) + // vLLM and SGLang serve whatever their `--model` / `--model-path` was: + // either a Hugging Face repo id verbatim (Qwen/Qwen3-8B), which the + // shared Hugging Face formatter already renders, or the path of a local + // model directory, which it would mangle into "models/my model". + // Neither needs engine-specific rules beyond telling those two apart. default: + if (isPathLikeModelId(formatted)) { + // Naming the last segment is only truthful on a name the cap + // left intact. The cap slices from the right, which is exactly + // where a path keeps its last segment, so on a capped path the + // rule would hand back a truncated *middle* segment as + // confidently as a real directory name. Hand back the truncated + // path instead — it reads as unfinished, which it is. + return capped ? formatted : formatLocalModelPath(formatted) + } if (formatted.includes('/')) { return formatHuggingFaceModelName(formatted) } @@ -53,6 +68,27 @@ export function formatModelDisplayName(name: string, engineType?: string | null) } } +/** + * A local model directory rather than a Hugging Face repo id. The leading slash + * is the whole test: a repo id is always `org/repo` with no leading separator, + * so nothing that would otherwise reach the Hugging Face formatter is captured + * here. + */ +function isPathLikeModelId(name: string): boolean { + return name.startsWith('/') +} + +/** + * The last segment of a local model path — `/models/my-model` renders as + * `my-model`. Kept verbatim rather than humanised the way a repo id is: a + * directory name is what the operator typed into "Model to serve", and it is + * only recognisable if we hand it back unchanged. + */ +function formatLocalModelPath(modelPath: string): string { + const segments = modelPath.split('/').filter(segment => segment.length > 0) + return segments.length > 0 ? segments[segments.length - 1] : modelPath +} + /** * Format an LM Studio / llama.cpp GGUF model path into a readable name. * "lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf" diff --git a/desktop/src/ui/utils/match-downloaded-model.ts b/desktop/src/ui/utils/match-downloaded-model.ts index 2edf8361..a1ad298e 100644 --- a/desktop/src/ui/utils/match-downloaded-model.ts +++ b/desktop/src/ui/utils/match-downloaded-model.ts @@ -39,6 +39,9 @@ const matchHfPullKeyOrName: DownloadedMatcher = (hubEntry, d) => { const MATCHERS: Partial> = { ollama: matchOllama, 'lm-studio': matchHfPullKeyOrName + // vLLM and SGLang declare no engineHub, so no hub row is ever rendered for + // them and they need no matcher. Adding one here without a hub would be dead + // code. } export function isHubEntryDownloaded( diff --git a/desktop/tests/modular/cluster-pairing-timeout.test.ts b/desktop/tests/modular/cluster-pairing-timeout.test.ts index 4811d208..eac88fc0 100644 --- a/desktop/tests/modular/cluster-pairing-timeout.test.ts +++ b/desktop/tests/modular/cluster-pairing-timeout.test.ts @@ -16,6 +16,11 @@ const mocks = vi.hoisted(() => ({ }, supervisor: { callProcess: vi.fn(), + // An outbound invite now adds the address as a manual node first, so it + // stays visible on a network that discovers nothing. These tests are + // about the pairing RPC timeout, so the broker reads as absent and that + // step is skipped. + hasProcess: vi.fn(() => false), markAutoCreatedSoloForInvite: vi.fn() } })) diff --git a/desktop/tests/modular/delete-model-restart.test.ts b/desktop/tests/modular/delete-model-restart.test.ts index 3563c5ee..d6f82931 100644 --- a/desktop/tests/modular/delete-model-restart.test.ts +++ b/desktop/tests/modular/delete-model-restart.test.ts @@ -27,7 +27,9 @@ const MANIFEST_DIR = path.resolve(process.cwd(), '../services/nvpair-engine-mana /** Manifest engine ids differ from our `EngineType` for LM Studio only. */ const ENGINE_TYPE_BY_MANIFEST_ID: Record = { ollama: 'ollama', - lmstudio: 'lm-studio' + lmstudio: 'lm-studio', + vllm: 'vllm', + sglang: 'sglang' } interface ManifestAction { diff --git a/desktop/tests/modular/lmstudio-stale-model.test.ts b/desktop/tests/modular/lmstudio-stale-model.test.ts index 8b37e862..b722abab 100644 --- a/desktop/tests/modular/lmstudio-stale-model.test.ts +++ b/desktop/tests/modular/lmstudio-stale-model.test.ts @@ -91,7 +91,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -202,7 +202,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -264,7 +264,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -323,7 +323,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', @@ -414,7 +414,7 @@ describe('LM Studio model reconciliation', () => { state.handleNotification({ source: 'lmstudio-proxy', method: 'node/discovered', - params: { id: nodeId, port: 1234 } + params: { id: nodeId, port: 1234, modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ source: 'broker', diff --git a/desktop/tests/modular/manual-node-ports.test.ts b/desktop/tests/modular/manual-node-ports.test.ts new file mode 100644 index 00000000..9f338b52 --- /dev/null +++ b/desktop/tests/modular/manual-node-ports.test.ts @@ -0,0 +1,131 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'fs' +import os from 'os' +import path from 'path' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +let userDataDir = '' + +vi.mock('electron', () => ({ BrowserWindow: { getAllWindows: () => [] } })) +vi.mock('@/electron/globals', () => ({ + getPaths: () => ({ getUserData: () => userDataDir }) +})) + +import { + addManualNodeEntry, + listManualNodeEntries, + manualPortsToWire, + removeManualNodeEntry, + resolveManualNodeKey +} from '@/electron/service-bridge/manual-nodes-store' + +// The durable manual-node list is what makes a peer on a network without +// multicast visible at all: nothing is ever discovered there, so the entry the +// application persists and replays is the node's only route into the directory. +// Its port overrides have to survive that round trip, or a node reachable on +// non-default ports comes back unreachable after a restart. +describe('manual node entries', () => { + beforeEach(() => { + userDataDir = fs.mkdtempSync(path.join(os.tmpdir(), 'pair-manual-nodes-')) + }) + + afterEach(() => { + fs.rmSync(userDataDir, { recursive: true, force: true }) + }) + + it('persists an address with no overrides and keys it by the address', () => { + const entry = addManualNodeEntry('gpu-box.tail1234.ts.net') + expect(entry).toEqual({ + id: 'gpu-box.tail1234.ts.net', + address: 'gpu-box.tail1234.ts.net', + name: 'gpu-box.tail1234.ts.net' + }) + expect(listManualNodeEntries()).toEqual([entry]) + }) + + it('round-trips port overrides through the persisted list', () => { + addManualNodeEntry('gpu-box.tail1234.ts.net', { nodeInfo: 24318, ollama: 21434 }) + const [entry] = listManualNodeEntries() + expect(entry.ports).toEqual({ nodeInfo: 24318, ollama: 21434 }) + }) + + it("round-trips the OpenAI-compatible engines' own ports", () => { + // vLLM (stock 8000) and SGLang (stock 30000) each run their own server + // behind the shared OpenAI-compatible proxy, so a node that moved either + // off its stock port has to say so in the entry that replays it. + addManualNodeEntry('gpu-box.tail1234.ts.net', { vllm: 8001, sglang: 30001 }) + const [entry] = listManualNodeEntries() + expect(entry.ports).toEqual({ vllm: 8001, sglang: 30001 }) + }) + + it('drops ports that are not usable, and the object when none are', () => { + addManualNodeEntry('a.example', { nodeInfo: 0, ollama: 70000, cluster: 14321 }) + expect(listManualNodeEntries()[0].ports).toEqual({ cluster: 14321 }) + + addManualNodeEntry('b.example', { nodeInfo: -1 }) + const b = listManualNodeEntries().find(entry => entry.address === 'b.example') + expect(b?.ports).toBeUndefined() + }) + + it('replaces rather than duplicates when the same address is added again', () => { + addManualNodeEntry('gpu-box.tail1234.ts.net', { ollama: 21434 }) + addManualNodeEntry('gpu-box.tail1234.ts.net', { ollama: 21435 }) + const entries = listManualNodeEntries() + expect(entries).toHaveLength(1) + expect(entries[0].ports).toEqual({ ollama: 21435 }) + }) + + it('resolves a host-name entry from the node addresses the backend reports', () => { + addManualNodeEntry('gpu-box.tail1234.ts.net') + // A node added by name is reported by that name, since a name is a + // dialable address like any other. + expect(resolveManualNodeKey(['gpu-box.tail1234.ts.net'])).toBe('gpu-box.tail1234.ts.net') + expect(resolveManualNodeKey(['192.0.2.10'])).toBeNull() + }) + + it('forgets an entry on removal', () => { + addManualNodeEntry('gpu-box.tail1234.ts.net') + removeManualNodeEntry('gpu-box.tail1234.ts.net') + expect(listManualNodeEntries()).toEqual([]) + }) + + it('ignores a persisted ports value that is not an object', () => { + const file = path.join(userDataDir, 'configs', 'manual-nodes.json') + fs.mkdirSync(path.dirname(file), { recursive: true }) + fs.writeFileSync(file, JSON.stringify([{ address: 'a.example', ports: 'nonsense' }])) + expect(listManualNodeEntries()[0].ports).toBeUndefined() + }) +}) + +// The application spells the overrides in camelCase and the service reads them +// in snake_case. The two spellings meet in one projection, so a rename cannot +// silently drop a field on the way to the backend. +describe('manualPortsToWire', () => { + it('projects every field onto the names node/add reads', () => { + expect( + manualPortsToWire({ + nodeInfo: 24318, + cluster: 24321, + ollama: 21434, + lmstudio: 2234, + vllm: 8001, + sglang: 30001 + }) + ).toEqual({ + node_info: 24318, + cluster: 24321, + ollama: 21434, + lmstudio: 2234, + vllm: 8001, + sglang: 30001 + }) + }) + + it('omits unset fields, and the object itself when nothing is set', () => { + expect(manualPortsToWire({ ollama: 21434 })).toEqual({ ollama: 21434 }) + expect(manualPortsToWire({})).toBeUndefined() + expect(manualPortsToWire(undefined)).toBeUndefined() + }) +}) diff --git a/desktop/tests/modular/manual-pair-peer-state.test.ts b/desktop/tests/modular/manual-pair-peer-state.test.ts new file mode 100644 index 00000000..588a7e44 --- /dev/null +++ b/desktop/tests/modular/manual-pair-peer-state.test.ts @@ -0,0 +1,130 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => ({ BrowserWindow: { getAllWindows: () => [] } })) +vi.mock('@/electron/window', () => ({ createOverviewWindow: vi.fn() })) + +import { getModularBridgeState } from '@/electron/service-bridge/modular-state' + +// A peer added by address on a network that discovers nothing arrives here as an +// ordinary directory node: the broker synthesizes the record the scanner would +// have produced. These tests pin that the renderer's view of it is the view of a +// discovered peer, with no special case anywhere — same card, same engines, same +// models, same telemetry — and that its address surviving as a host name rather +// than a literal changes none of it. +describe('a manually added PAIR peer in the bridge state', () => { + it('renders with the card, engines, models and telemetry a discovered peer gets', () => { + const state = getModularBridgeState() + state.handleNotification({ + source: 'broker', + method: 'discovery:nodes-changed', + params: { + nodes: [ + { + hostUuid: 'uuid-tailnet-peer', + name: 'gpu-box.tail1234.ts.net', + // A MagicDNS name, not a literal. Nothing downstream may + // require an IP: on a tailnet the name is what survives a + // node changing address. + ipAddress: 'gpu-box.tail1234.ts.net', + port: 14318, + trusted: true, + clustered: true, + models: ['llama3.2:latest', 'qwen2.5-7b-instruct'], + modelsByEngine: { + ollama: ['llama3.2:latest'], + lmstudio: ['qwen2.5-7b-instruct'] + }, + loadedByEngine: { ollama: ['llama3.2:latest'] } + } + ] + } + }) + + // The card: keyed by the peer's stable identity, named and addressed by + // what the operator typed. + const { nodes } = state.getNodesInitial() + const card = nodes['uuid-tailnet-peer'] + expect(card).toBeDefined() + expect(card.name).toBe('gpu-box.tail1234.ts.net') + expect(card.ipAddress).toBe('gpu-box.tail1234.ts.net') + + // Trust and membership, which is what distinguishes a paired peer from a + // stranger anywhere it is shown. + const available = state.getAvailableNodes().find(node => node.id === 'uuid-tailnet-peer') + expect(available).toMatchObject({ trusted: true, clustered: true }) + expect(available?.ipAddress).toBe('gpu-box.tail1234.ts.net') + + // Models, attributed to the engine that serves each one. + const engineState = state.getEngineInitialState() + const ollama = engineState.models.find( + entry => entry.nodeId === 'uuid-tailnet-peer' && entry.engineType === 'ollama' + ) + const lmStudio = engineState.models.find( + entry => entry.nodeId === 'uuid-tailnet-peer' && entry.engineType === 'lm-studio' + ) + expect(ollama?.models.map(model => model.name)).toEqual(['llama3.2:latest']) + expect(lmStudio?.models.map(model => model.name)).toEqual(['qwen2.5-7b-instruct']) + // Loaded state travels too, so a remote card can show what is resident. + expect(ollama?.models[0]?.status).toBe('loaded') + + // Telemetry: the peer is polled for its hardware at the address it was + // added by, on its node-info port, exactly as a discovered peer is. + const target = state + .getNodeInfoPollTargets() + .find(entry => entry.id === 'uuid-tailnet-peer') + expect(target).toBeDefined() + expect(target?.hosts).toContain('gpu-box.tail1234.ts.net') + expect(target?.port).toBe(14318) + expect(card.status).toBe('active') + + // And what comes back lands on the card. + state.mergeNodeInfoResponse('uuid-tailnet-peer', { + hostUuid: 'uuid-tailnet-peer', + GPUs: [ + { + name: 'NVIDIA GeForce RTX 4090', + vram_bytes: 25_769_803_776, + vram_used_bytes: 8_589_934_592, + utilization_percent: 37 + } + ], + cpu: { name: 'AMD Ryzen 9 5900X', cores: 12, utilization_percent: 9 }, + memory: { total_bytes: 68_719_476_736, used_bytes: 17_179_869_184 } + }) + const withHardware = state.getNodesInitial().nodes['uuid-tailnet-peer'] + expect(withHardware.topology.cpu.model).toBe('AMD Ryzen 9 5900X') + expect(withHardware.topology.cpu.cores).toBe(12) + expect(withHardware.topology.gpus.map(gpu => gpu.name)).toEqual(['NVIDIA GeForce RTX 4090']) + expect(withHardware.topology.gpus[0].vramTotal).toBe(25_769_803_776) + expect(withHardware.topology.ram).toBe(68_719_476_736) + }) + + it('keeps a host-name address rather than blanking it', () => { + const state = getModularBridgeState() + state.handleNotification({ + source: 'broker', + method: 'discovery:nodes-changed', + params: { + nodes: [ + { + hostUuid: 'uuid-name-only', + name: 'name-only.tail1234.ts.net', + ipAddress: 'name-only.tail1234.ts.net', + port: 14318, + trusted: true, + clustered: true + } + ] + } + }) + + const card = state.getNodesInitial().nodes['uuid-name-only'] + expect(card.ipAddress).toBe('name-only.tail1234.ts.net') + // Nothing collapses the address to an empty string on the way through: an + // empty address is what a node with nowhere to be dialed looks like. + expect(card.ipAddress).not.toBe('') + }) +}) diff --git a/desktop/tests/modular/node-address-rerank.test.ts b/desktop/tests/modular/node-address-rerank.test.ts index 454df5e1..1a042ca5 100644 --- a/desktop/tests/modular/node-address-rerank.test.ts +++ b/desktop/tests/modular/node-address-rerank.test.ts @@ -101,7 +101,8 @@ describe('broker address re-rank', () => { host: 'proxy-refresh-host', port: 1234, addresses: ['203.0.113.7'], - ip: '192.0.2.201' + ip: '192.0.2.201', + modelsByEngine: { lmstudio: [] } } }) diff --git a/desktop/tests/modular/openai-proxy-engines.test.ts b/desktop/tests/modular/openai-proxy-engines.test.ts new file mode 100644 index 00000000..1abf7c2c --- /dev/null +++ b/desktop/tests/modular/openai-proxy-engines.test.ts @@ -0,0 +1,158 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('electron', () => ({ BrowserWindow: { getAllWindows: () => [] } })) +vi.mock('@/electron/window', () => ({ createOverviewWindow: vi.fn() })) + +import { getModularBridgeState } from '@/electron/service-bridge/modular-state' + +/** + * The OpenAI-compatible proxy fronts more than one engine, so a node it reports + * is no longer necessarily LM Studio. The node's own per-engine model + * attribution says which engine it runs; reading the proxy's name instead would + * label every vLLM or SGLang node as LM Studio. + */ +function proxyNode(id: string, modelsByEngine: Record) { + return { + id, + host: id, + port: 1234, + addresses: ['192.0.2.60'], + ip: '192.0.2.60', + modelsByEngine + } +} + +describe('OpenAI proxy engine attribution', () => { + // The bridge state is a process singleton, so each case uses its own node + // ids rather than trying to reset it. + const state = getModularBridgeState() + + beforeEach(() => { + state.setSelfId('openai-proxy-engines-local') + }) + + it('labels a vLLM-only peer as vLLM, not LM Studio', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('vllm-peer', { vllm: ['Qwen/Qwen3-8B'] }) + }) + expect(state.isRemoteEngineRunning('vllm-peer', 'vllm')).toBe(true) + expect(state.isRemoteEngineRunning('vllm-peer', 'lm-studio')).toBe(false) + }) + + it('labels an SGLang-only peer as SGLang, not LM Studio', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('sglang-peer', { sglang: ['/models/my-model'] }) + }) + expect(state.isRemoteEngineRunning('sglang-peer', 'sglang')).toBe(true) + expect(state.isRemoteEngineRunning('sglang-peer', 'lm-studio')).toBe(false) + expect(state.isRemoteEngineRunning('sglang-peer', 'vllm')).toBe(false) + }) + + it('reports both engines for a peer running LM Studio and vLLM', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('dual-peer', { lmstudio: ['qwen2.5-7b'], vllm: ['Qwen/Qwen3-8B'] }) + }) + expect(state.isRemoteEngineRunning('dual-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('dual-peer', 'vllm')).toBe(true) + }) + + it('reports all three engines for a peer running every OpenAI-compatible one', () => { + // Nothing stops one host from running LM Studio, vLLM and SGLang side by + // side on three ports; the proxy fronts all of them and attributes each. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('triple-peer', { + lmstudio: ['qwen2.5-7b'], + vllm: ['Qwen/Qwen3-8B'], + sglang: ['/models/my-model'] + }) + }) + expect(state.isRemoteEngineRunning('triple-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('triple-peer', 'vllm')).toBe(true) + expect(state.isRemoteEngineRunning('triple-peer', 'sglang')).toBe(true) + }) + + it('counts an engine that is running with no models as present', () => { + // The proxy reports a key with an empty (JSON null) list for an engine + // that is up but holds nothing, which is not the same as absent. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('empty-peer', { vllm: [] }) + }) + expect(state.isRemoteEngineRunning('empty-peer', 'vllm')).toBe(true) + }) + + it('drops an engine the node stopped advertising', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('shrinking-peer', { lmstudio: ['a'], vllm: ['b'] }) + }) + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/updated', + params: proxyNode('shrinking-peer', { lmstudio: ['a'] }) + }) + expect(state.isRemoteEngineRunning('shrinking-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('shrinking-peer', 'vllm')).toBe(false) + }) + + it('drops SGLang while the other two keep running', () => { + // The update is authoritative for every engine the proxy fronts, so the + // one that vanished from the map must go down without disturbing the + // ones that stayed. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('shedding-peer', { lmstudio: ['a'], vllm: ['b'], sglang: ['c'] }) + }) + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/updated', + params: proxyNode('shedding-peer', { lmstudio: ['a'], vllm: ['b'] }) + }) + expect(state.isRemoteEngineRunning('shedding-peer', 'sglang')).toBe(false) + expect(state.isRemoteEngineRunning('shedding-peer', 'lm-studio')).toBe(true) + expect(state.isRemoteEngineRunning('shedding-peer', 'vllm')).toBe(true) + }) + + it('clears every engine the proxy fronts when the node is removed', () => { + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/discovered', + params: proxyNode('leaving-peer', { lmstudio: ['a'], vllm: ['b'], sglang: ['c'] }) + }) + // A removal payload carries the node id alone; the proxy sends it only + // once its last engine entry for that node is gone. + state.handleNotification({ + source: 'lmstudio-proxy', + method: 'node/removed', + params: { id: 'leaving-peer' } + }) + expect(state.isRemoteEngineRunning('leaving-peer', 'lm-studio')).toBe(false) + expect(state.isRemoteEngineRunning('leaving-peer', 'vllm')).toBe(false) + expect(state.isRemoteEngineRunning('leaving-peer', 'sglang')).toBe(false) + }) + + it('needs no attribution from a proxy that fronts one engine', () => { + // ollama-proxy routes Ollama and nothing else, so its source names the + // engine and its payload carries no per-engine map. + state.handleNotification({ + source: 'proxy', + method: 'node/discovered', + params: { id: 'ollama-peer', host: 'ollama-peer', port: 11434, ip: '192.0.2.61' } + }) + expect(state.isRemoteEngineRunning('ollama-peer', 'ollama')).toBe(true) + }) +}) diff --git a/desktop/tests/modular/remote-engine-install-state.test.ts b/desktop/tests/modular/remote-engine-install-state.test.ts index eed8aa89..634083ed 100644 --- a/desktop/tests/modular/remote-engine-install-state.test.ts +++ b/desktop/tests/modular/remote-engine-install-state.test.ts @@ -138,7 +138,8 @@ describe('remote engine install state', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.63'], - ip: '192.0.2.63' + ip: '192.0.2.63', + modelsByEngine: { lmstudio: [] } } }) @@ -179,7 +180,8 @@ describe('remote engine install state', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.65'], - ip: '192.0.2.65' + ip: '192.0.2.65', + modelsByEngine: { lmstudio: [] } } }) diff --git a/desktop/tests/modular/remote-engine-status.test.ts b/desktop/tests/modular/remote-engine-status.test.ts index 463e1ad6..d6500612 100644 --- a/desktop/tests/modular/remote-engine-status.test.ts +++ b/desktop/tests/modular/remote-engine-status.test.ts @@ -22,7 +22,8 @@ describe('remote engine status', () => { host: remoteNodeId, port: 1234, addresses: ['192.0.2.190'], - ip: '192.0.2.190' + ip: '192.0.2.190', + modelsByEngine: { lmstudio: [] } } }) expect(state.isRemoteEngineRunning(remoteNodeId, 'lm-studio')).toBe(true) diff --git a/desktop/tests/modular/sglang-engine.test.ts b/desktop/tests/modular/sglang-engine.test.ts new file mode 100644 index 00000000..c064e472 --- /dev/null +++ b/desktop/tests/modular/sglang-engine.test.ts @@ -0,0 +1,114 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'node:fs' +import path from 'node:path' +import { describe, expect, it } from 'vitest' +import { + EngineDefaultLinks, + EngineDisplayNames, + EngineTypes, + EnabledEngineTypes +} from '@/shared/constants/engines' +import { EngineCapabilities } from '@/ui/constants/engine-capabilities' +import { WELCOME_ENGINE_DEFAULT_SELECTED, getWelcomeEngineCandidates } from '@/ui/constants/welcome' +import { isEngineType } from '@/shared/utils/engines' +import { platformDisplayName } from '@/shared/utils/platform' +import { formatModelDisplayName } from '@/ui/utils/format-model-display-name' + +const MANIFEST_DIR = path.resolve(process.cwd(), '../services/nvpair-engine-manager/manifests') + +describe('SGLang engine registration', () => { + it('is a known, enabled engine type spelled the same as its manifest', () => { + expect(EngineTypes).toContain('sglang') + expect(EnabledEngineTypes).toContain('sglang') + expect(isEngineType('sglang')).toBe(true) + // The engine-manager id and our EngineType are the same string, so every + // name-translation helper is a pass-through. LM Studio is the only engine + // whose two spellings differ. + const manifest = JSON.parse( + fs.readFileSync(path.join(MANIFEST_DIR, 'sglang.json'), 'utf8') + ) as { engine: string; display_name: string } + expect(manifest.engine).toBe('sglang') + expect(EngineDisplayNames.sglang).toBe(manifest.display_name) + }) + + it('ships the SGLang documentation and install links', () => { + expect(EngineDefaultLinks.sglang.docsUrl).toBe('https://docs.sglang.ai/') + expect(EngineDefaultLinks.sglang.installUrl).toBe( + 'https://docs.sglang.ai/get_started/install.html' + ) + }) + + it('declares capabilities that match what SGLang can actually do', () => { + const caps = EngineCapabilities.sglang + // Linux only: SGLang targets Linux GPUs and its manifest ships Linux + // platform blocks only. + expect(caps.hasInstall).toEqual(['linux']) + expect(caps.hasEnginePort).toBe(true) + // One model per process, chosen before start and resident for the life of + // the process: nothing to eject, delete, or expire. + expect(caps.hasEject).toBe(false) + expect(caps.hasDeleteModel).toBe(false) + expect(caps.hasExpiry).toBe(false) + expect(caps.modelOpsWhenStopped).toBe(false) + // The served model is a start-time setting, which is what gives the + // engine settings its "Model to serve" field. + expect(caps.hasServedModel).toBe(true) + // No public catalog is wired up, so no hub source selector is offered. + expect(caps.engineHub).toBeUndefined() + }) + + it('is offered in onboarding on Linux only, and never pre-selected', () => { + expect(WELCOME_ENGINE_DEFAULT_SELECTED.sglang).toBe(false) + expect(getWelcomeEngineCandidates(platformDisplayName('linux'))).toContain('sglang') + expect(getWelcomeEngineCandidates(platformDisplayName('darwin'))).not.toContain('sglang') + expect(getWelcomeEngineCandidates(platformDisplayName('win32'))).not.toContain('sglang') + }) + + it('renders a Hugging Face repo id as a readable model name', () => { + // SGLang's `/v1/models` id is its `--served-model-name`, which defaults to + // `--model-path` verbatim. Given a repo id that is a Hugging Face id, and + // the shared formatter already renders those — the point of this case is + // that the engine falls into that path rather than an engine-specific one. + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'sglang')).toBe( + formatModelDisplayName('Qwen/Qwen3-8B', 'vllm') + ) + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'sglang')).not.toContain('/') + }) + + it('renders a local model directory path as its last segment', () => { + // The other id shape `--model-path` takes: a directory on the node. The + // Hugging Face formatter would strip the leading segment and turn the + // rest into "models/my model", so a path gets its own rule — the last + // segment, verbatim, because that is the name the operator typed. + expect(formatModelDisplayName('/models/my-model', 'sglang')).toBe('my-model') + expect(formatModelDisplayName('/srv/weights/llama-3.1-8b/', 'sglang')).toBe('llama-3.1-8b') + // vLLM takes local paths too, and renders them the same way. + expect(formatModelDisplayName('/models/my-model', 'vllm')).toBe('my-model') + }) + + it('does not pass off a truncated path segment as the directory name', () => { + // The 256-char input cap slices from the right, taking the last segment + // with it. Naming a truncated middle segment would look exactly like a + // real answer, so a capped path keeps the long string instead. + const long = `/mnt/nvme0/models/vendor/${'y'.repeat(250)}/my-model` + const out = formatModelDisplayName(long, 'sglang') + expect(out).not.toBe('my-model') + expect(out.startsWith('/mnt/nvme0/models/vendor/')).toBe(true) + // A path that fits the cap is unaffected. + expect(formatModelDisplayName('/mnt/nvme0/models/my-model', 'sglang')).toBe('my-model') + }) + + it('leaves the engines that format their own names untouched', () => { + // The path rule lives in the shared default branch, so this is the guard + // that it did not reach past the engines with rules of their own. + expect(formatModelDisplayName('llama3.2:latest', 'ollama')).toBe('llama3.2') + expect( + formatModelDisplayName( + 'lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf', + 'lm-studio' + ) + ).toBe('Meta Llama 3.1 8B Instruct (Q4_K_M)') + }) +}) diff --git a/desktop/tests/modular/uuid-node-keying.test.ts b/desktop/tests/modular/uuid-node-keying.test.ts index 5632b53f..808164ea 100644 --- a/desktop/tests/modular/uuid-node-keying.test.ts +++ b/desktop/tests/modular/uuid-node-keying.test.ts @@ -71,7 +71,8 @@ describe('UUID node keying', () => { host: 'merge-host-2', port: 1234, addresses: ['192.0.2.12'], - ip: '192.0.2.12' + ip: '192.0.2.12', + modelsByEngine: { lmstudio: [] } } }) state.handleNotification({ diff --git a/desktop/tests/modular/vllm-engine.test.ts b/desktop/tests/modular/vllm-engine.test.ts new file mode 100644 index 00000000..e1fec929 --- /dev/null +++ b/desktop/tests/modular/vllm-engine.test.ts @@ -0,0 +1,87 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import fs from 'node:fs' +import path from 'node:path' +import { describe, expect, it } from 'vitest' +import { + EngineDefaultLinks, + EngineDisplayNames, + EngineTypes, + EnabledEngineTypes +} from '@/shared/constants/engines' +import { EngineCapabilities } from '@/ui/constants/engine-capabilities' +import { WELCOME_ENGINE_DEFAULT_SELECTED, getWelcomeEngineCandidates } from '@/ui/constants/welcome' +import { isEngineType } from '@/shared/utils/engines' +import { platformDisplayName } from '@/shared/utils/platform' +import { formatModelDisplayName } from '@/ui/utils/format-model-display-name' + +const MANIFEST_DIR = path.resolve(process.cwd(), '../services/nvpair-engine-manager/manifests') + +describe('vLLM engine registration', () => { + it('is a known, enabled engine type spelled the same as its manifest', () => { + expect(EngineTypes).toContain('vllm') + expect(EnabledEngineTypes).toContain('vllm') + expect(isEngineType('vllm')).toBe(true) + // The engine-manager id and our EngineType are the same string, so every + // name-translation helper is a pass-through. LM Studio is the only engine + // whose two spellings differ. + const manifest = JSON.parse( + fs.readFileSync(path.join(MANIFEST_DIR, 'vllm.json'), 'utf8') + ) as { engine: string; display_name: string } + expect(manifest.engine).toBe('vllm') + expect(EngineDisplayNames.vllm).toBe(manifest.display_name) + }) + + it('ships the vLLM documentation and install links', () => { + expect(EngineDefaultLinks.vllm.docsUrl).toBe('https://docs.vllm.ai/') + expect(EngineDefaultLinks.vllm.installUrl).toBe( + 'https://docs.vllm.ai/en/latest/getting_started/installation/' + ) + }) + + it('declares capabilities that match what vLLM can actually do', () => { + const caps = EngineCapabilities.vllm + // Linux only: vLLM publishes no Windows build and no macOS GPU build, and + // its manifest ships Linux platform blocks only. + expect(caps.hasInstall).toEqual(['linux']) + expect(caps.hasEnginePort).toBe(true) + // One model per process, chosen before start and resident for the life of + // the process: nothing to eject, delete, or expire. + expect(caps.hasEject).toBe(false) + expect(caps.hasDeleteModel).toBe(false) + expect(caps.hasExpiry).toBe(false) + expect(caps.modelOpsWhenStopped).toBe(false) + // The served model is a start-time setting, which is what gives the + // engine settings its "Model to serve" field. + expect(caps.hasServedModel).toBe(true) + // No public catalog is wired up, so no hub source selector is offered. + expect(caps.engineHub).toBeUndefined() + }) + + it('shares the served-model capability with SGLang and nothing else', () => { + // The one-model-per-process engines, in EngineTypes order. Ollama and LM + // Studio host a library and must never grow this field. + const declaring = EngineTypes.filter(type => EngineCapabilities[type].hasServedModel) + expect(declaring).toEqual(['vllm', 'sglang']) + }) + + it('is offered in onboarding on Linux only, and never pre-selected', () => { + expect(WELCOME_ENGINE_DEFAULT_SELECTED.vllm).toBe(false) + expect(getWelcomeEngineCandidates(platformDisplayName('linux'))).toContain('vllm') + expect(getWelcomeEngineCandidates(platformDisplayName('darwin'))).not.toContain('vllm') + expect(getWelcomeEngineCandidates(platformDisplayName('win32'))).not.toContain('vllm') + }) + + it('renders a Hugging Face repo id as a readable model name', () => { + // vLLM's model ids are whatever `--model` was: a Hugging Face repo id + // here, which the shared formatter already handles — the point of this + // case is that the engine falls into that path rather than an + // engine-specific one. The local-directory shape it also accepts is + // covered in sglang-engine.test.ts, where that rule was added. + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'vllm')).toBe( + formatModelDisplayName('Qwen/Qwen3-8B', 'lm-studio') + ) + expect(formatModelDisplayName('Qwen/Qwen3-8B', 'vllm')).not.toContain('/') + }) +}) diff --git a/docs/architecture.mdx b/docs/architecture.mdx index f7e7ffed..8ceb275a 100644 --- a/docs/architecture.mdx +++ b/docs/architecture.mdx @@ -19,7 +19,7 @@ flowchart TB NodeA["Node
one machine running PAIR"] NodeB["Node"] EngineA1["Engine
Ollama"] - EngineA2["Engine
LM Studio"] + EngineA2["Engine
LM Studio / vLLM / SGLang"] EngineB1["Engine
Ollama"] ModelA1["Models
present on this engine"] ModelA2["Models"] @@ -40,9 +40,9 @@ flowchart TB one cluster. - A **node** is one machine. Nodes are peers where each runs the same services, and each can both serve requests and route them elsewhere. -- An **engine** is an inference server on a node, Ollama or LM Studio. A node can - run both, one, or neither, and a node with no running engine is not eligible to - serve. +- An **engine** is an inference server on a node: Ollama, LM Studio, vLLM, or + SGLang. A node can run several, one, or none, and a node with no running engine + is not eligible to serve. - **Models** belong to an engine on a specific node. Nothing is shared. The same model on two nodes is two independent copies and that duplication is what makes the two nodes interchangeable for a request. @@ -314,8 +314,8 @@ ordering reaches a proxy in three steps: The ranking combines *pending work and GPU pressure*. A workload counts as pending while it is queued or running, and it is attributed to the node it was -placed on. Both engines count together, so Ollama load affects LM Studio ordering -and vice versa. +placed on. Every engine counts together, so Ollama load affects LM Studio, vLLM, +and SGLang ordering and vice versa. GPU pressure is deliberately coarse. The scheduler smooths the busiest GPU's utilization, maps it to 0–3 pressure units at 40%, 70%, and 85%, and uses lower @@ -357,7 +357,8 @@ across the cluster as a whole. When an inference request contains a non-empty model, only nodes whose current inventory for that engine advertises the model enter the failover list. An empty inventory and an inventory that lists other models are both ineligible. Ollama's -implicit `:latest` tag is normalized; LM Studio model IDs match exactly. +implicit `:latest` tag is normalized; LM Studio, vLLM, and SGLang model IDs match +exactly. If no advertised owner is routable, the proxy returns an actionable local `502` without sending the request to an engine. It does not broaden the candidate list @@ -428,10 +429,10 @@ one place lower in the order. to an engine's own port is absent from workload events. GPU-heavy external work can still raise pressure, but CPU-only work and queued demand remain invisible. -**Both engines are counted as one pool.** Ollama and LM Studio load is summed, -and maximum GPU pressure applies to the whole node. That is conservative on a -typical single-GPU machine and can underuse a multi-GPU node where the engines -occupy different devices. +**Every engine is counted as one pool.** Ollama, LM Studio, vLLM, and SGLang +load is summed, and maximum GPU pressure applies to the whole node. That is +conservative on a typical single-GPU machine and can underuse a multi-GPU node +where the engines occupy different devices. **Every node ranks from its own view, and views lag.** There is no shared schedule. Two nodes dispatching at the same moment can briefly steer work to the @@ -462,9 +463,9 @@ port it listens on. ### Finding an Engine PAIR does not assume it installed the engine. Detection checks the manifest's -known install locations for each engine, so an Ollama or LM Studio you installed -yourself is found where it already is. "Installing" an engine that is already -present downloads nothing and reports it as installed. +known install locations for each engine, so an Ollama, LM Studio, vLLM, or +SGLang you installed yourself is found where it already is. "Installing" an +engine that is already present downloads nothing and reports it as installed. Starting is similarly deferential. If something is already serving the engine's port, PAIR **adopts** that instance instead of spawning a second copy, and reports @@ -521,7 +522,9 @@ A default installation listens on these ports: | Port | Listener | | --- | --- | | `11434` | Ollama-compatible proxy (Ollama itself moves to `11435`+) | -| `1234` | OpenAI-compatible proxy (LM Studio moves to `1235`+) | +| `1234` | OpenAI-compatible proxy for LM Studio, vLLM, and SGLang (LM Studio moves to `1235`+) | +| `8000` | vLLM engine (adopted or PAIR-started; the proxy never takes this port) | +| `30000` | SGLang engine (adopted or PAIR-started; the proxy never takes this port) | | `14318` | Node hardware and model inventory | | `14319` | Service-error synchronization between nodes | | `14320` | Workload propagation between nodes | @@ -646,11 +649,34 @@ its address. ### Manual Nodes -Some networks block or filter multicast, so discovery is not the only path in. -`nvpair-manual-nodes` takes an address you enter directly and probes it on a -fixed interval, and a manual node that answers is folded into the same directory -as a discovered one. It is initially keyed by the address you typed, and re-keyed -to the peer's real UUID as soon as that node reports it. +Some networks block or filter multicast, and some never carry it at all — a +Tailscale tailnet, a WireGuard tunnel, a routed link — so discovery is not the +only path in. `nvpair-manual-nodes` takes an address you enter directly, a name +or a literal, and probes it on a fixed interval. It is initially keyed by the +address you typed, and re-keyed to the peer's real UUID as soon as that node +reports it. + +What it finds decides how the node is used, and there are two answers: + +- **A bare inference host.** Ollama or LM Studio on a machine that does not run + PAIR. Its engines answer plain HTTP on their own ports, and it is bridged into + this node's proxies as a routing target. This is what manual nodes were + originally for. +- **Another PAIR node.** It answers on its node-info port with its identity, its + cluster principal, and the set of services it runs — the same set it would have + carried on an mDNS record that never arrives here. From that, this node + synthesizes the directory record discovery would have produced, and the peer is + a peer: inventory over mutual TLS from its engine manager, inference routed to + its proxy over mutual TLS, telemetry into the scheduler. Nothing downstream + knows it was typed rather than discovered. + +The distinction matters because a PAIR node's engine ports are proxy front doors +that refuse plaintext from anything but their own loopback. Probing them would +report a healthy peer as having no engines, so a node that identifies itself as a +PAIR node is never asked there. + +Refer to [Running PAIR across a Tailscale tailnet](remote-networks.mdx) for the +operator's walkthrough. ## Trust Boundaries diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index 81bf01d9..9ce1c481 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -6,8 +6,9 @@ SPDX-License-Identifier: Apache-2.0 # Managing Engines in NVIDIA Personal AI Router An **engine** is the local inference runtime Personal AI Router (PAIR) uses to -run models. Today that means Ollama or LM Studio on a given machine. PAIR can -install and run those engines for you, or work with a copy you already have. +run models. Today that means Ollama, LM Studio, vLLM, or SGLang on a given +machine. PAIR can install and run those engines for you, or work with a copy you +already have. This page explains what you can expect when you install, start, stop, update, or remove an engine. @@ -74,9 +75,9 @@ When you install an engine, consider the following: configures it, and owns it from then on. - A successful install **starts the engine** as part of the same flow, so you normally do not need a separate start step right after install. -- If Ollama or LM Studio is already running on the machine, PAIR can **adopt** - that install instead of downloading another copy. Adoption helps when the - usual engine port is already in use. +- If Ollama, LM Studio, vLLM, or SGLang is already running on the machine, PAIR + can **adopt** that install instead of downloading another copy. Adoption helps + when the usual engine port is already in use. To download models: @@ -128,10 +129,164 @@ controlled: the engine in its own application first if you want PAIR to manage it fully. - **LM Studio** publishes an official stop command, so PAIR can stop an adopted instance that way and restart it on the port you chose. +- **vLLM** is managed as a process, so it behaves like Ollama: PAIR will not move + an adopted one. +- **SGLang** is managed as a process too, so the same applies: an adopted SGLang + is neither moved nor stopped by PAIR. Refer to [Engines and Ports](architecture.mdx#engines-and-ports) in the architecture guide. +## vLLM + +vLLM works differently enough from the other engines to be worth reading before +you use it. + +**Linux only.** vLLM publishes no Windows build and no macOS GPU build, so PAIR +offers it on Linux and reports it as unavailable elsewhere. + +**Adopting an instance you already run is the normal path.** Most people run +vLLM themselves, either as `vllm serve ` or from the `vllm/vllm-openai` +container image. If something is already serving on port `8000`, turning vLLM on +in **Engine settings** adopts it rather than starting a second copy, and the +engine shows as running. PAIR does not stop an instance it did not start; stop +it the way you started it. + +**One model per process.** Unlike Ollama and LM Studio, a vLLM process serves +exactly one model, named when it starts. Set it in **Engine settings** under +**Model to serve**, using a Hugging Face model id such as `Qwen/Qwen3-8B`. PAIR +persists the choice and restarts the engine onto it. Starting vLLM with no model +chosen fails with a message saying so rather than starting something that cannot +answer. + +**The first start can take many minutes.** vLLM downloads the model's weights +and compiles CUDA graphs before it serves anything, so PAIR allows up to thirty +minutes for it to become ready. The engine shows as starting throughout; later +starts of the same model are much faster. + +**There are no model operations.** vLLM has no download, load, unload, or delete +surface — the engine fetches what it needs from the model id you configured, and +keeps it resident for the life of the process. The model list simply shows what +the running engine serves. + +**Installing it takes a while and uses a lot of disk.** Choosing install builds a +Python environment under PAIR's own engine directory and installs vLLM and its +CUDA dependencies into it, which is several gigabytes. Uninstalling removes that +environment; it does not remove downloaded weights, which live in the Hugging +Face cache at `~/.cache/huggingface`. + +**A vLLM instance that spans several machines looks like one engine to PAIR.** +vLLM itself can shard a model across the GPUs of two or more machines with its +own tensor parallelism (`--tensor-parallel-size N --nnodes M`), for example +across two DGX Spark systems joined by their direct high-speed link. PAIR +neither combines GPUs nor splits models; it adopts the instance vLLM formed as a +single engine on that instance's head node. Only the head node (`--node-rank 0`) +opens the API port; every other node runs with `--headless` and holds its share +of the weights without listening for requests. PAIR follows that shape: + +- Run PAIR on **every** machine in the vLLM group, so each reports hardware and + telemetry and takes part in the cluster. Pair them all with the rest of your + cluster the usual way. +- Turn vLLM on only on the head node. It adopts the instance on port `8000` and + advertises the model. On a worker node there is nothing on port `8000` to + adopt, so vLLM shows as off there, which is correct rather than a fault. Do not + start a second vLLM on a worker; its GPU is already in use by the head's + instance. +- Every request for that model routes to the head node. Because the group is one + instance, there is no second copy for the scheduler to balance against; run a + separate instance elsewhere if you want the same model on two owners. +- Never restart a worker's vLLM on its own. Losing a rank stops the whole + instance, and the head node's API goes with it. Restart the group in the order + vLLM's own documentation gives for multi-node serving. + +## SGLang + +[SGLang](https://docs.sglang.ai) is a second OpenAI-compatible inference server. +In PAIR it behaves much as vLLM does, so the section above applies almost +unchanged; what follows is the SGLang version of it, and calls out where the two +differ. + +**Linux only.** As with vLLM, PAIR offers SGLang on Linux and reports it as +unavailable elsewhere. + +**Adopting an instance you already run is the normal path.** Most people run +SGLang themselves, either as `sglang serve --model-path ` or from the +`lmsysorg/sglang` container image. If something is already serving on port +`30000`, turning SGLang on in **Engine settings** adopts it rather than starting +a second copy, and the engine shows as running. PAIR will not move or stop an +instance it did not start; stop it the way you started it. The +OpenAI-compatible proxy on `1234` fronts it exactly as it fronts LM Studio and +vLLM, and never takes port `30000` for itself. + +**One model per process.** An SGLang process serves exactly one model, chosen +before it starts. Set it in **Engine settings** under **Model to serve**, using +either a Hugging Face model id such as `Qwen/Qwen3-8B` or the path to a local +model directory. Starting SGLang with no model chosen fails with a message +saying so. The id the engine then advertises on `/v1/models`, and that you see +under **Endpoints**, is its `--served-model-name`, which defaults to the +`--model-path` verbatim — so a local directory is advertised under its own path, +for example `/models/my-model`. Name the model exactly that way in a +request, or start SGLang with `--served-model-name` set to something friendlier. + +**The port opens only once the model is loaded.** SGLang binds `30000` after +loading finishes rather than before, and PAIR checks readiness and liveness with +`GET /get_model_info`, SGLang's own metadata route. It deliberately does not use +`GET /health`: on current SGLang builds that route runs a real forward pass and +takes about a second, and PAIR probes an engine often enough that this would +cost real serving time. A node that is still loading therefore shows no SGLang +at all until it is ready, which is expected rather than a fault. A first start +can take many minutes, and PAIR allows up to thirty minutes for it; later starts +of the same model are much faster. + +**There are no model operations.** As with vLLM, SGLang has no download, load, +unload, or delete surface. The engine fetches what it needs for the model you +configured and keeps it resident for the life of the process, and the model list +simply shows what the running engine serves. + +**Installing it takes a while and uses a lot of disk.** Choosing install builds +a Python environment with `uv` under PAIR's own engine directory and installs +SGLang and its CUDA dependencies into it, which is several gigabytes. +Uninstalling removes that environment; it does not remove downloaded weights, +which live in the Hugging Face cache at `~/.cache/huggingface`. + +**On a machine PAIR does not run, SGLang is recognized by its own route.** A +bare inference host added by address answers on `30000` like any other +OpenAI-compatible server, so PAIR asks for `GET /get_model_info`, which SGLang +serves and the others do not, before treating the host as an SGLang node. + +**An SGLang instance that spans several machines looks like one engine to +PAIR.** SGLang itself can shard a model across the GPUs of two or more machines +with `--tp N --nnodes M`, giving each machine its own `--node-rank` and pointing +them all at the head's `--dist-init-addr`. As with vLLM, PAIR neither combines +GPUs nor splits models; it adopts the instance SGLang formed as a single engine +on that instance's head node. Two machines serving one model look like this, run +in the `lmsysorg/sglang` container: + +```bash +# head node +sglang serve --model-path /models/my-model --tp 2 --nnodes 2 --node-rank 0 \ + --dist-init-addr 192.0.2.10:20000 --host 0.0.0.0 --port 30000 +# second machine: the same command with --node-rank 1 +``` + +Only the head node (`--node-rank 0`) opens the API port; every other rank holds +its share of the weights and listens for nothing. PAIR follows that shape: + +- Run PAIR on **every** machine in the SGLang group, so each reports hardware and + telemetry and takes part in the cluster. Pair them all with the rest of your + cluster the usual way. +- Turn SGLang on only on the head node. It adopts the instance on port `30000` + and advertises the model. On a worker rank there is nothing on `30000` to + adopt, so SGLang shows as off there, which is correct rather than a fault. Do + not start a second SGLang on a worker; its GPU is already in use by the head's + instance. +- Every request for that model routes to the head node. Because the group is one + instance, there is no second copy for the scheduler to balance against; run a + separate instance elsewhere if you want the same model on two owners. +- Never restart a worker rank on its own. Losing a rank stops the whole instance, + and the head node's API goes with it. Restart the group in the order SGLang's + own documentation gives for multi-node serving. + ## Managing Engines on Other Machines After systems are paired, you can view another machine's engines from diff --git a/docs/overview.mdx b/docs/overview.mdx index 95ea262c..35565a2d 100644 --- a/docs/overview.mdx +++ b/docs/overview.mdx @@ -35,9 +35,9 @@ These terms have specific meanings in PAIR: is no server, controller, or primary node. - **Cluster** — the set of nodes you have paired together. A node belongs to at most one cluster, and it must leave before it can join another. -- **Engine** — the local inference server that runs models: Ollama or LM Studio. - PAIR can install, start, stop, and update an engine, or adopt one you already - run yourself. +- **Engine** — the local inference server that runs models: Ollama, LM Studio, + vLLM, or SGLang. PAIR can install, start, stop, and update an engine, or adopt + one you already run yourself. - **Model** — what you prepare on each node. Nodes do not share models, so a node can serve a request only for a model it already holds. Preparing the same model on several nodes is what makes those nodes interchangeable. @@ -121,7 +121,8 @@ For the trust boundaries in detail, refer to - A local endpoint for compatible AI applications and development tools. - LAN discovery plus manually configured nodes. -- Ollama-compatible and LM Studio/OpenAI-compatible routing proxies. +- Ollama-compatible and OpenAI-compatible routing proxies. The + OpenAI-compatible one fronts LM Studio, vLLM, and SGLang. - Pairing and cluster membership managed by the background services. - Model-aware, workload-informed routing of independent requests. - Encrypted routing between machines: a request sent to another node travels over diff --git a/docs/remote-networks.mdx b/docs/remote-networks.mdx new file mode 100644 index 00000000..29a31ba8 --- /dev/null +++ b/docs/remote-networks.mdx @@ -0,0 +1,210 @@ +{/* +SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 +*/} + +# Running PAIR across a Tailscale tailnet + +PAIR finds peers by multicast DNS, which never leaves a local link. Two machines +joined only by a Tailscale tailnet, a WireGuard tunnel, or a routed link will not +discover each other no matter how long you wait. Nothing is wrong with either +one: the announcement has nowhere to go. + +They can still be a cluster. You add the peer by address instead of waiting for +it to appear, and everything after that — pairing, inventory, telemetry, and +routing inference — works exactly as it does on a LAN, over the same +certificate-pinned mutual TLS. PAIR's trust is in the certificates, not in the +route the packets took. + +This page uses Tailscale because it is the common case. The same steps apply to +any network where the two machines can open TCP connections to each other but +multicast does not cross. + +## What changes on a tailnet + +**No discovery.** Nothing appears under **Add node** on its own. This is the only +part of PAIR that a tailnet actually breaks, and adding the peer by address is +the whole fix. + +**Addresses look unusual.** A Tailscale node's IPv4 address is in `100.64.0.0/10` +— carrier-grade NAT space, not a private LAN range — and its IPv6 address is a +unique local address under `fd7a:115c:a1e0::/48`. Both are ordinary addresses to +PAIR. When a node has a LAN address as well, PAIR prefers the LAN one, because a +peer on the same LAN reaches it faster; when the tailnet address is all a node +has, it is used without hesitation. + +**Use the MagicDNS name.** Tailscale gives every node a name like +`gpu-box.tail1234.ts.net`. Prefer it over the numeric address. PAIR re-resolves a +name on every check, so a node that is re-authenticated, re-created, or simply +handed a different address keeps working; a node added by IP address is stranded +the moment its address changes and has to be re-added by hand. + +**Latency is higher, and sometimes much higher.** A tailnet connection is usually +direct, but it falls back to a relay when both machines are behind restrictive +NATs. Inference still works; the first token just takes longer to arrive. + +## Add a node by host name, pair it, and check it + +Do this on one machine. Both must have PAIR running, and both must be on the same +tailnet. + +1. **Confirm the tailnet reaches the peer.** On the machine you are adding + *from*: + + ```bash + tailscale status # the peer should be listed and online + tailscale ping gpu-box # a direct or relayed path, either is fine + ``` + + On Windows, the command is `"C:\Program Files\Tailscale\tailscale.exe" status`. + On macOS, the CLI lives inside the app bundle at + `/Applications/Tailscale.app/Contents/MacOS/Tailscale`. + +2. **Open Add node** and type the peer's MagicDNS name — `gpu-box.tail1234.ts.net` + — into **Address or host name**. Enter the name on its own. Do not append a + port: PAIR adds each service's own port to what you type, so `host:14318` + becomes unreachable, and PAIR will tell you so rather than accepting it. + + If any of the peer's services are not on their default ports, open **Service + ports** and fill in only the ones that moved. Everything you leave blank keeps + its default. + +3. **Select Invite.** PAIR adds the node first and then starts pairing, so the + node appears in your node list within about ten seconds whether or not the + pairing succeeds. That is the fastest way to find out whether the address was + right. + +4. **Read the PIN to the other machine.** Someone there accepts the invitation + and types the six digits. This is the same PIN pairing as on a LAN. + +5. **Check the result.** The peer's card should show its GPU, CPU, and memory + within a probe cycle, and its models shortly after the pairing completes. + Models arrive only once you are paired: a peer serves its model list to + certificates it has pinned and to nobody else. + +6. **Route something to it.** Send a request to your local endpoint and confirm + the job is attributed to the remote node in the jobs view. + +Repeat from the other machine only if you want it to be able to *start* pairings +too; membership itself is mutual after one successful pairing. + +## Opening the right ports in a Tailscale ACL + +Tailscale's default policy allows everything between your own nodes, in which +case nothing here is required. If you have narrowed it, this grant opens exactly +what PAIR needs between machines tagged `tag:pair`, and nothing else: + +```json +{ + "tagOwners": { + "tag:pair": ["autogroup:admin"] + }, + "acls": [ + { + "action": "accept", + "src": ["tag:pair"], + "dst": ["tag:pair:1234,11434,14318,14319,14320,14321,14322,14323"] + } + ] +} +``` + +What each port is for: + +| Port | Service | Why a peer needs it | +| --- | --- | --- | +| `14318` | Node info | Hardware inventory, node identity, and the list of services this node runs. It is how a node added by address is recognized as a PAIR node at all. | +| `14321` | Cluster manager | PIN pairing, then the mutually authenticated membership channel. | +| `14322` | Engine manager | The peer's model inventory. Mutual TLS; a peer you have not paired with gets nothing. | +| `14323` | Engine control | Remote engine and model operations. Mutual TLS, cluster members only. | +| `11434` | Ollama-compatible proxy | Where inference is routed. Mutual TLS from a peer; plaintext is refused from anything but the machine's own loopback. | +| `1234` | OpenAI-compatible proxy | The same, for the OpenAI-compatible surface. | +| `14319` | Error sync | Cross-node error reporting. Mutual TLS. | +| `14320` | Workload manager | Job and telemetry exchange. Mutual TLS. | + +Two things this table is worth reading carefully for. First, `11434` and `1234` +on a PAIR node are **proxy front doors, not the engines themselves**. The engines +listen on loopback only and are never exposed. A peer connects to the front door +with its pinned certificate and the proxy passes the request to the local engine. +Second, `14318` is the one surface deliberately served in plaintext, so that a +machine which has not paired yet can still be seen. It carries hardware +inventory, a node identity, and port numbers — no prompts, no responses, no keys. + +Do not port-forward any of these on a public interface. The point of running PAIR +over a tailnet is that these ports are reachable only from machines you have +admitted to it. + +## Troubleshooting + +### The node appears but has no models + +Expected until the pairing completes. A peer serves its model list only over +mutual TLS to a certificate it has pinned. Check that the node card shows the +peer as paired; if the pairing was declined or expired, start it again from +**Add node** using the same address. + +If it is paired and models are still missing, the peer's engine-manager port +(`14322`) is most likely blocked. Confirm with: + +```bash +tailscale ping gpu-box +``` + +and check the ACL grant above. + +### 403 from the peer + +A `403` means the connection reached the peer and the peer declined to serve it. +Two causes, and they are worth telling apart: + +- **Not paired, or no longer paired.** The peer serves its inventory, engine + control, workloads, and inference only to certificates it pins. Removing a node + from a cluster takes effect on the very next request, so a node that worked + yesterday and 403s today may simply have been removed. Pair again. +- **Something dialed a proxy port in plaintext.** `11434` and `1234` accept + plaintext only from the machine's own loopback. If you are testing by hand with + `curl http://gpu-box.tail1234.ts.net:11434/`, a `403` is the correct answer and + not a fault. Route inference through your *local* endpoint and let PAIR make the + connection. + +### The host name does not resolve + +MagicDNS has to be enabled for the tailnet, and the machine has to be using +Tailscale's resolver. + +- **Everywhere:** confirm MagicDNS is on in the Tailscale admin console under + **DNS**, and that `tailscale status` shows the peer's full name. +- **Windows:** MagicDNS names sometimes fail to resolve when another VPN client + or a corporate DNS policy has claimed the resolver. Test with + `Resolve-DnsName gpu-box.tail1234.ts.net`. If it fails while `tailscale ping + gpu-box` succeeds, restart the Tailscale service, and if that does not help, + add the node by its `100.x.y.z` address instead. It works; it just will not + survive the node changing address. +- **Linux:** a host using `systemd-resolved` needs Tailscale's resolver + registered. `tailscale status` reporting the peer while + `getent hosts gpu-box.tail1234.ts.net` returns nothing points at that. + +### The node was found, then went unreachable + +PAIR checks a manual node every ten seconds and reports it after three +consecutive failures, so about thirty seconds of silence. A relayed tailnet path +can be slow enough to miss a check without the node being down; PAIR tolerates +that and keeps the peer in the routing pool across a short gap. + +A node added by IP address that never comes back has probably changed address. +Re-add it by its MagicDNS name so it can recover on its own next time. + +### Both machines are on a LAN *and* a tailnet + +Nothing to do. They will discover each other over the LAN, and PAIR prefers the +LAN address because it is the faster path. Adding the tailnet name as well is +harmless — the discovered record wins and the manual entry stands down. + +## Reading next + +- [Getting started](getting-started.mdx) — the LAN version of the same walkthrough +- [Architecture](architecture.mdx#manual-nodes) — how a node added by address is + folded into the same directory as a discovered one +- [Troubleshooting](troubleshooting.mdx) — everything that is not specific to a + remote network +- [SECURITY.md](../SECURITY.md) — what PAIR does and does not defend against diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index e4d8822c..a1949b4e 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -46,6 +46,12 @@ PAIR discovers peers on the local network. If **Add node** or - Allow PAIR through host firewalls on each system. - Retry with the peer's IP address in **Add node**. +**If the systems are joined by a VPN or overlay network rather than a local +link, discovery will never find them.** Multicast does not cross a tailnet, a +WireGuard tunnel, or a routed link, so an empty list is the expected result +rather than a fault. Add the peer by name instead: +[Running PAIR across a Tailscale tailnet](remote-networks.mdx). + ## Pairing Fails or Stalls A PIN belongs to one invitation attempt. A mistyped PIN, a canceled or expired diff --git a/services/lmstudio-proxy/README.md b/services/lmstudio-proxy/README.md index 71a8b70d..1eceedab 100644 --- a/services/lmstudio-proxy/README.md +++ b/services/lmstudio-proxy/README.md @@ -3,11 +3,17 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 --> -# LM Studio Proxy +# OpenAI-Compatible Proxy -A discovery-aware HTTP reverse proxy for LM Studio nodes on the local network. It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lm]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket). +A discovery-aware HTTP reverse proxy for every OpenAI-compatible engine on the local network — **LM Studio, vLLM and SGLang** today. They speak the same HTTP surface, so they share one router rather than one binary each; the binary, its port file, and its `lmstudio-proxy:` relay namespace keep their historical spelling because those are wire contracts. -> **Clone of `ollama-proxy`.** This proxy is a deliberate clone of [`ollama-proxy`](../ollama-proxy/README.md) so the two share identical routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lm` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags workloads `lmstudio`, and persists its port to its own file. It has no `--alias-address`, so its self-forward guard covers only its own listener. +It runs no mDNS browse of its own: its routing targets come from the broker's discovery relay (it sends `discovery:subscribe {services:[lm, vl, sg]}` and replaces its routing overlay from each pushed `discovery:nodes` snapshot) plus user-added manual nodes. It forwards HTTP requests to the selected node, aggregates the model-list route across candidate nodes, and exposes a bidirectional JSON-RPC 2.0 control channel over stdio (or an IPC socket). + +**Which engine owns a model.** A peer advertises `lm`, `vl` and `sg` at the same value — the port of its own OpenAI proxy — so a peer running several still resolves to one routing target. Which engine actually serves a given model comes from the node's per-engine model attribution (`modelsByEngine`, keyed by engine-manager name), not from the service key. That attribution is what tags each forwarded workload with the engine that ran it, and what the desktop reads to label a node. A model id advertised by more than one engine on a node resolves to the first in the proxy's engine order (LM Studio, then vLLM, then SGLang), so routing is deterministic. + +**Model ids are opaque.** A model id is compared for equality and never parsed. An LM Studio id looks like `qwen2.5-7b`, a vLLM one is usually a Hugging Face id such as `Qwen/Qwen3-8B`, and an SGLang one is its `--served-model-name`, which defaults to `--model-path` verbatim and so is routinely a bare local directory such as `/models/my-model`. Nothing here assumes any of those shapes. + +> **Sibling of `ollama-proxy`.** This proxy shares `ollama-proxy`'s routing, failover, CORS, and node-selection behavior — the CORS policy is literally the same code, `nvpair-shared/cors`, and is documented [there](../ollama-proxy/README.md#http-reverse-proxy). The differences are engine-specific: it subscribes to the discovery relay for `lm`, `vl` and `sg` nodes, forwards the OpenAI-compatible inference routes (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`), tags each workload with the engine that owns its model, and persists its port to its own file. It has no `--alias-address`, so its self-forward guard covers only its own listener. ## Build @@ -34,14 +40,14 @@ lmstudio-proxy [flags] ### HTTP Reverse Proxy -The proxy listens on `--port` (default 1234) and forwards incoming HTTP requests to the currently active LM Studio node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your OpenAI-compatible client at `http://localhost:1234` and the proxy handles routing. +The proxy listens on `--port` (default 1234) and forwards incoming HTTP requests to the currently active node — except the model-list route `GET /v1/models`, which is queried across every candidate node concurrently and merged into one de-duplicated inventory. Point your OpenAI-compatible client at `http://localhost:1234` and the proxy handles routing. -**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to the local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart. +**Cluster ingress.** The listener carries two personalities, demultiplexed by each connection's first byte. Plaintext HTTP is accepted only from loopback; a LAN caller is refused. When `--cluster-dir` shows this node is a cluster member, the same listener also terminates cluster mTLS: a peer whose client certificate matches one of this node's pins is forwarded straight to a local engine reported by `node/set-local-backend`, and is never re-routed onward to another node. With more than one local OpenAI engine the ingress picks the one that actually serves the requested model, read from the engines themselves (short-TTL cached) rather than from discovery, falling back to the single healthy engine when there is only one and answering `503` when neither resolves. A peer's `GET /v1/models` is answered by merging every local engine's inventory, so nothing this node serves is hidden from the cluster. Membership and pins are re-derived per request, so joining or leaving a cluster needs no restart. **Persisted port.** A port chosen at runtime via the `set-port` request (see below) is saved as `lmstudio-proxy-port.json` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) and **restored on startup**, taking precedence over `--port`/the default. One value is exempt: a stored `1235` is discarded and `--port` is used instead, so that port cannot be restored even when it was chosen deliberately via `set-port`. Any other stored port is honoured. The broker uses `--ignore-persisted-port` while reserving the managed `1234` facade. Node selection: -- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current LM Studio inventory advertises the exact requested model ID. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`. +- **Eligibility**: Before routing model-bearing inference, the proxy keeps only nodes whose current OpenAI-engine inventory advertises the exact requested model ID. An empty or non-matching inventory is excluded until a later discovery update; if no advertised owner is routable, the proxy returns a local `502`. - **Auto**: When no eligible node is explicitly selected, the proxy follows `node/set-priority` (see below), then discovered nodes in stable ID order. - **Priority (scheduler-driven)**: The Job Scheduler ranks the cluster least-loaded-first by pending workload plus smoothed GPU pressure and, via `nvpair-ui-broker`, pushes the ordered node list with those per-node counts to this proxy with `node/set-priority`. Auto routing sends the request to the listed node carrying the least estimated load. See [`nvpair-job-scheduler`](../nvpair-job-scheduler/README.md). - **Manual**: Use the `node/select` JSON-RPC method to pin traffic to a specific node. A manual pin **overrides the priority list only when that node is eligible** for the requested model. @@ -88,10 +94,11 @@ Nodes are represented throughout the protocol with this shape: |-------|------|-------------| | `id` | string | Stable per-host UUID from the discovery record (the ID you supply, for a manual node) | | `host` | string | Hostname, for display — routing never keys on it | -| `port` | int | LM Studio port from the discovery record's `lm` service entry | +| `port` | int | The node's OpenAI proxy port, from whichever of the `lm` / `vl` / `sg` service entries the record carries (they name the same port) | | `addresses` | string[] | Addresses to dial. A node fed by the discovery relay always carries exactly one canonical address; several only ever appear on a manual node | | `txt` | string[] | The discovery record's TXT pairs, carried verbatim | -| `models` | string[] | The node's LM Studio model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the exact requested model ID. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation | +| `modelsByEngine` | object | The node's inventory attributed per OpenAI engine (`lmstudio`, `vllm`, `sglang`), keyed by engine-manager engine name. A present key with an empty list means that engine is running and holds nothing; a missing key means the node does not run it. This is what says which engine a node runs and which one a model belongs to | +| `models` | string[] | The union of the above — the node's whole OpenAI-reachable model inventory from the discovery snapshot. Model-bearing inference is eligible only when this list advertises the exact requested model ID. An omitted or empty list excludes the node from that request until inventory updates; it remains available for non-inference routes and model-list aggregation | | `ip` | string | The single canonical LAN address to dial or display, resolved from the node's `ip=` TXT if present and otherwise the best-scored advertised IPv4. Stamped onto outbound `node/*` notifications so consumers agree with the address the proxy routes to | --- @@ -173,7 +180,7 @@ A proxied request finished, or was rejected before forwarding. `duration_ms` cov #### `workload:started` / `workload:completed` / `workload:errored` -One lifecycle transition per forwarded inference request, carrying a single `workloadInfo`. `engine` is always `lmstudio`; `originatedFrom` is left empty for the broker to stamp, and `scheduledOn` names the node that actually served (re-pointed if failover moved the request). The broker relays these to `nvpair-workload-manager`. The proxy never emits `workload:submitted` — it forwards immediately rather than queueing. +One lifecycle transition per forwarded inference request, carrying a single `workloadInfo`. `engine` names the OpenAI engine that owns the requested model on the node it was routed to — `lmstudio`, `vllm` or `sglang` — so a node running more than one attributes its work to the engine that actually ran it; `originatedFrom` is left empty for the broker to stamp, and `scheduledOn` names the node that actually served (re-pointed if failover moved the request). The broker relays these to `nvpair-workload-manager`. The proxy never emits `workload:submitted` — it forwards immediately rather than queueing. ```json {"jsonrpc":"2.0","method":"workload:started","params":{"workloadInfo":{"id":"17","model":"lmstudio-community/Qwen3-8B-GGUF","engine":"lmstudio","runId":"3ce8a1740b62df95","state":"running","originatedFrom":"","scheduledOn":"22222222-2222-2222-2222-222222222222","createdAt":1716998400000,"startedAt":1716998400000,"completedAt":null,"error":null,"requesterId":null}}} @@ -345,11 +352,11 @@ and the broker first steers the port clear of any running engine's port #### `node/add-manual` -Add a node manually (for networks where mDNS is blocked). If the node ID already exists as a manual node, it is updated. +Add a node manually (for networks where mDNS is blocked). `engine` is required and must be one of `lmstudio`, `vllm` or `sglang` — each sits on its own port, so the manual overlay is keyed by (engine, node) and one host may be added once per engine. An unrecognized engine is refused rather than creating an entry nothing can route to. If the (engine, node ID) pair already exists as a manual node, it is updated. **Request:** ```json -{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","host":"remote-server","port":1234,"addresses":["10.0.1.50"]}} +{"jsonrpc":"2.0","id":5,"method":"node/add-manual","params":{"id":"remote-server","engine":"vllm","host":"remote-server","port":8000,"addresses":["10.0.1.50"]}} ``` **Response:** @@ -365,7 +372,7 @@ Remove a previously added manual node. **Request:** ```json -{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server"}} +{"jsonrpc":"2.0","id":6,"method":"node/remove-manual","params":{"id":"remote-server","engine":"vllm"}} ``` **Response:** @@ -377,7 +384,7 @@ The proxy emits a `node/removed` notification and clears the active selection if #### `node/set-local-backend` -Tell the proxy which loopback engine this node's own traffic terminates on. The broker sends it once the local LM Studio address and health are known. It is the target the cluster mTLS ingress forwards to, and the substitute used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears it, and the ingress then answers `503`. +Tell the proxy which loopback engine this node's own traffic terminates on. Backends are held **per engine**, so one node may run LM Studio, vLLM and SGLang at once and clearing one leaves the others routing. The broker sends one per engine once that engine's local address and health are known. They are the targets the cluster mTLS ingress forwards to, and the substitutes used when discovery advertises this node's own proxy endpoint as a candidate. A zero `port` or `"healthy":false` effectively clears that engine's backend, and its ingress then answers `503`. **Request:** ```json @@ -441,4 +448,4 @@ The proxy shuts down gracefully on any of: ## Discovery -The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lm` (LM Studio) nodes (`discovery:subscribe {services:[lm]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set. +The proxy does not browse mDNS. On startup it subscribes to the broker's discovery relay for `lm` (LM Studio), `vl` (vLLM) and `sg` (SGLang) nodes (`discovery:subscribe {services:[lm, vl, sg]}`). Targets then arrive as `discovery:nodes` notifications carrying the relay's full filtered node set, and each snapshot replaces the routing overlay wholesale — a departed node is simply absent from the next one — while the diff against the previous overlay is what produces the `node/discovered`, `node/updated`, and `node/removed` notifications. User-added manual nodes are merged on top. Nodes are keyed by the discovery record's stable per-host UUID, so routing survives a machine being renamed. The single `_nvpair-node` browse that feeds the relay lives in the `nvpair-node-scanner` daemon (see its README) — this proxy is a pure consumer of the resulting routing set. diff --git a/services/lmstudio-proxy/activity_test.go b/services/lmstudio-proxy/activity_test.go index 49a63541..b9692654 100644 --- a/services/lmstudio-proxy/activity_test.go +++ b/services/lmstudio-proxy/activity_test.go @@ -58,6 +58,7 @@ func TestNoActivityReportedWithoutUpstreamBytes(t *testing.T) { Addresses: []string{"127.0.0.1"}, Port: closedPortFor(t), Models: []string{"qwen"}, + Engine: "lmstudio", }) p := NewProxy(NewCodec(rec), disc, 1235) diff --git a/services/lmstudio-proxy/discovery.go b/services/lmstudio-proxy/discovery.go index 9b05581b..ecd7ccdb 100644 --- a/services/lmstudio-proxy/discovery.go +++ b/services/lmstudio-proxy/discovery.go @@ -5,10 +5,11 @@ package main // Discovery is the proxy's routing-target set. The proxy runs no mDNS of its // own: routing targets are pushed down from the broker's discovery relay -// (discovery:nodes snapshots for the lm service) into the subscribed overlay, -// merged with user-added manual nodes. The proxy is itself advertised — as an lm -// service — by the node-scanner daemon's single _nvpair-node record, keyed off -// the engine port the broker's poller registers. +// (discovery:nodes snapshots for the OpenAI-compatible engine services — lm and +// vl) into the subscribed overlay, merged with user-added manual nodes. The +// proxy is itself advertised — under one key per local OpenAI engine — by the +// node-scanner daemon's single _nvpair-node record, which the broker's poller +// registers against this proxy's own listen port. // // The routable Node projection (IP / withPrimaryIP) and the manual-node // overlay live here; request-path reachability (TCP-probe + failover) lives in @@ -36,10 +37,24 @@ type Node struct { Addresses []string `json:"addresses"` TXT []string `json:"txt"` // Models is the latest model inventory carried by the broker's discovery - // snapshot. Model-bearing inference is eligible only when this list - // advertises the requested model; an empty list stays in discovery but is - // not an inference candidate until a later inventory update. + // snapshot — the union across every OpenAI engine the node runs. Model-bearing + // inference is eligible only when this list advertises the requested model; an + // empty list stays in discovery but is not an inference candidate until a + // later inventory update. Models []string `json:"models,omitempty"` + // ModelsByEngine attributes each model to the engine on this node that serves + // it, so a request can be tagged with the engine that will actually run it and + // so a node running LM Studio, vLLM and SGLang is not conflated into one + // inventory. Keyed by engine-manager engine name ("lmstudio", "vllm", + // "sglang"). + ModelsByEngine map[string][]string `json:"modelsByEngine,omitempty"` + // Engine names the single OpenAI engine a manual node was added for. A relay + // peer leaves it empty: one peer entry covers every engine that peer runs, + // because they all answer on that peer's one proxy port. A manual node is the + // exception — the user supplies an engine's own address, and LM Studio, vLLM + // and SGLang sit on different ports — so one manual node per engine is added, + // each carrying the engine it represents. + Engine string `json:"engine,omitempty"` // IP is the single canonical LAN address a consumer should dial/display for // this node, resolved via the shared netpick ranker: the node's // own ip= TXT if present, else the best-scored advertised IPv4. It is @@ -64,14 +79,33 @@ func (n Node) withPrimaryIP() Node { return n } +// routeKey is the node's key for per-target routing caches (the reachability +// chooser). It is the node ID for a relay peer, and engine-qualified for a +// manual node, because two manual entries can share an ID while pointing at +// different engine ports on that host — caching them under one key would let +// one engine's confirmed address be used to dial the other's. +// +// It is deliberately NOT the attribution handle: scheduledOn, node/select and +// the scheduler's priority list stay keyed by the bare node ID, which names the +// machine rather than one engine on it. +func (n Node) routeKey() string { + if n.Engine == "" { + return n.ID + } + return n.Engine + "\x00" + n.ID +} + +// manualKey is the manual overlay's storage key: one entry per (engine, node). +func manualKey(engine, id string) string { return engine + "\x00" + id } + // Discovery holds the proxy's routing targets: the relay-fed subscribed overlay // and the user-added manual overlay. type Discovery struct { mu sync.RWMutex manualNodes map[string]Node // subscribedNodes are routing targets pushed down by the broker's discovery - // relay (discovery:nodes snapshots for the lm service), keyed by node ID (the - // directory instance name). + // relay (discovery:nodes snapshots for the OpenAI engine services), keyed by + // node ID (the directory instance name). subscribedNodes map[string]Node } @@ -93,8 +127,11 @@ func (d *Discovery) Nodes() []Node { out = append(out, n) seen[id] = struct{}{} } - for id, n := range d.manualNodes { - if _, exists := seen[id]; !exists { + // Manual entries are keyed per (engine, node), so a host added for both + // engines contributes one entry each; a node the relay already covers is + // skipped entirely, exactly as before. + for _, n := range d.manualNodes { + if _, exists := seen[n.ID]; !exists { out = append(out, n) } } @@ -135,31 +172,61 @@ func (d *Discovery) SetSubscribed(nodes []Node) (discovered, updated, removed [] // warrants a node/updated. func nodeEqual(a, b Node) bool { return a.ID == b.ID && a.Host == b.Host && a.Port == b.Port && a.IP == b.IP && + a.Engine == b.Engine && slices.Equal(a.Addresses, b.Addresses) && slices.Equal(a.TXT, b.TXT) && - slices.Equal(a.Models, b.Models) + slices.Equal(a.Models, b.Models) && engineModelsEqual(a.ModelsByEngine, b.ModelsByEngine) } +// engineModelsEqual compares two per-engine inventories. It is part of nodeEqual +// because a node that moves a model from one engine to the other keeps the same +// union — without this the change would not raise a node/updated and consumers +// would keep the stale attribution. +func engineModelsEqual(a, b map[string][]string) bool { + if len(a) != len(b) { + return false + } + for engine, models := range a { + other, ok := b[engine] + if !ok || !slices.Equal(models, other) { + return false + } + } + return true +} + +// AddManual upserts a manual node for one engine. The same node ID added for +// several engines is one entry each, because each names that engine's own port. func (d *Discovery) AddManual(node Node) (added bool) { d.mu.Lock() defer d.mu.Unlock() - _, exists := d.manualNodes[node.ID] - d.manualNodes[node.ID] = node + key := manualKey(node.Engine, node.ID) + _, exists := d.manualNodes[key] + d.manualNodes[key] = node return !exists } -func (d *Discovery) RemoveManual(id string) (removed bool) { +// RemoveManual drops one engine's entry for a node. +func (d *Discovery) RemoveManual(engine, id string) (removed bool) { d.mu.Lock() defer d.mu.Unlock() - _, exists := d.manualNodes[id] + key := manualKey(engine, id) + _, exists := d.manualNodes[key] if exists { - delete(d.manualNodes, id) + delete(d.manualNodes, key) } return exists } +// IsManual reports whether any engine's manual entry exists for a node ID. It +// answers the routing question "did the user supply this address explicitly?", +// which is a property of the node, not of one engine on it. func (d *Discovery) IsManual(id string) bool { d.mu.RLock() defer d.mu.RUnlock() - _, exists := d.manualNodes[id] - return exists + for _, n := range d.manualNodes { + if n.ID == id { + return true + } + } + return false } diff --git a/services/lmstudio-proxy/e2e_test.go b/services/lmstudio-proxy/e2e_test.go index 5412ab7c..78b9bd6d 100644 --- a/services/lmstudio-proxy/e2e_test.go +++ b/services/lmstudio-proxy/e2e_test.go @@ -189,9 +189,9 @@ func TestE2EFailoverOverRealBinary(t *testing.T) { busyHost, busyPort := e2eSplitHostPort(t, busy.URL) goodHost, goodPort := e2eSplitHostPort(t, good.URL) - e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}}) + e2eSend(t, stdin, 1, "node/add-manual", map[string]any{"id": "busy", "engine": "lmstudio", "host": busyHost, "port": busyPort, "addresses": []string{busyHost}, "models": []string{"m"}}) e2eWaitResult(t, frames, "1", 5*time.Second) - e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}}) + e2eSend(t, stdin, 2, "node/add-manual", map[string]any{"id": "good", "engine": "lmstudio", "host": goodHost, "port": goodPort, "addresses": []string{goodHost}, "models": []string{"m"}}) e2eWaitResult(t, frames, "2", 5*time.Second) // Select the busy node so the failover path is deterministic. e2eSend(t, stdin, 3, "node/select", map[string]any{"id": "busy"}) diff --git a/services/lmstudio-proxy/engines.go b/services/lmstudio-proxy/engines.go new file mode 100644 index 00000000..dd2dec7e --- /dev/null +++ b/services/lmstudio-proxy/engines.go @@ -0,0 +1,239 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/url" + "slices" + "sort" + + "nvpair-shared/noderec" +) + +// openaiEngine binds one OpenAI-compatible engine's discovery service key to +// the engine-manager engine name that owns its models and tags its workloads. +// The two are separate contracts and are deliberately not derived from each +// other: the key is the mDNS TXT spelling, the name is what engine-manager's +// modelsByEngine map and the workload wire use. +type openaiEngine struct { + Service noderec.ServiceKey + Name string +} + +// openaiEngines is every engine this proxy fronts, in the order it resolves +// them. The order is the deterministic tie-break for a model id that more than +// one engine on the same node advertises — first listed wins — so routing and +// workload attribution can never depend on map iteration order. +// +// Every entry speaks the same OpenAI HTTP surface, which is what lets one proxy +// serve them all: the engines differ in how they are installed and managed, not +// in how a request is forwarded. +var openaiEngines = []openaiEngine{ + {Service: noderec.ServiceLMStudio, Name: "lmstudio"}, + {Service: noderec.ServiceVLLM, Name: "vllm"}, + {Service: noderec.ServiceSGLang, Name: "sglang"}, +} + +// subscribedServices is the discovery:subscribe service list: every engine key +// this proxy routes for. +func subscribedServices() []noderec.ServiceKey { + out := make([]noderec.ServiceKey, 0, len(openaiEngines)) + for _, e := range openaiEngines { + out = append(out, e.Service) + } + return out +} + +// engineNames returns the engine-manager names in resolution order. +func engineNames() []string { + out := make([]string, 0, len(openaiEngines)) + for _, e := range openaiEngines { + out = append(out, e.Name) + } + return out +} + +// isOpenAIEngine reports whether a name is one this proxy fronts. It gates the +// engine field on node/set-local-backend and node/add-manual so a typo is +// refused at the boundary rather than silently creating an unroutable entry. +func isOpenAIEngine(name string) bool { + return slices.Contains(engineNames(), name) +} + +// engineForModel returns the engine on this node that owns a model id, using +// openaiEngines order as the tie-break, and "" when none advertises it. A node +// with no per-engine attribution at all (a peer whose scanner has not enriched +// it yet) resolves to "" and is attributed by the caller. +func engineForModel(modelsByEngine map[string][]string, model string) string { + if model == "" { + return "" + } + for _, e := range openaiEngines { + if slices.Contains(modelsByEngine[e.Name], model) { + return e.Name + } + } + return "" +} + +// unionModels flattens a per-engine model map into one sorted, de-duplicated +// list — the node's whole OpenAI-reachable inventory, which is what eligibility +// filtering and the aggregated /v1/models answer both work from. Sorted so a +// node's Models list is stable across snapshots and cannot spuriously trip the +// node/updated diff. +func unionModels(modelsByEngine map[string][]string) []string { + seen := make(map[string]bool) + out := make([]string, 0) + for _, e := range openaiEngines { + for _, m := range modelsByEngine[e.Name] { + if m != "" && !seen[m] { + seen[m] = true + out = append(out, m) + } + } + } + sort.Strings(out) + return out +} + +// candidateEngine is the engine a forwarded request is attributed to. A +// candidate resolved from a model always names its owner; one resolved with no +// routing model (a control call) falls back to the first engine, matching the +// order routing itself resolves in. +func candidateEngine(c candidate) string { + if c.engine != "" { + return c.engine + } + return fallbackWorkloadEngine +} + +// appendCandidate adds a resolved candidate unless its backend host is already +// claimed or resolves back to this proxy. The self check is defensive: a local +// backend must never point at our own listener, which would loop. +func appendCandidate(out []candidate, seenHost map[string]bool, c candidate, selfPort int) []candidate { + if c.url == nil { + return out + } + if isSelfTarget(c.url, selfPort) { + slog.Debug("resolveCandidates: skipping self-target node", + "node_id", c.id, "engine", c.engine, "target", c.url.Host, "self_port", selfPort) + return out + } + if seenHost[c.url.Host] { + return out + } + seenHost[c.url.Host] = true + return append(out, c) +} + +// nodeEngineFor names the engine on a node that owns the routing model, or "" +// when there is no routing model or the node carries no attribution for it. A +// manual node has exactly one engine and answers with it directly. +func (p *Proxy) nodeEngineFor(n Node, model string) string { + if n.Engine != "" { + return n.Engine + } + return engineForModel(n.ModelsByEngine, model) +} + +// selfTargets are the local loopback engines a candidate for this node resolves +// to. With a routing model it is the one engine that serves it — read from the +// engines themselves, which is authoritative where the node's own discovery +// entry can lag. Without one (a control call or the aggregated model list) it is +// every healthy local engine, so nothing this node serves is hidden. +func (p *Proxy) selfTargets(n Node, model, discoveredEngine string) []localEngineTarget { + backends := p.localBackends() + if len(backends) == 0 { + slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID) + return nil + } + if model == "" { + return backends + } + engine := engineForModel(p.localModelsByEngine(context.Background(), backends), model) + if engine == "" { + engine = discoveredEngine + } + for _, b := range backends { + if b.Engine == engine { + return []localEngineTarget{b} + } + } + // The model is advertised for this node but no local engine claims it. Do not + // guess an engine that would answer 404: leave the node out and let another + // owner, or the no-owner rejection, decide. + slog.Debug("resolveCandidates: no local engine owns the requested model", "node_id", n.ID) + return nil +} + +// modelListTransportError marks a model-list failure that happened before any +// response arrived, so the caller can drop the confirmed address for that +// target. A protocol-level failure (bad status, unparseable body) proves the +// address is reachable and must not. +type modelListTransportError struct{ err error } + +func (e modelListTransportError) Error() string { return e.err.Error() } +func (e modelListTransportError) Unwrap() error { return e.err } + +func isTransportError(err error) bool { + var t modelListTransportError + return errors.As(err, &t) +} + +// fetchModelList reads one OpenAI /v1/models endpoint and returns its records +// with their ids. It is shared by the cluster-wide aggregation (over a peer's +// mTLS proxy) and the ingress aggregation (over this node's loopback engines), +// so both apply the same size cap and the same strict envelope validation. +func fetchModelList(ctx context.Context, client *http.Client, target *url.URL) ([]modelListItem, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, target.String(), nil) + if err != nil { + return nil, err + } + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + return nil, modelListTransportError{err} + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("upstream returned %s", resp.Status) + } + body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1)) + if err != nil { + return nil, err + } + if len(body) > maxModelListBytes { + return nil, fmt.Errorf("model list exceeds %d bytes", maxModelListBytes) + } + var envelope struct { + Data *[]json.RawMessage `json:"data"` + } + if err := json.Unmarshal(body, &envelope); err != nil { + return nil, err + } + if envelope.Data == nil { + return nil, fmt.Errorf("upstream response has no data array") + } + items := make([]modelListItem, 0, len(*envelope.Data)) + for _, raw := range *envelope.Data { + var identity struct { + ID string `json:"id"` + } + if err := json.Unmarshal(raw, &identity); err != nil { + return nil, fmt.Errorf("invalid model record: %w", err) + } + if identity.ID == "" { + return nil, fmt.Errorf("model record has no id") + } + items = append(items, modelListItem{key: identity.ID, raw: raw}) + } + return items, nil +} diff --git a/services/lmstudio-proxy/failover_test.go b/services/lmstudio-proxy/failover_test.go index 02e5361b..2ebf92fc 100644 --- a/services/lmstudio-proxy/failover_test.go +++ b/services/lmstudio-proxy/failover_test.go @@ -43,13 +43,16 @@ func nodeFor(t *testing.T, id, serverURL string) Node { if err != nil { t.Fatalf("port %q: %v", portStr, err) } - return Node{ID: id, Addresses: []string{host}, Port: port} + // Manual nodes always name the engine they were added for; the proxy keys its + // manual overlay by (engine, node) so one host can be added for both. + return Node{ID: id, Addresses: []string{host}, Port: port, Engine: "lmstudio"} } func nodeForModel(t *testing.T, id, serverURL, model string) Node { t.Helper() node := nodeFor(t, id, serverURL) node.Models = []string{model} + node.ModelsByEngine = map[string][]string{node.Engine: {model}} return node } diff --git a/services/lmstudio-proxy/ingress.go b/services/lmstudio-proxy/ingress.go index 2b70e697..0431ac7f 100644 --- a/services/lmstudio-proxy/ingress.go +++ b/services/lmstudio-proxy/ingress.go @@ -4,25 +4,34 @@ package main import ( + "bytes" + "context" "encoding/json" + "io" "log/slog" "net" "net/http" "net/http/httputil" "net/url" "strconv" + "sync" + "time" "nvpair-shared/cors" ) const engineIdentityProbeHeader = "X-NVPAIR-Engine-Identity-Probe" -// localBackend is the explicit loopback engine the cluster mTLS ingress +// localBackend is one explicit loopback engine the cluster mTLS ingress // forwards to. It is supplied by the broker over node/set-local-backend and is // deliberately NOT sourced from the discovery overlay: a request that arrived // over the LAN mTLS ingress can only ever be dumped on this node's own local // engine, never re-routed to a peer, so the ingress path is strictly terminal // and cannot recurse or amplify. +// +// Engine names which OpenAI engine this is. One node may run LM Studio, vLLM +// and SGLang at once, so the backends are held per engine and each is set and +// cleared independently — marking one unhealthy must not disturb the others. type localBackend struct { Engine string `json:"engine"` Host string `json:"host"` @@ -30,29 +39,68 @@ type localBackend struct { Healthy bool `json:"healthy"` } +// url is the loopback URL for this backend, and false when it is not set or not +// healthy. The host defaults to 127.0.0.1 and is always loopback. +func (b localBackend) url() (*url.URL, bool) { + if b.Port <= 0 || !b.Healthy { + return nil, false + } + host := b.Host + if host == "" { + host = "127.0.0.1" + } + return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true +} + // setLocalBackend records (or, with a zero port / unhealthy flag, effectively -// clears) the local engine the ingress serves. +// clears) one engine's local backend. Storing the unhealthy record rather than +// deleting the key keeps "known and down" distinguishable from "never +// advertised" in logs, while localBackendTarget treats both as unavailable. func (p *Proxy) setLocalBackend(b localBackend) { p.backendMu.Lock() - p.backend = b + if p.backends == nil { + p.backends = make(map[string]localBackend, len(openaiEngines)) + } + p.backends[b.Engine] = b p.backendMu.Unlock() } -// localBackendTarget returns the loopback URL of the current local engine, and -// false when none is set/healthy (the ingress then answers 503 rather than -// forwarding). The host defaults to 127.0.0.1 and is always loopback. -func (p *Proxy) localBackendTarget() (*url.URL, bool) { +// localBackendTarget returns the loopback URL of one engine's local backend, +// and false when that engine has none set or it is unhealthy (the ingress then +// answers 503 rather than forwarding). +func (p *Proxy) localBackendTarget(engine string) (*url.URL, bool) { p.backendMu.RLock() - b := p.backend + b, ok := p.backends[engine] p.backendMu.RUnlock() - if b.Port <= 0 || !b.Healthy { + if !ok { return nil, false } - host := b.Host - if host == "" { - host = "127.0.0.1" + return b.url() +} + +// localBackends returns every healthy local backend in openaiEngines order, +// paired with its engine name. It is the fan-out set for the ingress model list +// and the candidate set for this node's own entry in routing. +func (p *Proxy) localBackends() []localEngineTarget { + p.backendMu.RLock() + defer p.backendMu.RUnlock() + out := make([]localEngineTarget, 0, len(openaiEngines)) + for _, e := range openaiEngines { + b, ok := p.backends[e.Name] + if !ok { + continue + } + if u, ok := b.url(); ok { + out = append(out, localEngineTarget{Engine: e.Name, URL: u}) + } } - return &url.URL{Scheme: "http", Host: net.JoinHostPort(host, strconv.Itoa(b.Port))}, true + return out +} + +// localEngineTarget is one healthy local engine: its name and its loopback URL. +type localEngineTarget struct { + Engine string + URL *url.URL } // handlePlain is the plaintext personality: it accepts requests only from @@ -103,17 +151,66 @@ func (p *Proxy) handleClusterIngress(w http.ResponseWriter, r *http.Request) { "client certificate is not a pinned member of this node's cluster") return } - target, ok := p.localBackendTarget() - if !ok { + backends := p.localBackends() + if len(backends) == 0 { writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend", "no local inference backend is available on this node") return } + // A peer aggregating the cluster's inventory must see everything this node + // serves. Forwarding to one engine would hide the other's models and the + // peer would then never route a request this node could have answered, so + // the list is merged locally instead. Still terminal: the fan-out is to this + // node's own loopback engines only. + if r.Method == http.MethodGet && r.URL.Path == "/v1/models" { + p.serveLocalModelList(w, r, backends) + return + } + target, engine, ok := p.ingressTarget(r, backends) + if !ok { + writeIngressError(w, http.StatusServiceUnavailable, "no-model-owner", + "no local inference engine serves the requested model") + return + } slog.Debug("cluster ingress forwarding to local backend", - "peer", peer, "method", r.Method, "path", r.URL.Path, "target", target.Host) + "peer", peer, "engine", engine, "method", r.Method, "path", r.URL.Path, "target", target.Host) p.reverseProxyToLocal(w, r, target) } +// ingressTarget picks which local engine answers a peer's request. Resolution +// order, top down: +// +// 1. the engine that actually serves the requested model, read live from the +// local engines themselves (short-TTL cached) rather than from discovery — +// this node's own entry in the discovery overlay is enriched by the scanner +// and can lag or, on a freshly started node, be absent entirely, which would +// misroute every ingress request; +// 2. the only healthy backend, when there is exactly one — the common case, and +// the answer regardless of what the body names; +// 3. no target, which the caller reports as 503. +// +// The body is buffered and restored, so the reverse proxy still forwards it. +func (p *Proxy) ingressTarget(r *http.Request, backends []localEngineTarget) (*url.URL, string, bool) { + if len(backends) == 1 { + return backends[0].URL, backends[0].Engine, true + } + body, model := bufferBodyAndModel(r) + if body != nil { + r.Body = io.NopCloser(bytes.NewReader(body)) + r.ContentLength = int64(len(body)) + } + if model != "" { + if engine := engineForModel(p.localModelsByEngine(r.Context(), backends), model); engine != "" { + for _, b := range backends { + if b.Engine == engine { + return b.URL, b.Engine, true + } + } + } + } + return nil, "", false +} + // reverseProxyToLocal streams the request to the local engine, preserving // cancellation (the request context is the proxy's root context, so a client // disconnect or shutdown tears down the upstream call and stops generation). @@ -163,3 +260,109 @@ func writeIngressError(w http.ResponseWriter, status int, code, msg string) { } _, _ = w.Write(body) } + +// localModelsTTL bounds how stale the ingress engine attribution may be. Long +// enough that a burst of peer requests costs one loopback query per engine, +// short enough that a model swap is picked up within a few seconds. +const localModelsTTL = 5 * time.Second + +// localModelsCache memoizes the per-engine model ids read from this node's own +// engines, so ingress routing does not pay a loopback round trip per request. +type localModelsCache struct { + mu sync.Mutex + fetched time.Time + byEngine map[string][]string +} + +// localModelsByEngine returns which local engine serves which models, asking the +// engines themselves rather than the discovery overlay. Authoritative for this +// node and cheap: loopback, bounded, and cached for localModelsTTL. An engine +// that fails to answer contributes nothing, so a request for its model falls +// through to the caller's next resolution step. +func (p *Proxy) localModelsByEngine(ctx context.Context, backends []localEngineTarget) map[string][]string { + p.localModels.mu.Lock() + if time.Since(p.localModels.fetched) < localModelsTTL && p.localModels.byEngine != nil { + cached := p.localModels.byEngine + p.localModels.mu.Unlock() + return cached + } + p.localModels.mu.Unlock() + + fresh := make(map[string][]string, len(backends)) + for _, b := range backends { + ids, err := p.fetchModelIDs(ctx, b.URL) + if err != nil { + slog.Debug("local model list unavailable", "engine", b.Engine, "target", b.URL.Host, "err", err) + continue + } + fresh[b.Engine] = ids + } + + p.localModels.mu.Lock() + p.localModels.fetched = time.Now() + p.localModels.byEngine = fresh + p.localModels.mu.Unlock() + return fresh +} + +// fetchModelIDs reads one loopback engine's /v1/models and returns its ids. +func (p *Proxy) fetchModelIDs(ctx context.Context, target *url.URL) ([]string, error) { + records, err := p.fetchModelRecords(ctx, target) + if err != nil { + return nil, err + } + ids := make([]string, 0, len(records)) + for _, rec := range records { + ids = append(ids, rec.key) + } + return ids, nil +} + +// serveLocalModelList answers a peer's GET /v1/models by merging this node's +// own engines' inventories, first engine's metadata winning on a duplicate id. +func (p *Proxy) serveLocalModelList(w http.ResponseWriter, r *http.Request, backends []localEngineTarget) { + seen := make(map[string]bool) + models := make([]json.RawMessage, 0) + ok := false + for _, b := range backends { + records, err := p.fetchModelRecords(r.Context(), b.URL) + if err != nil { + slog.Debug("ingress model list candidate unavailable", "engine", b.Engine, "target", b.URL.Host, "err", err) + continue + } + ok = true + for _, rec := range records { + if !seen[rec.key] { + seen[rec.key] = true + models = append(models, rec.raw) + } + } + } + if !ok { + writeIngressError(w, http.StatusServiceUnavailable, "no-local-backend", + "no local inference backend answered its model list") + return + } + body, err := json.Marshal(struct { + Object string `json:"object"` + Data []json.RawMessage `json:"data"` + }{Object: "list", Data: models}) + if err != nil { + writeIngressError(w, http.StatusInternalServerError, "backend-error", "failed to encode model inventory") + return + } + cors.Apply(w.Header()) + w.Header().Set("Content-Type", "application/json") + w.Header().Set("X-Content-Type-Options", "nosniff") + w.WriteHeader(http.StatusOK) + _, _ = w.Write(body) +} + +// fetchModelRecords reads one loopback engine's /v1/models over the shared plain +// transport. Loopback only: the caller supplies a target from localBackends. +func (p *Proxy) fetchModelRecords(ctx context.Context, target *url.URL) ([]modelListItem, error) { + list := *target + list.Path = "/v1/models" + client := &http.Client{Timeout: modelListClient.Timeout, Transport: p.plainHTTPTransport()} + return fetchModelList(ctx, client, &list) +} diff --git a/services/lmstudio-proxy/multiengine_test.go b/services/lmstudio-proxy/multiengine_test.go new file mode 100644 index 00000000..5d46f577 --- /dev/null +++ b/services/lmstudio-proxy/multiengine_test.go @@ -0,0 +1,358 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "strings" + "testing" + "time" + + "nvpair-shared/noderec" +) + +// dirNode builds a relay DirectoryNode advertising the given engine services on +// one shared proxy port, with per-engine model attribution. +func dirNode(id string, port int, byEngine map[string][]string, services ...noderec.ServiceKey) noderec.DirectoryNode { + svc := make(map[noderec.ServiceKey]noderec.ServiceStatus, len(services)) + for _, s := range services { + svc[s] = noderec.ServiceStatus{Port: port} + } + return noderec.DirectoryNode{ + Name: id, + HostUUID: id, + IP: "192.0.2.10", + IPs: []string{"192.0.2.10"}, + Services: svc, + ModelsByEngine: byEngine, + } +} + +// TestSubscribedNodeCarriesEveryOpenAIEngine proves one peer advertising both +// lm and vl projects to a single routable node — they name the same proxy port +// — whose inventory is the union and whose attribution is kept per engine. +func TestSubscribedNodeCarriesEveryOpenAIEngine(t *testing.T) { + n, ok := subscribedToNode(dirNode("peer", 1234, map[string][]string{ + "lmstudio": {"qwen2.5-7b"}, + "vllm": {"Qwen/Qwen3-8B"}, + "ollama": {"llama3.2:1b"}, + }, noderec.ServiceLMStudio, noderec.ServiceVLLM)) + if !ok { + t.Fatal("peer advertising lm and vl was dropped") + } + if n.Port != 1234 { + t.Errorf("port = %d, want the shared proxy port 1234", n.Port) + } + want := []string{"Qwen/Qwen3-8B", "qwen2.5-7b"} + if strings.Join(n.Models, ",") != strings.Join(want, ",") { + t.Errorf("models = %v, want the sorted OpenAI union %v (Ollama-only models are not ours)", n.Models, want) + } + if got := engineForModel(n.ModelsByEngine, "Qwen/Qwen3-8B"); got != "vllm" { + t.Errorf("owner of Qwen/Qwen3-8B = %q, want vllm", got) + } + if got := engineForModel(n.ModelsByEngine, "qwen2.5-7b"); got != "lmstudio" { + t.Errorf("owner of qwen2.5-7b = %q, want lmstudio", got) + } + if got := engineForModel(n.ModelsByEngine, "llama3.2:1b"); got != "" { + t.Errorf("an Ollama-only model must not resolve to an OpenAI engine, got %q", got) + } +} + +// TestSubscribedNodeVLLMOnly proves a node that runs only vLLM is routable +// through this proxy, which is the whole point of the second service key. +func TestSubscribedNodeVLLMOnly(t *testing.T) { + n, ok := subscribedToNode(dirNode("vpeer", 1234, map[string][]string{ + "vllm": {"Qwen/Qwen3-8B"}, + }, noderec.ServiceVLLM)) + if !ok { + t.Fatal("vLLM-only peer was dropped") + } + if len(n.Models) != 1 || n.Models[0] != "Qwen/Qwen3-8B" { + t.Errorf("models = %v", n.Models) + } +} + +// TestSubscribedNodeWithNoOpenAIEngineIsDropped proves a node running only +// Ollama is not a candidate here, however many models it advertises. +func TestSubscribedNodeWithNoOpenAIEngineIsDropped(t *testing.T) { + if _, ok := subscribedToNode(dirNode("opeer", 11434, map[string][]string{ + "ollama": {"llama3.2:1b"}, + }, noderec.ServiceOllama)); ok { + t.Fatal("an Ollama-only node must not be an OpenAI routing candidate") + } +} + +// TestEngineForModelTieBreakIsDeterministic proves a model id served by both +// local engines resolves the same way every time, following openaiEngines order +// rather than map iteration order. +func TestEngineForModelTieBreakIsDeterministic(t *testing.T) { + byEngine := map[string][]string{ + "lmstudio": {"Qwen/Qwen3-8B"}, + "vllm": {"Qwen/Qwen3-8B"}, + } + for range 50 { + if got := engineForModel(byEngine, "Qwen/Qwen3-8B"); got != "lmstudio" { + t.Fatalf("tie-break = %q, want the first listed engine lmstudio", got) + } + } +} + +// TestMixedEngineCandidatesRouteToTheModelOwner proves eligibility and workload +// attribution both follow the per-engine inventory: a request for a vLLM-only +// model reaches the vLLM node and is tagged vllm, and vice versa. +func TestMixedEngineCandidatesRouteToTheModelOwner(t *testing.T) { + lmNode := nodeForModel(t, "lm-node", "http://192.0.2.11:1234", "qwen2.5-7b") + vlNode := nodeForModel(t, "vl-node", "http://192.0.2.12:8000", "Qwen/Qwen3-8B") + vlNode.Engine = "vllm" + vlNode.ModelsByEngine = map[string][]string{"vllm": {"Qwen/Qwen3-8B"}} + + disc := NewDiscovery() + disc.AddManual(lmNode) + disc.AddManual(vlNode) + p := testProxy(disc, 1235) + + for _, tc := range []struct{ model, wantID, wantEngine string }{ + {"qwen2.5-7b", "lm-node", "lmstudio"}, + {"Qwen/Qwen3-8B", "vl-node", "vllm"}, + } { + cands := p.resolveCandidates(tc.model) + if len(cands) != 1 { + t.Fatalf("%s: %d candidates, want only the owner", tc.model, len(cands)) + } + if cands[0].id != tc.wantID { + t.Errorf("%s routed to %q, want %q", tc.model, cands[0].id, tc.wantID) + } + if got := candidateEngine(cands[0]); got != tc.wantEngine { + t.Errorf("%s tagged %q, want %q", tc.model, got, tc.wantEngine) + } + } +} + +// TestManualNodeIsKeyedPerEngine proves one host added for both engines keeps +// two entries — LM Studio and vLLM sit on different ports, so collapsing them +// would lose one — and that removing one leaves the other routable. +func TestManualNodeIsKeyedPerEngine(t *testing.T) { + lm := nodeForModel(t, "host", "http://192.0.2.20:1234", "qwen2.5-7b") + vl := nodeForModel(t, "host", "http://192.0.2.20:8000", "Qwen/Qwen3-8B") + vl.Engine = "vllm" + vl.ModelsByEngine = map[string][]string{"vllm": {"Qwen/Qwen3-8B"}} + + disc := NewDiscovery() + if !disc.AddManual(lm) || !disc.AddManual(vl) { + t.Fatal("both engine entries should be new") + } + if got := len(disc.Nodes()); got != 2 { + t.Fatalf("nodes = %d, want one entry per engine", got) + } + if !disc.RemoveManual("lmstudio", "host") { + t.Fatal("removing the LM Studio entry should report removed") + } + if !disc.IsManual("host") { + t.Error("the node is still manual while its vLLM entry remains") + } + nodes := disc.Nodes() + if len(nodes) != 1 || nodes[0].Port != 8000 { + t.Fatalf("surviving node = %+v, want the vLLM entry on 8000", nodes) + } + if !disc.RemoveManual("vllm", "host") { + t.Fatal("removing the vLLM entry should report removed") + } + if disc.IsManual("host") { + t.Error("the node should be gone once its last engine entry is removed") + } +} + +// TestRouteKeyIsEngineQualifiedForManualNodes proves the reachability cache +// cannot carry one engine's confirmed address over to the other engine on the +// same host, while attribution stays keyed by the bare node id. +func TestRouteKeyIsEngineQualifiedForManualNodes(t *testing.T) { + lm := Node{ID: "host", Engine: "lmstudio"} + vl := Node{ID: "host", Engine: "vllm"} + if lm.routeKey() == vl.routeKey() { + t.Error("two engines on one host must not share a reachability key") + } + peer := Node{ID: "host"} + if peer.routeKey() != "host" { + t.Errorf("a relay peer's route key = %q, want the bare node id", peer.routeKey()) + } +} + +// TestIngressForwardsToTheEngineThatOwnsTheModel proves a peer's inference +// request lands on whichever local OpenAI engine actually serves the model, +// read from the engines themselves rather than from discovery. +func TestIngressForwardsToTheEngineThatOwnsTheModel(t *testing.T) { + var lmHits, vlHits int + lm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"qwen2.5-7b"}]}`)) + return + } + lmHits++ + _, _ = w.Write([]byte(`{"ok":"lm"}`)) + })) + defer lm.Close() + vl := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/v1/models" { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"Qwen/Qwen3-8B"}]}`)) + return + } + vlHits++ + _, _ = w.Write([]byte(`{"ok":"vl"}`)) + })) + defer vl.Close() + + p := testProxy(NewDiscovery(), 1235) + setBackendFromURL(t, p, "lmstudio", lm.URL) + setBackendFromURL(t, p, "vllm", vl.URL) + + backends := p.localBackends() + if len(backends) != 2 { + t.Fatalf("localBackends = %d, want both engines", len(backends)) + } + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"Qwen/Qwen3-8B"}`)) + target, engine, ok := p.ingressTarget(req, backends) + if !ok || engine != "vllm" { + t.Fatalf("ingress engine = %q ok=%v, want vllm", engine, ok) + } + p.reverseProxyToLocal(httptest.NewRecorder(), req, target) + if vlHits != 1 || lmHits != 0 { + t.Errorf("hits lm=%d vl=%d, want the request on vLLM only", lmHits, vlHits) + } + + // The body must survive the peek that found the model. + req2 := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{"model":"qwen2.5-7b"}`)) + target2, engine2, ok := p.ingressTarget(req2, backends) + if !ok || engine2 != "lmstudio" { + t.Fatalf("ingress engine = %q ok=%v, want lmstudio", engine2, ok) + } + var seen string + echo := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + buf := make([]byte, 64) + n, _ := r.Body.Read(buf) + seen = string(buf[:n]) + })) + defer echo.Close() + _ = target2 + setBackendFromURL(t, p, "lmstudio", echo.URL) + p.reverseProxyToLocal(httptest.NewRecorder(), req2, mustURL(t, echo.URL)) + if !strings.Contains(seen, "qwen2.5-7b") { + t.Errorf("forwarded body = %q, want the original request body", seen) + } +} + +// TestIngressModelListMergesEveryLocalEngine proves a peer aggregating this +// node's inventory sees both engines. Forwarding to one would hide the other's +// models and the peer would never route work this node could have run. +func TestIngressModelListMergesEveryLocalEngine(t *testing.T) { + lm := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"qwen2.5-7b"}]}`)) + })) + defer lm.Close() + vl := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"object":"list","data":[{"id":"Qwen/Qwen3-8B"}]}`)) + })) + defer vl.Close() + + p := testProxy(NewDiscovery(), 1235) + setBackendFromURL(t, p, "lmstudio", lm.URL) + setBackendFromURL(t, p, "vllm", vl.URL) + + rec := httptest.NewRecorder() + p.serveLocalModelList(rec, httptest.NewRequest(http.MethodGet, "/v1/models", nil), p.localBackends()) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d", rec.Code) + } + var envelope struct { + Data []struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &envelope); err != nil { + t.Fatal(err) + } + got := map[string]bool{} + for _, d := range envelope.Data { + got[d.ID] = true + } + if !got["qwen2.5-7b"] || !got["Qwen/Qwen3-8B"] { + t.Errorf("aggregated ids = %v, want both engines' models", got) + } +} + +// TestUnhealthyBackendDoesNotDisturbTheOther is the regression guard for the +// backends map: clearing one engine must leave the other routable. +func TestUnhealthyBackendDoesNotDisturbTheOther(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + p.setLocalBackend(localBackend{Engine: "lmstudio", Port: 1234, Healthy: true}) + p.setLocalBackend(localBackend{Engine: "vllm", Port: 8000, Healthy: true}) + + p.setLocalBackend(localBackend{Engine: "vllm", Port: 8000, Healthy: false}) + if _, ok := p.localBackendTarget("vllm"); ok { + t.Error("an unhealthy vLLM backend must not be a target") + } + if _, ok := p.localBackendTarget("lmstudio"); !ok { + t.Error("clearing vLLM must not disturb LM Studio") + } + if got := len(p.localBackends()); got != 1 { + t.Errorf("healthy backends = %d, want 1", got) + } + if _, ok := p.localBackendTarget("ollama"); ok { + t.Error("an engine this proxy never fronts must have no target") + } +} + +// TestOpenAIEnginesCoverEveryServiceKey guards the two halves of the engine +// table against drifting apart: every engine has a discovery key and a name, +// and the subscription list is exactly those keys. +func TestOpenAIEnginesCoverEveryServiceKey(t *testing.T) { + if len(subscribedServices()) != len(openaiEngines) { + t.Fatalf("subscribed services = %v, want one per engine", subscribedServices()) + } + for _, e := range openaiEngines { + if e.Service == "" || e.Name == "" { + t.Errorf("incomplete engine binding: %+v", e) + } + if !isOpenAIEngine(e.Name) { + t.Errorf("%q is in the table but not recognized", e.Name) + } + } + if isOpenAIEngine("ollama") { + t.Error("ollama is fronted by its own proxy, not this one") + } +} + +// setBackendFromURL registers an httptest server as one engine's local backend. +func setBackendFromURL(t *testing.T, p *Proxy, engine, serverURL string) { + t.Helper() + u := mustURL(t, serverURL) + host, portStr, err := net.SplitHostPort(u.Host) + if err != nil { + t.Fatalf("split %q: %v", u.Host, err) + } + port, err := strconv.Atoi(portStr) + if err != nil { + t.Fatalf("port %q: %v", portStr, err) + } + p.setLocalBackend(localBackend{Engine: engine, Host: host, Port: port, Healthy: true}) + // Each registration changes what the engines serve, so drop the memoized + // attribution rather than waiting out its TTL. + p.localModels.mu.Lock() + p.localModels.byEngine = nil + p.localModels.fetched = time.Time{} + p.localModels.mu.Unlock() +} + +func mustURL(t *testing.T, raw string) *url.URL { + t.Helper() + u, err := url.Parse(raw) + if err != nil { + t.Fatalf("parse %q: %v", raw, err) + } + return u +} diff --git a/services/lmstudio-proxy/overlay_test.go b/services/lmstudio-proxy/overlay_test.go new file mode 100644 index 00000000..d1a0cae5 --- /dev/null +++ b/services/lmstudio-proxy/overlay_test.go @@ -0,0 +1,90 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Routing to a node reachable only over an encrypted overlay such as a Tailscale +// tailnet. Such a node is never discovered — a tailnet carries no multicast — so +// it arrives as a manually added address, and that address is a 100.64/10 CGNAT +// literal, an IPv6 ULA, or a MagicDNS name. netpick demotes all three relative to +// a LAN literal; none of them may be excluded, or the node has nowhere to be +// dialed. + +package main + +import ( + "reflect" + "testing" +) + +func TestNodeCandidates_OverlayOnlyNodeIsRoutable(t *testing.T) { + tests := []struct { + name string + node Node + want []string + }{ + { + name: "cgnat only", + node: Node{Addresses: []string{"100.101.102.103"}, Port: 1234}, + want: []string{"100.101.102.103:1234"}, + }, + { + name: "ipv6 ula only", + node: Node{Addresses: []string{"fd7a:115c:a1e0::1701:b2c3"}, Port: 1234}, + want: []string{"[fd7a:115c:a1e0::1701:b2c3]:1234"}, + }, + { + name: "magicdns name only", + node: Node{Addresses: []string{"gpu-box.tail1234.ts.net"}, Port: 1234}, + want: []string{"gpu-box.tail1234.ts.net:1234"}, + }, + { + name: "magicdns name carried as the node's canonical ip= TXT", + node: Node{ + TXT: []string{"ip=gpu-box.tail1234.ts.net"}, + Addresses: []string{"gpu-box.tail1234.ts.net"}, + Port: 1234, + }, + want: []string{"gpu-box.tail1234.ts.net:1234"}, + }, + { + name: "lan literal still preferred when the node has both", + node: Node{Addresses: []string{"100.101.102.103", "192.0.2.10"}, Port: 1234}, + want: []string{"192.0.2.10:1234", "100.101.102.103:1234"}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := nodeCandidates(tc.node); !reflect.DeepEqual(got, tc.want) { + t.Errorf("nodeCandidates = %v, want %v", got, tc.want) + } + }) + } +} + +// A manual entry that names this host's own tailnet address must resolve to +// loopback rather than being dialed back around the tunnel to ourselves. The +// local-address set comes from netmon, which enumerates every interface without +// filtering, so an overlay adapter's address is in it exactly like a LAN one — +// this pins that, because a set built from the *publishing* picker instead would +// omit an unproven overlay address and let the entry loop. +func TestIsLocalAddress_CountsThisHostsOverlayAddresses(t *testing.T) { + const ourTailnetIP = "100.64.7.7" + + localAddrsMu.Lock() + prev := localAddrs + localAddrs = map[string]bool{"192.0.2.5": true, ourTailnetIP: true} + localAddrsMu.Unlock() + t.Cleanup(func() { + localAddrsMu.Lock() + localAddrs = prev + localAddrsMu.Unlock() + }) + + if !isLocalAddress(ourTailnetIP) { + t.Fatalf("isLocalAddress(%q) = false, want true", ourTailnetIP) + } + got := nodeCandidates(Node{Addresses: []string{ourTailnetIP}, Port: 1234}) + want := []string{"127.0.0.1:1234"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("nodeCandidates for our own tailnet address = %v, want %v", got, want) + } +} diff --git a/services/lmstudio-proxy/priority_test.go b/services/lmstudio-proxy/priority_test.go index 6661b30b..13d9c8f7 100644 --- a/services/lmstudio-proxy/priority_test.go +++ b/services/lmstudio-proxy/priority_test.go @@ -47,7 +47,7 @@ func contains(hay, needle string) bool { // probe) and it never trips the loopback rewrite or self-forward guard. The // octet is derived from the id's first byte so each id gets a distinct valid IP. func prNode(id string) Node { - return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234} + return Node{ID: id, Addresses: []string{"192.0.2." + strconv.Itoa(int(id[0]))}, Port: 1234, Engine: "lmstudio"} } // prProxy returns a proxy whose discovery holds the given node ids. diff --git a/services/lmstudio-proxy/proxy.go b/services/lmstudio-proxy/proxy.go index 6e619e77..7f4a8c99 100644 --- a/services/lmstudio-proxy/proxy.go +++ b/services/lmstudio-proxy/proxy.go @@ -21,6 +21,7 @@ import ( "net/url" "sort" "strconv" + "strings" "sync" "sync/atomic" "time" @@ -138,16 +139,19 @@ const ( workloadCompletedMethod = "workload:completed" workloadErroredMethod = "workload:errored" - // workloadEngine is the opaque engine identifier carried in every - // workload this proxy produces. This proxy only ever fronts LM Studio. - workloadEngine = "lmstudio" + // fallbackWorkloadEngine is the opaque engine identifier used when a + // forwarded request could not be attributed to one of this proxy's engines — + // a peer whose per-engine model attribution has not arrived yet. It is the + // first engine in openaiEngines, which is also the tie-break order routing + // uses, so the tag matches what the request was routed as. + fallbackWorkloadEngine = "lmstudio" ) // inferenceEndpoints is the set of request paths that count as cluster // workloads. Health checks, model listings (/v1/models), and other control // traffic are deliberately excluded so we don't flood the cluster with -// non-inference noise. LM Studio serves the OpenAI-compatible API, so these -// are the OpenAI inference routes. +// non-inference noise. Every engine behind this proxy serves the +// OpenAI-compatible API, so these are the OpenAI inference routes. var inferenceEndpoints = map[string]bool{ "/v1/chat/completions": true, "/v1/completions": true, @@ -335,12 +339,19 @@ type Proxy struct { // only loopback-plaintext local routing. Read-only after startup. mesh *clustertrust.Mesh - // backendMu guards backend, the explicit loopback engine the cluster mTLS - // ingress forwards to. The broker sets/clears it via node/set-local-backend; - // it is never sourced from discovery, so an ingress request can only ever - // reach this node's own local engine and can never be re-routed to a peer. + // backendMu guards backends, this node's own loopback engines keyed by + // engine-manager name. The broker sets/clears each via + // node/set-local-backend; they are never sourced from discovery, so an + // ingress request can only ever reach this node's own local engines and can + // never be re-routed to a peer. One node may run several at once, so each is + // set and cleared independently. backendMu sync.RWMutex - backend localBackend + backends map[string]localBackend + + // localModels caches which of this node's own engines serves which models, + // read from the engines themselves. It is what the mTLS ingress uses to pick + // the engine for a peer's request. See localModelsByEngine. + localModels localModelsCache selectedMu sync.RWMutex selectedID string @@ -475,12 +486,14 @@ func (p *Proxy) Run(ctx context.Context) error { return fmt.Errorf("failed to send ready notification: %w", err) } - // Routing targets come from the broker's discovery relay. Subscribe - // for lm nodes; they arrive as discovery:nodes snapshots (handled in - // handleMessage), each replacing the subscribed overlay. Non-fatal: if the - // parent isn't a relay-aware broker the proxy still routes to manual nodes. - slog.Debug("subscribing to discovery relay for routing targets", "service", string(noderec.ServiceLMStudio)) - if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: []noderec.ServiceKey{noderec.ServiceLMStudio}}); err != nil { + // Routing targets come from the broker's discovery relay. Subscribe for every + // OpenAI-compatible engine service (lm, vl); they arrive as discovery:nodes + // snapshots (handled in handleMessage), each replacing the subscribed + // overlay. Non-fatal: if the parent isn't a relay-aware broker the proxy + // still routes to manual nodes. + services := subscribedServices() + slog.Debug("subscribing to discovery relay for routing targets", "services", services) + if err := p.codec.Notify(noderec.MethodSubscribe, noderec.SubscribeParams{Services: services}); err != nil { slog.Warn("failed to subscribe to discovery relay", "err", err) } @@ -697,8 +710,16 @@ func (p *Proxy) emitWorkload(method string, w Workload) { // pinned to that peer's exact server cert. Empty peerUUID means a plain-HTTP // dial — the local backend (self) or an explicit manual node. type candidate struct { - id string - url *url.URL + id string + url *url.URL + // routeKey keys the reachability chooser. It equals id for a relay peer and + // is engine-qualified for a manual node, so the manual entries for one host's + // several engines never share a confirmed address. See Node.routeKey. + routeKey string + // engine is the OpenAI engine on that node this request is routed to. It is + // what the emitted workload is tagged with, so a dual-engine node's work is + // attributed to the engine that actually ran it. + engine string peerUUID string } @@ -828,12 +849,6 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidate target.Path = r.URL.Path target.RawPath = r.URL.RawPath target.RawQuery = r.URL.RawQuery - upstream, err := http.NewRequestWithContext(r.Context(), http.MethodGet, target.String(), nil) - if err != nil { - results[i].err = err - continue - } - upstream.Header.Set("Accept", "application/json") // A cluster-peer candidate is queried over mTLS to its promoted proxy; // self/manual candidates use the shared plain client. @@ -843,57 +858,18 @@ func (p *Proxy) serveModelList(w http.ResponseWriter, r *http.Request, candidate } wg.Add(1) - go func(i int, cand candidate, req *http.Request, client *http.Client) { + go func(i int, cand candidate, target url.URL, client *http.Client) { defer wg.Done() - resp, err := client.Do(req) + items, err := fetchModelList(r.Context(), client, &target) if err != nil { - p.targets.Forget(cand.id) - results[i].err = err - return - } - defer resp.Body.Close() - if resp.StatusCode != http.StatusOK { - results[i].err = fmt.Errorf("upstream returned %s", resp.Status) - return - } - body, err := io.ReadAll(io.LimitReader(resp.Body, maxModelListBytes+1)) - if err != nil { - results[i].err = err - return - } - if len(body) > maxModelListBytes { - results[i].err = fmt.Errorf("model list exceeds %d bytes", maxModelListBytes) - return - } - var envelope struct { - Data *[]json.RawMessage `json:"data"` - } - if err := json.Unmarshal(body, &envelope); err != nil { + if isTransportError(err) { + p.targets.Forget(cand.routeKey) + } results[i].err = err return } - if envelope.Data == nil { - results[i].err = fmt.Errorf("upstream response has no data array") - return - } - models := *envelope.Data - items := make([]modelListItem, 0, len(models)) - for _, raw := range models { - var identity struct { - ID string `json:"id"` - } - if err := json.Unmarshal(raw, &identity); err != nil { - results[i].err = fmt.Errorf("invalid model record: %w", err) - return - } - if identity.ID == "" { - results[i].err = fmt.Errorf("model record has no id") - return - } - items = append(items, modelListItem{key: identity.ID, raw: raw}) - } results[i] = modelListResult{items: items, ok: true} - }(i, cand, upstream, client) + }(i, cand, target, client) } wg.Wait() @@ -1042,7 +1018,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) { wl = &Workload{ ID: reqID, Model: model, - Engine: workloadEngine, + Engine: candidateEngine(candidates[0]), RunID: p.runID, State: "running", ScheduledOn: candidates[0].id, @@ -1213,7 +1189,7 @@ func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) { // can fail over to another of its published addresses // (multi-homed peer). The in-request failover below moves on to // the next node. - p.targets.Forget(cand.id) + p.targets.Forget(cand.routeKey) if !last { // Transport/dial error with candidates left: fail over. retry = true @@ -1369,7 +1345,9 @@ func (p *Proxy) resolveCandidates(model string) []candidate { out := make([]candidate, 0, len(nodes)) // Dedup by resolved backend host: the same physical node can appear under two // IDs (e.g. a manually-added entry and its relay-discovered record), and - // routing to the same engine twice is wasteful. + // routing to the same engine twice is wasteful. It is also what collapses a + // peer that advertises several engine keys back into one candidate: every + // key on a peer names that peer's single proxy port. seenHost := make(map[string]bool, len(nodes)) // placed tracks node IDs already considered so the priority-ordered and // fallback passes don't reconsider one (scheduler ordering). @@ -1383,19 +1361,23 @@ func (p *Proxy) resolveCandidates(model string) []candidate { if u == nil { return } + engine := p.nodeEngineFor(n, model) peerUUID := "" switch { case isSelfTarget(u, selfPort): - // Our own advertised endpoint (lm now points at this proxy). Serve - // it from the explicit local backend — the loopback engine — rather - // than dialing our own mTLS ingress, which would recurse. Ranking - // still used this node's real (discovered) model list above. - lb, ok := p.localBackendTarget() - if !ok { - slog.Debug("resolveCandidates: no local backend for self", "node_id", n.ID) - return + // Our own advertised endpoint (the engine keys now point at this + // proxy). Serve it from the explicit local backends — the loopback + // engines — rather than dialing our own mTLS ingress, which would + // recurse. Ranking still used this node's real (discovered) model + // list above. With no routing model there is nothing to attribute, so + // every healthy local engine becomes its own candidate and the + // aggregated model list covers them all. + for _, local := range p.selfTargets(n, model, engine) { + out = appendCandidate(out, seenHost, candidate{ + id: n.ID, routeKey: n.routeKey(), engine: local.Engine, url: local.URL, + }, selfPort) } - u = lb + return case p.mesh.HasPin(n.ClusterUUID): // A pinned cluster peer: reach it only over mTLS to its promoted // proxy (the lm port now advertises the proxy, not the engine). @@ -1419,21 +1401,13 @@ func (p *Proxy) resolveCandidates(model string) []candidate { "node_id", n.ID, "cluster_uuid", n.ClusterUUID) return } - // Defensive: the local backend must never resolve back to this proxy. - if isSelfTarget(u, selfPort) { - slog.Debug("resolveCandidates: skipping self-target node", - "node_id", n.ID, "target", u.Host, "self_port", selfPort) - return - } - if seenHost[u.Host] { - return - } - seenHost[u.Host] = true - out = append(out, candidate{ + out = appendCandidate(out, seenHost, candidate{ id: n.ID, + routeKey: n.routeKey(), + engine: engine, url: u, peerUUID: peerUUID, - }) + }, selfPort) } // Capability has already been enforced. An eligible explicit selection wins, @@ -1585,7 +1559,7 @@ func (p *Proxy) targetURL(n Node) *url.URL { if len(candidates) == 0 { return nil } - host := p.targets.Prefer(n.ID, candidates) + host := p.targets.Prefer(n.routeKey(), candidates) return &url.URL{Scheme: "http", Host: host} } @@ -1602,15 +1576,14 @@ func nodeCandidates(n Node) []string { port := strconv.Itoa(n.Port) sorted := netpick.Candidates(n.TXT, n.Addresses) if len(sorted) == 0 { - // A non-IP entry (a .local hostname) that netpick cannot parse. - hosts := n.Addresses - if len(hosts) == 0 { - if n.Host == "" { - return nil - } - hosts = []string{n.Host} + // The node published no dialable address of its own. Its discovery host + // name is a different source, not a re-reading of the same one — netpick + // admits IP literals and DNS names alike, so an address list holding a + // MagicDNS or .local name has already been ranked above. + if n.Host == "" { + return nil } - sorted = append([]string(nil), hosts...) + sorted = []string{n.Host} } seen := make(map[string]bool, len(sorted)) @@ -1826,8 +1799,26 @@ func upstreamUnreachableID(nodeID string) string { // LM Studio port the broker's engine poller registered, not the proxy's listen // port). func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { - svc, ok := n.Services[noderec.ServiceLMStudio] - if !ok || n.IP == "" { + if n.IP == "" { + return Node{}, false + } + // A node may advertise several OpenAI engine keys at once. They all carry the + // same value — the port of that node's one OpenAI-compatible proxy — so the + // node projects to a single routable entry whose per-engine model attribution + // records which engine actually serves what. + port := 0 + byEngine := make(map[string][]string, len(openaiEngines)) + for _, e := range openaiEngines { + svc, ok := n.Services[e.Service] + if !ok { + continue + } + if port == 0 { + port = svc.Port + } + byEngine[e.Name] = append([]string(nil), n.EngineModels(e.Name)...) + } + if port == 0 { return Node{}, false } // Key routing by the stable per-host UUID, not the hostname: candidate ids, @@ -1839,7 +1830,7 @@ func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { return Node{ ID: n.HostUUID, Host: n.Name, - Port: svc.Port, + Port: port, // The node's whole ranked address list, not just its canonical one: a // multi-homed peer's best address from its own vantage point may be a // direct-connect link this host cannot reach, and routing needs somewhere @@ -1848,11 +1839,12 @@ func subscribedToNode(n noderec.DirectoryNode) (Node, bool) { TXT: n.AddressTXT(), IP: n.IP, ClusterUUID: n.ClusterUUID, - // Filter on this node's LM Studio models only, not the cross-engine union, - // so a model a dual-engine node serves solely via Ollama isn't accepted as - // an LM Studio owner here (falls back to the union for a peer that sends - // no attribution — see DirectoryNode.EngineModels). - Models: append([]string(nil), n.EngineModels("lmstudio")...), + // Filter on this node's OpenAI-engine models only, not the cross-engine + // union, so a model a multi-engine node serves solely via Ollama isn't + // accepted as an owner here (each key falls back to the union for a peer + // that sends no attribution — see DirectoryNode.EngineModels). + Models: unionModels(byEngine), + ModelsByEngine: byEngine, }, true } @@ -1981,29 +1973,48 @@ func (p *Proxy) handleMessage(msg *Message) { p.codec.RespondError(msg.ID, -32602, "id, port, and at least one address are required") return } + // engine names which OpenAI engine on that host this address serves. + // LM Studio, vLLM and SGLang listen on different ports, so a node running + // more than one is added once per engine and the entries must not collide. + if !isOpenAIEngine(node.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) + return + } + // A manual node's whole inventory belongs to the engine it was added for. + node.ModelsByEngine = map[string][]string{node.Engine: append([]string(nil), node.Models...)} added := p.discovery.AddManual(node) if err := p.codec.Respond(msg.ID, map[string]bool{"added": added}); err != nil { log.Printf("failed to respond to node/add-manual: %v", err) } if added { - log.Printf("manual node added: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + log.Printf("manual node added: %s/%s (%s:%d)", node.Engine, node.ID, node.Addresses[0], node.Port) p.codec.Notify("node/discovered", node.withPrimaryIP()) } else { - log.Printf("manual node updated: %s (%s:%d)", node.ID, node.Addresses[0], node.Port) + log.Printf("manual node updated: %s/%s (%s:%d)", node.Engine, node.ID, node.Addresses[0], node.Port) p.codec.Notify("node/updated", node.withPrimaryIP()) } case "node/remove-manual": - var params SelectParams + var params struct { + ID string `json:"id"` + Engine string `json:"engine"` + } if err := json.Unmarshal(msg.Params, ¶ms); err != nil { - p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\"}") + p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"id\": \"...\", \"engine\": \"...\"}") + return + } + if !isOpenAIEngine(params.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) return } - removed := p.discovery.RemoveManual(params.ID) + removed := p.discovery.RemoveManual(params.Engine, params.ID) if err := p.codec.Respond(msg.ID, map[string]bool{"removed": removed}); err != nil { log.Printf("failed to respond to node/remove-manual: %v", err) } - if removed { + // The node itself is only gone once its last engine's entry is dropped; + // until then it is still a routing target for the other engine, so the + // selection and the node/removed event must not fire early. + if removed && !p.discovery.IsManual(params.ID) { log.Printf("manual node removed: %s", params.ID) p.selectedMu.Lock() if p.selectedID == params.ID { @@ -2022,6 +2033,10 @@ func (p *Proxy) handleMessage(msg *Message) { p.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"engine\",\"host\",\"port\",\"healthy\"}") return } + if !isOpenAIEngine(b.Engine) { + p.codec.RespondError(msg.ID, -32602, "engine must be one of "+strings.Join(engineNames(), ", ")) + return + } p.setLocalBackend(b) slog.Info("local backend updated", "engine", b.Engine, "host", b.Host, "port", b.Port, "healthy", b.Healthy) if err := p.codec.Respond(msg.ID, map[string]bool{"ok": true}); err != nil { diff --git a/services/lmstudio-proxy/sglang_test.go b/services/lmstudio-proxy/sglang_test.go new file mode 100644 index 00000000..6d71f5ae --- /dev/null +++ b/services/lmstudio-proxy/sglang_test.go @@ -0,0 +1,275 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "io" + "slices" + "strconv" + "strings" + "testing" + + "nvpair-shared/noderec" +) + +// sglangModel is the model id a real SGLang reports. Its --served-model-name +// defaults to --model-path verbatim, so an id can be a bare local directory — +// leading slash and all — rather than the Hugging Face "org/name" shape vLLM +// installs usually show. Nothing here may assume either. +const sglangModel = "/models/my-model" + +// TestSGLangIsAFrontedEngine proves the engine table carries SGLang on both of +// its halves: the engine-manager name that gates node/add-manual and tags +// workloads, and the discovery key the proxy subscribes to. A name accepted +// without its key would create entries no peer could ever be discovered for. +func TestSGLangIsAFrontedEngine(t *testing.T) { + if !isOpenAIEngine("sglang") { + t.Error("sglang must be recognized as an engine this proxy fronts") + } + if !slices.Contains(subscribedServices(), noderec.ServiceSGLang) { + t.Errorf("subscribed services = %v, want sg among them", subscribedServices()) + } + if got := engineNames(); len(got) != 3 || got[2] != "sglang" { + t.Errorf("engineNames = %v, want sglang last in resolution order", got) + } +} + +// TestSubscribedNodeSGLangOnly proves a peer that runs only SGLang is routable +// through this proxy, and that its inventory is read off the sglang attribution +// rather than any other engine's. +func TestSubscribedNodeSGLangOnly(t *testing.T) { + n, ok := subscribedToNode(dirNode("sgpeer", 1234, map[string][]string{ + "sglang": {sglangModel}, + "ollama": {"llama3.2:1b"}, + }, noderec.ServiceSGLang)) + if !ok { + t.Fatal("SGLang-only peer was dropped") + } + if n.Port != 1234 { + t.Errorf("port = %d, want the OpenAI proxy port 1234", n.Port) + } + if len(n.Models) != 1 || n.Models[0] != sglangModel { + t.Errorf("models = %v, want only the SGLang inventory", n.Models) + } + if got := engineForModel(n.ModelsByEngine, sglangModel); got != "sglang" { + t.Errorf("owner of %q = %q, want sglang", sglangModel, got) + } +} + +// TestSubscribedNodeCarriesAllThreeOpenAIEngines proves a peer advertising lm, +// vl and sg still projects to one routable node — all three name that peer's +// single proxy port — whose Models is the union and whose attribution stays per +// engine. +func TestSubscribedNodeCarriesAllThreeOpenAIEngines(t *testing.T) { + n, ok := subscribedToNode(dirNode("peer", 1234, map[string][]string{ + "lmstudio": {"qwen2.5-7b"}, + "vllm": {"Qwen/Qwen3-8B"}, + "sglang": {sglangModel}, + }, noderec.ServiceLMStudio, noderec.ServiceVLLM, noderec.ServiceSGLang)) + if !ok { + t.Fatal("peer advertising lm, vl and sg was dropped") + } + if n.Port != 1234 { + t.Errorf("port = %d, want the one shared proxy port 1234", n.Port) + } + want := []string{sglangModel, "Qwen/Qwen3-8B", "qwen2.5-7b"} + if strings.Join(n.Models, ",") != strings.Join(want, ",") { + t.Errorf("models = %v, want the sorted three-engine union %v", n.Models, want) + } + for model, engine := range map[string]string{ + "qwen2.5-7b": "lmstudio", + "Qwen/Qwen3-8B": "vllm", + sglangModel: "sglang", + } { + if got := engineForModel(n.ModelsByEngine, model); got != engine { + t.Errorf("owner of %q = %q, want %q", model, got, engine) + } + } +} + +// TestUnionModelsAcrossThreeEngines proves the aggregated inventory covers every +// fronted engine and de-duplicates a model more than one of them serves. A +// missing engine here is a node that silently advertises less than it can run. +func TestUnionModelsAcrossThreeEngines(t *testing.T) { + got := unionModels(map[string][]string{ + "lmstudio": {"shared-model", "qwen2.5-7b"}, + "vllm": {"shared-model", "Qwen/Qwen3-8B"}, + "sglang": {"shared-model", sglangModel}, + "ollama": {"llama3.2:1b"}, + }) + want := []string{sglangModel, "Qwen/Qwen3-8B", "qwen2.5-7b", "shared-model"} + if !slices.Equal(got, want) { + t.Errorf("unionModels = %v, want %v (Ollama is not ours)", got, want) + } +} + +// TestEngineTieBreakOrderWithThreeEngines proves a model id every OpenAI engine +// on a node claims resolves by openaiEngines order — lmstudio, then vllm, then +// sglang — every time, rather than by map iteration. +func TestEngineTieBreakOrderWithThreeEngines(t *testing.T) { + byEngine := map[string][]string{ + "lmstudio": {"shared-model"}, + "vllm": {"shared-model"}, + "sglang": {"shared-model"}, + } + for range 50 { + if got := engineForModel(byEngine, "shared-model"); got != "lmstudio" { + t.Fatalf("tie-break = %q, want the first listed engine lmstudio", got) + } + } + // With LM Studio out of the picture the next listed engine wins, which is + // what makes the order a rank rather than just a default. + delete(byEngine, "lmstudio") + if got := engineForModel(byEngine, "shared-model"); got != "vllm" { + t.Errorf("tie-break without lmstudio = %q, want vllm", got) + } + delete(byEngine, "vllm") + if got := engineForModel(byEngine, "shared-model"); got != "sglang" { + t.Errorf("tie-break without lmstudio or vllm = %q, want sglang", got) + } +} + +// TestLocalBackendsHoldAllThreeEngines proves one node can serve LM Studio, +// vLLM and SGLang at once and that clearing one leaves the other two routable. +// The backends are keyed by engine precisely so an engine going down is not a +// node going down. +func TestLocalBackendsHoldAllThreeEngines(t *testing.T) { + p := testProxy(NewDiscovery(), 1235) + p.setLocalBackend(localBackend{Engine: "lmstudio", Port: 1234, Healthy: true}) + p.setLocalBackend(localBackend{Engine: "vllm", Port: 8000, Healthy: true}) + p.setLocalBackend(localBackend{Engine: "sglang", Port: 30000, Healthy: true}) + if got := len(p.localBackends()); got != 3 { + t.Fatalf("healthy backends = %d, want all three engines", got) + } + target, ok := p.localBackendTarget("sglang") + if !ok { + t.Fatal("the SGLang backend must be a target") + } + if !strings.HasSuffix(target.Host, ":30000") { + t.Errorf("sglang target = %q, want SGLang's own port 30000", target.Host) + } + + p.setLocalBackend(localBackend{Engine: "sglang", Port: 30000, Healthy: false}) + if _, ok := p.localBackendTarget("sglang"); ok { + t.Error("an unhealthy SGLang backend must not be a target") + } + for _, engine := range []string{"lmstudio", "vllm"} { + if _, ok := p.localBackendTarget(engine); !ok { + t.Errorf("clearing SGLang must not disturb %s", engine) + } + } +} + +// TestAddManualAcceptsSGLang drives node/add-manual over the RPC surface with +// engine "sglang", which is the wire change a broker depends on: the engine +// field is validated against the fronted set, so an engine the table doesn't +// carry is refused at the boundary. It also proves the entry is keyed by +// (engine, node), so SGLang on 30000 sits alongside vLLM on 8000 for one host +// instead of replacing it. +func TestAddManualAcceptsSGLang(t *testing.T) { + rw := &captureRW{} + disc := NewDiscovery() + p := NewProxy(NewCodec(rw), disc, 1235) + + addManual(t, p, 1, map[string]any{ + "id": "host", "engine": "vllm", "host": "192.0.2.30", "port": 8000, + "addresses": []string{"192.0.2.30"}, "models": []string{"Qwen/Qwen3-8B"}, + }) + addManual(t, p, 2, map[string]any{ + "id": "host", "engine": "sglang", "host": "192.0.2.30", "port": 30000, + "addresses": []string{"192.0.2.30"}, "models": []string{sglangModel}, + }) + + for _, resp := range rw.replies(t) { + if resp.Error != nil { + t.Fatalf("node/add-manual rejected: %+v", resp.Error) + } + } + nodes := disc.Nodes() + if len(nodes) != 2 { + t.Fatalf("nodes = %+v, want one entry per engine on the same host", nodes) + } + var sg *Node + for i := range nodes { + if nodes[i].Engine == "sglang" { + sg = &nodes[i] + } + } + if sg == nil { + t.Fatalf("no sglang entry: %+v", nodes) + } + if sg.Port != 30000 { + t.Errorf("sglang port = %d, want SGLang's own 30000", sg.Port) + } + // A manual node's whole inventory belongs to the engine it was added for, so + // the model id — a bare path here — must resolve back to sglang and to no + // other engine. + if got := engineForModel(sg.ModelsByEngine, sglangModel); got != "sglang" { + t.Errorf("manual SGLang model owner = %q, want sglang", got) + } +} + +// TestAddManualRejectsAnUnfrontedEngine is the other half of the boundary +// check: a near-miss on an engine name must be refused rather than creating an +// entry nothing can route to, and the rejection must name the engines that are +// accepted so the caller can see sglang is among them. +func TestAddManualRejectsAnUnfrontedEngine(t *testing.T) { + rw := &captureRW{} + disc := NewDiscovery() + p := NewProxy(NewCodec(rw), disc, 1235) + + addManual(t, p, 1, map[string]any{ + "id": "host", "engine": "sglang-typo", "host": "192.0.2.30", "port": 30000, + "addresses": []string{"192.0.2.30"}, + }) + + replies := rw.replies(t) + if len(replies) != 1 || replies[0].Error == nil { + t.Fatalf("an unfronted engine was accepted: %+v", replies) + } + if !strings.Contains(replies[0].Error.Message, "sglang") { + t.Errorf("rejection %q should name the engines that are accepted", replies[0].Error.Message) + } + if len(disc.Nodes()) != 0 { + t.Errorf("a rejected add left an entry: %+v", disc.Nodes()) + } +} + +func addManual(t *testing.T, p *Proxy, id int, params map[string]any) { + t.Helper() + raw, err := json.Marshal(params) + if err != nil { + t.Fatal(err) + } + rawID := json.RawMessage(strconv.Itoa(id)) + p.handleMessage(&Message{JSONRPC: "2.0", ID: &rawID, Method: "node/add-manual", Params: raw}) +} + +// captureRW is a codec transport that records everything the proxy writes and +// reads EOF, so a request can be handed to handleMessage in-process and its +// reply read back. +type captureRW struct{ out bytes.Buffer } + +func (c *captureRW) Read([]byte) (int, error) { return 0, io.EOF } +func (c *captureRW) Write(p []byte) (int, error) { return c.out.Write(p) } + +func (c *captureRW) replies(t *testing.T) []Message { + t.Helper() + var out []Message + for _, line := range strings.Split(strings.TrimSpace(c.out.String()), "\n") { + if line == "" { + continue + } + var msg Message + if err := json.Unmarshal([]byte(line), &msg); err != nil { + t.Fatalf("unparseable frame %q: %v", line, err) + } + if msg.IsResponse() { + out = append(out, msg) + } + } + return out +} diff --git a/services/nvpair-cluster-manager/mdns_browser.go b/services/nvpair-cluster-manager/mdns_browser.go index 2950d719..a81ed178 100644 --- a/services/nvpair-cluster-manager/mdns_browser.go +++ b/services/nvpair-cluster-manager/mdns_browser.go @@ -103,12 +103,18 @@ func (b *Browser) Resolve(idOrUUID string) (hosts []string, port int, ok bool) { } // pickHosts returns the node's dialable addresses in ranked order, falling back -// to whatever it advertised when none of them rank. +// to its discovery host name when it published no address at all. netpick ranks +// IP literals and DNS names alike, so a node reachable only by name is already +// covered by the first branch; the host name is a separate source, not a second +// reading of the same one. func pickHosts(n discovery.Node) []string { if h := netpick.Candidates(n.TXT, n.Addresses); len(h) > 0 { return h } - return n.Addresses + if n.Host != "" { + return []string{n.Host} + } + return nil } // seed primes a known peer into the resolver map without a live event (tests). diff --git a/services/nvpair-engine-manager/README.md b/services/nvpair-engine-manager/README.md index 8c3dfa3c..f871638f 100644 --- a/services/nvpair-engine-manager/README.md +++ b/services/nvpair-engine-manager/README.md @@ -5,8 +5,8 @@ SPDX-License-Identifier: Apache-2.0 # nvpair-engine-manager -A config-driven control plane for local inference engines (Ollama today; -Intel/others via a dropped-in manifest). It manages everything about an +A config-driven control plane for local inference engines (Ollama, LM Studio, +vLLM and SGLang today; others via a dropped-in manifest). It manages everything about an engine **except serving inference**: detect, user-mode install, start/stop/restart, health, and config-declared actions. Adding an engine is a JSON manifest, not code. @@ -35,10 +35,11 @@ Requests (caller → service): | `engine:status` | `{ engine }` | `EngineStatus` | | `engine:install` | `{ engine, start?, port?, bind? }` | `EngineStatus` (after install; also starts it if `start:true`) | | `engine:uninstall` | `{ engine }` | `EngineStatus` (after removal) | -| `engine:start` | `{ engine, port?, bind? }` | `EngineStatus` (after readiness) | +| `engine:start` | `{ engine, port?, bind?, model? }` | `EngineStatus` (after readiness). `model` is a one-shot served-model override for this launch, mirroring `port` | | `engine:stop` | `{ engine }` | `EngineStatus` | | `engine:restart` | `{ engine }` | `EngineStatus` | | `engine:set-port` | `{ engine, port }` | `EngineStatus` (after rebind) | +| `engine:set-model` | `{ engine, model }` | `EngineStatus` (after restart). The persistent served-model setter for an engine that runs one model per process; an empty `model` removes the override | | `engine:action` | `{ engine, action, params }` | the engine's raw response. `action:"pull_model"` is streamed: it emits live `engine:pull-progress` notifications and returns the pull's terminal result (see below). An action whose manifest declares `restart_after` (LM Studio's `delete_model`) restarts a running engine before replying, so the response also means the engine is back and healthy | | `engine:logs` | `{ engine }` | `{ lines: [LogLine] }` | | `engine:errors` | — | `{ errors: [ServiceError] }` | @@ -51,7 +52,9 @@ Requests (caller → service): | `shutdown` | — | `null` | | `log/set-level` | `{ level }` | `{ level }` | -`EngineStatus` = `{ engine, display_name, installed, running, healthy, port }`. +`EngineStatus` = `{ engine, display_name, installed, running, healthy, port, model? }`. +`model` is the engine's configured served model, present only for an engine that +serves one model per process (vLLM, SGLang) and only once one has been chosen. Notifications (service → caller): `engine:ready{version}`, `engine:state-changed{EngineStatus}`, @@ -85,6 +88,20 @@ pinned cluster peer. See "Remote engine management" below. service side, so the read loop never blocks and their responses arrive when the op finishes. +`engine:set-model` is the persistent served-model setter, for an engine whose +launch command names the model it serves (vLLM's `vllm serve `, SGLang's +`sglang serve --model-path `). It +writes a `{ engine, runtime: { model } }` delta into the same per-user override +file `engine:set-port` uses — both setters read-modify-write it, so neither +clobbers the other, and the file is removed only once no override remains — and +applies the choice by restarting a running engine onto it. An empty model +removes the override, which leaves an engine that requires one unable to start +until a model is chosen again. A running, **adopted** engine is refused for the +same reason `engine:set-port` refuses one. Starting an engine whose launch +template substitutes `{model}` with none configured fails with an actionable +error rather than spawning a process that cannot serve anything. The one-shot +counterpart is `engine:start {model}`, which reverts on the next restart. + `engine:set-port` is the **persistent** port setter (distinct from the one-shot `engine:start {port}` override, which reverts on the next restart). It validates `1-65535`, persists the choice as a manifest override (a @@ -113,7 +130,10 @@ when one is pinned (an unpinned fetch runs with a loud warning). Start waits for the readiness probe, then runs a periodic health probe; an unexpected exit is reported. The bundled Ollama manifest allows up to ten minutes for startup because GPU discovery can exceed the previous 30-second -allowance on supported Windows systems. The deadline remains finite: if Ollama +allowance on supported Windows systems. The bundled vLLM and SGLang manifests +allow thirty: their first start downloads the model's weights and warms up the +GPU before either serves anything, and SGLang does not open its port at all until +the model is loaded. The deadline remains finite: if Ollama never serves its readiness endpoint, engine-manager stops the owned process and reports the failed start. Stop sends one stop signal and waits for the engine to exit, with no timeout: SIGTERM to the process group on Unix (graceful, no diff --git a/services/nvpair-engine-manager/actions.go b/services/nvpair-engine-manager/actions.go index 372ee104..286dbbc0 100644 --- a/services/nvpair-engine-manager/actions.go +++ b/services/nvpair-engine-manager/actions.go @@ -162,7 +162,7 @@ func (e *Executor) runRemovePathAction(ctx context.Context, st *engineState, act var pm map[string]any if err := json.Unmarshal(params, &pm); err == nil { for k, v := range pm { - if allowedPlaceholders[k] { + if reservedActionPlaceholders[k] { continue } vars[k] = fmt.Sprint(v) @@ -224,7 +224,7 @@ func (e *Executor) runCmdAction(ctx context.Context, st *engineState, act Action var pm map[string]any if err := json.Unmarshal(params, &pm); err == nil { for k, v := range pm { - if allowedPlaceholders[k] { + if reservedActionPlaceholders[k] { continue } vars[k] = fmt.Sprint(v) diff --git a/services/nvpair-engine-manager/executor.go b/services/nvpair-engine-manager/executor.go index 85ca5536..dc7dd118 100644 --- a/services/nvpair-engine-manager/executor.go +++ b/services/nvpair-engine-manager/executor.go @@ -33,6 +33,11 @@ type EngineStatus struct { Running bool `json:"running"` Healthy bool `json:"healthy"` Port int `json:"port,omitempty"` + // Model is the engine's configured served model, for an engine that runs one + // model per process (vLLM, SGLang). Empty for every other engine, and for one + // that needs a model but has none chosen yet — the state in which a start is + // refused. Set by engine:set-model; see runtime.model. + Model string `json:"model,omitempty"` } // engineState is the per-engine runtime state. diff --git a/services/nvpair-engine-manager/lifecycle.go b/services/nvpair-engine-manager/lifecycle.go index c10244b0..9b9d030f 100644 --- a/services/nvpair-engine-manager/lifecycle.go +++ b/services/nvpair-engine-manager/lifecycle.go @@ -57,6 +57,9 @@ func (e *Executor) waitDetect(engine string, want bool, timeout time.Duration) b type startOpts struct { Port int Bind string + // Model overrides runtime.model for this launch only, for an engine that + // serves one model per process. "" => the manifest's configured model. + Model string } // effectiveBind picks the listen address substituted as {host}: a per-call @@ -165,6 +168,16 @@ func (e *Executor) doStart(ctx context.Context, st *engineState, engine string, if !pathInstalled { return fmt.Errorf("engine %q is not installed", engine) } + model := rt.Model + if opts.Model != "" { + model = opts.Model + } + // An engine whose launch template substitutes {model} serves exactly one + // model per process and has nothing to serve until one is chosen. Fail with + // guidance rather than spawning a process that cannot answer. + if rt.referencesModel() && strings.TrimSpace(model) == "" { + return fmt.Errorf("cannot start %s: it serves one model per process, and no model is configured — choose the model to serve in Engine settings", st.manifest.DisplayName) + } vars := map[string]string{ "host": effectiveBind(rt.Bind, opts.Bind), "port": strconv.Itoa(port), @@ -173,6 +186,9 @@ func (e *Executor) doStart(ctx context.Context, st *engineState, engine string, if rt.CLI != "" { vars["cli"] = expandPath(rt.CLI) } + if model != "" { + vars["model"] = model + } st.mu.Lock() st.port = port @@ -227,7 +243,7 @@ func (e *Executor) bringUpProcess(ctx context.Context, st *engineState, engine s } vars["bin"] = binPath - args, err := resolveArgs(rt.Args, vars) + args, err := resolveArgs(rt.launchArgs(), vars) if err != nil { return err } diff --git a/services/nvpair-engine-manager/manager.go b/services/nvpair-engine-manager/manager.go index 80d572b4..01ea68f5 100644 --- a/services/nvpair-engine-manager/manager.go +++ b/services/nvpair-engine-manager/manager.go @@ -40,6 +40,10 @@ type opParam struct { Port int `json:"port,omitempty"` Bind string `json:"bind,omitempty"` Start bool `json:"start,omitempty"` + // Model is the one-shot served-model override for engine:start, mirroring + // Port: it applies to this launch only and is never persisted. Use + // engine:set-model for the persistent choice. + Model string `json:"model,omitempty"` } type actionParam struct { @@ -56,6 +60,16 @@ type setPortParam struct { Port int `json:"port"` } +// setModelParam is the engine:set-model input: the engine whose served model to +// change and the model id. It is the persistent counterpart of the one-shot +// engine:start {model} override — the choice is written as a manifest override +// exactly the way engine:set-port writes the port, and an empty model removes +// the override (back to the bundled default). +type setModelParam struct { + Engine string `json:"engine"` + Model string `json:"model"` +} + // Manager is the engine-manager's JSON-RPC front end. It dispatches the // engine:* surface to the Executor. Long-running operations (install, // start, stop, restart, action) run in their own goroutine so the read @@ -248,6 +262,9 @@ func (m *Manager) handleMessage(ctx context.Context, msg *Message) { case "engine:set-port": go m.runSetPort(ctx, msg) + case "engine:set-model": + go m.runSetModel(ctx, msg) + case "internal:set-reserved-port": var p struct { Port int `json:"port"` @@ -294,7 +311,7 @@ func (m *Manager) runOp(ctx context.Context, msg *Message) { return } start := func() error { - return m.exec.StartWith(ctx, p.Engine, startOpts{Port: p.Port, Bind: p.Bind}) + return m.exec.StartWith(ctx, p.Engine, startOpts{Port: p.Port, Bind: p.Bind, Model: p.Model}) } var err error switch msg.Method { @@ -338,6 +355,21 @@ func (m *Manager) runSetPort(ctx context.Context, msg *Message) { m.respondOrErr(msg, st, err) } +// runSetModel persists the model an engine serves (as a manifest override) and +// applies it, responding with the engine's resulting status. +func (m *Manager) runSetModel(ctx context.Context, msg *Message) { + var p setModelParam + if !m.parse(msg, &p) { + return + } + if p.Engine == "" { + m.codec.RespondError(msg.ID, -32602, "engine is required") + return + } + st, err := m.exec.SetModel(ctx, p.Engine, p.Model) + m.respondOrErr(msg, st, err) +} + // runModels answers engine:models with the union of running engines' model // lists plus per-engine attribution (modelsByEngine). It runs async (like // engine:action) because it makes one loopback HTTP call per running engine and diff --git a/services/nvpair-engine-manager/manifests/sglang.json b/services/nvpair-engine-manager/manifests/sglang.json new file mode 100644 index 00000000..b69d2af2 --- /dev/null +++ b/services/nvpair-engine-manager/manifests/sglang.json @@ -0,0 +1,51 @@ +{ + "engine": "sglang", + "display_name": "SGLang", + "manifest_version": 1, + "detect": ["{install_dir}/venv/bin/sglang", "~/.sglang/bin/sglang"], + "install": { + "fetch": { "url": "https://astral.sh/uv/install.sh" }, + "run": [ + "sh", + "-c", + "set -e; mkdir -p \"$1\"; UV_INSTALL_DIR=\"$1/uv\" UV_NO_MODIFY_PATH=1 sh \"$2\"; \"$1/uv/uv\" venv \"$1/venv\"; \"$1/uv/uv\" pip install --python \"$1/venv/bin/python\" \"sglang[all]\"", + "sh", + "{install_dir}", + "{download}" + ], + "mode": "user" + }, + "uninstall": { + "run": ["rm", "-rf", "{install_dir}"] + }, + "runtime": { + "mode": "process", + "bin": "{install_dir}/venv/bin/sglang", + "args": ["serve", "--model-path", "{model}", "--port", "{port}", "--host", "{host}"], + "extra_args": [], + "port": 30000, + "bind": "127.0.0.1", + "ready": { "http": "http://127.0.0.1:{port}/get_model_info", "status": 200, "timeout_s": 1800 }, + "health": { "http": "http://127.0.0.1:{port}/get_model_info", "status": 200, "interval_s": 10 } + }, + "platforms": { + "linux/amd64": {}, + "linux/arm64": {} + }, + "actions": { + "list_models": { + "description": "List the model this SGLang process serves (OpenAI-compatible).", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "loaded_models": { + "description": "List models resident in memory. SGLang serves exactly one model per process and keeps it resident, so this is the same list as list_models.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "chat": { + "description": "OpenAI-compatible chat completion (params: {\"model\": \"\", \"messages\": [...]}).", + "http": { "method": "POST", "path": "/v1/chat/completions", "body_schema": { "model": "string", "messages": "array" } } + } + } +} diff --git a/services/nvpair-engine-manager/manifests/vllm.json b/services/nvpair-engine-manager/manifests/vllm.json new file mode 100644 index 00000000..14fa7c99 --- /dev/null +++ b/services/nvpair-engine-manager/manifests/vllm.json @@ -0,0 +1,51 @@ +{ + "engine": "vllm", + "display_name": "vLLM", + "manifest_version": 1, + "detect": ["{install_dir}/venv/bin/vllm", "~/.vllm/bin/vllm"], + "install": { + "fetch": { "url": "https://astral.sh/uv/install.sh" }, + "run": [ + "sh", + "-c", + "set -e; mkdir -p \"$1\"; UV_INSTALL_DIR=\"$1/uv\" UV_NO_MODIFY_PATH=1 sh \"$2\"; \"$1/uv/uv\" venv \"$1/venv\"; \"$1/uv/uv\" pip install --python \"$1/venv/bin/python\" vllm", + "sh", + "{install_dir}", + "{download}" + ], + "mode": "user" + }, + "uninstall": { + "run": ["rm", "-rf", "{install_dir}"] + }, + "runtime": { + "mode": "process", + "bin": "{install_dir}/venv/bin/vllm", + "args": ["serve", "{model}", "--port", "{port}", "--host", "{host}"], + "extra_args": [], + "port": 8000, + "bind": "127.0.0.1", + "ready": { "http": "http://127.0.0.1:{port}/health", "status": 200, "timeout_s": 1800 }, + "health": { "http": "http://127.0.0.1:{port}/health", "status": 200, "interval_s": 10 } + }, + "platforms": { + "linux/amd64": {}, + "linux/arm64": {} + }, + "actions": { + "list_models": { + "description": "List the model this vLLM process serves (OpenAI-compatible).", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "loaded_models": { + "description": "List models resident in memory. vLLM serves exactly one model per process and keeps it resident, so this is the same list as list_models.", + "http": { "method": "GET", "path": "/v1/models" }, + "result": { "array": "data", "field": "id" } + }, + "chat": { + "description": "OpenAI-compatible chat completion (params: {\"model\": \"\", \"messages\": [...]}).", + "http": { "method": "POST", "path": "/v1/chat/completions", "body_schema": { "model": "string", "messages": "array" } } + } + } +} diff --git a/services/nvpair-engine-manager/registry.go b/services/nvpair-engine-manager/registry.go index b44ec6e0..3a84902e 100644 --- a/services/nvpair-engine-manager/registry.go +++ b/services/nvpair-engine-manager/registry.go @@ -25,11 +25,11 @@ import ( // schema growth stays backward compatible. const ManifestSchemaVersion = 1 -// allowedPlaceholders is the set of `{token}`s the runner can resolve -// at execution time. Validation rejects any other token so a typo in -// a manifest fails at load with a clear message rather than at run -// time with a mangled command line. -var allowedPlaceholders = map[string]bool{ +// reservedActionPlaceholders is the set of `{token}`s the runner always +// resolves itself from trusted state (the manifest, the chosen port, the +// install dir). An engine:action caller's params are refused these names, so a +// hostile param can never hijack one — notably {cli}, which is argv[0]. +var reservedActionPlaceholders = map[string]bool{ "bin": true, "cli": true, "host": true, @@ -39,6 +39,28 @@ var allowedPlaceholders = map[string]bool{ "models_dir": true, } +// allowedPlaceholders is the set of `{token}`s a manifest may template. +// Validation rejects any other token so a typo fails at load with a clear +// message rather than at run time with a mangled command line. It is the +// reserved set plus {model}: unlike the reserved names, model is a value the +// caller legitimately supplies per call on a model action (`lms get {model}`) +// as well as one the runner resolves from runtime.model at launch, so it is +// resolvable without being caller-proof. +var allowedPlaceholders = buildAllowedPlaceholders() + +func buildAllowedPlaceholders() map[string]bool { + out := make(map[string]bool, len(reservedActionPlaceholders)+1) + for k := range reservedActionPlaceholders { + out[k] = true + } + // model is the served model of an engine that runs one model per process + // (vLLM's `vllm serve `, SGLang's `sglang serve --model-path + // `): read from runtime.model at start, set persistently by + // engine:set-model, overridable per call by engine:start. + out["model"] = true + return out +} + var placeholderRe = regexp.MustCompile(`\{([a-zA-Z_][a-zA-Z0-9_]*)\}`) // engineNameRe restricts engine names to a safe charset — the name is @@ -113,17 +135,28 @@ type Fetch struct { // (e.g. LM Studio's `lms`); liveness = the readiness/health probe, // and Stop.Cmd brings it down. type Runtime struct { - Mode string `json:"mode,omitempty"` - Bin string `json:"bin,omitempty"` - Args []string `json:"args,omitempty"` - Env map[string]string `json:"env,omitempty"` - Port int `json:"port"` // 0 => auto-assign a free loopback port - Bind string `json:"bind,omitempty"` // listen addr, substituted as {host}; "" => 127.0.0.1 - Start [][]string `json:"start,omitempty"` // command mode: ordered bring-up commands + Mode string `json:"mode,omitempty"` + Bin string `json:"bin,omitempty"` + Args []string `json:"args,omitempty"` + // ExtraArgs are appended after Args on every launch. They exist so a + // per-user manifest override can add engine flags (vLLM's + // --gpu-memory-utilization, --max-model-len, …) without restating the + // bundled Args array, which a deep merge would replace wholesale. + ExtraArgs []string `json:"extra_args,omitempty"` + Env map[string]string `json:"env,omitempty"` + Port int `json:"port"` // 0 => auto-assign a free loopback port + Bind string `json:"bind,omitempty"` // listen addr, substituted as {host}; "" => 127.0.0.1 + Start [][]string `json:"start,omitempty"` // command mode: ordered bring-up commands // CLI is the engine's control-CLI path for this platform, referenced // elsewhere as {cli}. It lets the manifest's global actions resolve // to the correct per-OS binary (e.g. lms.exe vs lms). - CLI string `json:"cli,omitempty"` + CLI string `json:"cli,omitempty"` + // Model is the model this engine serves, for an engine that serves exactly + // one model per process (vLLM, SGLang). It is substituted as {model} in Args / + // ExtraArgs / Start. Empty means "not configured": starting an engine whose + // launch template references {model} then fails with an actionable error + // instead of spawning a process that cannot serve anything. + Model string `json:"model,omitempty"` Ready *Probe `json:"ready,omitempty"` Stop *StopSpec `json:"stop,omitempty"` Health *Probe `json:"health,omitempty"` @@ -394,29 +427,92 @@ func (r *Registry) LoadOverrideDir(dir string) error { return nil } -// bundledDefaultPort returns the host-platform runtime.port from the bundled -// (un-overridden) manifest for an engine, used to decide whether a chosen -// port is back at the default (so its override file can be dropped). -func (r *Registry) bundledDefaultPort(engine string) (int, bool) { +// bundledHostRuntime returns the host-platform runtime block from the bundled +// (un-overridden) manifest for an engine. It is how the persistent setters +// decide whether a chosen value is back at the bundled default, so that +// field's override can be dropped instead of pinned forever. +func (r *Registry) bundledHostRuntime(engine string) (Runtime, bool) { raw, ok := r.bundledRaw[engine] if !ok { - return 0, false + return Runtime{}, false } merged, err := applyPlatformDefaults(raw) if err != nil { - return 0, false + return Runtime{}, false } var m Manifest if err := json.Unmarshal(merged, &m); err != nil { - return 0, false + return Runtime{}, false } p, ok := m.HostPlatform() + if !ok { + return Runtime{}, false + } + return p.Runtime, true +} + +// bundledDefaultPort returns the host-platform runtime.port from the bundled +// (un-overridden) manifest for an engine, used to decide whether a chosen +// port is back at the default (so its override can be dropped). +func (r *Registry) bundledDefaultPort(engine string) (int, bool) { + rt, ok := r.bundledHostRuntime(engine) if !ok { return 0, false } - return p.Runtime.Port, true + return rt.Port, true } +// bundledDefaultModel returns the host-platform runtime.model from the bundled +// manifest — the counterpart of bundledDefaultPort for engine:set-model. +func (r *Registry) bundledDefaultModel(engine string) (string, bool) { + rt, ok := r.bundledHostRuntime(engine) + if !ok { + return "", false + } + return rt.Model, true +} + +// launchArgs is the full argv tail for a process-mode launch: the manifest's +// runtime.args followed by runtime.extra_args, before placeholder resolution. +func (r *Runtime) launchArgs() []string { + if len(r.ExtraArgs) == 0 { + return r.Args + } + out := make([]string, 0, len(r.Args)+len(r.ExtraArgs)) + out = append(out, r.Args...) + out = append(out, r.ExtraArgs...) + return out +} + +// referencesModel reports whether this runtime's launch template substitutes +// {model} anywhere — the signal that the engine serves one model per process +// and cannot start until a model has been chosen. +func (r *Runtime) referencesModel() bool { + if referencesModelToken(r.Bin) { + return true + } + for _, s := range r.launchArgs() { + if referencesModelToken(s) { + return true + } + } + for _, cmd := range r.Start { + for _, s := range cmd { + if referencesModelToken(s) { + return true + } + } + } + for _, v := range r.Env { + if referencesModelToken(v) { + return true + } + } + return false +} + +func referencesModelToken(s string) bool { return strings.Contains(s, "{model}") } + // mergeOntoBundled deep-merges an override manifest's raw JSON onto the // bundled manifest of the same engine (override wins). When the override // declares no engine, or no bundled base exists for it, the override is @@ -719,6 +815,7 @@ func (m *Manifest) templatedStrings() []string { } out = append(out, p.Runtime.Bin) out = append(out, p.Runtime.Args...) + out = append(out, p.Runtime.ExtraArgs...) for _, cmd := range p.Runtime.Start { out = append(out, cmd...) } diff --git a/services/nvpair-engine-manager/setmodel_test.go b/services/nvpair-engine-manager/setmodel_test.go new file mode 100644 index 00000000..228165cc --- /dev/null +++ b/services/nvpair-engine-manager/setmodel_test.go @@ -0,0 +1,289 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" +) + +// hostModel returns the effective host-platform runtime.model for an engine. +func hostModel(t *testing.T, reg *Registry, engine string) string { + t.Helper() + m, ok := reg.Get(engine) + if !ok { + t.Fatalf("engine %q not in registry", engine) + } + p, ok := m.HostPlatform() + if !ok { + t.Fatalf("engine %q has no host platform block", engine) + } + return p.Runtime.Model +} + +// vllmManifest is the bundled vLLM manifest re-keyed onto the running host so +// the Linux-only engine's lifecycle can be exercised on any developer machine. +// Only the platform key changes; every field under test is the bundled one. +func vllmManifest(t *testing.T) *Manifest { + t.Helper() + reg := NewRegistry() + if err := reg.LoadFS(bundledManifests, "manifests"); err != nil { + t.Fatalf("LoadFS bundled: %v", err) + } + m, ok := reg.Get("vllm") + if !ok { + t.Fatal("vllm manifest not bundled") + } + linux, ok := m.PlatformFor("linux", "amd64") + if !ok { + t.Fatal("vllm manifest has no linux/amd64 block") + } + clone := *m + clone.Platforms = map[string]Platform{runtime.GOOS + "/" + runtime.GOARCH: *linux} + return &clone +} + +// TestVLLMManifestLoadsAndValidates proves the bundled vLLM manifest passes +// schema validation, including the {model} placeholder in its launch args, and +// that it is offered on Linux only. +func TestVLLMManifestLoadsAndValidates(t *testing.T) { + reg := NewRegistry() + if err := reg.LoadFS(bundledManifests, "manifests"); err != nil { + t.Fatalf("LoadFS bundled: %v", err) + } + m, ok := reg.Get("vllm") + if !ok { + t.Fatal("vllm manifest not bundled") + } + if m.DisplayName != "vLLM" { + t.Errorf("display_name = %q, want %q", m.DisplayName, "vLLM") + } + want := map[string]bool{"linux/amd64": true, "linux/arm64": true} + for key := range m.Platforms { + if !want[key] { + t.Errorf("unexpected platform %q: vLLM ships on Linux only", key) + } + delete(want, key) + } + for key := range want { + t.Errorf("missing platform %q", key) + } + + p, ok := m.PlatformFor("linux", "amd64") + if !ok { + t.Fatal("no linux/amd64 block") + } + if p.Runtime.modeOrDefault() != "process" { + t.Errorf("runtime.mode = %q, want process (vllm serve is a foreground server, not a bring-up command)", p.Runtime.modeOrDefault()) + } + if p.Runtime.Port != 8000 { + t.Errorf("runtime.port = %d, want 8000", p.Runtime.Port) + } + if !p.Runtime.referencesModel() { + t.Error("vLLM's launch template must substitute {model}") + } + if p.Runtime.Ready == nil || p.Runtime.Ready.TimeoutS != 1800 { + t.Errorf("runtime.ready timeout = %+v, want 1800s (first start downloads weights)", p.Runtime.Ready) + } + if p.Runtime.Health == nil || !strings.HasSuffix(p.Runtime.Health.HTTP, "/health") { + t.Errorf("runtime.health = %+v, want the /health endpoint", p.Runtime.Health) + } + for _, name := range []string{"list_models", "loaded_models", "chat"} { + if _, ok := m.Actions[name]; !ok { + t.Errorf("missing action %q", name) + } + } + for _, name := range []string{"pull_model", "load_model", "unload_model", "delete_model"} { + if _, ok := m.Actions[name]; ok { + t.Errorf("action %q must not be declared: vLLM has no model management surface", name) + } + } +} + +// TestVLLMHasNoHostPlatformOffHostLinux proves an engine with no block for the +// running platform is simply not offered there: it still lists (so the UI can +// say "unavailable on this system") but every lifecycle op refuses. +func TestVLLMHasNoHostPlatformOffHostLinux(t *testing.T) { + if runtime.GOOS == "linux" { + t.Skip("vLLM has a platform block on Linux; this covers the other hosts") + } + ex := newBundledExecutor(t, t.TempDir()) + var found *EngineStatus + for _, st := range ex.GetInstalled() { + if st.Engine == "vllm" { + s := st + found = &s + } + } + if found == nil { + t.Fatal("vllm missing from engine:get-installed") + } + if found.Installed || found.Running { + t.Errorf("vllm reported present on %s: %+v", runtime.GOOS, *found) + } + if _, err := ex.state("vllm"); err == nil { + t.Error("expected vllm to have no host platform block on this OS") + } + if err := ex.Start(context.Background(), "vllm"); err == nil { + t.Error("expected Start to refuse an engine with no host platform block") + } +} + +// TestStartWithoutConfiguredModelIsActionable proves an engine whose launch +// template needs {model} refuses to spawn until a model is chosen, with a +// message that tells the user what to do rather than leaking a placeholder. +func TestStartWithoutConfiguredModelIsActionable(t *testing.T) { + m := vllmManifest(t) + // Move off 8000 so an unrelated listener on this host cannot pre-empt the + // model check with a port-occupied error. + key := runtime.GOOS + "/" + runtime.GOARCH + plat := m.Platforms[key] + free, err := freePort() + if err != nil { + t.Fatal(err) + } + plat.Runtime.Port = free + // Point detect at a real file so the refusal is about the missing model, + // not about the engine being uninstalled. + plat.Detect = []string{fakeEngineBin} + m.Platforms[key] = plat + + ex := newTestExecutor(t, m) + err = ex.Start(context.Background(), "vllm") + if err == nil { + t.Fatal("expected Start to refuse with no model configured") + } + for _, want := range []string{"vLLM", "one model per process", "Engine settings"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error %q does not mention %q", err.Error(), want) + } + } + if strings.Contains(err.Error(), "{model}") { + t.Errorf("error leaks the raw placeholder: %q", err.Error()) + } +} + +// TestSetModelPersistsAsManifestOverride proves engine:set-model survives a +// "restart" through the same override-dir merge engine:set-port uses, and that +// clearing it removes the override again. It drives ollama because the writer is +// engine-agnostic and ollama has a platform block on every host; vLLM's own +// Linux-only manifest is covered by the manifest test above. +func TestSetModelPersistsAsManifestOverride(t *testing.T) { + dir := t.TempDir() + ex := newBundledExecutor(t, dir) + if err := ex.persistModel("ollama", "Qwen/Qwen3-8B"); err != nil { + t.Fatalf("persistModel: %v", err) + } + if got := hostModel(t, loadWithOverrides(t, dir), "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model after reload = %q, want Qwen/Qwen3-8B", got) + } + + if err := ex.persistModel("ollama", ""); err != nil { + t.Fatalf("persistModel clear: %v", err) + } + if got := hostModel(t, loadWithOverrides(t, dir), "ollama"); got != "" { + t.Errorf("model after clear = %q, want empty", got) + } + if _, err := os.Stat(filepath.Join(dir, "ollama.json")); !os.IsNotExist(err) { + t.Errorf("override file should be gone once no override remains: %v", err) + } +} + +// TestSetModelAndSetPortDoNotClobberEachOther is the regression guard for the +// two persistent setters sharing one override file: each writes its own +// runtime field and must leave the other's alone, in either order. +func TestSetModelAndSetPortDoNotClobberEachOther(t *testing.T) { + dir := t.TempDir() + ex := newBundledExecutor(t, dir) + bundledPort := hostPort(t, ex.reg, "ollama") + + if err := ex.persistModel("ollama", "Qwen/Qwen3-8B"); err != nil { + t.Fatalf("persistModel: %v", err) + } + if err := ex.persistPort("ollama", 9001); err != nil { + t.Fatalf("persistPort: %v", err) + } + reg := loadWithOverrides(t, dir) + if got := hostModel(t, reg, "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model lost when the port was set: got %q", got) + } + if got := hostPort(t, reg, "ollama"); got != 9001 { + t.Errorf("port = %d, want 9001", got) + } + + // Back to the bundled default port: the port override drops, the model stays. + if err := ex.persistPort("ollama", bundledPort); err != nil { + t.Fatalf("persistPort default: %v", err) + } + reg = loadWithOverrides(t, dir) + if got := hostModel(t, reg, "ollama"); got != "Qwen/Qwen3-8B" { + t.Errorf("model lost when the port returned to default: got %q", got) + } + if got := hostPort(t, reg, "ollama"); got != bundledPort { + t.Errorf("port = %d, want the bundled %d", got, bundledPort) + } + + raw, err := os.ReadFile(filepath.Join(dir, "ollama.json")) + if err != nil { + t.Fatalf("read override: %v", err) + } + var doc struct { + Runtime map[string]any `json:"runtime"` + } + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatal(err) + } + if _, ok := doc.Runtime["port"]; ok { + t.Errorf("default port should not stay pinned in the override: %s", raw) + } +} + +// TestSetModelAppliesToTheLaunchArgs proves the configured model reaches the +// spawned process's argv, and that runtime.extra_args is appended after it. +func TestSetModelAppliesToTheLaunchArgs(t *testing.T) { + m := vllmManifest(t) + key := runtime.GOOS + "/" + runtime.GOARCH + plat := m.Platforms[key] + // Drive the fake engine instead of a real vLLM: it takes FAKE_ADDR, so the + // vLLM argv is carried purely to prove substitution, and extra_args adds a + // flag the fake ignores. + plat.Detect = []string{fakeEngineBin} + plat.Runtime.Bin = fakeEngineBin + plat.Runtime.Model = "Qwen/Qwen3-8B" + plat.Runtime.Args = []string{"echo", "{model}", "--port", "{port}"} + plat.Runtime.ExtraArgs = []string{"--gpu-memory-utilization", "0.9"} + plat.Runtime.Ready = nil + plat.Runtime.Health = nil + m.Platforms[key] = plat + + ex := newTestExecutor(t, m) + st, err := ex.state("vllm") + if err != nil { + t.Fatal(err) + } + if err := ex.Start(context.Background(), "vllm"); err != nil { + t.Fatalf("start: %v", err) + } + t.Cleanup(func() { _ = ex.Stop("vllm") }) + + // `echo` prints its args and exits, so the captured stdout is the resolved argv. + captured := func() string { + var line string + for _, l := range st.logs.snapshot() { + line += l.Text + "\n" + } + return line + } + waitFor(t, 5*time.Second, func() bool { return strings.Contains(captured(), "Qwen/Qwen3-8B") }) + line := captured() + if !strings.Contains(line, "--gpu-memory-utilization 0.9") { + t.Errorf("extra_args not appended to the launch argv: %q", line) + } +} diff --git a/services/nvpair-engine-manager/setport.go b/services/nvpair-engine-manager/setport.go index 5a9554c2..98fca571 100644 --- a/services/nvpair-engine-manager/setport.go +++ b/services/nvpair-engine-manager/setport.go @@ -10,6 +10,7 @@ import ( "fmt" "os" "path/filepath" + "strings" ) func canMoveAdoptedEngine(rt Runtime) bool { @@ -93,50 +94,133 @@ func (e *Executor) SetPort(ctx context.Context, engine string, port int) (Engine return e.snapshot(engine, st), nil } -// persistPort writes (or removes) the per-engine manifest override that pins -// runtime.port so the chosen port survives a restart. For a bundled engine it -// writes only the {engine, runtime:{port}} delta (which deep-merges onto the -// bundled manifest at load), and removes the override entirely when the port -// is back at the bundled default — keeping the override set minimal. For a -// non-bundled engine (a full manifest that lives only in the override dir) it -// merges the port into the existing file rather than clobbering it. Atomic -// (tmp + rename). +// persistPort pins runtime.port in the per-engine manifest override so the +// chosen port survives a restart. Back at the bundled default, the port key is +// dropped again rather than pinned forever. See persistRuntimeField. func (e *Executor) persistPort(engine string, port int) error { + def, bundled := e.reg.bundledDefaultPort(engine) + return e.persistRuntimeField(engine, "port", port, bundled && def == port) +} + +// persistModel pins runtime.model in the same override file persistPort writes. +func (e *Executor) persistModel(engine, model string) error { + def, bundled := e.reg.bundledDefaultModel(engine) + return e.persistRuntimeField(engine, "model", model, bundled && def == model) +} + +// persistRuntimeField writes one runtime. value into the per-engine +// manifest override that deep-merges onto the bundled manifest at load, so the +// manifest stays the single source of truth for the setting and it is restored +// on the next start with no separate store. +// +// The file is always read-modify-written rather than replaced with a one-key +// delta: port and model are set independently, and a wholesale write would +// silently drop whichever the caller wasn't changing. atDefault removes just +// that field; the file itself is unlinked only once no override remains, which +// keeps the override set minimal without discarding a sibling setting. Atomic +// (tmp + rename). +func (e *Executor) persistRuntimeField(engine, field string, value any, atDefault bool) error { if e.overrideDir == "" { - return fmt.Errorf("no config directory available to persist the port") + return fmt.Errorf("no config directory available to persist the %s", field) } if err := os.MkdirAll(e.overrideDir, 0o755); err != nil { return fmt.Errorf("create override dir: %w", err) } path := filepath.Join(e.overrideDir, engine+".json") - delta := map[string]any{"engine": engine, "runtime": map[string]any{"port": port}} - - if def, ok := e.reg.bundledDefaultPort(engine); ok { - // Bundled engine: back to default ⇒ drop the override; else persist - // just the delta so bundled upgrades to everything else still apply. - if def == port { - if err := os.Remove(path); err != nil && !os.IsNotExist(err) { - return fmt.Errorf("remove override: %w", err) - } - return nil + + doc := map[string]any{} + existing, err := os.ReadFile(path) + switch { + case err == nil: + if err := json.Unmarshal(existing, &doc); err != nil { + return fmt.Errorf("parse override %s: %w", path, err) } - return writeJSONAtomic(path, delta) + case !os.IsNotExist(err): + return fmt.Errorf("read override: %w", err) } - // Non-bundled engine: the full manifest lives only here, so merge the - // port into it rather than overwriting the file with a partial. - existing, err := os.ReadFile(path) + rt, _ := doc["runtime"].(map[string]any) + if rt == nil { + rt = map[string]any{} + } + if atDefault { + delete(rt, field) + } else { + rt[field] = value + } + doc["engine"] = engine + doc["runtime"] = rt + + // A bundled engine's override exists only to carry deltas, so an override + // with none left is removed. A non-bundled engine's full manifest lives + // only here and is never removed. + if _, bundled := e.reg.bundledDefaultPort(engine); bundled && len(rt) == 0 { + if err := os.Remove(path); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("remove override: %w", err) + } + return nil + } + return writeJSONAtomic(path, doc) +} + +// SetModel persists the model an engine serves as a manifest override and +// applies it: a running engine is restarted onto the new model, since an engine +// that templates {model} into its launch command can only serve the model it +// was started with. Empty removes the override (back to the bundled default), +// which leaves an engine that requires a model unable to start until one is +// chosen again. Held under the engine's op lock, like SetPort. +// +// A running, adopted engine is refused for the same reason SetPort refuses one: +// NVPAIR cannot restart a process it did not start, and the externally-managed +// instance would keep serving its own model while the manifest claimed another. +func (e *Executor) SetModel(ctx context.Context, engine, model string) (EngineStatus, error) { + model = strings.TrimSpace(model) + st, err := e.state(engine) if err != nil { - if os.IsNotExist(err) { - return writeJSONAtomic(path, delta) + return EngineStatus{}, err + } + st.opMu.Lock() + defer st.opMu.Unlock() + + st.mu.Lock() + wasRunning := st.running + adopted := st.adopted + st.mu.Unlock() + + if wasRunning && adopted && !canMoveAdoptedEngine(st.plat.Runtime) { + return EngineStatus{}, fmt.Errorf("cannot change the model %s serves: it is running under external management (NVPAIR adopted it rather than starting it), so NVPAIR cannot restart it — stop it in its own app first, then set the model", engine) + } + + oldModel := st.plat.Runtime.Model + if wasRunning { + if err := e.doStop(st, engine); err != nil { + return EngineStatus{}, err } - return fmt.Errorf("read override: %w", err) } - var m map[string]any - if err := json.Unmarshal(existing, &m); err != nil { - return fmt.Errorf("parse override %s: %w", path, err) + + if err := e.persistModel(engine, model); err != nil { + if !wasRunning { + return EngineStatus{}, err + } + st.mu.Lock() + st.plat.Runtime.Model = oldModel + st.mu.Unlock() + restartErr := e.doStart(ctx, st, engine, startOpts{}) + return EngineStatus{}, errors.Join(err, restartErr) + } + + st.mu.Lock() + st.plat.Runtime.Model = model + st.mu.Unlock() + + if wasRunning { + if err := e.doStart(ctx, st, engine, startOpts{}); err != nil { + return EngineStatus{}, err + } + } else { + e.emitState(engine) } - return writeJSONAtomic(path, deepMerge(m, delta)) + return e.snapshot(engine, st), nil } // writeJSONAtomic marshals v and writes it to path via a tmp file + rename so diff --git a/services/nvpair-engine-manager/status.go b/services/nvpair-engine-manager/status.go index 75419cb7..674f9c46 100644 --- a/services/nvpair-engine-manager/status.go +++ b/services/nvpair-engine-manager/status.go @@ -124,7 +124,7 @@ func (e *Executor) Errors() []serviceError { func (e *Executor) snapshot(engine string, st *engineState) EngineStatus { st.mu.Lock() defer st.mu.Unlock() - return EngineStatus{ + status := EngineStatus{ Engine: engine, DisplayName: st.manifest.DisplayName, Installed: st.installed, @@ -132,6 +132,10 @@ func (e *Executor) snapshot(engine string, st *engineState) EngineStatus { Healthy: st.healthy, Port: st.port, } + if st.plat != nil { + status.Model = st.plat.Runtime.Model + } + return status } // reconcilePresence reconciles filesystem detection with a fixed-port engine diff --git a/services/nvpair-engine-manager/testdata/fakeengine/main.go b/services/nvpair-engine-manager/testdata/fakeengine/main.go index 76626c20..e1ca9fdb 100644 --- a/services/nvpair-engine-manager/testdata/fakeengine/main.go +++ b/services/nvpair-engine-manager/testdata/fakeengine/main.go @@ -146,6 +146,15 @@ func main() { log.Fatalf("write FAKE_PID_FILE %q: %v", path, err) } } + // vLLM mode: a vLLM process serves exactly one model, named on its command + // line, and keeps it resident. FAKE_ENGINE_MODEL reseeds the registry with + // just that model so the OpenAI list/loaded endpoints report vLLM's shape. + if m := os.Getenv("FAKE_ENGINE_MODEL"); m != "" { + modelsMu.Lock() + models = map[string]bool{m: true} + loaded = map[string]bool{m: true} + modelsMu.Unlock() + } if raw := os.Getenv("FAKE_START_DELAY"); raw != "" { delay, err := time.ParseDuration(raw) if err != nil { @@ -291,6 +300,17 @@ func main() { _ = json.NewEncoder(w).Encode(map[string]any{"model": bstr(b, "model"), "object": "chat.completion", "choices": []map[string]any{{"index": 0, "message": map[string]string{"role": "assistant", "content": "ok"}, "finish_reason": "stop"}}}) }) + // vLLM-shape readiness/health: a bare 200 on /health, and a /version that + // answers JSON with a version field. /version is what distinguishes vLLM + // from any other OpenAI-compatible server on a probe (LM Studio has no + // such route), so nvpair-manual-nodes uses it as its disambiguator. + mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + }) + mux.HandleFunc("/version", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{"version": "0.0.0-fake"}) + }) mux.HandleFunc("/api/error", func(w http.ResponseWriter, r *http.Request) { http.Error(w, "boom", http.StatusInternalServerError) }) diff --git a/services/nvpair-job-scheduler/README.md b/services/nvpair-job-scheduler/README.md index 8e810467..b55a5551 100644 --- a/services/nvpair-job-scheduler/README.md +++ b/services/nvpair-job-scheduler/README.md @@ -57,8 +57,8 @@ rank thrash. Invalid, missing, or older-than-10-second telemetry contributes a neutral pressure of 1. Nodes are sorted by `pending + gpuPressure`, then lower GPU pressure, then -stable node ID. Pending counts include **both** engines together, so Ollama load -affects the LM Studio ordering and vice versa. +stable node ID. Pending counts include **every** engine together, so Ollama load +affects the LM Studio, vLLM and SGLang ordering and vice versa. Rankings are recomputed when the node set, catalog, or effective pressure changes, and reconciled on the interval timer. A ranking is only emitted when @@ -66,7 +66,8 @@ the order, pending counts, or pressure actually changed. ## Output -One `schedule:priority` notification per engine (`ollama`, `lmstudio`): +One `schedule:priority` notification per engine (`ollama`, `lmstudio`, `vllm`, +`sglang`): ```json { @@ -83,9 +84,10 @@ One `schedule:priority` notification per engine (`ollama`, `lmstudio`): } ``` -The broker relays each snapshot to the matching proxy as `node/set-priority`. -Both engines currently receive the same node-wide ordering; the per-engine -envelope exists so the routing contract can diverge later without a wire change. +The broker relays each snapshot to the matching proxy as `node/set-priority`; +`lmstudio`, `vllm` and `sglang` all resolve to the one OpenAI-compatible proxy. Every +engine currently receives the same node-wide ordering; the per-engine envelope +exists so the routing contract can diverge later without a wire change. Each proxy then adds its own reservations for in-flight requests whose workload feedback has not arrived yet, so a burst of concurrent requests does not all diff --git a/services/nvpair-job-scheduler/schedule.go b/services/nvpair-job-scheduler/schedule.go index 1afe2fe4..9afc2fec 100644 --- a/services/nvpair-job-scheduler/schedule.go +++ b/services/nvpair-job-scheduler/schedule.go @@ -12,9 +12,10 @@ import ( "nvpair-shared/schedulerwire" ) -// schedulerEngines is the fixed set of engine-specific output contracts. Both -// receive the same node-wide ranking because their work shares node resources. -var schedulerEngines = []string{"ollama", "lmstudio"} +// schedulerEngines is the fixed set of engine-specific output contracts. They +// all receive the same node-wide ranking because their work shares node +// resources — a node's GPU is one queue however many engines front it. +var schedulerEngines = []string{"ollama", "lmstudio", "vllm", "sglang"} // NodeRank is retained as the scheduler's public status type while the wire // definition is shared with the broker and proxies. diff --git a/services/nvpair-job-scheduler/sglangengine_test.go b/services/nvpair-job-scheduler/sglangengine_test.go new file mode 100644 index 00000000..635b196c --- /dev/null +++ b/services/nvpair-job-scheduler/sglangengine_test.go @@ -0,0 +1,37 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import "testing" + +// TestSGLangWorkloadsCountTowardTheNodeWideRanking proves SGLang work shares the +// node's queue depth with the other engines rather than being ranked separately: +// one GPU serves them all, so a node busy with SGLang must rank below an idle +// one for every engine, Ollama included. It is the same guarantee the vLLM case +// makes, asserted for the engine tag SGLang workloads actually carry. +func TestSGLangWorkloadsCountTowardTheNodeWideRanking(t *testing.T) { + rec := &capRW{} + m := mgrWith(rec, []string{"busy", "idle"}, + workload{ID: "1", Engine: "sglang", RunID: "s1", State: "running", OriginatedFrom: "x", ScheduledOn: "busy"}, + workload{ID: "2", Engine: "sglang", RunID: "s2", State: "queued", OriginatedFrom: "x", ScheduledOn: "busy"}, + ) + order, ranks := m.rank() + if len(order) == 0 || order[len(order)-1] != "busy" { + t.Fatalf("order = %v, want the SGLang-busy node last", order) + } + if pendingOf(ranks, "busy") != 2 { + t.Fatalf("SGLang work was not counted node-wide: %+v", ranks) + } + + m.recomputeAll(false) + for _, engine := range schedulerEngines { + got := rec.orders(engine) + if len(got) != 1 { + t.Fatalf("%s emissions = %d, want 1", engine, len(got)) + } + if got[0][len(got[0])-1] != "busy" { + t.Errorf("%s order = %v, want the busy node last", engine, got[0]) + } + } +} diff --git a/services/nvpair-job-scheduler/vllmengine_test.go b/services/nvpair-job-scheduler/vllmengine_test.go new file mode 100644 index 00000000..170501b7 --- /dev/null +++ b/services/nvpair-job-scheduler/vllmengine_test.go @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "slices" + "testing" +) + +// TestSchedulerEnginesCoversEveryEngine proves each engine gets its own +// priority contract. A proxy only applies the snapshot published for the engine +// it is routing, so an engine missing from this list would route with no +// scheduler ordering at all and every request would fall back to the stable-ID +// pass. +func TestSchedulerEnginesCoversEveryEngine(t *testing.T) { + for _, want := range []string{"ollama", "lmstudio", "vllm", "sglang"} { + if !slices.Contains(schedulerEngines, want) { + t.Errorf("schedulerEngines is missing %q: %v", want, schedulerEngines) + } + } +} + +// TestVLLMWorkloadsCountTowardTheNodeWideRanking proves vLLM work shares the +// node's queue depth with the other engines rather than being ranked +// separately: one GPU serves them all, so a node busy with vLLM must rank below +// an idle one for every engine, including Ollama. +func TestVLLMWorkloadsCountTowardTheNodeWideRanking(t *testing.T) { + rec := &capRW{} + m := mgrWith(rec, []string{"busy", "idle"}, + workload{ID: "1", Engine: "vllm", RunID: "v1", State: "running", OriginatedFrom: "x", ScheduledOn: "busy"}, + workload{ID: "2", Engine: "vllm", RunID: "v2", State: "queued", OriginatedFrom: "x", ScheduledOn: "busy"}, + ) + order, ranks := m.rank() + if len(order) == 0 || order[len(order)-1] != "busy" { + t.Fatalf("order = %v, want the vLLM-busy node last", order) + } + if pendingOf(ranks, "busy") != 2 { + t.Fatalf("vLLM work was not counted node-wide: %+v", ranks) + } + + m.recomputeAll(false) + for _, engine := range schedulerEngines { + got := rec.orders(engine) + if len(got) != 1 { + t.Fatalf("%s emissions = %d, want 1", engine, len(got)) + } + if got[0][len(got[0])-1] != "busy" { + t.Errorf("%s order = %v, want the busy node last", engine, got[0]) + } + } +} diff --git a/services/nvpair-manual-nodes/README.md b/services/nvpair-manual-nodes/README.md index 5fb648a6..815962f3 100644 --- a/services/nvpair-manual-nodes/README.md +++ b/services/nvpair-manual-nodes/README.md @@ -5,7 +5,14 @@ SPDX-License-Identifier: Apache-2.0 # nvpair-manual-nodes -A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable. Accepts node addresses via JSON-RPC, probes each for Ollama, LM Studio, and node-info, and emits status events. +A Go service for managing manually configured nodes on networks where mDNS discovery is unavailable — a filtered LAN, or an overlay network such as a Tailscale tailnet, which carries no multicast at all. Accepts node addresses via JSON-RPC, probes each one, and emits status events. + +There are two kinds of manual node, and one probe tells them apart. **node-info is asked first**, and its answer decides everything else: + +- **A bare inference host** — Ollama, LM Studio, vLLM, or SGLang on a machine that does not run PAIR. It serves no `/v1/node-info`, so its engines are probed on their own ports in plain HTTP and a supervising broker bridges it into the local proxies as a routing target. This is what manual nodes were originally for. +- **A PAIR node** — it answers `/v1/node-info` *with a `services` map*. It is then reported with `pair_node: true`, its cluster principal, its service map, and its model inventory read from its engine manager over cluster mTLS. A supervising broker folds it into the discovery directory as if it had been found over mDNS. + +A PAIR node is **never** probed on its engine ports. On such a node `:11434` and `:1234` are the proxy facades, which refuse plaintext from anything but loopback, so a probe there is a guaranteed `403` that would report a healthy peer as having no engines. ## Communication @@ -50,8 +57,15 @@ Emitted when a manually added node has been probed and its initial status determ "lmstudio_up":true, "lmstudio_port":1234, "lmstudio_models":["qwen2.5-7b-instruct"], + "vllm_up":true, + "vllm_port":8000, + "vllm_models":["Qwen/Qwen3-8B"], + "sglang_up":true, + "sglang_port":30000, + "sglang_models":["/models/my-model"], "node_info_up":true, "node_info_port":14318, + "pair_node":false, "gpus":[{"name":"NVIDIA GeForce RTX 3080","utilization_percent":37}], "telemetryValid":true, "msSince":120, @@ -60,7 +74,16 @@ Emitted when a manually added node has been probed and its initial status determ } ``` -Each node is probed for both inference engines: Ollama on its default `:11434` (`GET /` + `/api/tags`) and LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list). `lmstudio_up` / `lmstudio_port` / `lmstudio_models` mirror the `ollama_*` fields and let a supervising broker bridge the node into `lmstudio-proxy` the same way it bridges Ollama into `ollama-proxy`. A node can run either engine, both, or neither. +Each node is probed for every inference engine: Ollama on its default `:11434` (`GET /` + `/api/tags`), LM Studio on its default `:1234` (`GET /v1/models`, which doubles as the liveness check and the model list), vLLM on its default `:8000`, and SGLang on its default `:30000`. The `lmstudio_*`, `vllm_*` and `sglang_*` fields mirror the `ollama_*` ones and let a supervising broker bridge the node into the right proxy the same way it bridges Ollama into `ollama-proxy`. A node can run any combination of them, or none. + +All three OpenAI-compatible engines serve the same `/v1/models`, so the model list alone cannot tell them apart on a host running more than one. Each therefore has a second route that identifies it: + +- **vLLM** serves `GET /version` returning a JSON `version` field, which LM Studio does not. +- **SGLang** serves `GET /get_model_info` returning a JSON `model_path`, which neither of the others does. + +Both routes must answer before a node is reported as running that engine, and the vLLM probe additionally refuses a server that answers `/get_model_info`: that route is SGLang's own, so a server answering it is SGLang whatever else it serves. + +An SGLang model id is its `--served-model-name`, which defaults to `--model-path` verbatim — often a bare local directory such as `/models/my-model` rather than a Hugging Face id. It is reported as given and never parsed. ### `node/updated` @@ -91,11 +114,14 @@ A hostname is preferred over an IP literal: probe clients disable keep-alives sp | Param | Required | Description | |---|---|---| -| `address` | Yes | IP address or hostname of the node, with no port | +| `address` | Yes | IP address or host name of the node, with no port. A `host:port` string is **rejected** with an actionable error: every probe appends its own service port, so such an entry could never be reached | | `name` | No | Friendly name (used as node ID; defaults to `manual:
`) | -| `tls_port` | No | Probe node-info over HTTPS on this port instead of plain HTTP on `14318`. Echoed back as `tls_enabled` | +| `ports` | No | Per-service port overrides: `{node_info, cluster, ollama, lmstudio, vllm, sglang}`. An unset field keeps that service's default. Persisted and echoed back as `ports`. | +| `tls_port` | No | Probe node-info over HTTPS on this port instead of plain HTTP. Takes precedence over `ports.node_info`, since it names an HTTPS listener and therefore names its port. Echoed back as `tls_enabled` | | `mtls` | No | Stored and echoed back as `mtls_required`. The probe transport itself is chosen by `tls_port` and live cluster membership, so this field records intent rather than driving it | +An address may be a MagicDNS name (`gpu-box.tail1234.ts.net`), a `.local` name, an IPv4 literal, or an IPv6 literal in plain or bracketed form. + Response: the initial node status object. ### `node/remove` @@ -132,15 +158,19 @@ Changes the log level at runtime. Accepted as either a request (answered with `{ ## Probing -Each manual node is probed every 10 seconds, with a 3-second timeout per leg, for: +Each manual node is probed every 10 seconds, with a 3-second timeout per leg. **node-info is asked first**, because its answer decides which other legs may run at all: -- **Ollama** on port 11434: health check (`GET /`) and model list (`GET /api/tags`) -- **LM Studio** on port 1234: `GET /v1/models`, which doubles as the liveness check and the model list -- **Node Info** on port 14318, or `tls_port` over HTTPS: hardware inventory and identity (`GET /v1/node-info`) +- **Node Info** on port 14318 (or `ports.node_info`, or `tls_port` over HTTPS): hardware inventory, identity, cluster principal, and service map (`GET /v1/node-info`) +- Then, **only for a bare host** (node-info reported no service map): + - **Ollama** on port 11434 (or `ports.ollama`): health check (`GET /`) and model list (`GET /api/tags`) + - **LM Studio** on port 1234 (or `ports.lmstudio`): `GET /v1/models`, which doubles as the liveness check and the model list + - **vLLM** on port 8000 (or `ports.vllm`): `GET /version` (the disambiguator from LM Studio) followed by `GET /v1/models` for the model list, and refused if it also answers SGLang's `/get_model_info` + - **SGLang** on port 30000 (or `ports.sglang`): `GET /get_model_info` (the disambiguator from any other OpenAI-compatible server) followed by `GET /v1/models` for the model list +- Or, **only for a PAIR node**, its model inventory from its engine manager (`GET /v1/models` on the `em` port from the service map) over cluster mTLS, pinned to the peer's cluster principal. No pin, no models: a peer does not serve its inventory to a stranger, so the node appears with its hardware and gains its models once paired. A node can have any combination of these, or none if the target is unreachable. Status changes trigger `node/updated` events. Because change detection compares CPU, memory, and GPU values, a node running node-info emits a `node/updated` on most probe cycles as utilization moves. -The three engine ports are compiled in: only the node-info leg's port can be moved, via `tls_port`. A remote engine on a non-default port is not discovered. +`pair_node` holds across a failure episode rather than being recomputed per probe. One missed node-info answer is routine across an overlay network, and without that tolerance the gap would probe the peer's proxy facades in plaintext and withdraw it from every consumer for a cycle. It reverts past `probeFailThreshold` consecutive failures, so a node that genuinely stops being a PAIR node is not remembered as one. ## Shutdown diff --git a/services/nvpair-manual-nodes/manager.go b/services/nvpair-manual-nodes/manager.go index 55a4040a..a9f716c6 100644 --- a/services/nvpair-manual-nodes/manager.go +++ b/services/nvpair-manual-nodes/manager.go @@ -13,12 +13,15 @@ import ( "net" "net/http" "strconv" + "strings" "sync" "time" "nvpair-shared/applog" "nvpair-shared/clustertrust" "nvpair-shared/errors" + "nvpair-shared/netpick" + "nvpair-shared/noderec" ) // Version is stamped at build time via -ldflags "-X main.Version=...". @@ -43,6 +46,19 @@ const ( probeFailThreshold = 3 ) +// Default service ports for a manual node. A manual node is remote, so its ports +// are assumed rather than resolved (the engine manager governs only the local +// engine) — an entry's "ports" object overrides any of them for a host that runs +// something somewhere else. +const ( + defaultOllamaPort = 11434 + defaultLMStudioPort = 1234 + defaultVLLMPort = 8000 + defaultSGLangPort = 30000 + defaultNodeInfoPort = 14318 + defaultClusterPort = 14321 +) + type GPUInfo struct { Name string `json:"name"` VramBytes uint64 `json:"vram_bytes,omitempty"` @@ -78,6 +94,26 @@ type NodeInfoResponse struct { // machine if it's also discovered over mDNS. Empty when the remote // predates this field or isn't a NVPAIR node-info server. HostUUID string `json:"hostUuid,omitempty"` + // ClusterUUID is the remote's cluster principal. Tri-state on the wire and + // therefore a pointer here: absent means the remote does not know its own + // membership, present-and-empty means it belongs to no cluster, and a value is + // that principal. Reading absent as unclustered would have this node clear a + // correct annotation and dial a clustered peer's mTLS surfaces in plaintext. + ClusterUUID *string `json:"clusterUuid,omitempty"` + // Services is the remote's {service key: port} set. Its presence is what + // identifies the remote as a PAIR node: a bare Ollama or LM Studio host serves + // no /v1/node-info at all, and a PAIR node too old to report the set is + // treated as a bare host, which is what it can be reached as. + Services map[noderec.ServiceKey]int `json:"services,omitempty"` +} + +// peerModels is the inventory read from a paired PAIR node's engine-manager. It +// is the same body the node-scanner fetches for a discovered peer, so a manual +// peer's models reach the fleet through the same shape. +type peerModels struct { + Models []string `json:"models"` + ModelsByEngine map[string][]string `json:"modelsByEngine"` + LoadedByEngine map[string][]string `json:"loadedByEngine"` } // ManualEntry is the user-supplied identity of a manually added @@ -93,6 +129,82 @@ type ManualEntry struct { Name string `json:"name"` TLSPort int `json:"tls_port,omitempty"` MTLS bool `json:"mtls,omitempty"` + // Ports overrides the assumed port of any single service on this host. It + // exists because the defaults are assumptions about a machine this process + // cannot introspect: a peer may run its engine on a second port, a whole PAIR + // node may be reachable on a forwarded range, and two nodes may share one + // loopback in a test. A zero or absent field keeps that service's default, so + // an entry overrides only what it means to. + Ports *ManualPorts `json:"ports,omitempty"` +} + +// ManualPorts is the per-service port override set for one manual entry. Field +// names follow the service names manual-nodes already uses rather than the +// compact mDNS TXT keys, because this is what an operator types. +// +// Every field here has a probe leg behind it (see engineLegs and +// probeNodeInfo), so an override changes where this process actually looks. +type ManualPorts struct { + NodeInfo int `json:"node_info,omitempty"` + Cluster int `json:"cluster,omitempty"` + Ollama int `json:"ollama,omitempty"` + LMStudio int `json:"lmstudio,omitempty"` + VLLM int `json:"vllm,omitempty"` + SGLang int `json:"sglang,omitempty"` +} + +// resolved returns this entry's ports with every unset field filled from the +// default for that service, so probe code reads one value per service and never +// repeats the defaulting rule. +func (e ManualEntry) resolved() ManualPorts { + p := ManualPorts{} + if e.Ports != nil { + p = *e.Ports + } + p.NodeInfo = portOr(p.NodeInfo, defaultNodeInfoPort) + p.Cluster = portOr(p.Cluster, defaultClusterPort) + p.Ollama = portOr(p.Ollama, defaultOllamaPort) + p.LMStudio = portOr(p.LMStudio, defaultLMStudioPort) + p.VLLM = portOr(p.VLLM, defaultVLLMPort) + p.SGLang = portOr(p.SGLang, defaultSGLangPort) + return p +} + +func portOr(override, fallback int) int { + if override > 0 { + return override + } + return fallback +} + +// validateManualAddress rejects what cannot work as a manual address, with the +// reason rather than a silent permanently-down entry. +// +// A "host:port" string is the common mistake: every probe appends its own +// service port to this value, so such an entry dials host:port:port and reads +// down forever. Per-service ports belong in "ports". Bare IPv6 literals are +// accepted in both plain and bracketed form; net.JoinHostPort re-brackets on the +// way out. +func validateManualAddress(addr string) (string, error) { + addr = strings.TrimSpace(addr) + if addr == "" { + return "", fmt.Errorf("address is required") + } + if unbracketed, ok := strings.CutPrefix(addr, "["); ok { + if inner, closed := strings.CutSuffix(unbracketed, "]"); closed { + addr = inner + } + } + if net.ParseIP(addr) != nil { + return addr, nil + } + if host, port, err := net.SplitHostPort(addr); err == nil && host != "" && port != "" { + return "", fmt.Errorf("address %q carries a port; give the host on its own (%q) and put per-service ports in \"ports\" — a service port is appended to this value, so a host:port entry can never be reached", addr, host) + } + if !netpick.Hostname(addr) { + return "", fmt.Errorf("address %q is neither an IP address nor a host name", addr) + } + return addr, nil } // ManualNodeStatus mirrors a manual entry plus the latest probe @@ -112,9 +224,21 @@ type ManualNodeStatus struct { // LM Studio is probed on its default OpenAI-API port the same way Ollama // is on 11434, so a manually-added node running LM Studio can be bridged // into lmstudio-proxy by a supervising broker. - LMStudioUp bool `json:"lmstudio_up"` - LMStudioPort int `json:"lmstudio_port"` - LMStudioModels []string `json:"lmstudio_models,omitempty"` + LMStudioUp bool `json:"lmstudio_up"` + LMStudioPort int `json:"lmstudio_port"` + LMStudioModels []string `json:"lmstudio_models,omitempty"` + // vLLM is probed on its default OpenAI-API port alongside LM Studio. Both + // speak the same API, so /v1/models alone cannot tell them apart; the probe + // additionally requires vLLM's own /version, which LM Studio does not serve. + VLLMUp bool `json:"vllm_up"` + VLLMPort int `json:"vllm_port"` + VLLMModels []string `json:"vllm_models,omitempty"` + // SGLang is probed on its own default port alongside the other two OpenAI + // servers. It is told apart by GET /get_model_info, a route neither LM Studio + // nor vLLM serves; see probeSGLang. + SGLangUp bool `json:"sglang_up"` + SGLangPort int `json:"sglang_port"` + SGLangModels []string `json:"sglang_models,omitempty"` NodeInfoUp bool `json:"node_info_up"` NodeInfoPort int `json:"node_info_port"` TLSEnabled bool `json:"tls_enabled,omitempty"` @@ -128,6 +252,29 @@ type ManualNodeStatus struct { // manual node carries the same permanent identity the rest of the system // keys on. Empty when node-info didn't report one. HostUUID string `json:"hostUuid,omitempty"` + // Ports echoes the entry's port overrides so a caller can render what it + // configured without holding its own copy. + Ports *ManualPorts `json:"ports,omitempty"` + // PairNode is true when node-info answered AND reported a service map — the + // remote is a PAIR node, not a bare inference host. It is the switch between + // the two kinds of manual node: a PAIR node is folded into the directory as a + // peer and reached through its proxies over cluster mTLS, and a bare host is + // bridged into the local proxies by its raw engine ports. The engine fields + // below stay zero for a PAIR node, because its 11434 / 1234 are proxy facades + // that refuse plaintext from anything but loopback. + PairNode bool `json:"pair_node"` + // ClusterUUID is the remote's cluster principal, carried through with its + // three states intact (see NodeInfoResponse.ClusterUUID). A consumer keys the + // peer's certificate pin on it. + ClusterUUID *string `json:"cluster_uuid,omitempty"` + // Services is the remote's {service key: port} set, verbatim from node-info. + Services map[noderec.ServiceKey]int `json:"services,omitempty"` + // Models / ModelsByEngine / LoadedByEngine are a paired PAIR node's inventory, + // read from its engine manager over cluster mTLS. Empty until this node holds + // a pin for the peer: model names are not served to a stranger. + Models []string `json:"models,omitempty"` + ModelsByEngine map[string][]string `json:"models_by_engine,omitempty"` + LoadedByEngine map[string][]string `json:"loaded_by_engine,omitempty"` } type ReadyParams struct { @@ -138,9 +285,9 @@ type trackedNode struct { entry ManualEntry status ManualNodeStatus - // consecutiveFails counts back-to-back probes where neither - // service answered (OllamaUp && NodeInfoUp both false). Reset - // to 0 on any probe where at least one service responded. + // consecutiveFails counts back-to-back probes where no service + // answered — no engine and no node-info. Reset to 0 on any probe + // where at least one service responded. // Used to gate probe-failed errors:report emits at // probeFailThreshold so a single transient failure doesn't // generate UI noise. @@ -247,53 +394,71 @@ func (m *Manager) probeAll(ctx context.Context) { } } +// lastProbe returns a node's previous status and consecutive-failure count, and +// whether it is still tracked at all (it can be removed mid-probe). +func (m *Manager) lastProbe(id string) (ManualNodeStatus, int, bool) { + m.mu.RLock() + defer m.mu.RUnlock() + tn, ok := m.nodes[id] + if !ok { + return ManualNodeStatus{}, 0, false + } + return tn.status, tn.consecutiveFails, true +} + func (m *Manager) probeNode(entry ManualEntry) { addr := entry.Address id := nodeID(entry) + ports := entry.resolved() - ollamaUp, ollamaModels := m.probeOllama(addr, 11434) - lmStudioUp, lmStudioModels := m.probeLMStudio(addr, lmStudioPort) - - // Pick scheme + port + client based on the entry's TLS hint. - // The operator decides which scheme this manual node uses; we - // don't probe both. TLSPort > 0 means HTTPS on that port via - // the TLS client (which carries the operator's client cert, - // if configured). Otherwise it's plain HTTP on the historical - // 14318. - scheme := "http" - nodeInfoPort := 14318 - probeClient := m.client - if entry.TLSPort > 0 { - scheme = "https" - nodeInfoPort = entry.TLSPort - probeClient = m.tlsClient - // Clustered: a TLS manual node is a cluster peer whose node-info is - // pin-gated mTLS with no plaintext listener. Dial it with our cluster - // leaf, accepting any currently-pinned server cert (a manual node has no - // cluster-uuid= TXT to key a specific pin on). Refresh first so a cluster - // joined, or a peer paired, after startup is seen; falls back to the BYO - // tlsClient while unclustered. - m.mesh.Refresh() - if cfg, ok := m.mesh.ClientTLSConfigAny(); ok { - probeClient = &http.Client{Timeout: probeTimeout, Transport: &http.Transport{TLSClientConfig: cfg, DisableKeepAlives: true}} - } + // node-info is asked FIRST, because its answer decides what kind of node this + // is and therefore what else may be probed at all. + // + // A PAIR node's 11434, 1234, 8000 and 30000 are its proxy facades, not its + // engines: the engines bind loopback and the facades refuse plaintext from + // anything but loopback. Probing them would 403 every cycle and report a + // healthy peer as having no engines, so a node that identifies itself as PAIR + // is never probed there. Its engines are read from its engine manager instead, + // and it is routed to through those same facades over cluster mTLS. + nodeInfoUp, info := m.probeNodeInfo(entry, ports) + + // What this node was on the previous cycle, read before anything else runs: + // the sticky decision below has to be made before the engine legs, not after. + last, lastFails, tracked := m.lastProbe(id) + if !tracked { + return + } + + // A single missed node-info answer must not turn a peer back into a stranger. + // Three seconds is a routine gap across an overlay network — a relayed path, a + // wake from sleep, a peer under load — and without hysteresis that gap would + // probe the peer's proxy facades in plaintext, blank its service map and + // withdraw it from every consumer, only to restore it 10 seconds later. + // Discovery tolerates three consecutive misses before evicting an mDNS node; + // this is the same tolerance, bounded by the same counter, so a node that + // genuinely stops being a PAIR node still reverts within one failure episode. + pairNode := nodeInfoUp && len(info.Services) > 0 + sticky := !pairNode && last.PairNode && lastFails < probeFailThreshold + if sticky { + pairNode = true } - nodeInfoUp, info := m.probeNodeInfo(probeClient, scheme, addr, nodeInfoPort) newStatus := ManualNodeStatus{ ID: id, Name: entry.Name, Address: addr, - OllamaUp: ollamaUp, - OllamaPort: 11434, - OllamaModels: ollamaModels, - LMStudioUp: lmStudioUp, - LMStudioPort: lmStudioPort, - LMStudioModels: lmStudioModels, + OllamaPort: ports.Ollama, + LMStudioPort: ports.LMStudio, + VLLMPort: ports.VLLM, + SGLangPort: ports.SGLang, NodeInfoUp: nodeInfoUp, - NodeInfoPort: nodeInfoPort, + NodeInfoPort: m.nodeInfoProbePort(entry, ports), TLSEnabled: entry.TLSPort > 0, MTLSRequired: entry.TLSPort > 0 && entry.MTLS, + Ports: entry.Ports, + PairNode: pairNode, + ClusterUUID: info.ClusterUUID, + Services: info.Services, GPUs: info.GPUs, CPU: info.CPU, Memory: info.Memory, @@ -302,7 +467,34 @@ func (m *Manager) probeNode(entry ManualEntry) { HostUUID: info.HostUUID, } - reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.NodeInfoUp + if sticky { + // Carry the peer's identity forward across the gap, exactly as HostUUID is + // carried below: what a consumer holds must not oscillate because one + // probe timed out. NodeInfoUp stays false, so the node still reads as + // unreachable and the failure counter still runs. + newStatus.ClusterUUID = last.ClusterUUID + newStatus.Services = last.Services + newStatus.Models = last.Models + newStatus.ModelsByEngine = last.ModelsByEngine + newStatus.LoadedByEngine = last.LoadedByEngine + newStatus.GPUs = last.GPUs + newStatus.CPU = last.CPU + newStatus.Memory = last.Memory + } else if pairNode { + models := m.fetchPeerModels(addr, info) + newStatus.Models = models.Models + newStatus.ModelsByEngine = models.ModelsByEngine + newStatus.LoadedByEngine = models.LoadedByEngine + } else { + // A bare inference host: the engines themselves are on these ports, in + // plaintext, which is what manual nodes were originally for. + for _, leg := range m.engineLegs(ports) { + up, models := leg.probe(addr, leg.port) + leg.apply(&newStatus, up, models) + } + } + + reachable := newStatus.OllamaUp || newStatus.LMStudioUp || newStatus.VLLMUp || newStatus.SGLangUp || newStatus.NodeInfoUp m.mu.Lock() tn, exists := m.nodes[id] @@ -331,10 +523,20 @@ func (m *Manager) probeNode(entry ManualEntry) { changed := prev.OllamaUp != newStatus.OllamaUp || prev.LMStudioUp != newStatus.LMStudioUp || + prev.VLLMUp != newStatus.VLLMUp || + prev.SGLangUp != newStatus.SGLangUp || prev.NodeInfoUp != newStatus.NodeInfoUp || prev.HostUUID != newStatus.HostUUID || + prev.PairNode != newStatus.PairNode || + !clusterUUIDEqual(prev.ClusterUUID, newStatus.ClusterUUID) || + !servicesEqual(prev.Services, newStatus.Services) || + !sliceEqual(prev.Models, newStatus.Models) || + !byEngineEqual(prev.ModelsByEngine, newStatus.ModelsByEngine) || + !byEngineEqual(prev.LoadedByEngine, newStatus.LoadedByEngine) || !sliceEqual(prev.OllamaModels, newStatus.OllamaModels) || !sliceEqual(prev.LMStudioModels, newStatus.LMStudioModels) || + !sliceEqual(prev.VLLMModels, newStatus.VLLMModels) || + !sliceEqual(prev.SGLangModels, newStatus.SGLangModels) || !gpusEqual(prev.GPUs, newStatus.GPUs) || !cpuEqual(prev.CPU, newStatus.CPU) || !memoryEqual(prev.Memory, newStatus.Memory) || @@ -397,11 +599,52 @@ func probeFailedID(nodeID string) string { return "manual-nodes:probe-failed:" + nodeID } -// lmStudioPort is LM Studio's default OpenAI-API server port, probed the same -// way Ollama is hardcoded to 11434. A manual node is remote, so (like Ollama) -// we assume the engine's default port rather than resolving it via the engine -// manager (which only governs the local engine). -const lmStudioPort = 1234 +// engineLeg is one plain-HTTP inference-engine probe against a bare host. The +// legs are a table rather than a sequence of calls so adding an engine is one +// entry: its port, how it is probed, and where its result lands on the status. +type engineLeg struct { + name string + port int + probe func(addr string, port int) (bool, []string) + apply func(s *ManualNodeStatus, up bool, models []string) +} + +func (m *Manager) engineLegs(ports ManualPorts) []engineLeg { + return []engineLeg{ + { + name: "ollama", + port: ports.Ollama, + probe: m.probeOllama, + apply: func(s *ManualNodeStatus, up bool, models []string) { + s.OllamaUp, s.OllamaModels = up, models + }, + }, + { + name: "lmstudio", + port: ports.LMStudio, + probe: m.probeLMStudio, + apply: func(s *ManualNodeStatus, up bool, models []string) { + s.LMStudioUp, s.LMStudioModels = up, models + }, + }, + { + name: "vllm", + port: ports.VLLM, + probe: m.probeVLLM, + apply: func(s *ManualNodeStatus, up bool, models []string) { + s.VLLMUp, s.VLLMModels = up, models + }, + }, + { + name: "sglang", + port: ports.SGLang, + probe: m.probeSGLang, + apply: func(s *ManualNodeStatus, up bool, models []string) { + s.SGLangUp, s.SGLangModels = up, models + }, + }, + } +} // probeLMStudio checks LM Studio's OpenAI-compatible server on addr:port. A // single GET /v1/models doubles as the liveness check and the model list (the @@ -446,6 +689,168 @@ func (m *Manager) probeLMStudio(addr string, port int) (bool, []string) { return true, models } +// probeVLLM checks vLLM's OpenAI-compatible server on addr:port. LM Studio +// serves the same /v1/models, so a model list alone would let one engine be +// reported as the other on a host running both. vLLM additionally serves +// GET /version returning {"version": "..."} and LM Studio does not, so that +// route is the disambiguator: both must answer before the node is reported as +// running vLLM. +// +// It then refuses a server that also answers GET /get_model_info. That route is +// SGLang's own, so a server answering it is SGLang whatever else it serves, and +// vLLM must not claim it — a defensive guard rather than an observed collision, +// since SGLang answers no /version and so never reaches this check. An error or +// timeout on /get_model_info means "not SGLang, carry on as vLLM": treating an +// unanswered probe as a rejection would flap a slow-but-healthy vLLM to down +// every cycle. +// +// Returns whether it is up and the model ids it serves. +func (m *Manager) probeVLLM(addr string, port int) (bool, []string) { + if !m.probeVLLMVersion(addr, port) { + return false, nil + } + if m.probeSGLangModelInfo(addr, port) { + slog.Debug("manual probe vllm rejected: the server answers SGLang's /get_model_info", + "addr", addr, "port", port) + return false, nil + } + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/models" + start := time.Now() + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe vllm failed", + "addr", addr, "port", port, "duration_ms", time.Since(start).Milliseconds(), "err", err) + return false, nil + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + slog.Debug("manual probe vllm non-OK", + "addr", addr, "port", port, "status", resp.StatusCode, + "duration_ms", time.Since(start).Milliseconds()) + return false, nil + } + var result struct { + Data *[]struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.NewDecoder(resp.Body).Decode(&result); err != nil || result.Data == nil { + // /version identified it as vLLM, but its model list is not the OpenAI + // shape. Nothing can be routed to it, so it is not usable here. + slog.Debug("manual probe vllm model list unusable", "addr", addr, "port", port, "err", err) + return false, nil + } + models := make([]string, 0, len(*result.Data)) + for _, d := range *result.Data { + if d.ID != "" { + models = append(models, d.ID) + } + } + slog.Debug("manual probe vllm up", + "addr", addr, "port", port, "models", len(models), + "duration_ms", time.Since(start).Milliseconds()) + return true, models +} + +// probeVLLMVersion reports whether addr:port answers vLLM's GET /version with a +// JSON body carrying a version field. +func (m *Manager) probeVLLMVersion(addr string, port int) bool { + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/version" + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe vllm version failed", "addr", addr, "port", port, "err", err) + return false + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return false + } + var body struct { + Version string `json:"version"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return false + } + return body.Version != "" +} + +// probeSGLang checks SGLang's OpenAI-compatible server on addr:port. LM Studio +// and vLLM serve the same /v1/models, so a model list alone cannot tell any of +// the three apart; SGLang additionally serves GET /get_model_info, which neither +// of the others does, so that route is the disambiguator: both must answer +// before the node is reported as running SGLang. +// +// The model ids come from /v1/models as for the other OpenAI engines. SGLang's +// id is its --served-model-name, which defaults to --model-path verbatim, so it +// may be a Hugging Face id or a bare local directory. Nothing here interprets +// the shape. Returns whether it is up and the model ids it serves. +func (m *Manager) probeSGLang(addr string, port int) (bool, []string) { + if !m.probeSGLangModelInfo(addr, port) { + return false, nil + } + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/models" + start := time.Now() + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe sglang failed", + "addr", addr, "port", port, "duration_ms", time.Since(start).Milliseconds(), "err", err) + return false, nil + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + slog.Debug("manual probe sglang non-OK", + "addr", addr, "port", port, "status", resp.StatusCode, + "duration_ms", time.Since(start).Milliseconds()) + return false, nil + } + var result struct { + Data *[]struct { + ID string `json:"id"` + } `json:"data"` + } + if err := json.NewDecoder(resp.Body).Decode(&result); err != nil || result.Data == nil { + // /get_model_info identified it as SGLang, but its model list is not the + // OpenAI shape. Nothing can be routed to it, so it is not usable here. + slog.Debug("manual probe sglang model list unusable", "addr", addr, "port", port, "err", err) + return false, nil + } + models := make([]string, 0, len(*result.Data)) + for _, d := range *result.Data { + if d.ID != "" { + models = append(models, d.ID) + } + } + slog.Debug("manual probe sglang up", + "addr", addr, "port", port, "models", len(models), + "duration_ms", time.Since(start).Milliseconds()) + return true, models +} + +// probeSGLangModelInfo reports whether addr:port answers SGLang's +// GET /get_model_info with a JSON body carrying a model_path. It is both +// SGLang's identity check and, inverted, vLLM's guard against claiming an +// SGLang, so any failure — unreachable, non-200, unparseable, empty +// model_path — answers "not SGLang" rather than propagating an error. +func (m *Manager) probeSGLangModelInfo(addr string, port int) bool { + url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/get_model_info" + resp, err := m.client.Get(url) + if err != nil { + slog.Debug("manual probe sglang model-info failed", "addr", addr, "port", port, "err", err) + return false + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return false + } + var body struct { + ModelPath string `json:"model_path"` + } + if err := json.NewDecoder(resp.Body).Decode(&body); err != nil { + return false + } + return body.ModelPath != "" +} + func (m *Manager) probeOllama(addr string, port int) (bool, []string) { url := "http://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/" start := time.Now() @@ -496,7 +901,83 @@ func (m *Manager) fetchOllamaModels(addr string, port int) []string { return names } -func (m *Manager) probeNodeInfo(client *http.Client, scheme, addr string, port int) (bool, NodeInfoResponse) { +// nodeInfoProbePort is the port node-info is actually reached on: the entry's +// tls_port when one is set (it names an HTTPS listener, so it also names the +// port), otherwise the resolved plain node-info port. +func (m *Manager) nodeInfoProbePort(entry ManualEntry, ports ManualPorts) int { + if entry.TLSPort > 0 { + return entry.TLSPort + } + return ports.NodeInfo +} + +// nodeInfoClient picks the scheme, port and transport for the node-info probe. +// The operator decides which scheme a manual node uses; we do not probe both. +// tls_port > 0 means HTTPS on that port via the TLS client (carrying the +// operator's client cert, if configured), or over cluster mTLS while this node is +// a cluster member — a clustered peer running node-info standalone serves its +// inventory only to pinned peers. Otherwise it is plain HTTP on the resolved +// node-info port, which is what a peer under the broker serves: node-info is the +// one inter-node surface deliberately left readable by any peer. +func (m *Manager) nodeInfoClient(entry ManualEntry, ports ManualPorts) (*http.Client, string, int) { + if entry.TLSPort == 0 { + return m.client, "http", ports.NodeInfo + } + // Refresh first so a cluster joined, or a peer paired, after startup is seen; + // falls back to the BYO tlsClient while unclustered. + m.mesh.Refresh() + if cfg, ok := m.mesh.ClientTLSConfigAny(); ok { + return &http.Client{Timeout: probeTimeout, Transport: &http.Transport{TLSClientConfig: cfg, DisableKeepAlives: true}}, "https", entry.TLSPort + } + return m.tlsClient, "https", entry.TLSPort +} + +// fetchPeerModels reads a paired PAIR node's model inventory from its engine +// manager, the same body the node-scanner fetches for a discovered peer. +// +// It is pinned mTLS or nothing: engine-manager's LAN surface serves plaintext +// only over loopback, and a peer we hold no pin for is a stranger. An empty +// result is therefore the correct answer before pairing, not a failure — the node +// still appears with its hardware, and its models arrive once it is paired. +func (m *Manager) fetchPeerModels(addr string, info NodeInfoResponse) peerModels { + port, ok := info.Services[noderec.ServiceEngineManager] + if !ok || port <= 0 { + return peerModels{} + } + if info.ClusterUUID == nil || *info.ClusterUUID == "" { + slog.Debug("manual peer model fetch skipped: peer reports no cluster principal", "addr", addr) + return peerModels{} + } + m.mesh.Refresh() + cfg, ok := m.mesh.ClientTLSConfig(*info.ClusterUUID) + if !ok { + slog.Debug("manual peer model fetch skipped: no pin held for peer", "addr", addr) + return peerModels{} + } + client := &http.Client{Timeout: probeTimeout, Transport: &http.Transport{TLSClientConfig: cfg, DisableKeepAlives: true}} + url := "https://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/models" + resp, err := client.Get(url) + if err != nil { + slog.Debug("manual peer model fetch failed", "addr", addr, "port", port, "err", err) + return peerModels{} + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + slog.Debug("manual peer model fetch non-OK", "addr", addr, "port", port, "status", resp.StatusCode) + return peerModels{} + } + var models peerModels + if err := json.NewDecoder(resp.Body).Decode(&models); err != nil { + slog.Debug("manual peer model fetch decode failed", "addr", addr, "port", port, "err", err) + return peerModels{} + } + slog.Debug("manual peer models fetched", "addr", addr, "port", port, "models", len(models.Models)) + return models +} + +func (m *Manager) probeNodeInfo(entry ManualEntry, ports ManualPorts) (bool, NodeInfoResponse) { + client, scheme, port := m.nodeInfoClient(entry, ports) + addr := entry.Address url := scheme + "://" + net.JoinHostPort(addr, strconv.Itoa(port)) + "/v1/node-info" start := time.Now() resp, err := client.Get(url) @@ -531,18 +1012,19 @@ func (m *Manager) probeNodeInfo(client *http.Client, scheme, addr string, port i func (m *Manager) addNode(entry ManualEntry) ManualNodeStatus { id := nodeID(entry) - nodeInfoPort := 14318 - if entry.TLSPort > 0 { - nodeInfoPort = entry.TLSPort - } + ports := entry.resolved() status := ManualNodeStatus{ ID: id, Name: entry.Name, Address: entry.Address, - OllamaPort: 11434, - NodeInfoPort: nodeInfoPort, + OllamaPort: ports.Ollama, + LMStudioPort: ports.LMStudio, + VLLMPort: ports.VLLM, + SGLangPort: ports.SGLang, + NodeInfoPort: m.nodeInfoProbePort(entry, ports), TLSEnabled: entry.TLSPort > 0, MTLSRequired: entry.TLSPort > 0 && entry.MTLS, + Ports: entry.Ports, } m.mu.Lock() @@ -648,10 +1130,12 @@ func (m *Manager) handleMessage(msg *Message) { m.codec.RespondError(msg.ID, -32602, "invalid params: expected {\"address\": \"...\"}") return } - if entry.Address == "" { - m.codec.RespondError(msg.ID, -32602, "address is required") + address, err := validateManualAddress(entry.Address) + if err != nil { + m.codec.RespondError(msg.ID, -32602, err.Error()) return } + entry.Address = address status := m.addNode(entry) if err := m.codec.Respond(msg.ID, status); err != nil { log.Printf("failed to respond to node/add: %v", err) @@ -701,6 +1185,41 @@ func nodeID(entry ManualEntry) string { return "manual:" + entry.Address } +// clusterUUIDEqual compares the tri-state principal by value, so a peer that +// went from "unknown" to "unclustered" reports as changed rather than as the same +// empty string. +func clusterUUIDEqual(a, b *string) bool { + if a == nil || b == nil { + return a == b + } + return *a == *b +} + +func servicesEqual(a, b map[noderec.ServiceKey]int) bool { + if len(a) != len(b) { + return false + } + for k, v := range a { + if b[k] != v { + return false + } + } + return true +} + +func byEngineEqual(a, b map[string][]string) bool { + if len(a) != len(b) { + return false + } + for k, v := range a { + other, ok := b[k] + if !ok || !sliceEqual(v, other) { + return false + } + } + return true +} + func sliceEqual(a, b []string) bool { if len(a) != len(b) { return false diff --git a/services/nvpair-manual-nodes/manager_test.go b/services/nvpair-manual-nodes/manager_test.go index e52d9d5d..74fd509d 100644 --- a/services/nvpair-manual-nodes/manager_test.go +++ b/services/nvpair-manual-nodes/manager_test.go @@ -140,7 +140,7 @@ func TestProbeLMStudioReportsModels(t *testing.T) { m, _, rt := newTestManager() configureHealthyLMStudio(rt, "node.local", []string{"qwen2.5-7b", "llama-3.1-8b"}) - up, models := m.probeLMStudio("node.local", lmStudioPort) + up, models := m.probeLMStudio("node.local", defaultLMStudioPort) if !up { t.Fatal("expected lmstudio up") } @@ -148,7 +148,7 @@ func TestProbeLMStudioReportsModels(t *testing.T) { t.Fatalf("models = %#v", models) } - downUp, downModels := m.probeLMStudio("absent.local", lmStudioPort) + downUp, downModels := m.probeLMStudio("absent.local", defaultLMStudioPort) if downUp || downModels != nil { t.Fatalf("expected absent lmstudio down, got up=%v models=%#v", downUp, downModels) } diff --git a/services/nvpair-manual-nodes/pairnode_test.go b/services/nvpair-manual-nodes/pairnode_test.go new file mode 100644 index 00000000..57cdd407 --- /dev/null +++ b/services/nvpair-manual-nodes/pairnode_test.go @@ -0,0 +1,334 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Two kinds of manual node, told apart by one probe. +// +// A bare Ollama / LM Studio box is what manual nodes were originally for: its +// engines answer plain HTTP on their own ports and it serves no /v1/node-info. +// A PAIR node is the other kind, and it is the one an operator adds when the only +// route between two machines is an overlay network such as a Tailscale tailnet, +// where multicast never arrives and nothing is ever discovered. Its 11434 and +// 1234 carry proxy facades that refuse plaintext from anything but loopback, so +// probing them reports a healthy peer as having no engines. + +package main + +import ( + "encoding/json" + "errors" + "net" + "net/http" + "strconv" + "strings" + "sync/atomic" + "testing" + + "nvpair-shared/noderec" +) + +func strptr(s string) *string { return &s } + +// errFakeTimeout stands in for a probe that got no answer. +var errFakeTimeout = errors.New("simulated probe timeout") + +// probeSync registers an entry and probes it on this goroutine, so a test reads a +// settled status instead of racing addNode's background probe. +func probeSync(m *Manager, entry ManualEntry) { + m.mu.Lock() + m.nodes[nodeID(entry)] = &trackedNode{entry: entry} + m.mu.Unlock() + m.probeNode(entry) +} + +func statusFor(t *testing.T, m *Manager, id string) ManualNodeStatus { + t.Helper() + m.mu.RLock() + defer m.mu.RUnlock() + tn, ok := m.nodes[id] + if !ok { + t.Fatalf("no tracked node %q", id) + } + return tn.status +} + +// configurePairNode answers /v1/node-info on addr:port as a PAIR node: with a +// service map, which is what identifies it. +func configurePairNode(rt *fakeRoundTripper, addr string, port int, info NodeInfoResponse) { + host := net.JoinHostPort(addr, strconv.Itoa(port)) + rt.set(http.MethodGet, host, "/v1/node-info", func(*http.Request) (*http.Response, error) { + data, _ := json.Marshal(info) + return httpJSON(http.StatusOK, string(data)) + }) +} + +func pairNodeInfo() NodeInfoResponse { + return NodeInfoResponse{ + GPUs: []GPUInfo{{Name: "NVIDIA GeForce RTX 4090", UtilizationPercent: 12}}, + TelemetryValid: true, + HostUUID: "peer-host-uuid", + ClusterUUID: strptr("peer-cluster-uuid"), + Services: map[noderec.ServiceKey]int{ + noderec.ServiceNodeInfo: 14318, + noderec.ServiceOllama: 11434, + noderec.ServiceEngineManager: 14322, + noderec.ServiceEngineControl: 14323, + noderec.ServiceCluster: 14321, + }, + } +} + +// The regression gate: a PAIR node is never probed on its engine ports. A +// plaintext connection there is a guaranteed 403 from the peer's proxy facade, +// and treating that as "engine down" is what made a manually added PAIR peer read +// as having nothing to route to. +func TestProbeNode_PairNodeIsNotProbedOnItsEnginePorts(t *testing.T) { + m, _, rt := newTestManager() + + var engineProbes atomic.Int32 + for _, port := range []string{"11434", "1234"} { + host := net.JoinHostPort("gpu-box.tail1234.ts.net", port) + for _, path := range []string{"/", "/api/tags", "/v1/models"} { + rt.set(http.MethodGet, host, path, func(*http.Request) (*http.Response, error) { + engineProbes.Add(1) + return httpJSON(http.StatusForbidden, `{"error":"loopback-only"}`) + }) + } + } + configurePairNode(rt, "gpu-box.tail1234.ts.net", 14318, pairNodeInfo()) + + entry := ManualEntry{Address: "gpu-box.tail1234.ts.net"} + probeSync(m, entry) + + status := statusFor(t, m, nodeID(entry)) + if !status.PairNode { + t.Fatal("a node-info answer carrying a service map must mark the node as a PAIR node") + } + if n := engineProbes.Load(); n != 0 { + t.Fatalf("%d plaintext engine probes were made against a PAIR node, want 0", n) + } + if status.OllamaUp || status.LMStudioUp { + t.Fatalf("a PAIR node must report no raw engines (ollama_up=%v lmstudio_up=%v)", status.OllamaUp, status.LMStudioUp) + } + if status.Services[noderec.ServiceEngineManager] != 14322 { + t.Fatalf("services = %v, want the peer's em port carried through", status.Services) + } + if status.ClusterUUID == nil || *status.ClusterUUID != "peer-cluster-uuid" { + t.Fatalf("cluster_uuid = %v, want the peer's principal", status.ClusterUUID) + } + if status.HostUUID != "peer-host-uuid" { + t.Fatalf("hostUuid = %q, want the peer's identity", status.HostUUID) + } +} + +// One missed node-info answer must not turn a peer back into a stranger. Three +// seconds is a routine gap across an overlay network, and without hysteresis that +// gap would probe the peer's proxy facades in plaintext, blank its service map, +// and withdraw it from every consumer for a cycle. Discovery tolerates three +// consecutive misses before evicting an mDNS node; this tolerates the same. +func TestProbeNode_PairNodeSurvivesAMissedProbe(t *testing.T) { + m, _, rt := newTestManager() + + var engineProbes atomic.Int32 + for _, port := range []string{"11434", "1234"} { + host := net.JoinHostPort("gpu-box.tail1234.ts.net", port) + for _, path := range []string{"/", "/api/tags", "/v1/models"} { + rt.set(http.MethodGet, host, path, func(*http.Request) (*http.Response, error) { + engineProbes.Add(1) + return httpJSON(http.StatusForbidden, `{"error":"loopback-only"}`) + }) + } + } + + answering := true + rt.set(http.MethodGet, net.JoinHostPort("gpu-box.tail1234.ts.net", "14318"), "/v1/node-info", + func(*http.Request) (*http.Response, error) { + if !answering { + return nil, errFakeTimeout + } + data, _ := json.Marshal(pairNodeInfo()) + return httpJSON(http.StatusOK, string(data)) + }) + + entry := ManualEntry{Address: "gpu-box.tail1234.ts.net"} + probeSync(m, entry) + + answering = false + for i := 0; i < probeFailThreshold; i++ { + m.probeNode(entry) + status := statusFor(t, m, nodeID(entry)) + if !status.PairNode { + t.Fatalf("probe %d: a missed node-info answer must not un-pair a peer", i+1) + } + if status.NodeInfoUp { + t.Fatalf("probe %d: the node must still read as unreachable", i+1) + } + if status.Services[noderec.ServiceEngineManager] != 14322 { + t.Fatalf("probe %d: services = %v, want the peer's map carried across the gap", i+1, status.Services) + } + if status.ClusterUUID == nil || *status.ClusterUUID != "peer-cluster-uuid" { + t.Fatalf("probe %d: the peer's principal must survive the gap", i+1) + } + } + if n := engineProbes.Load(); n != 0 { + t.Fatalf("%d plaintext engine probes during the gap, want 0", n) + } + + // Past the tolerance it does revert: a node that genuinely stopped being a + // PAIR node must not be remembered as one forever. + m.probeNode(entry) + if statusFor(t, m, nodeID(entry)).PairNode { + t.Fatal("past the failure threshold the node must revert to a bare host") + } +} + +// A node-info answer with no service map is a host too old to report one, or one +// that is not a PAIR node at all. Either way the only thing it can be reached as +// is a bare inference host, so the plain engine probes still run. +func TestProbeNode_NodeInfoWithoutServicesStaysABareHost(t *testing.T) { + m, _, rt := newTestManager() + configureHealthyNode(rt, "10.0.0.9", []string{"llama3.2:latest"}, NodeInfoResponse{ + HostUUID: "bare-host", + TelemetryValid: true, + }) + + entry := ManualEntry{Address: "10.0.0.9"} + probeSync(m, entry) + + status := statusFor(t, m, nodeID(entry)) + if status.PairNode { + t.Fatal("a node-info answer without a service map must not read as a PAIR node") + } + if !status.OllamaUp || len(status.OllamaModels) != 1 { + t.Fatalf("bare host = ollama_up:%v models:%v, want the engine probed", status.OllamaUp, status.OllamaModels) + } +} + +// A paired peer's model inventory is pinned mTLS or nothing. Without a pin the +// node still appears with its hardware; its models arrive once it is paired. +func TestFetchPeerModels_RequiresAPin(t *testing.T) { + m, _, _ := newTestManager() + + if got := m.fetchPeerModels("gpu-box.tail1234.ts.net", pairNodeInfo()); len(got.Models) != 0 { + t.Fatalf("models = %v, want none while this node holds no pin for the peer", got.Models) + } + // An unclustered peer serves its engine manager over loopback only, so there + // is nothing to ask for either. + info := pairNodeInfo() + info.ClusterUUID = strptr("") + if got := m.fetchPeerModels("gpu-box.tail1234.ts.net", info); len(got.Models) != 0 { + t.Fatalf("models = %v, want none for an unclustered peer", got.Models) + } +} + +func TestValidateManualAddress(t *testing.T) { + for _, tc := range []struct { + in string + want string + }{ + {"10.0.0.9", "10.0.0.9"}, + {"gpu-box.tail1234.ts.net", "gpu-box.tail1234.ts.net"}, + {"gpu-box", "gpu-box"}, + {"100.101.102.103", "100.101.102.103"}, + {"fd7a:115c:a1e0::1", "fd7a:115c:a1e0::1"}, + {"[fd7a:115c:a1e0::1]", "fd7a:115c:a1e0::1"}, + {" 10.0.0.9 ", "10.0.0.9"}, + } { + got, err := validateManualAddress(tc.in) + if err != nil { + t.Errorf("validateManualAddress(%q) = error %v, want %q", tc.in, err, tc.want) + continue + } + if got != tc.want { + t.Errorf("validateManualAddress(%q) = %q, want %q", tc.in, got, tc.want) + } + } + + // A host:port entry used to be accepted and then read permanently down, + // because every probe appends its own port to the value. + for _, bad := range []string{"gpu-box.tail1234.ts.net:14318", "10.0.0.9:11434", "[fd7a::1]:14318"} { + got, err := validateManualAddress(bad) + if err == nil { + t.Errorf("validateManualAddress(%q) = %q, want a rejection", bad, got) + continue + } + if !strings.Contains(err.Error(), "ports") { + t.Errorf("rejection of %q = %q, want it to point at the ports object", bad, err) + } + } + + for _, bad := range []string{"", " ", "gpu box", "under_score.example"} { + if _, err := validateManualAddress(bad); err == nil { + t.Errorf("validateManualAddress(%q) succeeded, want a rejection", bad) + } + } +} + +func TestManualPorts_OverrideEveryServiceAndDefaultTheRest(t *testing.T) { + full := ManualEntry{Ports: &ManualPorts{ + NodeInfo: 24318, Cluster: 24321, Ollama: 21434, LMStudio: 2234, VLLM: 8001, SGLang: 30001, + }}.resolved() + want := ManualPorts{NodeInfo: 24318, Cluster: 24321, Ollama: 21434, LMStudio: 2234, VLLM: 8001, SGLang: 30001} + if full != want { + t.Fatalf("resolved = %+v, want %+v", full, want) + } + + partial := ManualEntry{Ports: &ManualPorts{Ollama: 21434}}.resolved() + if partial.Ollama != 21434 { + t.Errorf("ollama = %d, want the override", partial.Ollama) + } + if partial.NodeInfo != defaultNodeInfoPort || partial.LMStudio != defaultLMStudioPort || + partial.Cluster != defaultClusterPort || partial.VLLM != defaultVLLMPort || + partial.SGLang != defaultSGLangPort { + t.Errorf("resolved = %+v, want every unset field defaulted", partial) + } + + if none := (ManualEntry{}).resolved(); none.NodeInfo != defaultNodeInfoPort || none.Ollama != defaultOllamaPort { + t.Errorf("resolved with no overrides = %+v, want the defaults", none) + } +} + +// Two PAIR nodes can share one loopback when each is addressed by its own ports. +// That is what makes a manual entry usable for a forwarded range, and what lets a +// cross-process test stand up a peer without a second machine. +func TestProbeNode_PortOverridesAddressTheRightService(t *testing.T) { + m, _, rt := newTestManager() + configurePairNode(rt, "127.0.0.1", 24318, pairNodeInfo()) + + entry := ManualEntry{Address: "127.0.0.1", Name: "peer-b", Ports: &ManualPorts{NodeInfo: 24318}} + probeSync(m, entry) + + status := statusFor(t, m, "peer-b") + if !status.NodeInfoUp { + t.Fatal("node-info on the overridden port must be probed") + } + if status.NodeInfoPort != 24318 { + t.Fatalf("node_info_port = %d, want the override echoed", status.NodeInfoPort) + } + if status.Ports == nil || status.Ports.NodeInfo != 24318 { + t.Fatalf("ports = %+v, want the entry's overrides echoed back", status.Ports) + } +} + +// The engine legs are a table so an engine is one entry rather than a new pair of +// hardcoded calls. A bare host with a relocated engine is reached at its port. +func TestEngineLegs_FollowThePortOverrides(t *testing.T) { + m, _, rt := newTestManager() + configureHealthyLMStudio(rt, "10.0.0.9", []string{"qwen2.5-7b-instruct"}) + rt.set(http.MethodGet, net.JoinHostPort("10.0.0.9", "21434"), "/", func(*http.Request) (*http.Response, error) { + return httpJSON(http.StatusOK, `{}`) + }) + rt.set(http.MethodGet, net.JoinHostPort("10.0.0.9", "21434"), "/api/tags", func(*http.Request) (*http.Response, error) { + return httpJSON(http.StatusOK, `{"models":[{"name":"llama3.2:latest"}]}`) + }) + + entry := ManualEntry{Address: "10.0.0.9", Name: "relocated", Ports: &ManualPorts{Ollama: 21434}} + probeSync(m, entry) + + status := statusFor(t, m, "relocated") + if !status.OllamaUp || status.OllamaPort != 21434 { + t.Fatalf("ollama = up:%v port:%d, want the relocated engine found", status.OllamaUp, status.OllamaPort) + } + if !status.LMStudioUp || status.LMStudioPort != defaultLMStudioPort { + t.Fatalf("lmstudio = up:%v port:%d, want the default port still used", status.LMStudioUp, status.LMStudioPort) + } +} diff --git a/services/nvpair-manual-nodes/sglangprobe_test.go b/services/nvpair-manual-nodes/sglangprobe_test.go new file mode 100644 index 00000000..12dc6258 --- /dev/null +++ b/services/nvpair-manual-nodes/sglangprobe_test.go @@ -0,0 +1,208 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "testing" +) + +// sglangModelPath is the model id a real SGLang reports: --served-model-name +// defaults to --model-path verbatim, so a container started with +// `--model-path /models/my-model` advertises exactly that. Nothing in the +// probe may assume the Hugging Face "org/name" shape. +const sglangModelPath = "/models/my-model" + +// sglangStub serves the routes probeSGLang requires. modelPath == "" omits the +// model_path field, standing in for an OpenAI-compatible server that is not +// SGLang. servesVersion additionally answers vLLM's GET /version, which a real +// SGLang does not — that combination exists only to exercise probeVLLM's guard. +func sglangStub(t *testing.T, modelPath string, models []string, servesVersion bool) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/get_model_info": + if modelPath == "" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{ + "model_path": modelPath, + "tokenizer_path": modelPath, + "is_generation": true, + }) + case "/version": + if !servesVersion { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"version": "0.5.0"}) + case "/v1/models": + data := make([]map[string]any, 0, len(models)) + for _, id := range models { + data = append(data, map[string]any{"id": id, "object": "model", "owned_by": "sglang", "root": id}) + } + _ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data}) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return srv +} + +// TestProbeSGLangReportsModels covers the SGLang probe against a real HTTP +// server: a node answering both /get_model_info and /v1/models reports up with +// its model ids, path-shaped id and all. +func TestProbeSGLangReportsModels(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := sglangStub(t, sglangModelPath, []string{sglangModelPath}, false) + host, port := stubHostPort(t, srv) + + up, models := m.probeSGLang(host, port) + if !up { + t.Fatal("expected sglang up") + } + if len(models) != 1 || models[0] != sglangModelPath { + t.Fatalf("models = %#v, want the served model path carried verbatim", models) + } +} + +// TestProbeSGLangRejectsAnOpenAIServerThatIsNotSGLang is the disambiguator +// guard: LM Studio and vLLM serve the same /v1/models but neither serves +// /get_model_info, so an OpenAI server without it must never be reported as +// SGLang. +func TestProbeSGLangRejectsAnOpenAIServerThatIsNotSGLang(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := sglangStub(t, "", []string{"qwen2.5-7b"}, false) + host, port := stubHostPort(t, srv) + + if up, models := m.probeSGLang(host, port); up || models != nil { + t.Fatalf("an OpenAI server without /get_model_info was reported as SGLang: up=%v models=%#v", up, models) + } +} + +// TestProbeSGLangAbsentNodeIsDown covers the unreachable case. +func TestProbeSGLangAbsentNodeIsDown(t *testing.T) { + m, _, _ := newTestManager() + if up, models := m.probeSGLang("absent.local", defaultSGLangPort); up || models != nil { + t.Fatalf("expected absent sglang down, got up=%v models=%#v", up, models) + } +} + +// TestVLLMAndSGLangProbesDoNotClaimEachOther pins both directions of the +// disambiguation. A vLLM (serves /version, 404 on /get_model_info) is vLLM and +// not SGLang. A server that answers /get_model_info is SGLang and must not be +// claimed by probeVLLM even when it also answers /version — that combination is +// hypothetical (a real SGLang answers no /version, so it is already rejected on +// that leg alone) and the case exists to document the guard: /get_model_info is +// SGLang's own route, so a server answering it is SGLang whatever else it +// serves. +func TestVLLMAndSGLangProbesDoNotClaimEachOther(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + + t.Run("vllm stub", func(t *testing.T) { + // vllmStub serves /version and 404s everything else, /get_model_info + // included — exactly what a real vLLM does. + host, port := stubHostPort(t, vllmStub(t, "0.11.0", []string{"Qwen/Qwen3-8B"})) + up, models := m.probeVLLM(host, port) + if !up || len(models) != 1 || models[0] != "Qwen/Qwen3-8B" { + t.Errorf("probeVLLM on a vLLM = up:%v models:%#v, want it accepted", up, models) + } + if up, models := m.probeSGLang(host, port); up || models != nil { + t.Errorf("a vLLM was reported as SGLang: up=%v models=%#v", up, models) + } + }) + + t.Run("server answering get_model_info", func(t *testing.T) { + host, port := stubHostPort(t, sglangStub(t, sglangModelPath, []string{sglangModelPath}, true)) + if up, models := m.probeVLLM(host, port); up || models != nil { + t.Errorf("a server answering /get_model_info was reported as vLLM: up=%v models=%#v", up, models) + } + up, models := m.probeSGLang(host, port) + if !up || len(models) != 1 || models[0] != sglangModelPath { + t.Errorf("probeSGLang = up:%v models:%#v, want it accepted", up, models) + } + }) + + t.Run("real sglang serves no version", func(t *testing.T) { + // The shape the live engine actually has: /get_model_info answers, + // /version 404s. probeVLLM rejects it on the /version leg before the + // guard is even reached, which is why the guard is defense in depth. + host, port := stubHostPort(t, sglangStub(t, sglangModelPath, []string{sglangModelPath}, false)) + if up, _ := m.probeVLLM(host, port); up { + t.Error("an SGLang was reported as vLLM") + } + if up, _ := m.probeSGLang(host, port); !up { + t.Error("expected the SGLang to be accepted by its own probe") + } + }) +} + +// TestSGLangCountsAsReachable proves a node that runs only SGLang is not treated +// as unreachable, which would otherwise raise a probe-failed error for a healthy +// node and drop it from routing. It asserts against the manager's own +// reachability decision rather than a re-derived expression, so a leg missing +// from probeNode cannot pass here. +func TestSGLangCountsAsReachable(t *testing.T) { + m, rw, _ := newTestManager() + m.client = http.DefaultClient + srv := sglangStub(t, sglangModelPath, []string{sglangModelPath}, false) + host, port := stubHostPort(t, srv) + dead := closedPort(t) + + // Only the SGLang leg has anything to reach: every other service is pointed + // at a closed port, so the node's reachability rests on SGLang alone. Pointing + // them at the stub instead would not do — it answers /v1/models, which is all + // LM Studio's deliberately lenient probe asks for. + status := m.addNode(ManualEntry{Address: host, Ports: &ManualPorts{ + NodeInfo: dead, Cluster: dead, Ollama: dead, LMStudio: dead, VLLM: dead, SGLang: port, + }}) + got := decodeParams[ManualNodeStatus](t, readCaptureUntil(t, rw, methodIs("node/discovered"))) + + if !got.SGLangUp { + t.Fatalf("sglang leg did not come up: %+v", got) + } + if got.OllamaUp || got.LMStudioUp || got.VLLMUp || got.NodeInfoUp { + t.Fatalf("a leg pointed at a closed port reported up: %+v", got) + } + if len(got.SGLangModels) != 1 || got.SGLangModels[0] != sglangModelPath { + t.Errorf("sglang models = %v, want the served model path", got.SGLangModels) + } + if got.SGLangPort != port { + t.Errorf("sglang port = %d, want the override %d", got.SGLangPort, port) + } + + // The manager's own reachability decision, not one re-derived here: a node + // running nothing but SGLang must not accumulate probe failures, which would + // raise a probe-failed error for a healthy node and drop it from routing. + m.mu.RLock() + fails := m.nodes[status.ID].consecutiveFails + m.mu.RUnlock() + if fails != 0 { + t.Errorf("consecutiveFails = %d, want 0: an SGLang-only node is reachable", fails) + } +} + +// closedPort returns a port on loopback that nothing is listening on, so a probe +// against it fails immediately with a connection refusal rather than waiting out +// a timeout. +func closedPort(t *testing.T) int { + t.Helper() + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + port := l.Addr().(*net.TCPAddr).Port + if err := l.Close(); err != nil { + t.Fatal(err) + } + return port +} diff --git a/services/nvpair-manual-nodes/vllmprobe_test.go b/services/nvpair-manual-nodes/vllmprobe_test.go new file mode 100644 index 00000000..2c46ff80 --- /dev/null +++ b/services/nvpair-manual-nodes/vllmprobe_test.go @@ -0,0 +1,119 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "strconv" + "testing" +) + +// vllmStub serves the two routes probeVLLM requires. version == "" omits the +// version field, standing in for an OpenAI-compatible server that is not vLLM. +func vllmStub(t *testing.T, version string, models []string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/version": + if version == "" { + http.NotFound(w, r) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"version": version}) + case "/v1/models": + data := make([]map[string]string, 0, len(models)) + for _, id := range models { + data = append(data, map[string]string{"id": id, "object": "model"}) + } + _ = json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data}) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func stubHostPort(t *testing.T, srv *httptest.Server) (string, int) { + t.Helper() + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatal(err) + } + port, err := strconv.Atoi(u.Port()) + if err != nil { + t.Fatalf("port %q: %v", u.Port(), err) + } + return u.Hostname(), port +} + +// TestProbeVLLMReportsModels covers the vLLM probe against a real HTTP server: +// a node answering both /version and /v1/models reports up with its model ids. +func TestProbeVLLMReportsModels(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := vllmStub(t, "0.11.0", []string{"Qwen/Qwen3-8B"}) + host, port := stubHostPort(t, srv) + + up, models := m.probeVLLM(host, port) + if !up { + t.Fatal("expected vllm up") + } + if len(models) != 1 || models[0] != "Qwen/Qwen3-8B" { + t.Fatalf("models = %#v", models) + } +} + +// TestProbeVLLMRejectsAnOpenAIServerThatIsNotVLLM is the disambiguator guard: +// LM Studio serves the same /v1/models but has no /version, so an OpenAI server +// without one must never be reported as vLLM. +func TestProbeVLLMRejectsAnOpenAIServerThatIsNotVLLM(t *testing.T) { + m, _, _ := newTestManager() + m.client = http.DefaultClient + srv := vllmStub(t, "", []string{"qwen2.5-7b"}) + host, port := stubHostPort(t, srv) + + if up, models := m.probeVLLM(host, port); up || models != nil { + t.Fatalf("an OpenAI server without /version was reported as vLLM: up=%v models=%#v", up, models) + } +} + +// TestProbeVLLMAbsentNodeIsDown covers the unreachable case. +func TestProbeVLLMAbsentNodeIsDown(t *testing.T) { + m, _, _ := newTestManager() + if up, models := m.probeVLLM("absent.local", defaultVLLMPort); up || models != nil { + t.Fatalf("expected absent vllm down, got up=%v models=%#v", up, models) + } +} + +// TestVLLMCountsAsReachable proves a node that runs only vLLM is not treated as +// unreachable, which would otherwise raise a probe-failed error for a healthy +// node and drop it from routing. +func TestVLLMCountsAsReachable(t *testing.T) { + for _, tc := range []struct { + name string + status ManualNodeStatus + want bool + }{ + {name: "vllm only", status: ManualNodeStatus{VLLMUp: true}, want: true}, + {name: "lmstudio only", status: ManualNodeStatus{LMStudioUp: true}, want: true}, + {name: "sglang only", status: ManualNodeStatus{SGLangUp: true}, want: true}, + {name: "nothing", status: ManualNodeStatus{}, want: false}, + } { + t.Run(tc.name, func(t *testing.T) { + // Mirrors probeNode's reachability expression. It has to be kept in + // step by hand; TestSGLangCountsAsReachable asserts against the + // manager's own decision, which is what catches a leg that was added + // here but never wired into probeNode. + got := tc.status.OllamaUp || tc.status.LMStudioUp || tc.status.VLLMUp || + tc.status.SGLangUp || tc.status.NodeInfoUp + if got != tc.want { + t.Errorf("reachable = %v, want %v", got, tc.want) + } + }) + } +} diff --git a/services/nvpair-node-info/README.md b/services/nvpair-node-info/README.md index 8ed76948..f4017e46 100644 --- a/services/nvpair-node-info/README.md +++ b/services/nvpair-node-info/README.md @@ -11,7 +11,7 @@ A Go service that exposes this machine's hardware inventory (GPUs, CPU, physical Two surfaces: -- **HTTP(S)** — serves the node inventory at `/v1/node-info`. Plaintext HTTP on `:14318` by default; optional HTTPS (with optional mTLS) on `:14319` when a cert/key pair is supplied. +- **HTTP(S)** — serves the node inventory and this node's service map at `/v1/node-info`. Plaintext HTTP on `:14318` by default; optional HTTPS (with optional mTLS) on `:14319` when a cert/key pair is supplied. - **stdio JSON-RPC 2.0** — newline-delimited, used for lifecycle/control (`log/set-level`, `nodeinfo:set-cluster-identity`) and shutdown via stdin EOF. The service is normally launched as a subprocess by the broker. It does **not** advertise itself over mDNS. Discovery is centralized in the `nvpair-node-scanner` daemon: the broker registers this service's `ni` port with the daemon, which carries it on the node's one `_nvpair-node` record and fetches `/v1/node-info` over plain HTTP to enrich each node. @@ -71,7 +71,8 @@ Returns the merged static identity (collected once at startup) and the latest dy "used_bytes": 12884901888 }, "hostUuid": "8661676a-0d1c-4bd3-ac5e-4d370e6f1a9c", - "clusterUuid": "" + "clusterUuid": "", + "services": { "ni": 14318, "ol": 11434, "lm": 1234, "em": 14322, "ec": 14323, "cl": 14321 } } ``` @@ -81,6 +82,8 @@ Field notes: - `telemetryValid` and `msSince` describe the node-wide GPU utilization snapshot. `telemetryValid` is `true` after the collector has produced a usable GPU sample; `msSince` is that sample's age in milliseconds at response time. A failed collection retains the last usable sample and lets its age increase. Before the first usable sample, and on platforms without dynamic GPU telemetry, the response reports `telemetryValid:false` and `msSince:0`; consumers must ignore the age while validity is false. - `clusterUuid` is the cluster principal this node currently holds. It has three distinct states on the wire: **absent** means unknown, **present and empty** means this node belongs to no cluster, and a value is that principal. A consumer must not read absent as unclustered — that is how a node too old to report the field answers, and also how this node answers before its parent has told it anything, so acting on it would clear a correct annotation elsewhere in the fleet. - Under the broker, `clusterUuid` is pushed in over stdin (`nodeinfo:set-cluster-identity`) because node-info is spawned with no cluster dir and so cannot read membership itself; the field stays absent until the first push arrives. Standalone with `--cluster-dir`, it reads membership from the trust store per request instead and is therefore always known. The two sources are mutually exclusive by deployment, not a fallback chain. +- `services` is this node's `{service key: port}` set — the same set the node-scanner carries on this host's mDNS record, keyed by the same compact `nvpair-shared/noderec` service keys. It exists for the same reason `clusterUuid` does, and more so: that record is the only other place the set lives, and multicast does not cross a routed or overlay network. A peer that reached this node by a typed address reads it here and learns that this is a PAIR node and where each of its services listens. Its presence is what tells such a peer to route through this node's proxies over mutual TLS rather than probing its engine ports in plaintext. Absent means the parent has not pushed the set yet — not that this node runs nothing. +- Under the broker, `services` is pushed in over stdin (`nodeinfo:set-services`) on spawn and on every registration change, from the same cache the broker replays to the node-scanner, so the HTTP answer and the mDNS record are one derivation rather than two. The set is always sent whole: a service that stopped is expressed by its key being absent. - `clusterUuid` exists so a peer can learn this node's membership without its mDNS record. Membership otherwise travels only as the `cluster-uuid=` TXT key, which a consumer reads once per record *change*; a consumer that misses that change keeps the previous value indefinitely, and one still holding a departed node's principal will suppress the invite that would bring it back. - All dynamic fields and the `cpu` / `memory` objects use `omitempty`: a value the service couldn't read is dropped from the JSON entirely rather than reported as a misleading literal zero. A genuinely idle CPU renders the same as "unknown" — that ambiguity is intentional and benign. - `vram_bytes` is reported through DXGI on Windows, `nvidia-smi` on Linux, and IORegistry on macOS. On a unified-memory NVIDIA GPU such as DGX Spark, Linux uses total physical system memory for `vram_bytes` and the independently sampled system-memory usage for `vram_used_bytes`. On Apple Silicon, `vram_bytes` is total physical unified memory and `vram_used_bytes` is the GPU driver's mapped allocation (`Alloc system memory`), not whole-system RAM usage or the momentarily active subset. diff --git a/services/nvpair-node-info/cluster_identity_test.go b/services/nvpair-node-info/cluster_identity_test.go index 1cbc141c..fb4b8785 100644 --- a/services/nvpair-node-info/cluster_identity_test.go +++ b/services/nvpair-node-info/cluster_identity_test.go @@ -97,7 +97,7 @@ func TestBuildResponseClusterUUIDWireStates(t *testing.T) { raw := func(clusterUUID *string) map[string]any { t.Helper() var out map[string]any - if err := json.Unmarshal(buildResponse(nil, nil, 0, statsSnapshot{}, "host", clusterUUID), &out); err != nil { + if err := json.Unmarshal(buildResponse(nil, nil, 0, statsSnapshot{}, "host", clusterUUID, nil), &out); err != nil { t.Fatalf("decode: %v", err) } return out diff --git a/services/nvpair-node-info/main.go b/services/nvpair-node-info/main.go index 0b1f34c0..b4305221 100644 --- a/services/nvpair-node-info/main.go +++ b/services/nvpair-node-info/main.go @@ -99,6 +99,18 @@ type NodeInfoResponse struct { // evidence would have a peer clear a correct annotation and offer an invite // its target will reject. ClusterUUID *string `json:"clusterUuid,omitempty"` + // Services is this node's {service key: port} set — node-info, both proxies, + // engine manager, engine control, errors, workloads, cluster manager — the + // same set the node-scanner carries on this host's mDNS record. + // + // It is here because that record is the only other place it exists, and + // multicast does not cross a routed or overlay network. A peer that reached + // this node by a typed address (a Tailscale MagicDNS name, say) can read this + // and learn that the node is a PAIR node and where each of its services + // listens, which is everything a discovered peer knows. Absent means the + // parent has not pushed the set yet, or that this is not a PAIR node at all — + // a bare inference host answers no /v1/node-info. + Services map[noderec.ServiceKey]int `json:"services,omitempty"` } // clusterIdentity is the cluster principal this node reports, kept current by @@ -147,6 +159,61 @@ func handleClusterIdentity(msg applog.StdinMessage, identity *clusterIdentity) { slog.Info("cluster identity updated", "clustered", params.ClusterUUID != "") } +// serviceMap is this node's {service key: port} set, kept current by the parent +// broker over stdin (noderec.MethodSetServices). node-info cannot derive it: the +// ports belong to sibling processes the broker owns and re-assigns. Guarded +// because the stdin reader and every HTTP handler touch it. +type serviceMap struct { + mu sync.RWMutex + services map[noderec.ServiceKey]int +} + +// set replaces the whole set. The broker always sends it complete, so a service +// that stopped is expressed by its key being absent — merging would leave a +// departed service advertised forever. +func (s *serviceMap) set(services map[noderec.ServiceKey]int) { + next := make(map[noderec.ServiceKey]int, len(services)) + for k, port := range services { + if k != "" && port > 0 { + next[k] = port + } + } + s.mu.Lock() + defer s.mu.Unlock() + s.services = next +} + +// get returns a copy, or nil when nothing has been pushed. A copy because the +// value is marshaled outside the lock on every request. +func (s *serviceMap) get() map[noderec.ServiceKey]int { + s.mu.RLock() + defer s.mu.RUnlock() + if len(s.services) == 0 { + return nil + } + out := make(map[noderec.ServiceKey]int, len(s.services)) + for k, v := range s.services { + out[k] = v + } + return out +} + +// handleSetServices applies a MethodSetServices notification. Like the cluster +// identity push, a malformed payload is dropped rather than latching a wrong +// set: the broker re-pushes on every change, so the next one corrects us. +func handleSetServices(msg applog.StdinMessage, services *serviceMap) { + if msg.Method != noderec.MethodSetServices { + return + } + var params noderec.ServicesParams + if err := json.Unmarshal(msg.Params, ¶ms); err != nil { + slog.Warn("ignoring malformed service map push", "err", err) + return + } + services.set(params.Services) + slog.Info("service map updated", "services", len(params.Services)) +} + // detectGPUs lives in gpu_windows.go (DXGI), gpu_linux.go (nvidia-smi with a // ghw fallback), gpu_darwin.go (IORegistry), and gpu_other.go (ghw fallback). // detectCPU lives in cpu_detect.go; memory detection is split so macOS can use @@ -169,11 +236,11 @@ func handleClusterIdentity(msg applog.StdinMessage, identity *clusterIdentity) { // cpuStatic is nil when static CPU introspection failed; memTotal is zero when // physical-memory introspection failed. Both conditions omit their respective // top-level object from the JSON entirely. -func buildResponse(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap statsSnapshot, hostUUID string, clusterUUID *string) []byte { - return buildResponseAt(gpus, cpuStatic, memTotal, snap, hostUUID, clusterUUID, time.Now()) +func buildResponse(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap statsSnapshot, hostUUID string, clusterUUID *string, services map[noderec.ServiceKey]int) []byte { + return buildResponseAt(gpus, cpuStatic, memTotal, snap, hostUUID, clusterUUID, services, time.Now()) } -func buildResponseAt(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap statsSnapshot, hostUUID string, clusterUUID *string, now time.Time) []byte { +func buildResponseAt(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap statsSnapshot, hostUUID string, clusterUUID *string, services map[noderec.ServiceKey]int, now time.Time) []byte { outGPUs := mergeGPUInventory(gpus, snap.GPUInventory) for i := range outGPUs { gpu := &outGPUs[i] @@ -195,6 +262,7 @@ func buildResponseAt(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap s MSSince: msSince, HostUUID: hostUUID, ClusterUUID: clusterUUID, + Services: services, } if cpuStatic != nil { cpu := *cpuStatic @@ -392,6 +460,7 @@ func main() { // push the answer is genuinely unknown and the field is omitted. The two // sources are mutually exclusive by construction. identity := &clusterIdentity{} + services := &serviceMap{} clusterPrincipal := func() *string { if !clusterGated { uuid, told := identity.get() @@ -409,7 +478,7 @@ func main() { mux := http.NewServeMux() mux.HandleFunc("/v1/node-info", nodeInfoHandler(mesh, func() []byte { - return buildResponse(gpus, cpu, memTotal, collector.Snapshot(), hostUUID, clusterPrincipal()) + return buildResponse(gpus, cpu, memTotal, collector.Snapshot(), hostUUID, clusterPrincipal(), services.get()) })) // Listener layout (set up below depending on flags). Exactly one of the two @@ -549,6 +618,7 @@ func main() { go applog.StdinRPC(notifier, func(msg applog.StdinMessage) { handleClusterIdentity(msg, identity) + handleSetServices(msg, services) }, func() { log.Print("stdin closed, shutting down") cancel() diff --git a/services/nvpair-node-info/services_test.go b/services/nvpair-node-info/services_test.go new file mode 100644 index 00000000..3e35f068 --- /dev/null +++ b/services/nvpair-node-info/services_test.go @@ -0,0 +1,119 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// The {service: port} set reported on /v1/node-info. It is what lets a peer that +// reached this node by a typed address — one on a Tailscale tailnet, where no +// mDNS record ever arrives — learn that this is a PAIR node and where each of its +// services listens. + +package main + +import ( + "encoding/json" + "testing" + + "nvpair-shared/applog" + "nvpair-shared/noderec" +) + +func servicesPush(t *testing.T, services map[noderec.ServiceKey]int) applog.StdinMessage { + t.Helper() + params, err := json.Marshal(noderec.ServicesParams{Services: services}) + if err != nil { + t.Fatalf("marshal services params: %v", err) + } + return applog.StdinMessage{Method: noderec.MethodSetServices, Params: params} +} + +func TestServiceMap_AbsentUntilPushed(t *testing.T) { + services := &serviceMap{} + if got := services.get(); got != nil { + t.Fatalf("service map before any push = %v, want nil", got) + } + + var out map[string]any + if err := json.Unmarshal(buildResponse(nil, nil, 0, statsSnapshot{}, "host", nil, services.get()), &out); err != nil { + t.Fatalf("decode response: %v", err) + } + if _, present := out["services"]; present { + // A node that has not been told its own ports must not claim it has none: + // a peer would read that as "not a PAIR node" and fall back to probing + // raw engine ports that answer 403. + t.Fatal("services must be absent from the response until the parent pushes the set") + } +} + +func TestServiceMap_ReportedAfterPush(t *testing.T) { + services := &serviceMap{} + handleSetServices(servicesPush(t, map[noderec.ServiceKey]int{ + noderec.ServiceNodeInfo: 14318, + noderec.ServiceOllama: 11434, + noderec.ServiceEngineManager: 14322, + noderec.ServiceEngineControl: 14323, + }), services) + + var typed NodeInfoResponse + if err := json.Unmarshal(buildResponse(nil, nil, 0, statsSnapshot{}, "host", nil, services.get()), &typed); err != nil { + t.Fatalf("decode response: %v", err) + } + if got := typed.Services[noderec.ServiceOllama]; got != 11434 { + t.Fatalf("ol port = %d, want 11434", got) + } + if got := typed.Services[noderec.ServiceEngineControl]; got != 14323 { + t.Fatalf("ec port = %d, want 14323", got) + } + if len(typed.Services) != 4 { + t.Fatalf("services = %v, want 4 entries", typed.Services) + } +} + +// The set is replaced, never merged: a service that stopped is expressed by its +// key being absent, exactly as an unregister is on the discovery record. Merging +// would leave a departed service advertised to peers forever. +func TestServiceMap_PushReplacesRatherThanMerges(t *testing.T) { + services := &serviceMap{} + handleSetServices(servicesPush(t, map[noderec.ServiceKey]int{ + noderec.ServiceOllama: 11434, + noderec.ServiceLMStudio: 1234, + }), services) + handleSetServices(servicesPush(t, map[noderec.ServiceKey]int{ + noderec.ServiceOllama: 11434, + }), services) + + got := services.get() + if _, present := got[noderec.ServiceLMStudio]; present { + t.Fatalf("services = %v, want the departed lm key gone", got) + } +} + +func TestServiceMap_DropsUnusableEntriesAndMalformedPushes(t *testing.T) { + services := &serviceMap{} + handleSetServices(servicesPush(t, map[noderec.ServiceKey]int{ + noderec.ServiceOllama: 11434, + "": 9000, // no key + noderec.ServiceErrors: 0, // no port + }), services) + if got := services.get(); len(got) != 1 || got[noderec.ServiceOllama] != 11434 { + t.Fatalf("services = %v, want only the ol entry", got) + } + + // A malformed payload is dropped rather than latching: the broker re-pushes + // on every change, so the next one corrects us. + handleSetServices(applog.StdinMessage{ + Method: noderec.MethodSetServices, + Params: json.RawMessage(`{"services":"not a map"}`), + }, services) + if got := services.get(); len(got) != 1 { + t.Fatalf("services after a malformed push = %v, want the previous set kept", got) + } +} + +func TestServiceMap_IgnoresOtherMethods(t *testing.T) { + services := &serviceMap{} + msg := servicesPush(t, map[noderec.ServiceKey]int{noderec.ServiceOllama: 11434}) + msg.Method = noderec.MethodSetClusterIdentity + handleSetServices(msg, services) + if got := services.get(); got != nil { + t.Fatalf("services = %v, want nil for an unrelated method", got) + } +} diff --git a/services/nvpair-node-info/stats_test.go b/services/nvpair-node-info/stats_test.go index ad785017..05cf1e80 100644 --- a/services/nvpair-node-info/stats_test.go +++ b/services/nvpair-node-info/stats_test.go @@ -215,6 +215,7 @@ func TestBuildResponseTelemetryFreshness(t *testing.T) { }, "", nil, + nil, now, ) var typed NodeInfoResponse @@ -247,7 +248,7 @@ func TestBuildResponseTelemetryFreshness(t *testing.T) { // with no "cpu" key at all, not `"cpu":null`. func buildResponseDecode(t *testing.T, static []GPUInfo, cpu *CPUInfo, memTotal uint64, snap statsSnapshot) (NodeInfoResponse, map[string]any) { t.Helper() - body := buildResponse(static, cpu, memTotal, snap, "", nil) + body := buildResponse(static, cpu, memTotal, snap, "", nil, nil) var typed NodeInfoResponse if err := json.Unmarshal(body, &typed); err != nil { t.Fatalf("typed decode: %v", err) diff --git a/services/nvpair-node-scanner/README.md b/services/nvpair-node-scanner/README.md index ae2376e0..1b7f0351 100644 --- a/services/nvpair-node-scanner/README.md +++ b/services/nvpair-node-scanner/README.md @@ -9,7 +9,7 @@ SPDX-License-Identifier: Apache-2.0 Concretely, the daemon: -- **Advertises** this node's one `_nvpair-node._tcp` record, built from a registry of the local services its parent (the broker) registers — `ni`/`ol`/`lm`/`er`/`wl`/`cl`/`em`/`ec` ports plus the node identity (`uuid=`, `cluster-uuid=` when clustered, `ip=`). One record per node covers every service. +- **Advertises** this node's one `_nvpair-node._tcp` record, built from a registry of the local services its parent (the broker) registers — `ni`/`ol`/`lm`/`vl`/`sg`/`er`/`wl`/`cl`/`em`/`ec` ports plus the node identity (`uuid=`, `cluster-uuid=` when clustered, `ip=`). One record per node covers every service. - **Browses** `_nvpair-node._tcp` for every node on the LAN (including itself) and maintains a queryable directory keyed by host UUID. - **Enriches** each discovered node: GPU/CPU/memory inventory from its `ni` (node-info) port over plain HTTP, and the model list from its `em` (engine-manager) port (`GET /v1/models`) over cluster mTLS — a peer's model inventory is cluster data, so it is fetched only when this node holds a pin for the principal that peer advertises, while this node's own list is read over loopback in plaintext — the flat union, the per-engine breakdown (`modelsByEngine`), and the per-engine set of models loaded in memory (`loadedByEngine`), all carried onto the node's directory entry so remote cards can show loaded state. Each enrichment has a last-good cache so a transient fetch miss doesn't blank the node's card. Node-info enrichment can optionally be moved onto HTTPS (see the TLS flags below) — off by default and gated only on operator flags, never inferred per-node. - **Samples scheduling telemetry** from healthy nodes every two seconds with diff --git a/services/nvpair-tui/README.md b/services/nvpair-tui/README.md index e5b92ca7..529de356 100644 --- a/services/nvpair-tui/README.md +++ b/services/nvpair-tui/README.md @@ -30,7 +30,7 @@ Tabs: | **Overview** | Broker liveness/version/uptime (`ping`) and a per-worker health table derived from the broker's `supervisor:subprocess-crashed:*` errors. | | **Errors** | The service-error datastore (`errors:get-initial` + live `errors:update`); `c` clears the selected entry. | | **Nodes** | mDNS-discovered Ollama nodes (`discovery:subscribe` / `discovery:nodes-changed`). | -| **Proxies** | Ollama and LM Studio reverse proxies: status, discovered upstreams, select a node (`enter`/`a`), set the listen port (`p`). | +| **Proxies** | The Ollama and OpenAI-compatible (LM Studio, vLLM, SGLang) reverse proxies: status, discovered upstreams, select a node (`enter`/`a`), set the listen port (`p`). | | **Workloads** | Live cluster workloads (`workloads:subscribe` / `workloads:upsert` / `workloads:remove`). | | **Engines** | Local inference engines: install (`i`), start (`s`), stop (`x`), restart (`r`), uninstall (`u`). | | **Cluster** | Pairing + membership: invite by address (`i`, shows the six-digit PIN — the first invite auto-founds a cluster of one), accept (`a`) / decline (`d`) an inbound invite, remove a member (`r`), leave (`L`). | diff --git a/services/nvpair-tui/ui/proxies.go b/services/nvpair-tui/ui/proxies.go index 072c1a8b..62367988 100644 --- a/services/nvpair-tui/ui/proxies.go +++ b/services/nvpair-tui/ui/proxies.go @@ -24,11 +24,13 @@ type proxyNode struct { Port int `json:"port"` } -// proxyEngine is one of the two reverse proxies the broker fronts. Both +// proxyEngine is one of the reverse proxies the broker fronts. They all // speak the same routing/failover contract; only the JSON-RPC prefix and -// label differ. +// label differ. One proxy may front several engines — the OpenAI-compatible +// one serves LM Studio, vLLM and SGLang — so the label names the engines, not a +// binary. type proxyEngine struct { - label string // "Ollama" / "LM Studio" + label string // "Ollama" / "LM Studio / vLLM / SGLang" prefix string // "proxy" / "lmstudio-proxy" ready bool port int @@ -91,7 +93,7 @@ func newProxiesView(client *rpc.Client) *proxiesView { portInput: ti, engines: []*proxyEngine{ {label: "Ollama", prefix: "proxy", table: newTable(nil)}, - {label: "LM Studio", prefix: "lmstudio-proxy", table: newTable(nil)}, + {label: "LM Studio / vLLM / SGLang", prefix: "lmstudio-proxy", table: newTable(nil)}, }, } return v diff --git a/services/nvpair-ui-broker/README.md b/services/nvpair-ui-broker/README.md index d2b0e174..ddfdf2da 100644 --- a/services/nvpair-ui-broker/README.md +++ b/services/nvpair-ui-broker/README.md @@ -21,14 +21,14 @@ namespace: | `nvpair-node-scanner` | Discovery daemon: advertises this host's one `_nvpair-node._tcp` record and browses the LAN | `discovery:*` | | `nvpair-node-info` | Local GPU / CPU / memory inventory over HTTP at `/v1/node-info` | — (HTTP only) | | `ollama-proxy` | Ollama-compatible inference proxy and router | `proxy:*` | -| `lmstudio-proxy` | The LM Studio counterpart, supervised identically | `lmstudio-proxy:*` | +| `lmstudio-proxy` | The OpenAI-compatible counterpart (LM Studio, vLLM, SGLang), supervised identically | `lmstudio-proxy:*` | | `nvpair-engine-manager` | Local engine and model control plane; also serves `GET /v1/models` to peers | `engine:*` | | `nvpair-cluster-manager` | Node identity, trusted-node store, PIN pairing | `cluster:*`, `nodes:*` | | `nvpair-workload-manager` | Cluster workload relay between this node and peers | `workloads:*` | | `nvpair-job-scheduler` | Ranks nodes by pending work and GPU pressure for the proxies | _internal_ | | `nvpair-errors` | Service-error datastore and cross-node sync | `errors:*` | | `nvpair-node-settings` | Typed per-node settings store | `settings/*` | -| `nvpair-manual-nodes` | User-added nodes, merged into the discovery snapshot | `node/*`, `nodes/list` | +| `nvpair-manual-nodes` | User-added nodes, merged into the discovery snapshot; a user-added PAIR node is folded into the discovery relay as a peer | `node/*`, `nodes/list` | Only the scanner is required. Every other worker is optional: a missing binary leaves the broker running without that capability rather than failing to start. @@ -37,9 +37,14 @@ lifecycle, and relay rules. Two responsibilities live in the broker itself rather than in a worker: -- **Engine advertising.** The broker polls local Ollama and LM Studio every 5 s - and registers each running engine's port (`ol` / `lm`) with the discovery - daemon, so both are carried in this host's single `_nvpair-node` record. The +- **Engine advertising.** The broker polls local Ollama, LM Studio, vLLM and + SGLang every 5 s and registers each running engine's port + (`ol` / `lm` / `vl` / `sg`) with the discovery daemon, so all of them are + carried in this host's single `_nvpair-node` record. `lm`, `vl` and `sg` all + carry the port of the one OpenAI-compatible proxy, so a host serving from + several of those engines advertises the same port several times; which engine + owns a given model comes from engine-manager model attribution, not from the + key. The model list is not part of that record — it is served over HTTP by `nvpair-engine-manager` on the `em` service and fetched by a peer's daemon during discovery enrichment. @@ -77,7 +82,7 @@ Bidirectional newline-delimited JSON-RPC 2.0 — same conventions as every other | `--settings-path ` | `./nvpair-node-settings[.exe]` in the CWD | Explicit path to the `nvpair-node-settings` binary the broker spawns for the typed settings store. Same optional semantics as `--node-info-path` | | `--cluster-manager-path ` | `./nvpair-cluster-manager[.exe]` in the CWD | Explicit path to the `nvpair-cluster-manager` binary the broker spawns for cluster pairing / membership. Same optional semantics as `--node-info-path` | | `--scheduler-path ` | `./nvpair-job-scheduler[.exe]` in the CWD | Explicit path to the `nvpair-job-scheduler` binary the broker spawns for responsive, node-wide workload and GPU-pressure ranking. Same optional semantics as `--node-info-path` | -| `--cluster-dir ` | `cluster/` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) | Cluster config dir (`node.crt`/`node.key` + `trusted/`, minted by `nvpair-cluster-manager`). Threaded to the cluster-scoped workers (`nvpair-errors`, `nvpair-workload-manager`, `nvpair-node-scanner`, `nvpair-manual-nodes`, and `nvpair-engine-manager`), each of which derives its membership from it continuously — so a create, join, or leave takes effect in place and the broker does **not** restart them. The broker also passes the parent of this path to `nvpair-cluster-manager` as `--config-dir`, so the only writer of the cluster dir and the workers reading it cannot resolve different directories. `nvpair-node-info` is excluded — it stays plain HTTP even when clustered (see the repository-root `SECURITY.md`). Defaults so cluster mTLS auto-activates with nothing to pass; with an empty or cert-less dir this node is not a member, so it serves and dials no inter-node cluster traffic at all | +| `--cluster-dir ` | `cluster/` in the per-user data dir (`%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows, `~/.config/Nvidia Corporation/Personal AI Router` on Linux) | Cluster config dir (`node.crt`/`node.key` + `trusted/`, minted by `nvpair-cluster-manager`). Threaded to the cluster-scoped workers (`nvpair-errors`, `nvpair-workload-manager`, `nvpair-node-scanner`, `nvpair-manual-nodes`, and `nvpair-engine-manager`), each of which derives its membership from it continuously — so a create, join, or leave takes effect in place and the broker does **not** restart them. The broker also passes the parent of this path to `nvpair-cluster-manager` as `--config-dir`, so the only writer of the cluster dir and the workers reading it cannot resolve different directories. `nvpair-node-info` is excluded — it stays plain HTTP even when clustered (see the repository-root `SECURITY.md`), which is also what lets a peer that has not paired yet learn this node exists. Defaults so cluster mTLS auto-activates with nothing to pass; with an empty or cert-less dir this node is not a member, so it serves and dials no inter-node cluster traffic at all | | `--log-level ` | _(env `NVPAIR_LOG_LEVEL` or `info`)_ | `debug` \| `info` \| `warn` \| `error` | | `--version` | | Print version and exit | @@ -115,7 +120,7 @@ Two classes of proxy notification are **not** re-emitted under the `proxy:` name - **Inbound (peers -> manager -> broker).** The manager translates peer-origin lifecycle events into `workloads:upsert` and peer-origin removals into `workloads:remove` on stdout. The broker applies each accepted transition to the same store, fans it to the scheduler, and relays it to clients subscribed via `workloads:subscribe`. - **Local echo.** Local-origin proxy workloads are also emitted to the same `workloads:*` client stream (lifecycle translated to `workloads:upsert`), so a subscribed client sees a coherent cluster-wide view — its own workloads alongside peers'. -**`nvpair-job-scheduler`** consumes the accepted workload stream, compact GPU telemetry, and discovery snapshot. It smooths fresh utilization into pressure 0–3, uses neutral pressure 1 for invalid/missing/older-than-10-second samples, and orders by `pending + gpuPressure`, then pressure, then stable UUID. Load is node-wide across Ollama and LM Studio because both normally contend for the same resources. Each engine-specific `schedule:priority` carries `{engine,nodes,ranks}` and refreshes when order, pending counts, or pressure changes. The broker caches, generation-orders, and replays the full `{nodes,ranks}` snapshot to the matching proxy, where a newly delivered snapshot resets optimistic reservation deltas. On scheduler spawn/restart the broker replays active workloads and telemetry before discovery, then resumes all three live feeds. +**`nvpair-job-scheduler`** consumes the accepted workload stream, compact GPU telemetry, and discovery snapshot. It smooths fresh utilization into pressure 0–3, uses neutral pressure 1 for invalid/missing/older-than-10-second samples, and orders by `pending + gpuPressure`, then pressure, then stable UUID. Load is node-wide across Ollama, LM Studio, vLLM, and SGLang because they normally contend for the same resources. Each engine-specific `schedule:priority` carries `{engine,nodes,ranks}` and refreshes when order, pending counts, or pressure changes. The broker caches, generation-orders, and replays the full `{nodes,ranks}` snapshot to the matching proxy, where a newly delivered snapshot resets optimistic reservation deltas. On scheduler spawn/restart the broker replays active workloads and telemetry before discovery, then resumes all three live feeds. `schedule:priority` and `node/set-priority` are internal worker contracts: the broker does not expose either notification to its connected client. @@ -475,9 +480,17 @@ Any `settings/*` request is forwarded to `nvpair-node-settings` and its response #### `node/add` / `node/remove` / `nodes/list` (manual nodes) -Relayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, tls_port?, mtls? }`) registers a user-added node and probes it; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. A `nvpair-manual-nodes` restart loses the in-memory entries because neither that worker nor the broker persists an authoritative copy, so clients must re-add manual nodes after a restart. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised. +Relayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, ports?, tls_port?, mtls? }`) registers a user-added node and probes it. `address` must carry no port — `ports` (`{ node_info, cluster, ollama, lmstudio, vllm, sglang }`) moves any single service off its default; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. A `nvpair-manual-nodes` restart loses the in-memory entries because neither that worker nor the broker persists an authoritative copy, so clients must re-add manual nodes after a restart. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised. -**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), and one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`) — a node running both is bridged into both. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before. +**A manual node reaches inference one of two ways, never both**, decided by what `nvpair-manual-nodes` reports for it. + +**A bare inference host** (`pair_node: false`) — Ollama, LM Studio, vLLM or SGLang on a machine that does not run PAIR — is bridged into the proxies via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (from `ollama_port`), and one whose `lmstudio_*`, `vllm_*` or `sglang_*` status is up into the OpenAI proxy once per engine (from that engine's port), so a host running several of them contributes one entry each. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op. + +**A PAIR node** (`pair_node: true`, its node-info reported a `services` map) is instead folded into the **discovery relay**: the broker synthesizes the `DirectoryNode` the scanner would have produced — the typed address as its canonical one, the peer's cluster principal, its service map, its hardware and its models — and applies it to the same relay every consumer subscribes to. Both proxies, the scheduler, `nvpair-engine-manager`'s remote operations, the workload relay and the errors peer sync then treat it exactly as they treat a discovered pinned peer, and it is dialed over cluster mTLS to its own proxy ports. The raw-engine bridge is *withdrawn* for such a node rather than left alongside: those ports are the peer's proxy facades and refuse plaintext from anything but their own loopback, so a second plaintext candidate could only ever 403. + +The synthesis stands down in three cases, so it never fights another writer or points at this machine: the node's `hostUuid` is this node's own (a manual entry naming ourselves must not become a routing target), the scanner already claims that key (the daemon's record is authoritative and carries evidence the synthesis cannot), or the node is not a PAIR node. A withdrawal removes only what the broker itself synthesized. + +This is what makes a peer on a network that carries no multicast — a Tailscale tailnet, a WireGuard tunnel, a routed link — an ordinary cluster member. See [Running PAIR across a Tailscale tailnet](../../docs/remote-networks.mdx). #### `cluster:` / `nodes:` (generic relay) diff --git a/services/nvpair-ui-broker/advertiser.go b/services/nvpair-ui-broker/advertiser.go index c3f2e7d1..a1a42131 100644 --- a/services/nvpair-ui-broker/advertiser.go +++ b/services/nvpair-ui-broker/advertiser.go @@ -22,6 +22,12 @@ const ( // real port is resolved per poll via localEnginePort. defaultOllamaPort = 11434 defaultLMStudioPort = 1234 + // defaultVLLMPort is vLLM's stock OpenAI-API port, used on the same + // fallback-only terms as the two above. + defaultVLLMPort = 8000 + // defaultSGLangPort is SGLang's stock --port, used on the same fallback-only + // terms as the three above. + defaultSGLangPort = 30000 // engineManagerHTTPPort is the fixed LAN port the broker tells // nvpair-engine-manager to serve its HTTP surface (/v1/models) on, and the port @@ -181,6 +187,106 @@ func (b *Broker) reconcileAdvertiseLMStudio(client *http.Client) { } } +// runAutoAdvertiseVLLM is the vLLM sibling of runAutoAdvertiseLMStudio. vLLM +// speaks the same OpenAI API, so it is fronted by the same proxy; only the +// discovery service key (vl) and the engine name differ. There is no managed +// facade for vLLM — nothing serves :8000 on the proxy's behalf — so this loop +// has none of the facade/backend-cache handling the LM Studio one carries. +func (b *Broker) runAutoAdvertiseVLLM(ctx context.Context) { + client := &http.Client{Timeout: 2 * time.Second} + ticker := time.NewTicker(autoAdvertiseInterval) + defer ticker.Stop() + + b.reconcileAdvertiseVLLM(client) + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + b.reconcileAdvertiseVLLM(client) + } + } +} + +// reconcileAdvertiseVLLM brings this node's vl registration into line with the +// local vLLM server: it advertises the OpenAI proxy's port (never the engine's) +// and hands the engine's loopback port to that same proxy via +// node/set-local-backend, which keys its backends by engine so this never +// disturbs LM Studio's. +func (b *Broker) reconcileAdvertiseVLLM(client *http.Client) { + enginePort, probe := b.localEnginePort("vllm", defaultVLLMPort) + b.reconcileAdvertiseVLLMAt(client, enginePort, probe) +} + +// reconcileAdvertiseVLLMAt is reconcileAdvertiseVLLM with the engine port +// already resolved, so the registration and backend decisions can be exercised +// without an engine-manager. The health probe is the last term of the guard, so +// a collision or an authoritative "not running" short-circuits before any +// request is made — which is what lets a caller pass a nil client to assert +// exactly that. +func (b *Broker) reconcileAdvertiseVLLMAt(client *http.Client, enginePort int, probe bool) { + proxyPort := b.lmstudioProxyListenPort() + up := probe && proxyPort != 0 && enginePort != proxyPort && checkVLLMHealth(client, enginePort) + if up { + b.registerService(noderec.RegisterParams{Service: noderec.ServiceVLLM, Port: proxyPort}) + b.setProxyLocalBackend(b.getOpenAIProxy(), "vllm", enginePort, true) + } else { + b.unregisterService(noderec.ServiceVLLM) + b.setProxyLocalBackend(b.getOpenAIProxy(), "vllm", enginePort, false) + } +} + +// runAutoAdvertiseSGLang is the SGLang sibling of runAutoAdvertiseVLLM. SGLang +// speaks the same OpenAI API, so it too is fronted by the one OpenAI proxy; +// only the discovery service key (sg) and the engine name differ. Like vLLM it +// has no managed facade — nothing serves :30000 on the proxy's behalf — so this +// loop carries none of LM Studio's facade/backend-cache handling. +func (b *Broker) runAutoAdvertiseSGLang(ctx context.Context) { + client := &http.Client{Timeout: 2 * time.Second} + ticker := time.NewTicker(autoAdvertiseInterval) + defer ticker.Stop() + + b.reconcileAdvertiseSGLang(client) + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + b.reconcileAdvertiseSGLang(client) + } + } +} + +// reconcileAdvertiseSGLang brings this node's sg registration into line with the +// local SGLang server: it advertises the OpenAI proxy's port (never the +// engine's) and hands the engine's loopback port to that same proxy via +// node/set-local-backend, which keys its backends by engine so this never +// disturbs LM Studio's or vLLM's. +func (b *Broker) reconcileAdvertiseSGLang(client *http.Client) { + enginePort, probe := b.localEnginePort("sglang", defaultSGLangPort) + b.reconcileAdvertiseSGLangAt(client, enginePort, probe) +} + +// reconcileAdvertiseSGLangAt is reconcileAdvertiseSGLang with the engine port +// already resolved, so the registration and backend decisions can be exercised +// without an engine-manager. The health probe is the last term of the guard, so +// a collision or an authoritative "not running" short-circuits before any +// request is made — which is what lets a caller pass a nil client to assert +// exactly that. +func (b *Broker) reconcileAdvertiseSGLangAt(client *http.Client, enginePort int, probe bool) { + proxyPort := b.lmstudioProxyListenPort() + up := probe && proxyPort != 0 && enginePort != proxyPort && checkSGLangHealth(client, enginePort) + if up { + b.registerService(noderec.RegisterParams{Service: noderec.ServiceSGLang, Port: proxyPort}) + b.setProxyLocalBackend(b.getOpenAIProxy(), "sglang", enginePort, true) + } else { + b.unregisterService(noderec.ServiceSGLang) + b.setProxyLocalBackend(b.getOpenAIProxy(), "sglang", enginePort, false) + } +} + // proxyLocalBackend is the node/set-local-backend payload: the loopback engine // the proxy's cluster mTLS ingress forwards to, and the proxy's own self // candidate on the local routing path. @@ -249,6 +355,13 @@ func (b *Broker) proxyListenPort() int { return 0 } +// getOpenAIProxy returns the proxy that fronts every OpenAI-compatible engine. +// One process serves LM Studio, vLLM and SGLang: they speak the same HTTP +// surface, so they need one router, and node/set-local-backend keys its backends +// by engine so all of them are held at once. Named for the role rather than the +// binary, which keeps its historical lmstudio-proxy spelling as a wire contract. +func (b *Broker) getOpenAIProxy() *proxyProcess { return b.getLMStudioProxy() } + // lmstudioProxyListenPort is the LM Studio sibling of proxyListenPort. It // prevents the compatibility fallback from mistaking a proxy moved onto :1234 // for the actual engine. @@ -287,3 +400,36 @@ func checkLMStudioHealth(client *http.Client, port int) bool { resp.Body.Close() return resp.StatusCode == http.StatusOK } + +// checkVLLMHealth reports whether a local vLLM server is answering on the given +// port. vLLM exposes a dedicated /health, which is a cheaper and more precise +// liveness signal than its model list — and, unlike /v1/models, one the OpenAI +// proxy's own facade does not answer, so the proxy can never be mistaken for the +// engine. The port is resolved per poll (see localEnginePort), not hardcoded. +func checkVLLMHealth(client *http.Client, port int) bool { + resp, err := client.Get(fmt.Sprintf("http://localhost:%d/health", port)) + if err != nil { + return false + } + resp.Body.Close() + return resp.StatusCode == http.StatusOK +} + +// checkSGLangHealth reports whether a local SGLang server is answering on the +// given port. It asks for /get_model_info, SGLang's own metadata route, rather +// than /health: on current SGLang builds /health runs a real forward pass and +// takes about a second, and this loop runs every five seconds, so probing it +// would spend a fifth of the engine's time answering PAIR. /get_model_info +// answers in well under a millisecond, and because SGLang binds its port only +// once the model is loaded, a 200 from it means ready to serve, not merely +// started. It is also a route neither vLLM nor the OpenAI proxy's own facade +// answers, so nothing else can be mistaken for the engine. The port is resolved +// per poll (see localEnginePort), not hardcoded. +func checkSGLangHealth(client *http.Client, port int) bool { + resp, err := client.Get(fmt.Sprintf("http://localhost:%d/get_model_info", port)) + if err != nil { + return false + } + resp.Body.Close() + return resp.StatusCode == http.StatusOK +} diff --git a/services/nvpair-ui-broker/broker.go b/services/nvpair-ui-broker/broker.go index 0d189578..92efe8b2 100644 --- a/services/nvpair-ui-broker/broker.go +++ b/services/nvpair-ui-broker/broker.go @@ -75,7 +75,8 @@ type AvailableNode struct { // is the key clients should dedup/track nodes by; id/name stay the hostname // for display. Two machines sharing a hostname are distinct by HostUUID. It // is the same value the cluster surface exposes as nodeUuid. - // Omitted for manual nodes, which carry no UUID and are keyed by id. + // A manually added node carries one too: its real UUID once its node-info + // reports one, and its manual id until then, so it is never keyless. HostUUID string `json:"hostUuid,omitempty"` IPAddress string `json:"ipAddress"` // IPAddresses is every address this node published, in its own ranked order @@ -305,6 +306,12 @@ type Broker struct { manualMu sync.Mutex manualNodeKeys map[string]string manualNodeStatuses map[string]manualNodeStatusEntry + // manualRelayKeys are the directory keys this broker synthesized for manual + // PAIR nodes. The relay directory is keyed by hostUuid and the scanner writes + // the same keys, so a withdrawal has to know what it put there: removing a + // record the daemon owns would evict a live discovered peer from every + // consumer until its next browse event. + manualRelayKeys map[string]bool // schedMu guards each engine's cached priority and generation. Per-engine // delivery locks serialize asynchronous node/set-priority calls; a stale @@ -382,6 +389,7 @@ func NewBroker(codec *Codec, paths workerPaths) *Broker { relayDir: relay.NewDirectory(), regCache: relay.NewRegistrationCache(), manualNodeKeys: make(map[string]string), + manualRelayKeys: make(map[string]bool), manualNodeStatuses: make(map[string]manualNodeStatusEntry), workloads: workloadstore.New(), ollamaPortReady: make(chan struct{}), @@ -400,6 +408,7 @@ func (b *Broker) registerService(p noderec.RegisterParams) { if sc := b.getScanner(); sc != nil { go sc.pushRegister(p) } + go b.pushServicesToNodeInfo() } // unregisterService removes a local service from the cache and the daemon. @@ -410,6 +419,32 @@ func (b *Broker) unregisterService(svc noderec.ServiceKey) { if sc := b.getScanner(); sc != nil { go sc.pushUnregister(svc) } + go b.pushServicesToNodeInfo() +} + +// localServices projects the registration cache onto the {service: port} map +// node-info reports. One derivation, so what a peer reads over HTTP and what it +// would have read off this host's mDNS record are the same set. +func (b *Broker) localServices() map[noderec.ServiceKey]int { + regs := b.regCache.Snapshot() + services := make(map[noderec.ServiceKey]int, len(regs)) + for _, p := range regs { + services[p.Service] = p.Port + } + return services +} + +// pushServicesToNodeInfo sends node-info this node's current service map. +// Best-effort on a node-info that isn't running (never spawned, or mid-restart): +// the next spawn pushes again. +func (b *Broker) pushServicesToNodeInfo() { + np := b.getNodeInfo() + if np == nil { + return + } + if err := np.SetServices(b.localServices()); err != nil { + slog.Warn("failed to push service map to node-info", "err", err) + } } // get*/set* are the workersMu-guarded accessors for the supervised worker @@ -505,16 +540,21 @@ func (b *Broker) restoreEnabledEnginesAfterPortGate(ctx context.Context) bool { return true } -func (b *Broker) runEngineAvailabilityAfterPortGates( - ctx context.Context, - runOllama func(context.Context), - runLMStudio func(context.Context), -) bool { +// runEngineAvailabilityAfterPortGates starts every engine-advertise loop once +// the managed port gates have settled, so no loop can advertise a port that is +// still being moved. The last loop runs on this goroutine; the rest get their +// own. Variadic because the set grows with each engine PAIR fronts. +func (b *Broker) runEngineAvailabilityAfterPortGates(ctx context.Context, loops ...func(context.Context)) bool { if !b.restoreEnabledEnginesAfterPortGate(ctx) { return false } - go runOllama(ctx) - runLMStudio(ctx) + if len(loops) == 0 { + return true + } + for _, run := range loops[:len(loops)-1] { + go run(ctx) + } + loops[len(loops)-1](ctx) return true } @@ -626,6 +666,11 @@ func (b *Broker) spawnNodeInfo() (supervisedHandle, error) { // /v1/node-info but holds no cluster dir to read it from, so this push is the // only source. It runs on every spawn, which also covers a supervised restart. b.pushClusterIdentityToNodeInfo() + // And the service map, for the same reason: node-info is the only surface a + // peer that never saw this host's mDNS record can ask which services it runs. + // Pushed here as well as from registerService so a restart re-seeds the set + // the workers registered before node-info came back. + b.pushServicesToNodeInfo() // Register node-info's service so the daemon advertises ni= on _nvpair-node. // node-info binds the fixed :14318 (force_ports is inert), so the broker // knows its port. Idempotent across restarts. @@ -1263,7 +1308,7 @@ func (b *Broker) forwardManualNodesNotification(method string, params json.RawMe // the alias's key changed (node-info revealed its real UUID) the // old key is reprojected from a surviving alias or released. func (b *Broker) upsertManualNode(s manualNodeStatus) { - en := manualToEnriched(s) + en := b.manualEnriched(s) key := en.storeKey() receivedAt := time.Now() @@ -1275,6 +1320,9 @@ func (b *Broker) upsertManualNode(s manualNodeStatus) { b.store.Upsert(en, sourceManual) b.ingestTelemetryAt(sourceManual, manualNodeTelemetry(s, key), receivedAt) + // A PAIR node joins the discovery relay as if it had been found over mDNS, so + // every consumer treats it as the pinned peer it is. + b.applyManualDirectory(s, key) // Bridge a reachable manual node into each engine's proxy (ollama-proxy / // lmstudio-proxy) so inference can route to it; an unreachable engine is // pulled back out. No-op for a proxy the broker doesn't supervise. @@ -1318,13 +1366,15 @@ func (b *Broker) reprojectOrRelease(key string) { survivor, ok := b.survivingAliasLocked(key) b.manualMu.Unlock() if ok { - b.store.Upsert(manualToEnriched(survivor.status), sourceManual) + b.store.Upsert(b.manualEnriched(survivor.status), sourceManual) b.bridgeManualNode(survivor.status, key) + b.applyManualDirectory(survivor.status, key) b.ingestTelemetryAt(sourceManual, manualNodeTelemetry(survivor.status, key), survivor.receivedAt) return } b.store.Remove(key, sourceManual) b.removeManualNodeFromProxies(key) + b.releaseManualDirectory(key) b.removeTelemetry(sourceManual, key) } @@ -1371,6 +1421,9 @@ func (b *Broker) clearManualNodesState() { // doesn't keep a stale manual target the crashed prober can no // longer vouch for. Clients re-add manual nodes after the restart. b.removeManualNodeFromProxies(key) + // And out of the discovery relay, for the same reason: a synthesized PAIR + // peer is only as good as the prober that keeps vouching for it. + b.releaseManualDirectory(key) } } @@ -1462,8 +1515,9 @@ func (b *Broker) proxyForEngine(engine string) *proxyProcess { switch engine { case "ollama": return b.getProxy() - case "lmstudio": - return b.getLMStudioProxy() + case "lmstudio", "vllm", "sglang": + // Every OpenAI-compatible engine is fronted by the one OpenAI proxy. + return b.getOpenAIProxy() default: return nil } @@ -1775,11 +1829,13 @@ func (b *Broker) Serve(ctx context.Context) error { b.finishLMStudioProxyTerminal() } - // Restore engines and begin both advertising loops only after both proxy + // Restore engines and begin every advertising loop only after both proxy // startup attempts have established either readiness or a terminal outcome. // This prevents a restored engine from taking a persisted proxy port before // the broker can resolve ownership. - go b.runEngineAvailabilityAfterPortGates(ctx, b.runAutoAdvertise, b.runAutoAdvertiseLMStudio) + go b.runEngineAvailabilityAfterPortGates(ctx, + b.runAutoAdvertise, b.runAutoAdvertiseLMStudio, b.runAutoAdvertiseVLLM, + b.runAutoAdvertiseSGLang) // nvpair-workload-manager is another auxiliary worker: it relays local // workload lifecycle events to peer nodes and surfaces peer events diff --git a/services/nvpair-ui-broker/discovery.go b/services/nvpair-ui-broker/discovery.go index 8f216533..23048c85 100644 --- a/services/nvpair-ui-broker/discovery.go +++ b/services/nvpair-ui-broker/discovery.go @@ -39,8 +39,9 @@ type EnrichedNode struct { // It's the discovery-store key, so a PC rename — which changes the hostname // (ID) but not the UUID — updates the existing entry in place instead of // leaving a ghost under the old name. It stays off the wire; the - // client-facing id/name remain the hostname. Empty for manual nodes, which - // fall back to keying by their own ID. + // client-facing id/name remain the hostname. A manual node carries one too: + // its real UUID once its node-info reports one, and its manual id until then + // (see manualToEnriched), so the store key is never empty. HostUUID string `json:"-"` Host string `json:"host"` Port int `json:"port"` @@ -273,6 +274,19 @@ func (s *discoveryStore) Remove(key string, source nodeSource) bool { return gone } +// hasSource reports whether the given source currently claims key. It answers +// "does the daemon already own this record?" for the manual-node synthesis, +// which must not write a directory entry the scanner is authoritative for. +func (s *discoveryStore) hasSource(key string, source nodeSource) bool { + if key == "" { + return false + } + s.mu.RLock() + defer s.mu.RUnlock() + sn, ok := s.nodes[key] + return ok && sn.hasSource(source) +} + // Snapshot returns the narrow wire-format view used by // discovery:get-nodes and discovery:nodes-changed, sorted by id for // stable rendering. The rich EnrichedNode payload stays in the store @@ -672,3 +686,32 @@ func writeClusterIdentityFrame(mu *sync.Mutex, w io.Writer, clusterUUID string) _, err = w.Write(data) return err } + +// writeServicesFrame marshals a newline-delimited nodeinfo:set-services +// notification and writes it to a child's stdin under mu. The set is always sent +// whole: a service that stopped is expressed by its key being absent, exactly as +// an unregister is on the discovery record. +func writeServicesFrame(mu *sync.Mutex, w io.Writer, services map[noderec.ServiceKey]int) error { + if services == nil { + services = map[noderec.ServiceKey]int{} + } + frame := struct { + JSONRPC string `json:"jsonrpc"` + Method string `json:"method"` + Params noderec.ServicesParams `json:"params"` + }{ + JSONRPC: "2.0", + Method: noderec.MethodSetServices, + Params: noderec.ServicesParams{Services: services}, + } + data, err := json.Marshal(frame) + if err != nil { + return err + } + data = append(data, '\n') + + mu.Lock() + defer mu.Unlock() + _, err = w.Write(data) + return err +} diff --git a/services/nvpair-ui-broker/main.go b/services/nvpair-ui-broker/main.go index ced0d783..741b63b7 100644 --- a/services/nvpair-ui-broker/main.go +++ b/services/nvpair-ui-broker/main.go @@ -33,7 +33,7 @@ func main() { settingsPath := flag.String("settings-path", "", "path to nvpair-node-settings binary (default: ./nvpair-node-settings in the current working directory)") clusterMgrPath := flag.String("cluster-manager-path", "", "path to nvpair-cluster-manager binary (default: ./nvpair-cluster-manager in the current working directory)") schedulerPath := flag.String("scheduler-path", "", "path to nvpair-job-scheduler binary (default: ./nvpair-job-scheduler in the current working directory)") - clusterDirFlag := flag.String("cluster-dir", "", "cluster config dir (node.crt/node.key + trusted/) the broker passes to its mDNS workers (nvpair-errors, nvpair-workload-manager, nvpair-node-info, nvpair-node-scanner, nvpair-manual-nodes) to enable cluster-scoped inter-node mTLS; defaults to the per-user Nvidia Corporation/Personal AI Router cluster/ dir, where nvpair-cluster-manager mints them") + clusterDirFlag := flag.String("cluster-dir", "", "cluster config dir (node.crt/node.key + trusted/) the broker passes to its inter-node workers (nvpair-errors, nvpair-workload-manager, nvpair-node-scanner, nvpair-manual-nodes, nvpair-engine-manager, both proxies) to enable cluster-scoped inter-node mTLS. nvpair-node-info is deliberately NOT among them: its inventory is the one inter-node surface kept plain so any peer can read it; defaults to the per-user Nvidia Corporation/Personal AI Router cluster/ dir, where nvpair-cluster-manager mints them") showVersion := flag.Bool("version", false, "print version and exit") resolveLevel := applog.RegisterFlag(nil, slog.LevelInfo) flag.Parse() diff --git a/services/nvpair-ui-broker/manual_pairnode_test.go b/services/nvpair-ui-broker/manual_pairnode_test.go new file mode 100644 index 00000000..796b1d86 --- /dev/null +++ b/services/nvpair-ui-broker/manual_pairnode_test.go @@ -0,0 +1,196 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Folding a manually added PAIR node into the discovery relay. +// +// A peer reachable only across an overlay network such as a Tailscale tailnet is +// never discovered — no multicast crosses it — but it is not a different kind of +// node. Once its node-info reports a service map, the broker synthesizes the +// directory record the scanner would have produced, so both inference proxies, +// the scheduler, engine-manager's remote operations, the workload relay and the +// errors peer sync all see a pinned peer rather than a special case. + +package main + +import ( + "testing" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +func pairStatus() manualNodeStatus { + principal := "peer-cluster-uuid" + return manualNodeStatus{ + ID: "gpu-box.tail1234.ts.net", + Address: "gpu-box.tail1234.ts.net", + NodeInfoPort: 14318, + HostUUID: "peer-host-uuid", + PairNode: true, + ClusterUUID: &principal, + Services: map[noderec.ServiceKey]int{ + noderec.ServiceNodeInfo: 14318, + noderec.ServiceOllama: 11434, + noderec.ServiceLMStudio: 1234, + noderec.ServiceEngineManager: 14322, + noderec.ServiceEngineControl: 14323, + noderec.ServiceCluster: 14321, + }, + Models: []string{"llama3.2:latest"}, + ModelsByEngine: map[string][]string{"ollama": {"llama3.2:latest"}}, + TelemetryValid: true, + } +} + +func newPairPeerTestBroker(nodeID string) *Broker { + return &Broker{ + nodeID: nodeID, + store: newDiscoveryStore(), + relayDir: relay.NewDirectory(), + manualRelayKeys: make(map[string]bool), + } +} + +func TestManualDirectoryNode_CarriesEverythingAPeerNeedsToBeRouted(t *testing.T) { + b := newPairPeerTestBroker("this-host") + s := pairStatus() + + node, ok := b.manualDirectoryNode(s, s.HostUUID) + if !ok { + t.Fatal("a PAIR node with a service map must enter the directory") + } + if node.HostUUID != "peer-host-uuid" { + t.Errorf("hostUuid = %q, want the peer's identity", node.HostUUID) + } + // The typed address is this node's canonical one. It is a MagicDNS name here, + // which is the case that used to be dropped: a name is dialable, and it + // re-resolves, so it outlives every literal the peer holds. + if node.IP != "gpu-box.tail1234.ts.net" { + t.Errorf("ip = %q, want the address the operator typed", node.IP) + } + if got := node.CandidateIPs(); len(got) != 1 || got[0] != "gpu-box.tail1234.ts.net" { + t.Errorf("candidate addresses = %v, want the typed address", got) + } + if got := node.AddressTXT(); len(got) != 1 || got[0] != "ip=gpu-box.tail1234.ts.net" { + t.Errorf("address TXT = %v, want ip=
", got) + } + // The cluster principal is what a consumer pins the peer's certificate on. + // Without it every mTLS surface is dialed in plaintext and answers 403. + if node.ClusterUUID != "peer-cluster-uuid" || !node.Clustered() { + t.Errorf("clusterUuid = %q, want the peer's principal", node.ClusterUUID) + } + for svc, want := range map[noderec.ServiceKey]int{ + noderec.ServiceOllama: 11434, + noderec.ServiceLMStudio: 1234, + noderec.ServiceEngineManager: 14322, + noderec.ServiceEngineControl: 14323, + } { + got, ok := node.Services[svc] + if !ok || got.Port != want { + t.Errorf("service %s = %d (present:%v), want %d", svc, got.Port, ok, want) + } + } + if len(node.Models) != 1 || node.Models[0] != "llama3.2:latest" { + t.Errorf("models = %v, want the peer's inventory", node.Models) + } +} + +// A bare Ollama / LM Studio box has no proxy and no engine manager, so it is not +// a peer. It stays on the raw-engine bridge, which is what manual nodes were for. +func TestManualDirectoryNode_BareHostStaysOutOfTheDirectory(t *testing.T) { + b := newPairPeerTestBroker("this-host") + s := manualNodeStatus{ + ID: "10.0.0.9", Address: "10.0.0.9", HostUUID: "bare-host", + OllamaUp: true, OllamaPort: 11434, OllamaModels: []string{"llama3.2:latest"}, + } + if _, ok := b.manualDirectoryNode(s, s.HostUUID); ok { + t.Fatal("a bare inference host must not be synthesized as a directory peer") + } +} + +// A manual entry naming this machine must never become a routing target. The +// test is identity, not address: the same host reached by its overlay name is +// still us, and an address comparison would miss it. +func TestManualDirectoryNode_RefusesThisHost(t *testing.T) { + b := newPairPeerTestBroker("peer-host-uuid") + if _, ok := b.manualDirectoryNode(pairStatus(), "peer-host-uuid"); ok { + t.Fatal("a manual entry naming this host must not enter the directory") + } +} + +// When the daemon already holds the node, its record wins: it carries the peer's +// full ranked address list and its liveness probes, which this synthesis cannot. +func TestManualDirectoryNode_YieldsToTheScanner(t *testing.T) { + b := newPairPeerTestBroker("this-host") + b.store.Upsert(EnrichedNode{ID: "gpu-box", HostUUID: "peer-host-uuid"}, sourceScanner) + if _, ok := b.manualDirectoryNode(pairStatus(), "peer-host-uuid"); ok { + t.Fatal("the scanner's record is authoritative; the manual synthesis must stand down") + } +} + +func TestApplyManualDirectory_AddsAndWithdraws(t *testing.T) { + b := newPairPeerTestBroker("this-host") + s := pairStatus() + + b.applyManualDirectory(s, s.HostUUID) + if got := b.relayDir.Snapshot(noderec.ServiceOllama); len(got) != 1 || got[0].HostUUID != "peer-host-uuid" { + t.Fatalf("relay directory = %v, want the synthesized peer", got) + } + + // It went down, or turned out not to be a PAIR node: withdraw it, or the + // proxies keep a routing target nothing vouches for. + down := s + down.PairNode = false + b.applyManualDirectory(down, s.HostUUID) + if got := b.relayDir.Snapshot(""); len(got) != 0 { + t.Fatalf("relay directory = %v, want the withdrawn peer gone", got) + } +} + +// The relay directory is keyed by hostUuid and the scanner writes the same keys. +// A withdrawal must remove only what this broker synthesized, or a live +// discovered peer disappears from every consumer until its next browse event. +func TestReleaseManualDirectory_NeverWithdrawsADiscoveredPeer(t *testing.T) { + b := newPairPeerTestBroker("this-host") + b.relayDir.Apply(noderec.NotifyNodeDiscovered, noderec.DirectoryNode{ + HostUUID: "discovered-peer", + Name: "gpu-box", + IP: "192.168.1.10", + Services: map[noderec.ServiceKey]noderec.ServiceStatus{noderec.ServiceOllama: {Port: 11434}}, + }) + + b.releaseManualDirectory("discovered-peer") + if got := b.relayDir.Snapshot(""); len(got) != 1 { + t.Fatalf("relay directory = %v, want the discovered peer untouched", got) + } +} + +// A machine the scanner took over mid-life must not be withdrawn either: the +// manual claim goes away, the daemon's record stays. +func TestReleaseManualDirectory_YieldsWhenTheScannerTookOver(t *testing.T) { + b := newPairPeerTestBroker("this-host") + s := pairStatus() + b.applyManualDirectory(s, s.HostUUID) + b.store.Upsert(EnrichedNode{ID: "gpu-box", HostUUID: s.HostUUID}, sourceScanner) + + b.releaseManualDirectory(s.HostUUID) + if got := b.relayDir.Snapshot(""); len(got) != 1 { + t.Fatalf("relay directory = %v, want the record kept for the scanner", got) + } +} + +func TestManualToEnriched_PairNodeCarriesItsInventoryAndMembership(t *testing.T) { + en := manualToEnriched(pairStatus()) + if !en.Clustered { + t.Error("a peer reporting a cluster principal must project as clustered") + } + if len(en.Models) != 1 || en.Models[0] != "llama3.2:latest" { + t.Errorf("models = %v, want the peer's inventory", en.Models) + } + if len(en.Addresses) != 1 || en.Addresses[0] != "gpu-box.tail1234.ts.net" { + t.Errorf("addresses = %v, want the typed address", en.Addresses) + } + if len(en.TXT) != 1 || en.TXT[0] != "ip=gpu-box.tail1234.ts.net" { + t.Errorf("txt = %v, want ip=
so netpick ranks it", en.TXT) + } +} diff --git a/services/nvpair-ui-broker/manualnodes.go b/services/nvpair-ui-broker/manualnodes.go index 47fe0ff1..28032e52 100644 --- a/services/nvpair-ui-broker/manualnodes.go +++ b/services/nvpair-ui-broker/manualnodes.go @@ -9,6 +9,7 @@ import ( "log/slog" "time" + "nvpair-shared/clustertrust" "nvpair-shared/noderec" ) @@ -16,9 +17,9 @@ import ( // broker needs. Its JSON tags match the producer's so node/discovered| // updated|removed payloads unmarshal straight into it (the GPU/CPU/memory // sub-objects reuse the broker's discovery types, whose tags are identical). -// The ollama_* / lmstudio_* fields drive the per-engine manual→proxy bridge -// (bridgeManualNode); the rest project into the discovery store via -// manualToEnriched. +// The ollama_* / lmstudio_* / vllm_* / sglang_* fields drive the per-engine +// manual→proxy bridge (bridgeManualNode); the rest project into the discovery +// store via manualToEnriched. type manualNodeStatus struct { ID string `json:"id"` Address string `json:"address"` @@ -28,6 +29,12 @@ type manualNodeStatus struct { LMStudioUp bool `json:"lmstudio_up"` LMStudioPort int `json:"lmstudio_port"` LMStudioModels []string `json:"lmstudio_models,omitempty"` + VLLMUp bool `json:"vllm_up"` + VLLMPort int `json:"vllm_port"` + VLLMModels []string `json:"vllm_models,omitempty"` + SGLangUp bool `json:"sglang_up"` + SGLangPort int `json:"sglang_port"` + SGLangModels []string `json:"sglang_models,omitempty"` NodeInfoPort int `json:"node_info_port"` GPUs []GPUInfo `json:"gpus"` CPU *CPUInfo `json:"cpu"` @@ -39,6 +46,39 @@ type manualNodeStatus struct { // identity as mDNS-discovered nodes (and dedup with itself when the same // machine is also discovered). Empty until the node-info probe succeeds. HostUUID string `json:"hostUuid,omitempty"` + // PairNode is true when the prober's node-info leg answered AND reported a + // service map: the remote is a PAIR node, not a bare inference host. It is the + // switch between the two ways a manual node reaches inference. A bare host is + // bridged into the local proxies by its raw engine ports; a PAIR node is + // folded into the discovery relay instead and reached through its own proxies + // over cluster mTLS, because its 11434 / 1234 are proxy facades that refuse + // plaintext from anything but loopback. + PairNode bool `json:"pair_node"` + // ClusterUUID is the remote's cluster principal, tri-state (see the prober's + // NodeInfoResponse): absent means it does not know, present-and-empty means it + // belongs to no cluster. It is the key a consumer pins the peer's certificate + // on, so getting it wrong is the difference between mTLS and a 403. + ClusterUUID *string `json:"cluster_uuid,omitempty"` + // Services is the remote's {service key: port} set, read from its node-info. + // It is everything this host would otherwise have read off the peer's mDNS + // record, which never arrives across a routed or overlay network. + Services map[noderec.ServiceKey]int `json:"services,omitempty"` + // Models / ModelsByEngine / LoadedByEngine are a paired PAIR node's inventory, + // read from its engine manager over cluster mTLS. Empty until this node holds + // a pin for the peer. + Models []string `json:"models,omitempty"` + ModelsByEngine map[string][]string `json:"models_by_engine,omitempty"` + LoadedByEngine map[string][]string `json:"loaded_by_engine,omitempty"` +} + +// clusterUUID flattens the tri-state principal for the consumers that only need +// a value. Unknown and unclustered both answer "" — which is correct for every +// use here, since both mean "we hold no principal to pin on". +func (s manualNodeStatus) clusterUUID() string { + if s.ClusterUUID == nil { + return "" + } + return *s.ClusterUUID } type manualNodeStatusEntry struct { @@ -87,20 +127,138 @@ func manualToEnriched(s manualNodeStatus) EnrichedNode { GPUs: s.GPUs, CPU: s.CPU, Memory: s.Memory, - Models: mergeModels(s.OllamaModels, s.LMStudioModels), + Clustered: s.clusterUUID() != "", + Models: mergeModels(s.Models, s.OllamaModels, s.LMStudioModels), ModelsByEngine: manualModelsByEngine(s), + LoadedByEngine: s.LoadedByEngine, } if s.Address != "" { en.Addresses = []string{s.Address} + en.TXT = []string{noderec.KeyIP + "=" + s.Address} } return en } +// manualEnriched is manualToEnriched plus the one field it cannot derive from +// the status alone: whether this node pins the peer's certificate. That answer +// lives in the trust store, and it is read live rather than cached, because it +// changes the moment a pairing completes or a member is removed. +func (b *Broker) manualEnriched(s manualNodeStatus) EnrichedNode { + en := manualToEnriched(s) + en.Trusted = b.holdsPinFor(s.clusterUUID()) + return en +} + +// manualDirectoryNode synthesizes the directory record a manual PAIR node would +// have had if it had been discovered, so every consumer of the discovery relay — +// both inference proxies, the scheduler's inventory, engine-manager's remote +// operations, the workload relay and the errors peer sync — treats it exactly +// like a pinned peer found over mDNS. That is the whole point: a peer reachable +// only across an overlay network is not a second kind of node, it is the same +// node with no multicast between here and there. +// +// ok is false when the node must NOT enter the directory: +// - it is a bare inference host (no service map): it has no proxy or engine +// manager to be a peer with, and is bridged into the local proxies instead. +// - it is this host (its hostUuid is ours): a manual entry naming ourselves +// must never become a routing target, or the proxy would forward to its own +// ingress. Identity is the exact test; an address comparison would miss the +// overlay name for this same machine. +// - the scanner already holds the node: the daemon's record is authoritative +// and carries evidence this synthesis cannot (its full ranked address list, +// its liveness probes), and two writers on one key would fight. +func (b *Broker) manualDirectoryNode(s manualNodeStatus, key string) (noderec.DirectoryNode, bool) { + if !s.PairNode || len(s.Services) == 0 || key == "" || s.Address == "" { + return noderec.DirectoryNode{}, false + } + if key == b.nodeID { + slog.Debug("manual node names this host; not folding it into the directory", "id", s.ID) + return noderec.DirectoryNode{}, false + } + if b.store.hasSource(key, sourceScanner) { + return noderec.DirectoryNode{}, false + } + services := make(map[noderec.ServiceKey]noderec.ServiceStatus, len(s.Services)) + for svc, port := range s.Services { + if svc != "" && port > 0 { + services[svc] = noderec.ServiceStatus{Port: port} + } + } + if len(services) == 0 { + return noderec.DirectoryNode{}, false + } + clusterUUID := s.clusterUUID() + return noderec.DirectoryNode{ + HostUUID: key, + Name: s.ID, + // The address the operator typed is this node's canonical one: it is the + // only route we know works, and unlike a discovered peer there is no + // published ranking to defer to. + IP: s.Address, + ClusterUUID: clusterUUID, + Trusted: b.holdsPinFor(clusterUUID), + Services: services, + GPUs: s.GPUs, + CPU: s.CPU, + Memory: s.Memory, + Models: s.Models, + ModelsByEngine: s.ModelsByEngine, + LoadedByEngine: s.LoadedByEngine, + LastSeen: time.Now().Unix(), + }, true +} + +// holdsPinFor reports whether this node pins the given cluster principal's +// certificate — the same question the scanner answers for a browsed peer, asked +// against live membership rather than a cached annotation. +func (b *Broker) holdsPinFor(clusterUUID string) bool { + if clusterUUID == "" || b.clusterDir == "" { + return false + } + mesh := clustertrust.Open(b.clusterDir) + mesh.Refresh() + return mesh.HasPin(clusterUUID) +} + +// applyManualDirectory folds a manual PAIR node into the discovery relay, or +// withdraws it when it stopped qualifying (it went down, it turned out to be a +// bare host, or the scanner took the record over). +func (b *Broker) applyManualDirectory(s manualNodeStatus, key string) { + node, ok := b.manualDirectoryNode(s, key) + if !ok { + b.releaseManualDirectory(key) + return + } + b.manualMu.Lock() + b.manualRelayKeys[key] = true + b.manualMu.Unlock() + b.relayDir.Apply(noderec.NotifyNodeUpdated, node) +} + +// releaseManualDirectory withdraws a record this broker synthesized. It withdraws +// only what it put there: the relay directory is keyed by hostUuid and the +// scanner writes the same keys, so removing one the daemon owns would evict a +// live discovered peer from every consumer until its next browse event. +func (b *Broker) releaseManualDirectory(key string) { + if key == "" { + return + } + b.manualMu.Lock() + synthesized := b.manualRelayKeys[key] + delete(b.manualRelayKeys, key) + b.manualMu.Unlock() + if !synthesized || b.store.hasSource(key, sourceScanner) { + return + } + b.relayDir.Apply(noderec.NotifyNodeRemoved, noderec.DirectoryNode{HostUUID: key}) +} + // manualModelsByEngine builds the per-engine attribution for a manual node from // the per-engine lists the prober already collected, keyed by the same -// engine-manager engine names discovered nodes use ("ollama", "lmstudio") so the -// two discovery sources present ModelsByEngine identically. An engine with no -// models adds no key; returns nil when neither engine reports any. +// engine-manager engine names discovered nodes use ("ollama", "lmstudio", +// "vllm", "sglang") so the two discovery sources present ModelsByEngine +// identically. An engine with no models adds no key; returns nil when no engine +// reports any. func manualModelsByEngine(s manualNodeStatus) map[string][]string { byEngine := map[string][]string{} if len(s.OllamaModels) > 0 { @@ -109,6 +267,12 @@ func manualModelsByEngine(s manualNodeStatus) map[string][]string { if len(s.LMStudioModels) > 0 { byEngine["lmstudio"] = s.LMStudioModels } + if len(s.VLLMModels) > 0 { + byEngine["vllm"] = s.VLLMModels + } + if len(s.SGLangModels) > 0 { + byEngine["sglang"] = s.SGLangModels + } if len(byEngine) == 0 { return nil } @@ -139,7 +303,12 @@ func mergeModels(lists ...[]string) []string { // It mirrors the proxy's Node wire shape (id/host/port/addresses[/txt]); // the proxy requires a non-empty address list and a port to forward to. type proxyManualNode struct { - ID string `json:"id"` + ID string `json:"id"` + // Engine names which engine on that host this address serves. The OpenAI + // proxy fronts more than one engine on different ports, so it keys its manual + // overlay by (engine, node) and would otherwise collapse a host running both + // into a single entry. ollama-proxy fronts one engine and ignores the field. + Engine string `json:"engine"` Host string `json:"host"` Port int `json:"port"` Addresses []string `json:"addresses"` @@ -147,11 +316,18 @@ type proxyManualNode struct { Models []string `json:"models,omitempty"` } +// proxyManualRef identifies a manual node to remove from one proxy engine. +type proxyManualRef struct { + ID string `json:"id"` + Engine string `json:"engine"` +} + // bridgeManualNode keeps every supervised proxy's manual-node set in step with // a manual node's per-engine reachability: a node whose Ollama is up is bridged -// into ollama-proxy and one whose LM Studio is up into lmstudio-proxy -// (idempotent — each proxy upserts on a repeat), while an engine that is not -// (or no longer) reachable is removed from its proxy. Each leg is a no-op when +// into ollama-proxy and one whose LM Studio, vLLM or SGLang is up into the +// OpenAI proxy, once per engine (idempotent — each proxy upserts on a repeat), +// while an engine that is not (or no longer) reachable is removed from its +// proxy. Each leg is a no-op when // that proxy isn't supervised — the bridge only applies when the broker owns // both ends. // @@ -160,8 +336,20 @@ type proxyManualNode struct { // daemon to carry — so without this explicit add the proxies can't route // inference to them even though both workers are broker-owned. func (b *Broker) bridgeManualNode(s manualNodeStatus, key string) { + if s.PairNode { + // A PAIR node reaches the proxies through the discovery relay instead, + // which is what gets it dialed over cluster mTLS against its pinned + // certificate. Bridging it here as well would add a second candidate for + // the same node that the proxy dials in plaintext — straight into the + // peer's loopback-only refusal — so the raw-engine bridge is withdrawn + // rather than left alongside. + b.removeManualNodeFromProxies(key) + return + } b.bridgeToProxy(b.getProxy(), "ollama", s, key, s.OllamaUp, s.OllamaPort, s.OllamaModels) - b.bridgeToProxy(b.getLMStudioProxy(), "lmstudio", s, key, s.LMStudioUp, s.LMStudioPort, s.LMStudioModels) + b.bridgeToProxy(b.getOpenAIProxy(), "lmstudio", s, key, s.LMStudioUp, s.LMStudioPort, s.LMStudioModels) + b.bridgeToProxy(b.getOpenAIProxy(), "vllm", s, key, s.VLLMUp, s.VLLMPort, s.VLLMModels) + b.bridgeToProxy(b.getOpenAIProxy(), "sglang", s, key, s.SGLangUp, s.SGLangPort, s.SGLangModels) } // bridgeToProxy adds the node to p when its engine is reachable, or removes it @@ -177,6 +365,7 @@ func (b *Broker) bridgeToProxy(p *proxyProcess, engine string, s manualNodeStatu if up && s.Address != "" && port > 0 { node := proxyManualNode{ ID: key, + Engine: engine, Host: s.Address, Port: port, Addresses: []string{s.Address}, @@ -187,15 +376,17 @@ func (b *Broker) bridgeToProxy(p *proxyProcess, engine string, s manualNodeStatu } // Engine unreachable (down, or this node doesn't run it): make sure the // proxy isn't left holding a stale manual entry it would try to route to. - b.callProxyManual(p, engine, "node/remove-manual", map[string]string{"id": key}, key) + b.callProxyManual(p, engine, "node/remove-manual", proxyManualRef{ID: key, Engine: engine}, key) } // removeManualNodeFromProxies drops a manual node from every supervised proxy. // Idempotent: a no-op for a proxy where the node was never bridged or that // isn't supervised (the proxy's RemoveManual just reports removed=false). func (b *Broker) removeManualNodeFromProxies(id string) { - b.callProxyManual(b.getProxy(), "ollama", "node/remove-manual", map[string]string{"id": id}, id) - b.callProxyManual(b.getLMStudioProxy(), "lmstudio", "node/remove-manual", map[string]string{"id": id}, id) + b.callProxyManual(b.getProxy(), "ollama", "node/remove-manual", proxyManualRef{ID: id, Engine: "ollama"}, id) + for _, engine := range []string{"lmstudio", "vllm", "sglang"} { + b.callProxyManual(b.getOpenAIProxy(), engine, "node/remove-manual", proxyManualRef{ID: id, Engine: engine}, id) + } } // callProxyManual issues a best-effort node/add-manual|remove-manual to a diff --git a/services/nvpair-ui-broker/nodeinfo.go b/services/nvpair-ui-broker/nodeinfo.go index af890547..4092ed29 100644 --- a/services/nvpair-ui-broker/nodeinfo.go +++ b/services/nvpair-ui-broker/nodeinfo.go @@ -13,6 +13,8 @@ import ( "os" "os/exec" "sync" + + "nvpair-shared/noderec" ) // maxNodeInfoLine caps one stdout frame from node-info. Its frames are a handful @@ -58,6 +60,15 @@ func (n *nodeInfoProcess) SetClusterIdentity(clusterUUID string) error { return writeClusterIdentityFrame(&n.stdinMu, n.stdin, clusterUUID) } +// SetServices tells node-info this node's whole {service: port} set, so +// /v1/node-info can report it. The set otherwise exists only on this host's mDNS +// record, and multicast does not cross a routed or overlay network — a peer that +// reached this node across a Tailscale tailnet has node-info and nothing else to +// ask. Sent on spawn and on every registration change. +func (n *nodeInfoProcess) SetServices(services map[noderec.ServiceKey]int) error { + return writeServicesFrame(&n.stdinMu, n.stdin, services) +} + // Done implements supervisedHandle: the returned channel closes once the // node-info process has exited (cmd.Wait returned). func (n *nodeInfoProcess) Done() <-chan struct{} { return n.done } diff --git a/services/nvpair-ui-broker/nodeinfo_services_test.go b/services/nvpair-ui-broker/nodeinfo_services_test.go new file mode 100644 index 00000000..4696bf27 --- /dev/null +++ b/services/nvpair-ui-broker/nodeinfo_services_test.go @@ -0,0 +1,106 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "encoding/json" + "strings" + "sync" + "testing" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +// TestWriteServicesFrame pins the wire form of the service-map push node-info +// decodes. It is the only way a peer that never received this host's mDNS record +// — anything across a routed or overlay network — learns which services this node +// runs, so a renamed method or a missing newline silently reduces such a peer to +// probing raw engine ports that answer 403. +func TestWriteServicesFrame(t *testing.T) { + var buf bytes.Buffer + var mu sync.Mutex + services := map[noderec.ServiceKey]int{ + noderec.ServiceNodeInfo: 14318, + noderec.ServiceOllama: 11434, + } + if err := writeServicesFrame(&mu, &buf, services); err != nil { + t.Fatalf("write: %v", err) + } + + line := buf.String() + if !strings.HasSuffix(line, "\n") { + t.Error("frame is not newline-terminated; node-info reads line-delimited frames") + } + if strings.Count(line, "\n") != 1 { + t.Errorf("frame contains %d newlines, want exactly one", strings.Count(line, "\n")) + } + + var frame struct { + JSONRPC string `json:"jsonrpc"` + Method string `json:"method"` + ID json.RawMessage `json:"id"` + Params noderec.ServicesParams `json:"params"` + } + if err := json.Unmarshal([]byte(line), &frame); err != nil { + t.Fatalf("decode frame %q: %v", line, err) + } + if frame.JSONRPC != "2.0" { + t.Errorf("jsonrpc = %q, want 2.0", frame.JSONRPC) + } + if frame.Method != noderec.MethodSetServices { + t.Errorf("method = %q, want %q", frame.Method, noderec.MethodSetServices) + } + // A notification, not a request: node-info's stdout is drained to io.Discard, + // so an id-bearing frame would strand a reply. + if len(frame.ID) != 0 { + t.Errorf("frame carries an id (%s); the push must be a notification", frame.ID) + } + if frame.Params.Services[noderec.ServiceOllama] != 11434 { + t.Errorf("services = %v, want ol=11434", frame.Params.Services) + } +} + +// An empty set is a real value: it is how "every service went away" is expressed, +// and merging on the far side would leave a departed service advertised forever. +func TestWriteServicesFrame_EmptySetIsStillSent(t *testing.T) { + var buf bytes.Buffer + var mu sync.Mutex + if err := writeServicesFrame(&mu, &buf, nil); err != nil { + t.Fatalf("write: %v", err) + } + if !strings.Contains(buf.String(), `"services":{}`) { + t.Errorf("frame did not carry an empty services object: %s", buf.String()) + } +} + +// The map node-info reports and the ports the scanner advertises come from one +// cache, so a peer reading /v1/node-info and a peer reading the mDNS record agree. +func TestLocalServices_MirrorsTheRegistrationCache(t *testing.T) { + b := &Broker{regCache: relay.NewRegistrationCache()} + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceNodeInfo, Port: 14318}) + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceOllama, Port: 11434}) + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceEngineControl, Port: 14323}) + + got := b.localServices() + want := map[noderec.ServiceKey]int{ + noderec.ServiceNodeInfo: 14318, + noderec.ServiceOllama: 11434, + noderec.ServiceEngineControl: 14323, + } + if len(got) != len(want) { + t.Fatalf("localServices = %v, want %v", got, want) + } + for k, v := range want { + if got[k] != v { + t.Errorf("localServices[%s] = %d, want %d", k, got[k], v) + } + } + + b.regCache.Unregister(noderec.ServiceOllama) + if _, present := b.localServices()[noderec.ServiceOllama]; present { + t.Error("an unregistered service must leave the map a peer reads") + } +} diff --git a/services/nvpair-ui-broker/sglangadvertise_test.go b/services/nvpair-ui-broker/sglangadvertise_test.go new file mode 100644 index 00000000..1ee54cf4 --- /dev/null +++ b/services/nvpair-ui-broker/sglangadvertise_test.go @@ -0,0 +1,243 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "strconv" + "testing" + "time" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +// TestSGLangAdvertisesTheProxyPortNotTheEnginePort proves a healthy local SGLang +// registers the sg service at the OpenAI proxy's listen port — peers must reach +// this node through the proxy, never the engine — while the engine's own +// loopback port is handed to that proxy as the sglang backend. +func TestSGLangAdvertisesTheProxyPortNotTheEnginePort(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/get_model_info" { + http.Error(w, "not sglang", http.StatusNotFound) + return + } + w.WriteHeader(http.StatusOK) + })) + defer engine.Close() + _, portStr, err := net.SplitHostPort(engine.Listener.Addr().String()) + if err != nil { + t.Fatal(err) + } + enginePort, err := strconv.Atoi(portStr) + if err != nil { + t.Fatal(err) + } + + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + // No engine-manager, so localEnginePort answers with the fallback; point the + // fallback at the stub by probing it directly. + client := &http.Client{Timeout: 2 * time.Second} + if !checkSGLangHealth(client, enginePort) { + t.Fatal("stub did not answer /get_model_info") + } + + b.reconcileAdvertiseSGLangAt(client, enginePort, true) + + snapshot := b.regCache.Snapshot() + reg, ok := registrationFor(snapshot, noderec.ServiceSGLang) + if !ok { + t.Fatalf("sg was not registered: %+v", snapshot) + } + if reg.Port != proxyPort { + t.Errorf("sg advertised port = %d, want the proxy's %d", reg.Port, proxyPort) + } + got := waitBackend(t, backends, "sglang") + if got.Port != enginePort || !got.Healthy { + t.Errorf("sglang local backend = %+v, want the healthy engine port %d", got, enginePort) + } +} + +// TestSGLangUnregistersWhenTheEngineIsDown proves the sg key is withdrawn and +// the proxy's sglang backend cleared when the engine stops answering, without +// any health request reaching a listener that is not SGLang. +func TestSGLangUnregistersWhenTheEngineIsDown(t *testing.T) { + b, backends := openAIProxyFixture(t, 1234) + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceSGLang, Port: 1234}) + + b.reconcileAdvertiseSGLangAt(nil, defaultSGLangPort, false) + + if _, ok := registrationFor(b.regCache.Snapshot(), noderec.ServiceSGLang); ok { + t.Error("sg stayed registered with the engine down") + } + got := waitBackend(t, backends, "sglang") + if got.Healthy { + t.Errorf("sglang backend = %+v, want cleared", got) + } +} + +// TestSGLangNeverAdvertisesItsOwnProxy is the self-forward guard: when the +// resolved engine port equals the proxy's own listener, the node must not +// advertise sg and must not be handed its own listener as a backend. +func TestSGLangNeverAdvertisesItsOwnProxy(t *testing.T) { + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + + // A nil client is intentional: the collision check must short-circuit before + // any health request could mistake the proxy for the engine. + b.reconcileAdvertiseSGLangAt(nil, proxyPort, true) + + if got := b.regCache.Snapshot(); len(got) != 0 { + t.Fatalf("the OpenAI proxy was advertised as SGLang: %+v", got) + } + if got := waitBackend(t, backends, "sglang"); got.Healthy { + t.Errorf("proxy listener was retained as the SGLang backend: %+v", got) + } +} + +// TestManualSGLangNodeIsBridgedWithItsEngine proves a manual node whose +// sglang_up is set is bridged into the OpenAI proxy tagged sglang, and that the +// vLLM entry on the same host is a separate entry on its own port rather than +// being overwritten. +func TestManualSGLangNodeIsBridgedWithItsEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + node proxyManualNode + } + calls := make(chan call, 16) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var n proxyManualNode + _ = json.Unmarshal(msg.Params, &n) + calls <- call{method: msg.Method, node: n} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", + Address: "192.0.2.7", + VLLMUp: true, + VLLMPort: 8000, + VLLMModels: []string{"Qwen/Qwen3-8B"}, + SGLangUp: true, + SGLangPort: 30000, + SGLangModels: []string{"/models/my-model"}, + }, "host-uuid") + + byEngine := map[string]proxyManualNode{} + deadline := time.After(2 * time.Second) + for len(byEngine) < 2 { + select { + case c := <-calls: + if c.method == "node/add-manual" { + byEngine[c.node.Engine] = c.node + } + case <-deadline: + t.Fatalf("only bridged %v", byEngine) + } + } + got := byEngine["sglang"] + if got.Port != 30000 || got.ID != "host-uuid" { + t.Errorf("sglang bridge = %+v, want SGLang's port on the node's operational key", got) + } + // A model id that is a bare local path is what SGLang actually reports, so it + // has to survive the bridge unaltered. + if len(got.Models) != 1 || got.Models[0] != "/models/my-model" { + t.Errorf("sglang models = %v, want the path-shaped id carried verbatim", got.Models) + } + if got := byEngine["vllm"]; got.Port != 8000 { + t.Errorf("vllm bridge = %+v, want its own port preserved", got) + } +} + +// TestManualNodeWithoutSGLangIsRemovedFromThatEngine proves an unreachable +// SGLang leg clears only its own entry, so the other engines on that host keep +// routing. +func TestManualNodeWithoutSGLangIsRemovedFromThatEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + ref proxyManualRef + } + calls := make(chan call, 16) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var ref proxyManualRef + _ = json.Unmarshal(msg.Params, &ref) + calls <- call{method: msg.Method, ref: ref} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", Address: "192.0.2.7", + VLLMUp: true, VLLMPort: 8000, + }, "host-uuid") + + deadline := time.After(2 * time.Second) + for { + select { + case c := <-calls: + if c.method == "node/remove-manual" && c.ref.Engine == "sglang" { + if c.ref.ID != "host-uuid" { + t.Fatalf("remove ref = %+v", c.ref) + } + return + } + if c.method == "node/remove-manual" && c.ref.Engine == "vllm" { + t.Fatal("a reachable vLLM must not be removed") + } + case <-deadline: + t.Fatal("SGLang entry was never cleared") + } + } +} + +// TestManualModelsByEngineCarriesSGLang proves a manual node's SGLang inventory +// is attributed under the same engine name discovered nodes use, and that an +// engine reporting nothing still adds no key. +func TestManualModelsByEngineCarriesSGLang(t *testing.T) { + got := manualModelsByEngine(manualNodeStatus{ + OllamaModels: []string{"llama3.2:1b"}, + SGLangModels: []string{"/models/my-model"}, + }) + if len(got["sglang"]) != 1 || got["sglang"][0] != "/models/my-model" { + t.Errorf("modelsByEngine = %+v, want an sglang key", got) + } + for _, engine := range []string{"lmstudio", "vllm"} { + if _, ok := got[engine]; ok { + t.Errorf("an engine with no models must add no key: %+v", got) + } + } +} diff --git a/services/nvpair-ui-broker/vllmadvertise_test.go b/services/nvpair-ui-broker/vllmadvertise_test.go new file mode 100644 index 00000000..6a26f3b7 --- /dev/null +++ b/services/nvpair-ui-broker/vllmadvertise_test.go @@ -0,0 +1,309 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "net" + "net/http" + "net/http/httptest" + "strconv" + "testing" + "time" + + "nvpair-shared/noderec" + "nvpair-ui-broker/relay" +) + +// openAIProxyFixture returns a broker with a fake OpenAI proxy on the given +// listen port, plus a channel receiving every node/set-local-backend it is sent. +func openAIProxyFixture(t *testing.T, listenPort int) (*Broker, <-chan proxyLocalBackend) { + t.Helper() + proxyClient, proxyServer := net.Pipe() + t.Cleanup(func() { proxyClient.Close(); proxyServer.Close() }) + proxy := &proxyProcess{ + peer: NewPeer(NewCodec(proxyClient)), + ready: true, + port: listenPort, + } + go proxy.peer.Serve(nil, nil) + + backends := make(chan proxyLocalBackend, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var got proxyLocalBackend + if json.Unmarshal(msg.Params, &got) == nil { + backends <- got + } + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + return b, backends +} + +func waitBackend(t *testing.T, ch <-chan proxyLocalBackend, engine string) proxyLocalBackend { + t.Helper() + deadline := time.After(2 * time.Second) + for { + select { + case got := <-ch: + if got.Engine == engine { + return got + } + case <-deadline: + t.Fatalf("no node/set-local-backend for %q", engine) + } + } +} + +// TestVLLMAdvertisesTheProxyPortNotTheEnginePort proves a healthy local vLLM +// registers the vl service at the OpenAI proxy's listen port — peers must reach +// this node through the proxy, never the engine — while the engine's own +// loopback port is handed to that proxy as the vllm backend. +func TestVLLMAdvertisesTheProxyPortNotTheEnginePort(t *testing.T) { + engine := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/health" { + http.Error(w, "not vllm", http.StatusNotFound) + return + } + w.WriteHeader(http.StatusOK) + })) + defer engine.Close() + _, portStr, err := net.SplitHostPort(engine.Listener.Addr().String()) + if err != nil { + t.Fatal(err) + } + enginePort, err := strconv.Atoi(portStr) + if err != nil { + t.Fatal(err) + } + + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + // No engine-manager, so localEnginePort answers with the fallback; point the + // fallback at the stub by probing it directly. + client := &http.Client{Timeout: 2 * time.Second} + if !checkVLLMHealth(client, enginePort) { + t.Fatal("stub did not answer /health") + } + + b.reconcileAdvertiseVLLMAt(client, enginePort, true) + + snapshot := b.regCache.Snapshot() + reg, ok := registrationFor(snapshot, noderec.ServiceVLLM) + if !ok { + t.Fatalf("vl was not registered: %+v", snapshot) + } + if reg.Port != proxyPort { + t.Errorf("vl advertised port = %d, want the proxy's %d", reg.Port, proxyPort) + } + got := waitBackend(t, backends, "vllm") + if got.Port != enginePort || !got.Healthy { + t.Errorf("vllm local backend = %+v, want the healthy engine port %d", got, enginePort) + } +} + +// TestVLLMUnregistersWhenTheEngineIsDown proves the vl key is withdrawn and the +// proxy's vllm backend cleared when the engine stops answering, without any +// health request reaching a listener that is not vLLM. +func TestVLLMUnregistersWhenTheEngineIsDown(t *testing.T) { + b, backends := openAIProxyFixture(t, 1234) + b.regCache.Register(noderec.RegisterParams{Service: noderec.ServiceVLLM, Port: 1234}) + + b.reconcileAdvertiseVLLMAt(nil, defaultVLLMPort, false) + + if _, ok := registrationFor(b.regCache.Snapshot(), noderec.ServiceVLLM); ok { + t.Error("vl stayed registered with the engine down") + } + got := waitBackend(t, backends, "vllm") + if got.Healthy { + t.Errorf("vllm backend = %+v, want cleared", got) + } +} + +// TestVLLMNeverAdvertisesItsOwnProxy is the self-forward guard: when the +// resolved engine port equals the proxy's own listener, the node must not +// advertise vl and must not be handed its own listener as a backend. +func TestVLLMNeverAdvertisesItsOwnProxy(t *testing.T) { + const proxyPort = 1234 + b, backends := openAIProxyFixture(t, proxyPort) + + // A nil client is intentional: the collision check must short-circuit before + // any health request could mistake the proxy for the engine. + b.reconcileAdvertiseVLLMAt(nil, proxyPort, true) + + if got := b.regCache.Snapshot(); len(got) != 0 { + t.Fatalf("the OpenAI proxy was advertised as vLLM: %+v", got) + } + if got := waitBackend(t, backends, "vllm"); got.Healthy { + t.Errorf("proxy listener was retained as the vLLM backend: %+v", got) + } +} + +// TestOpenAIProxyServesEveryOpenAIEngine proves every OpenAI engine resolves to +// the one proxy process, which is what lets a node advertise lm, vl and sg at +// the same port. +func TestOpenAIProxyServesEveryOpenAIEngine(t *testing.T) { + b, _ := openAIProxyFixture(t, 1234) + openai := b.proxyForEngine("lmstudio") + if openai == nil { + t.Fatal("proxyForEngine(lmstudio) = nil") + } + for _, engine := range []string{"vllm", "sglang"} { + if got := b.proxyForEngine(engine); got != openai { + t.Errorf("proxyForEngine(%s) = %v, want the same OpenAI proxy process", engine, got) + } + } + if b.proxyForEngine("ollama") == openai { + t.Error("Ollama must keep its own proxy") + } +} + +// TestManualVLLMNodeIsBridgedWithItsEngine proves a manual node whose vllm_up is +// set is bridged into the OpenAI proxy tagged vllm, and that an LM Studio entry +// on the same host is a separate entry rather than overwriting it. +func TestManualVLLMNodeIsBridgedWithItsEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + node proxyManualNode + } + calls := make(chan call, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var n proxyManualNode + _ = json.Unmarshal(msg.Params, &n) + calls <- call{method: msg.Method, node: n} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", + Address: "192.0.2.7", + LMStudioUp: true, + LMStudioPort: 1234, + LMStudioModels: []string{"qwen2.5-7b"}, + VLLMUp: true, + VLLMPort: 8000, + VLLMModels: []string{"Qwen/Qwen3-8B"}, + }, "host-uuid") + + byEngine := map[string]proxyManualNode{} + deadline := time.After(2 * time.Second) + for len(byEngine) < 2 { + select { + case c := <-calls: + if c.method == "node/add-manual" { + byEngine[c.node.Engine] = c.node + } + case <-deadline: + t.Fatalf("only bridged %v", byEngine) + } + } + if got := byEngine["vllm"]; got.Port != 8000 || got.ID != "host-uuid" { + t.Errorf("vllm bridge = %+v, want the vLLM port on the node's operational key", got) + } + if got := byEngine["lmstudio"]; got.Port != 1234 { + t.Errorf("lmstudio bridge = %+v, want its own port preserved", got) + } +} + +// TestManualNodeWithoutVLLMIsRemovedFromThatEngine proves an unreachable engine +// leg clears only its own entry, so the other engine on that host keeps routing. +func TestManualNodeWithoutVLLMIsRemovedFromThatEngine(t *testing.T) { + proxyClient, proxyServer := net.Pipe() + defer proxyClient.Close() + defer proxyServer.Close() + proxy := &proxyProcess{peer: NewPeer(NewCodec(proxyClient)), ready: true, port: 1234} + go proxy.peer.Serve(nil, nil) + + type call struct { + method string + ref proxyManualRef + } + calls := make(chan call, 8) + go func() { + codec := NewCodec(proxyServer) + for { + msg, err := codec.Read() + if err != nil { + return + } + var ref proxyManualRef + _ = json.Unmarshal(msg.Params, &ref) + calls <- call{method: msg.Method, ref: ref} + _ = codec.Respond(msg.ID, map[string]bool{"ok": true}) + } + }() + + b := &Broker{regCache: relay.NewRegistrationCache()} + b.setLMStudioProxy(proxy) + b.bridgeManualNode(manualNodeStatus{ + ID: "n1", Address: "192.0.2.7", + LMStudioUp: true, LMStudioPort: 1234, + }, "host-uuid") + + deadline := time.After(2 * time.Second) + for { + select { + case c := <-calls: + if c.method == "node/remove-manual" && c.ref.Engine == "vllm" { + if c.ref.ID != "host-uuid" { + t.Fatalf("remove ref = %+v", c.ref) + } + return + } + if c.method == "node/remove-manual" && c.ref.Engine == "lmstudio" { + t.Fatal("a reachable LM Studio must not be removed") + } + case <-deadline: + t.Fatal("vLLM entry was never cleared") + } + } +} + +// TestManualModelsByEngineCarriesVLLM proves a manual node's vLLM inventory is +// attributed under the same engine name discovered nodes use. +func TestManualModelsByEngineCarriesVLLM(t *testing.T) { + got := manualModelsByEngine(manualNodeStatus{ + OllamaModels: []string{"llama3.2:1b"}, + VLLMModels: []string{"Qwen/Qwen3-8B"}, + }) + if len(got["vllm"]) != 1 || got["vllm"][0] != "Qwen/Qwen3-8B" { + t.Errorf("modelsByEngine = %+v, want a vllm key", got) + } + if _, ok := got["lmstudio"]; ok { + t.Errorf("an engine with no models must add no key: %+v", got) + } +} + +func registrationFor(regs []noderec.RegisterParams, svc noderec.ServiceKey) (noderec.RegisterParams, bool) { + for _, r := range regs { + if r.Service == svc { + return r, true + } + } + return noderec.RegisterParams{}, false +} diff --git a/services/ollama-proxy/overlay_test.go b/services/ollama-proxy/overlay_test.go new file mode 100644 index 00000000..dc4d4ac6 --- /dev/null +++ b/services/ollama-proxy/overlay_test.go @@ -0,0 +1,90 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Routing to a node reachable only over an encrypted overlay such as a Tailscale +// tailnet. Such a node is never discovered — a tailnet carries no multicast — so +// it arrives as a manually added address, and that address is a 100.64/10 CGNAT +// literal, an IPv6 ULA, or a MagicDNS name. netpick demotes all three relative to +// a LAN literal; none of them may be excluded, or the node has nowhere to be +// dialed. + +package main + +import ( + "reflect" + "testing" +) + +func TestNodeCandidates_OverlayOnlyNodeIsRoutable(t *testing.T) { + tests := []struct { + name string + node Node + want []string + }{ + { + name: "cgnat only", + node: Node{Addresses: []string{"100.101.102.103"}, Port: 11434}, + want: []string{"100.101.102.103:11434"}, + }, + { + name: "ipv6 ula only", + node: Node{Addresses: []string{"fd7a:115c:a1e0::1701:b2c3"}, Port: 11434}, + want: []string{"[fd7a:115c:a1e0::1701:b2c3]:11434"}, + }, + { + name: "magicdns name only", + node: Node{Addresses: []string{"gpu-box.tail1234.ts.net"}, Port: 11434}, + want: []string{"gpu-box.tail1234.ts.net:11434"}, + }, + { + name: "magicdns name carried as the node's canonical ip= TXT", + node: Node{ + TXT: []string{"ip=gpu-box.tail1234.ts.net"}, + Addresses: []string{"gpu-box.tail1234.ts.net"}, + Port: 11434, + }, + want: []string{"gpu-box.tail1234.ts.net:11434"}, + }, + { + name: "lan literal still preferred when the node has both", + node: Node{Addresses: []string{"100.101.102.103", "192.0.2.10"}, Port: 11434}, + want: []string{"192.0.2.10:11434", "100.101.102.103:11434"}, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := nodeCandidates(tc.node); !reflect.DeepEqual(got, tc.want) { + t.Errorf("nodeCandidates = %v, want %v", got, tc.want) + } + }) + } +} + +// A manual entry that names this host's own tailnet address must resolve to +// loopback rather than being dialed back around the tunnel to ourselves. The +// local-address set comes from netmon, which enumerates every interface without +// filtering, so an overlay adapter's address is in it exactly like a LAN one — +// this pins that, because a set built from the *publishing* picker instead would +// omit an unproven overlay address and let the entry loop. +func TestIsLocalAddress_CountsThisHostsOverlayAddresses(t *testing.T) { + const ourTailnetIP = "100.64.7.7" + + localAddrsMu.Lock() + prev := localAddrs + localAddrs = map[string]bool{"192.0.2.5": true, ourTailnetIP: true} + localAddrsMu.Unlock() + t.Cleanup(func() { + localAddrsMu.Lock() + localAddrs = prev + localAddrsMu.Unlock() + }) + + if !isLocalAddress(ourTailnetIP) { + t.Fatalf("isLocalAddress(%q) = false, want true", ourTailnetIP) + } + got := nodeCandidates(Node{Addresses: []string{ourTailnetIP}, Port: 11434}) + want := []string{"127.0.0.1:11434"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("nodeCandidates for our own tailnet address = %v, want %v", got, want) + } +} diff --git a/services/ollama-proxy/proxy.go b/services/ollama-proxy/proxy.go index ad4ac7a2..76b0f134 100644 --- a/services/ollama-proxy/proxy.go +++ b/services/ollama-proxy/proxy.go @@ -1859,15 +1859,14 @@ func nodeCandidates(n Node) []string { port := strconv.Itoa(n.Port) sorted := netpick.Candidates(n.TXT, n.Addresses) if len(sorted) == 0 { - // A non-IP entry (a .local hostname) that netpick cannot parse. - hosts := n.Addresses - if len(hosts) == 0 { - if n.Host == "" { - return nil - } - hosts = []string{n.Host} - } - sorted = append([]string(nil), hosts...) + // The node published no dialable address of its own. Its discovery host + // name is a different source, not a re-reading of the same one — netpick + // admits IP literals and DNS names alike, so an address list holding a + // MagicDNS or .local name has already been ranked above. + if n.Host == "" { + return nil + } + sorted = []string{n.Host} } seen := make(map[string]bool, len(sorted)) diff --git a/services/readme.md b/services/readme.md index ee3523d5..5fff8c20 100644 --- a/services/readme.md +++ b/services/readme.md @@ -11,9 +11,11 @@ local network: each node advertises itself over mDNS as one consolidated node offers and where to reach them. What a discovered node can actually serve is a separate question, answered after -discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, both, -or neither, and its model inventory is fetched over HTTP from its engine-manager -rather than crammed into mDNS TXT records, which are too small to carry it. +discovery. A node may be running [Ollama](https://ollama.com/), LM Studio, +[vLLM](https://docs.vllm.ai/), [SGLang](https://docs.sglang.ai/), any combination +of them, or none, and its model inventory is fetched over HTTP from its +engine-manager rather than crammed into mDNS TXT records, which are too small to +carry it. Locally, each node exposes compatibility proxies — Ollama-compatible and OpenAI-compatible — so an unmodified client on that machine can reach any capable @@ -41,7 +43,7 @@ This tree builds thirteen Go binaries. `nvpair-ui-broker` is the parent service | --- | --- | | `nvpair-ui-broker` | Parent service and JSON-RPC API surface used by the bundled UI and other clients. Supervises workers, relays consolidated discovery, and coordinates routing and scheduling. | | `ollama-proxy` | Ollama-compatible HTTP reverse proxy. Routes only to advertised model owners, with owner failover and scheduler priorities. | -| `lmstudio-proxy` | LM Studio counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes with equivalent owner-only routing and failover behavior. | +| `lmstudio-proxy` | OpenAI-compatible counterpart to `ollama-proxy`, forwarding OpenAI-compatible inference routes for **every** such engine — LM Studio, vLLM and SGLang — with equivalent owner-only routing and failover behavior. One process fronts them all; the binary name is a wire contract and keeps its historical spelling. | | `nvpair-node-info` | Local HTTP service on `:14318` exposing GPU, CPU, and memory inventory at `/v1/node-info`. | | `nvpair-node-scanner` | Consolidated discovery daemon. Advertises and browses `_nvpair-node._tcp`, maintains the node directory, and enriches peers with hardware and model information over HTTP. | | `nvpair-manual-nodes` | Manages user-added nodes that don't appear via mDNS; probes them every 10 s. | @@ -59,7 +61,7 @@ The mDNS responder is our own rather than the host's, because Windows ships none The broker feeds every accepted local or peer workload transition plus compact GPU telemetry to the scheduler. Queued and running work is counted by destination -node across Ollama and LM Studio together. Fresh maximum-GPU utilization is +node across every engine together. Fresh maximum-GPU utilization is smoothed into pressure 0–3; missing or stale telemetry is neutral. Rankings use `pending + gpuPressure`, and each proxy adds local reservations before choosing, so bursts spread without waiting for workload feedback. @@ -69,7 +71,7 @@ so bursts spread without waiting for workload feedback. ``` nvpair-ui-broker/ Parent service / JSON-RPC API surface ollama-proxy/ Ollama-compatible routing proxy -lmstudio-proxy/ OpenAI-compatible routing proxy for LM Studio +lmstudio-proxy/ OpenAI-compatible routing proxy (LM Studio, vLLM, SGLang) nvpair-node-info/ Local GPU-inventory HTTP service nvpair-node-scanner/ Consolidated _nvpair-node._tcp discovery daemon nvpair-manual-nodes/ Manual-node manager diff --git a/services/shared/netpick/netpick.go b/services/shared/netpick/netpick.go index 641a289d..f29fd878 100644 --- a/services/shared/netpick/netpick.go +++ b/services/shared/netpick/netpick.go @@ -2,9 +2,15 @@ // SPDX-License-Identifier: Apache-2.0 // Package netpick is the single source of truth for choosing which of a node's -// IPv4 addresses to publish and which to dial, so every NVPAIR service agrees on -// the same answer instead of reducing a multi-homed host's address set with its -// own ad-hoc rule. +// addresses to publish and which to dial, so every NVPAIR service agrees on the +// same answer instead of reducing a multi-homed host's address set with its own +// ad-hoc rule. +// +// "Address" here means anything a dialer can use: an IP literal or a DNS name. +// The local picker publishes IPv4 literals, because that is what a node can +// observe about itself; the remote side additionally accepts names, because a +// node reachable only over an encrypted overlay may have no address a peer can +// assume — a Tailscale node's MagicDNS name outlives every literal it holds. // // Selection is evidence-driven, not inferred from what a subnet number is // assumed to mean. A private range says nothing about whether the fleet lives on @@ -54,9 +60,83 @@ const ( scoreIPv6Global = 20 scoreIPv6ULA = 30 scorePublic = 40 // routable, but rarely how LAN peers reach each other - scorePrivate = 100 + // scoreHostname rates a DNS name. A name is not an address class at all: it + // carries no prefix to judge, and it re-resolves on every dial, so it + // survives a renumbering that strands every literal in the list. It sits + // above the public and IPv6 classes and below the private blocks, which is + // where a LAN peer's own literal still deserves to win. + scoreHostname = 60 + scorePrivate = 100 +) + +// maxHostnameLen and maxLabelLen are the DNS name limits (RFC 1035 2.3.4). +const ( + maxHostnameLen = 253 + maxLabelLen = 63 ) +// Hostname reports whether addr is a DNS name a dialer can hand to a resolver: +// one or more dot-separated labels of letters, digits, and interior hyphens, +// with an optional trailing root dot. +// +// It exists because an overlay network can give a node a name and no address a +// peer may assume. A Tailscale node's MagicDNS name (host.tailnet.ts.net) is the +// only identifier that stays correct across a re-authentication, and an mDNS peer +// can likewise be reachable only as host.local. Neither parses as an IP, so a +// candidate list that admits addresses alone would report such a node as having +// nowhere to be dialed. +// +// It deliberately rejects a "host:port" string: a colon is not a legal name +// character, and every consumer appends its own service port, so accepting one +// would produce "host:port:port". +func Hostname(addr string) bool { + addr = strings.TrimSpace(addr) + if addr == "" || len(addr) > maxHostnameLen || net.ParseIP(addr) != nil { + return false + } + // One trailing root dot is legal and canonical; anything else empty is not. + addr = strings.TrimSuffix(addr, ".") + if addr == "" { + return false + } + for _, label := range strings.Split(addr, ".") { + if len(label) == 0 || len(label) > maxLabelLen { + return false + } + for i := 0; i < len(label); i++ { + c := label[i] + switch { + case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9': + case c == '-' && i > 0 && i < len(label)-1: + default: + return false + } + } + } + return true +} + +// dialable reports whether addr is something a consumer can hand to a dialer at +// all — an IP literal or a DNS name. It is the single admission test for every +// candidate list here, so "which addresses does this node have" has one answer +// rather than one per caller. +func dialable(addr string) bool { + addr = strings.TrimSpace(addr) + return net.ParseIP(addr) != nil || Hostname(addr) +} + +// scoreAddr rates any dialable entry, name or literal. A name has no address +// class to judge, so it takes scoreHostname; everything else falls to scoreIP. +func scoreAddr(addr string) int { + if ip := net.ParseIP(strings.TrimSpace(addr)); ip != nil { + return scoreIP(ip) + } + if Hostname(addr) { + return scoreHostname + } + return scoreUnparseable +} + // dockerDefaultBridge reports whether ip is in 172.17/16, Docker's default bridge // subnet. It is the one private address that is not a host's own: every Docker // host has 172.17.0.1, so it identifies no machine and must never be published as @@ -105,10 +185,10 @@ func scoreIP(ip net.IP) int { return scoreIPv6Global } -// RankRemote returns the parseable addresses sorted best-first for dialing a -// remote peer. The sort is stable and ties break on the address string, so a -// multi-homed peer's chosen address does not flap between equally-ranked -// candidates from scan to scan. +// RankRemote returns the dialable entries — IP literals and DNS names alike — +// sorted best-first for dialing a remote peer. The sort is stable and ties break +// on the address string, so a multi-homed peer's chosen address does not flap +// between equally-ranked candidates from scan to scan. // // With the RFC1918 blocks scored equally, most of a peer's addresses now tie and // resolve by string order. That is intentional: this ranking is a last resort for @@ -121,8 +201,8 @@ func RankRemote(addrs []string) []string { } ranked := make([]scored, 0, len(addrs)) for _, a := range addrs { - if ip := net.ParseIP(strings.TrimSpace(a)); ip != nil { - ranked = append(ranked, scored{addr: a, score: scoreIP(ip)}) + if dialable(a) { + ranked = append(ranked, scored{addr: a, score: scoreAddr(a)}) } } sort.SliceStable(ranked, func(i, j int) bool { @@ -150,9 +230,9 @@ func IPFromTXT(txt []string) string { } // IPsFromTXT returns the ranked candidate list a node published in "ips=", in -// the node's own order, dropping entries that are not valid IPs. Empty when the -// key is absent — a node that publishes only "ip=" is not distinguishable here -// from one with a single address, and both are handled the same way. +// the node's own order, dropping entries nothing could dial. Empty when the key +// is absent — a node that publishes only "ip=" is not distinguishable here from +// one with a single address, and both are handled the same way. func IPsFromTXT(txt []string) []string { for _, kv := range txt { v, ok := strings.CutPrefix(kv, noderec.KeyIPs+"=") @@ -162,7 +242,7 @@ func IPsFromTXT(txt []string) []string { var out []string for _, part := range strings.Split(v, noderec.IPsSeparator) { part = strings.TrimSpace(part) - if part != "" && net.ParseIP(part) != nil { + if dialable(part) { out = append(out, part) } } @@ -185,16 +265,24 @@ func IPsFromTXT(txt []string) []string { // they are a fallback for a node whose published list is stale or truncated, not // a competing opinion, so they never displace a ranked entry. // +// An entry is kept when a dialer could use it: an IP literal, or a DNS name (see +// Hostname). Admitting names is what lets a node reachable only through an +// encrypted overlay — a Tailscale peer known by its MagicDNS name, or an mDNS +// peer known only as host.local — appear here at all rather than as a node with +// nowhere to be dialed. +// // The result is capped at noderec.MaxAdvertisedIPs. Nothing here comes from a // source this process controls — the TXT keys and the resolved address set both // arrive from an unauthenticated mDNS record — and every entry costs a dialer one -// connect timeout, so the bound has to hold on the reading side too. +// connect timeout, so the bound has to hold on the reading side too. A name is no +// more trusted than a literal from the same record: both are claims a consumer +// confirms by connecting, and a name resolves under the host's own resolver. func Candidates(txt []string, addrs []string) []string { var out []string seen := make(map[string]bool) add := func(a string) { a = strings.TrimSpace(a) - if a == "" || seen[a] || len(out) >= noderec.MaxAdvertisedIPs || net.ParseIP(a) == nil { + if a == "" || seen[a] || len(out) >= noderec.MaxAdvertisedIPs || !dialable(a) { return } seen[a] = true diff --git a/services/shared/netpick/netpick_test.go b/services/shared/netpick/netpick_test.go index 23380385..721682e1 100644 --- a/services/shared/netpick/netpick_test.go +++ b/services/shared/netpick/netpick_test.go @@ -54,7 +54,10 @@ func TestRankRemote_PrivateBeatsPublic(t *testing.T) { } func TestRankRemote_DropsUnparseableAndStableTie(t *testing.T) { - got := RankRemote([]string{"not-an-ip", "192.168.0.9", "192.168.0.3", ""}) + // "not an ip" holds a space and "gpu:11434" a colon; neither is a legal DNS + // name, so neither survives. A bare "not-an-ip" WOULD survive — it is a valid + // single-label name — which is the point of the hostname admission. + got := RankRemote([]string{"not an ip", "gpu:11434", "192.168.0.9", "192.168.0.3", ""}) want := []string{"192.168.0.3", "192.168.0.9"} // equal score -> string order, junk dropped if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { t.Fatalf("RankRemote = %v, want %v", got, want) @@ -70,7 +73,7 @@ func TestPrimary_TXTWins(t *testing.T) { } func TestPrimary_InvalidTXTFallsBackToRanked(t *testing.T) { - if got := Primary([]string{"ip=garbage", "other=x"}, []string{"10.0.0.1", "192.168.1.5"}); got != "10.0.0.1" { + if got := Primary([]string{"ip=not a name", "other=x"}, []string{"10.0.0.1", "192.168.1.5"}); got != "10.0.0.1" { t.Fatalf("Primary fallback = %q, want the top-ranked advertised address", got) } } @@ -79,7 +82,7 @@ func TestPrimary_Empty(t *testing.T) { if got := Primary(nil, nil); got != "" { t.Fatalf("Primary(nil,nil) = %q, want empty", got) } - if got := Primary(nil, []string{"junk"}); got != "" { + if got := Primary(nil, []string{"not a name"}); got != "" { t.Fatalf("Primary with only junk = %q, want empty", got) } } @@ -94,7 +97,7 @@ func TestIPFromTXT(t *testing.T) { } func TestIPsFromTXT(t *testing.T) { - got := IPsFromTXT([]string{"uuid=abc", "ips=10.172.54.70,192.168.240.2, 192.168.240.6 ,junk"}) + got := IPsFromTXT([]string{"uuid=abc", "ips=10.172.54.70,192.168.240.2, 192.168.240.6 ,not a name"}) want := []string{"10.172.54.70", "192.168.240.2", "192.168.240.6"} if len(got) != len(want) { t.Fatalf("IPsFromTXT = %v, want %v (junk dropped, whitespace trimmed)", got, want) @@ -520,3 +523,87 @@ func TestRouteSourceIP_Smoke(t *testing.T) { t.Fatalf("routeSourceIP = %q, want empty or a non-loopback IPv4", got) } } + +// --- Overlay networks: a peer reachable only over an encrypted overlay --- +// +// A Tailscale tailnet carries no multicast, so such a peer is never discovered; +// it is added by address. Its addresses are a 100.64/10 CGNAT literal, an IPv6 +// ULA out of fd7a:115c:a1e0::/48, or a MagicDNS name. Each of those is demoted +// by scoreIP relative to a LAN literal — deliberately, because a LAN peer's own +// literal is the better answer when both exist — but a node whose ONLY address +// is one of them must still be dialable. Demotion, never exclusion. + +func TestCandidates_CGNATOnlyNodeIsDialable(t *testing.T) { + got := Candidates([]string{"ip=100.101.102.103"}, []string{"100.101.102.103"}) + if len(got) != 1 || got[0] != "100.101.102.103" { + t.Fatalf("Candidates for a CGNAT-only node = %v, want [100.101.102.103]", got) + } + if Primary([]string{"ip=100.101.102.103"}, nil) != "100.101.102.103" { + t.Fatal("Primary must answer a CGNAT-only node's single address") + } +} + +func TestCandidates_IPv6ULAOnlyNodeIsDialable(t *testing.T) { + const ula = "fd7a:115c:a1e0::1701:b2c3" // Tailscale's IPv6 ULA prefix + got := Candidates([]string{"ip=" + ula}, []string{ula}) + if len(got) != 1 || got[0] != ula { + t.Fatalf("Candidates for a ULA-only node = %v, want [%s]", got, ula) + } +} + +func TestCandidates_HostnameOnlyNodeIsDialable(t *testing.T) { + const magicDNS = "gpu-box.tail1234.ts.net" + got := Candidates(nil, []string{magicDNS}) + if len(got) != 1 || got[0] != magicDNS { + t.Fatalf("Candidates for a MagicDNS-only node = %v, want [%s]", got, magicDNS) + } + if Primary([]string{"ip=" + magicDNS}, nil) != magicDNS { + t.Fatal("Primary must answer a node whose canonical address is a DNS name") + } +} + +func TestRankRemote_LANLiteralStillOutranksOverlayAddresses(t *testing.T) { + got := RankRemote([]string{"gpu-box.tail1234.ts.net", "100.101.102.103", "192.168.1.10"}) + if len(got) != 3 || got[0] != "192.168.1.10" { + t.Fatalf("RankRemote = %v, want the LAN literal first", got) + } + // A name outranks a CGNAT literal: it re-resolves, so it survives the + // renumbering that strands the literal. + if got[1] != "gpu-box.tail1234.ts.net" || got[2] != "100.101.102.103" { + t.Fatalf("RankRemote = %v, want the name ahead of the CGNAT literal", got) + } +} + +func TestHostname(t *testing.T) { + valid := []string{ + "gpu-box.tail1234.ts.net", + "gpu-box", + "node.local", + "node.local.", // one trailing root dot is canonical + "a1-b2.example.com", + } + for _, h := range valid { + if !Hostname(h) { + t.Errorf("Hostname(%q) = false, want true", h) + } + } + invalid := []string{ + "", + "192.168.1.1", // an IP literal is not a name + "fd7a:115c:a1e0::1", // nor is an IPv6 literal + "gpu-box.tail1234.ts.net:14318", // a colon: ports are appended by the caller + "gpu box", // space + "-lead.example", // leading hyphen in a label + "trail-.example", // trailing hyphen in a label + "a..b", // empty label + "under_score.example", // underscore + ".", // root alone + strings.Repeat("a", 64), // label over 63 bytes + strings.Repeat("a.", 200), // name over 253 bytes + } + for _, h := range invalid { + if Hostname(h) { + t.Errorf("Hostname(%q) = true, want false", h) + } + } +} diff --git a/services/shared/noderec/noderec.go b/services/shared/noderec/noderec.go index 112e7fe3..1c865c55 100644 --- a/services/shared/noderec/noderec.go +++ b/services/shared/noderec/noderec.go @@ -10,7 +10,7 @@ // whose TXT map carries a schema version, the node's identity, its LAN address, // and one compact key per local service port, e.g.: // -// v=1;uuid=;cluster-uuid=;ip=192.168.1.10;ni=14318;ol=11434;lm=1234;er=14319;wl=14320;cl=14321;em=14322 +// v=1;uuid=;cluster-uuid=;ip=192.168.1.10;ni=14318;ol=11434;lm=1234;vl=8000;sg=30000;er=14319;wl=14320;cl=14321;em=14322 // // Design decisions this package encodes: // - SRV port is a fixed, NON-authoritative constant; consumers ignore it and @@ -87,6 +87,16 @@ const ( ServiceNodeInfo ServiceKey = "ni" ServiceOllama ServiceKey = "ol" ServiceLMStudio ServiceKey = "lm" + // ServiceVLLM is a node's vLLM engine, reached through the same + // OpenAI-compatible proxy that fronts LM Studio. A node may advertise lm, vl + // and sg at once (all point at that one proxy port); which engine owns a given + // model comes from the engine-manager model attribution, not from the key. + ServiceVLLM ServiceKey = "vl" + // ServiceSGLang is a node's SGLang engine, reached through that same proxy. + // Like lm and vl, its advertised value is the proxy's listen port and never + // the engine's own, so a node running all three still projects to a single + // routing target. + ServiceSGLang ServiceKey = "sg" ServiceErrors ServiceKey = "er" ServiceWorkload ServiceKey = "wl" ServiceCluster ServiceKey = "cl" @@ -104,9 +114,9 @@ const ( // serviceKeyOrder is the deterministic emit order for service ports in TXT. var serviceKeyOrder = []ServiceKey{ - ServiceNodeInfo, ServiceOllama, ServiceLMStudio, - ServiceErrors, ServiceWorkload, ServiceCluster, ServiceEngineManager, - ServiceEngineControl, + ServiceNodeInfo, ServiceOllama, ServiceLMStudio, ServiceVLLM, + ServiceSGLang, ServiceErrors, ServiceWorkload, ServiceCluster, + ServiceEngineManager, ServiceEngineControl, } // Transport is the connection policy for a service, derived (not advertised). @@ -326,6 +336,23 @@ const ( // record keeps its last observed value indefinitely. MethodSetClusterIdentity = "nodeinfo:set-cluster-identity" + // MethodSetServices tells nvpair-node-info which services this node runs and + // on which ports — the same {key: port} set the broker registers with + // nvpair-node-scanner — so it can report them on /v1/node-info. + // + // It exists because that set is otherwise published only on this host's mDNS + // record, and multicast does not cross a routed or overlay network. A peer on + // a Tailscale tailnet never sees the record, so without this it can learn + // that a node exists (it was typed in) but not that the node is a PAIR node, + // nor where its engine-manager, proxies or cluster manager listen. node-info + // is the one inter-node surface kept plain, which makes it the one place such + // a peer can ask. + // + // The broker owns the set (it is the process that assigns and re-assigns + // those ports) and re-pushes it on every change, so node-info reports one + // live value rather than deriving a second one. + MethodSetServices = "nodeinfo:set-services" + // NotifyObservedAddresses is nvpair-node-info -> broker: the local addresses // peers have actually reached this node on, learned from its own accepted // connections. @@ -398,6 +425,14 @@ type ClusterIdentityParams struct { ClusterUUID string `json:"clusterUuid"` } +// ServicesParams carries this node's whole {service: port} set for +// MethodSetServices. The set is always sent complete rather than as a delta: a +// service that stopped is expressed by its key being absent, which is the same +// thing an unregister means on the discovery record. +type ServicesParams struct { + Services map[ServiceKey]int `json:"services"` +} + // ObservedAddressesParams carries the local addresses remote peers have reached // this node on, for NotifyObservedAddresses and MethodSetObservedAddresses. The // set is always complete: a receiver replaces what it holds, so an address a peer diff --git a/services/shared/noderec/noderec_test.go b/services/shared/noderec/noderec_test.go index 8b0757e5..10086933 100644 --- a/services/shared/noderec/noderec_test.go +++ b/services/shared/noderec/noderec_test.go @@ -130,6 +130,32 @@ func TestTXTDeterministicOrder(t *testing.T) { } } +// TestTXTEmitsEveryEngineKeyInOrder pins where the OpenAI-compatible engine +// keys sit in the emit order. All three name one node's single proxy port, so a +// record carrying them is what a peer running LM Studio, vLLM and SGLang at once +// publishes; the order is part of the wire format's determinism guarantee, and a +// key appended to serviceKeyOrder in the wrong place would move a key nothing +// else in the tree notices. +func TestTXTEmitsEveryEngineKeyInOrder(t *testing.T) { + r := NodeRecord{ + HostUUID: "h", + Services: map[ServiceKey]int{ + ServiceSGLang: 1234, + ServiceOllama: 11434, + ServiceLMStudio: 1234, + ServiceVLLM: 1234, + }, + } + got := r.TXT() + want := []string{"v=1", "uuid=h", "ol=11434", "lm=1234", "vl=1234", "sg=1234"} + if !reflect.DeepEqual(got, want) { + t.Fatalf("TXT order = %v, want %v", got, want) + } + if p, ok := ParseTXT(got).Port(ServiceSGLang); !ok || p != 1234 { + t.Errorf("sg round-trip = %d,%v want 1234,true", p, ok) + } +} + func TestTXTDefaultsSchema(t *testing.T) { r := NodeRecord{HostUUID: "h", Services: map[ServiceKey]int{}} if got := r.TXT()[0]; got != "v="+SchemaVersion { @@ -147,6 +173,8 @@ func TestTransportPolicy(t *testing.T) { {ServiceNodeInfo, TransportPlain, false, false}, {ServiceOllama, TransportPlain, false, false}, {ServiceLMStudio, TransportPlain, false, false}, + {ServiceVLLM, TransportPlain, false, false}, + {ServiceSGLang, TransportPlain, false, false}, {ServiceEngineManager, TransportPlain, false, false}, {ServiceErrors, TransportMTLSWhenClustered, true, false}, {ServiceWorkload, TransportMTLSWhenClustered, true, false}, diff --git a/services/shared/splitlisten/splitlisten_test.go b/services/shared/splitlisten/splitlisten_test.go index 52e0412a..758a9be4 100644 --- a/services/shared/splitlisten/splitlisten_test.go +++ b/services/shared/splitlisten/splitlisten_test.go @@ -237,11 +237,11 @@ func testServerTLSConfig(t *testing.T) *tls.Config { // scriptedListener is a net.Listener whose Accept behavior is driven by acceptFn. type scriptedListener struct { - addr net.Addr - acceptFn func(call int) (net.Conn, error) - mu sync.Mutex - calls int - closed chan struct{} + addr net.Addr + acceptFn func(call int) (net.Conn, error) + mu sync.Mutex + calls int + closed chan struct{} closeOnce sync.Once } diff --git a/services/tests/broker_test.go b/services/tests/broker_test.go index 22944f5b..235b9015 100644 --- a/services/tests/broker_test.go +++ b/services/tests/broker_test.go @@ -41,9 +41,12 @@ import ( type availableNode struct { ID string `json:"id"` Name string `json:"name"` + HostUUID string `json:"hostUuid"` IPAddress string `json:"ipAddress"` Port int `json:"port"` LastSeen int64 `json:"lastSeen"` + Trusted bool `json:"trusted"` + Clustered bool `json:"clustered"` Models []string `json:"models,omitempty"` ModelsByEngine map[string][]string `json:"modelsByEngine,omitempty"` LoadedByEngine map[string][]string `json:"loadedByEngine,omitempty"` diff --git a/services/tests/manual_pair_peer_test.go b/services/tests/manual_pair_peer_test.go new file mode 100644 index 00000000..7ac36bf1 --- /dev/null +++ b/services/tests/manual_pair_peer_test.go @@ -0,0 +1,496 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +// Cross-process gate for a PAIR peer that can only be reached by a typed +// address — the shape of every peer on a Tailscale tailnet, where no multicast +// crosses and nothing is ever discovered. +// +// Node A is a real broker with its real workers. Node B is a real ollama-proxy +// serving cluster mTLS in front of a fake engine, plus the two HTTP surfaces a +// peer exposes: a plain node-info reporting its identity and service map, and a +// pin-gated engine manager serving its model list. The two nodes cross-pin, as +// they would after pairing. +// +// A is then told about B with nothing but `node/add`, and must end up treating it +// exactly as if it had discovered it: +// +// - B appears in A's directory with its identity, hardware and models, the +// models having been read over cluster mTLS from B's engine manager; +// - A's proxy routes an inference request to B over cluster mTLS and B's +// engine serves it; +// - and A never opens a plaintext connection to B's engine ports. That is the +// regression gate: on a PAIR node those ports are proxy facades that refuse +// plaintext, so probing them reported a healthy peer as having no engines. +// +// Two real brokers cannot share one loopback — every broker-owned port is a +// compiled-in constant — so B is a stub peer on ephemeral ports, addressed +// through node/add's per-service port overrides. Everything the assertions turn +// on (the mTLS identities, the pins, the proxy, the transport choice) is real. + +package tests + +import ( + "bytes" + "crypto/tls" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "os" + "os/exec" + "path/filepath" + "strconv" + "sync/atomic" + "testing" + "time" + + "nvpair-shared/clustertrust" + "nvpair-shared/jsonrpc" +) + +// connectionTrap is a listener that accepts and immediately closes, counting +// every connection. It stands in for a port A must never touch. +type connectionTrap struct { + port int + count atomic.Int32 +} + +func startConnectionTrap(t *testing.T) *connectionTrap { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("trap listen: %v", err) + } + trap := &connectionTrap{port: ln.Addr().(*net.TCPAddr).Port} + go func() { + for { + conn, err := ln.Accept() + if err != nil { + return + } + trap.count.Add(1) + _ = conn.Close() + } + }() + t.Cleanup(func() { _ = ln.Close() }) + return trap +} + +// startPeerNodeInfo serves B's /v1/node-info in plaintext, the way a PAIR node +// under a broker does. Its service map is what identifies B as a PAIR node and +// tells A where B's proxy and engine manager listen. +func startPeerNodeInfo(t *testing.T, hostUUID, clusterUUID string, services map[string]int) int { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("node-info listen: %v", err) + } + port := ln.Addr().(*net.TCPAddr).Port + services["ni"] = port + mux := http.NewServeMux() + mux.HandleFunc("/v1/node-info", func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "GPUs": []map[string]any{{"name": "NVIDIA GeForce RTX 4090", "utilization_percent": 11}}, + "telemetryValid": true, + "msSince": 120, + "hostUuid": hostUUID, + "clusterUuid": clusterUUID, + "services": services, + }) + }) + srv := &http.Server{Handler: mux} + go func() { _ = srv.Serve(ln) }() + t.Cleanup(func() { _ = srv.Close() }) + return port +} + +// startPeerEngineManager serves B's /v1/models over cluster mTLS, refusing any +// caller B does not pin — the same gate the real engine manager applies. +func startPeerEngineManager(t *testing.T, mesh *clustertrust.Mesh, models []string) int { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("engine-manager listen: %v", err) + } + port := ln.Addr().(*net.TCPAddr).Port + mux := http.NewServeMux() + mux.HandleFunc("/v1/models", func(w http.ResponseWriter, r *http.Request) { + if _, ok := mesh.VerifyClientPin(r); !ok { + http.Error(w, "forbidden: not a pinned cluster peer", http.StatusForbidden) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "models": models, + "modelsByEngine": map[string][]string{"ollama": models}, + "loadedByEngine": map[string][]string{"ollama": {}}, + }) + }) + cfg := mesh.ServerTLSConfig() + if cfg == nil { + t.Fatal("the stub peer must be able to serve cluster mTLS") + } + srv := &http.Server{Handler: mux, TLSConfig: cfg} + go func() { _ = srv.Serve(tls.NewListener(ln, cfg)) }() + t.Cleanup(func() { _ = srv.Close() }) + return port +} + +// brokerProc drives a real broker over stdio. +type brokerProc struct { + t *testing.T + cmd *exec.Cmd + stdin io.WriteCloser + msgs <-chan jsonrpc.Message + buf []jsonrpc.Message + nextID int +} + +// clusterManagerPort is the broker-owned cluster-manager port. It is a +// compiled-in constant, so only one broker on this machine can hold it, and a +// test that needs its *own* broker's pairing listener has to wait for whatever +// held it last to let go. +const clusterManagerPort = 14321 + +// awaitPortFree blocks until nothing accepts on addr. The broker's ports are +// fixed constants and a previous test's broker can still be tearing its workers +// down when the next one starts; without this the new broker's cluster-manager +// silently fails to bind and a pairing completion is posted to the wrong process. +func awaitPortFree(t *testing.T, addr string) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for time.Now().Before(deadline) { + conn, err := net.DialTimeout("tcp", addr, time.Second) + if err != nil { + return + } + _ = conn.Close() + time.Sleep(200 * time.Millisecond) + } + t.Fatalf("%s was still held after 30s; a previous test's broker has not exited", addr) +} + +// awaitPortListening blocks until addr accepts, so a pairing is not sent before +// the listener that has to answer it exists. +func awaitPortListening(t *testing.T, addr string) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for time.Now().Before(deadline) { + conn, err := net.DialTimeout("tcp", addr, time.Second) + if err == nil { + _ = conn.Close() + return + } + time.Sleep(200 * time.Millisecond) + } + t.Fatalf("nothing is listening on %s after 30s", addr) +} + +func startBrokerWithClusterDir(t *testing.T, clusterDir string) *brokerProc { + t.Helper() + cfg := t.TempDir() + cmd := exec.Command(brokerBin, + "--cluster-dir", clusterDir, + "--scanner-path", scannerBin, + "--node-info-path", nodeInfoBin, + "--proxy-path", proxyBin, + "--lmstudio-proxy-path", lmstudioProxyBin, + "--workload-manager-path", workloadMgrBin, + "--errors-path", errorsBin, + "--engine-manager-path", engineMgrBin, + "--manual-nodes-path", manualNodesBin, + "--settings-path", nodeSettingsBin, + "--cluster-manager-path", clusterMgrBin, + "--scheduler-path", schedulerBin, + ) + cmd.Env = append(os.Environ(), + "HOME="+cfg, "XDG_CONFIG_HOME="+cfg, "APPDATA="+cfg, "LOCALAPPDATA="+cfg, + ) + cmd.Stderr = os.Stderr + stdin, err := cmd.StdinPipe() + if err != nil { + t.Fatalf("broker stdin pipe: %v", err) + } + stdout, err := cmd.StdoutPipe() + if err != nil { + t.Fatalf("broker stdout pipe: %v", err) + } + if err := cmd.Start(); err != nil { + t.Fatalf("start broker: %v", err) + } + b := &brokerProc{t: t, cmd: cmd, stdin: stdin, msgs: startMsgReader(stdout), nextID: 1} + t.Cleanup(func() { + _ = stdin.Close() + done := make(chan struct{}) + go func() { _ = cmd.Wait(); close(done) }() + select { + case <-done: + case <-time.After(8 * time.Second): + _ = cmd.Process.Kill() + <-done + } + }) + return b +} + +func (b *brokerProc) call(method string, params any) jsonrpc.Message { + b.t.Helper() + id := b.nextID + b.nextID++ + req := map[string]any{"jsonrpc": "2.0", "id": id, "method": method} + if params != nil { + req["params"] = params + } + raw, _ := json.Marshal(req) + raw = append(raw, '\n') + if _, err := b.stdin.Write(raw); err != nil { + b.t.Fatalf("write %s: %v", method, err) + } + resp := b.pump(func(m jsonrpc.Message) bool { return m.Method == "" && idEquals(m.ID, id) }, 20*time.Second) + if resp.Error != nil { + b.t.Fatalf("%s returned error %d: %s", method, resp.Error.Code, resp.Error.Message) + } + return resp +} + +func (b *brokerProc) pump(want func(jsonrpc.Message) bool, timeout time.Duration) jsonrpc.Message { + b.t.Helper() + for i, m := range b.buf { + if want(m) { + b.buf = append(b.buf[:i], b.buf[i+1:]...) + return m + } + } + timer := time.NewTimer(timeout) + defer timer.Stop() + for { + select { + case m, ok := <-b.msgs: + if !ok { + b.t.Fatal("broker stdout closed unexpectedly") + } + if want(m) { + return m + } + b.buf = append(b.buf, m) + case <-timer.C: + b.t.Fatal("timed out waiting on a broker message") + } + } +} + +func (b *brokerProc) nodes() []availableNode { + b.t.Helper() + var result struct { + Nodes []availableNode `json:"nodes"` + } + if err := json.Unmarshal(b.call("discovery:get-nodes", nil).Result, &result); err != nil { + b.t.Fatalf("decode discovery:get-nodes: %v", err) + } + return result.Nodes +} + +// awaitNode polls the directory until a node matching want appears. +func (b *brokerProc) awaitNode(hostUUID string, want func(availableNode) bool, why string) availableNode { + b.t.Helper() + deadline := time.Now().Add(45 * time.Second) + var last []availableNode + for time.Now().Before(deadline) { + last = b.nodes() + for _, n := range last { + if n.HostUUID == hostUUID && want(n) { + return n + } + } + time.Sleep(300 * time.Millisecond) + } + b.t.Fatalf("node %s never %s; directory = %+v", hostUUID, why, last) + return availableNode{} +} + +func TestManualPairPeerBehavesLikeADiscoveredPeer(t *testing.T) { + baseA, baseB := t.TempDir(), t.TempDir() + dirA := filepath.Join(baseA, "cluster") + dirB := filepath.Join(baseB, "cluster") + + // B runs a real cluster-manager so the pairing below is the real EAP-NOOB + // exchange, and the pins both sides end up holding are the ones it mints. + portB := freePort(t) + cmB := startCM(t, baseB, portB) + t.Cleanup(cmB.stop) + + // A is a real broker with its own real cluster-manager. Its pairing port is a + // compiled-in constant, so wait for a previous test's broker to release it + // before starting, and for this one's listener to exist before pairing. + awaitPortFree(t, net.JoinHostPort("127.0.0.1", strconv.Itoa(clusterManagerPort))) + brokerA := startBrokerWithClusterDir(t, dirA) + brokerA.call("discovery:subscribe", nil) + brokerA.call("proxy:subscribe", nil) + awaitPortListening(t, net.JoinHostPort("127.0.0.1", strconv.Itoa(clusterManagerPort))) + awaitPortListening(t, net.JoinHostPort("127.0.0.1", strconv.Itoa(portB))) + + // Pair A to B by address and port alone — no nodeId, because A has not + // discovered B and never will. This is the invite an operator sends after + // typing a peer's overlay address into Add node. + pairFromBroker(t, brokerA, cmB, portB) + + bInfo := decodeResult[cmNodeID](t, cmB.call("cluster:get-node-id", nil)) + if bInfo.NodeUUID == "" { + t.Fatal("the peer must report a node uuid after pairing") + } + uuidB := bInfo.NodeUUID + + meshB := clustertrust.Open(dirB) + meshB.Refresh() + if !meshB.Clustered() { + t.Fatal("the peer must read as a cluster member after pairing") + } + + // B: a real ollama-proxy over cluster mTLS in front of a fake engine. + engineHost, enginePort, generates := startFakeOllama(t) + proxyB := startProxyProc(t, dirB, freePort(t)) + t.Cleanup(proxyB.stop) + proxyB.setLocalBackend("ollama", engineHost, enginePort, true) + + // B's two HTTP surfaces, and two traps standing where a bare host's engines + // would be. On a PAIR node those ports carry proxy facades; A must not touch + // them. + emPort := startPeerEngineManager(t, meshB, []string{"m:latest"}) + ollamaTrap := startConnectionTrap(t) + lmStudioTrap := startConnectionTrap(t) + niPort := startPeerNodeInfo(t, uuidB, uuidB, map[string]int{ + "ol": proxyB.port, + "em": emPort, + "cl": portB, + }) + + // Everything A is told about B. No discovery, no mDNS: one address and the + // ports its services sit on. + brokerA.call("node/add", map[string]any{ + "address": "127.0.0.1", + "name": "peer-b", + "ports": map[string]any{ + "node_info": niPort, + "ollama": ollamaTrap.port, + "lmstudio": lmStudioTrap.port, + }, + }) + + t.Run("B joins A's directory as a trusted peer with its models", func(t *testing.T) { + node := brokerA.awaitNode(uuidB, func(n availableNode) bool { + return len(n.Models) > 0 + }, "appeared with a model list") + if !node.Trusted { + t.Errorf("node.trusted = false, want true: A holds B's pin") + } + if !node.Clustered { + t.Errorf("node.clustered = false, want true: B reported a cluster principal") + } + if node.Models[0] != "m:latest" { + t.Errorf("models = %v, want B's inventory read over cluster mTLS", node.Models) + } + if node.IPAddress != "127.0.0.1" { + t.Errorf("ipAddress = %q, want the address the operator typed", node.IPAddress) + } + }) + + t.Run("A routes inference to B over cluster mTLS", func(t *testing.T) { + port := awaitRoutablePeerAtProxy(t, brokerA, uuidB) + before := atomic.LoadInt32(generates) + resp := postInference(t, fmt.Sprintf("http://127.0.0.1:%d/api/generate", port), + []byte(`{"model":"m:latest","prompt":"hi","stream":false}`)) + defer resp.Body.Close() + payload, _ := io.ReadAll(resp.Body) + if resp.StatusCode != http.StatusOK { + t.Fatalf("A->B inference status = %d, body = %s", resp.StatusCode, payload) + } + if !bytes.Contains(payload, []byte("hello from the backend")) { + t.Fatalf("response did not come from B's engine: %s", payload) + } + if got := atomic.LoadInt32(generates); got != before+1 { + t.Fatalf("B engine generate count = %d, want %d", got, before+1) + } + }) + + // The regression gate. A PAIR node's engine ports are proxy facades that + // refuse plaintext from anything but loopback; probing them reports a healthy + // peer as having no engines, and bridging them gives the proxy a second route + // to the same node that can only ever 403. + t.Run("A never opens a plaintext connection to B's engine ports", func(t *testing.T) { + if n := ollamaTrap.count.Load(); n != 0 { + t.Errorf("%d plaintext connections to B's ollama port, want 0", n) + } + if n := lmStudioTrap.count.Load(); n != 0 { + t.Errorf("%d plaintext connections to B's lmstudio port, want 0", n) + } + }) +} + +// pairFromBroker drives a PIN pairing from the broker's own cluster-manager to a +// standalone peer, addressed by nothing but an address and a port. It is the +// invite an operator sends after typing a peer's overlay address into Add node: +// no nodeId, because the peer was never discovered and never will be. +func pairFromBroker(t *testing.T, b *brokerProc, joiner *cmProc, joinerPort int) { + t.Helper() + var invite struct { + InviteID string `json:"inviteId"` + State string `json:"state"` + Pin string `json:"pin"` + } + resp := b.call("cluster:invite-node", map[string]any{ + "address": "127.0.0.1", + "port": joinerPort, + }) + if err := json.Unmarshal(resp.Result, &invite); err != nil { + t.Fatalf("decode invite: %v", err) + } + if invite.State != "pending" || len(invite.Pin) != 6 { + t.Fatalf("invite-node = %+v, want pending with a six-digit pin", invite) + } + joiner.waitNotify("cluster:invite-received") + var accepted struct { + State string `json:"state"` + } + if err := json.Unmarshal(joiner.call("cluster:respond-to-invite", map[string]any{ + "inviteId": invite.InviteID, "accept": true, "pin": invite.Pin, + }).Result, &accepted); err != nil { + t.Fatalf("decode respond-to-invite: %v", err) + } + if accepted.State != "paired" { + t.Fatalf("respond-to-invite state = %q, want paired", accepted.State) + } +} + +// awaitRoutablePeerAtProxy waits until the broker's ollama-proxy lists the peer +// as a routing target and returns the proxy's listen port. +func awaitRoutablePeerAtProxy(t *testing.T, b *brokerProc, hostUUID string) int { + t.Helper() + deadline := time.Now().Add(45 * time.Second) + var lastNodes json.RawMessage + for time.Now().Before(deadline) { + var status struct { + Port int `json:"port"` + } + if err := json.Unmarshal(b.call("proxy:get-status", nil).Result, &status); err == nil && status.Port > 0 { + var listed struct { + Nodes []struct { + ID string `json:"id"` + } `json:"nodes"` + } + resp := b.call("proxy:nodes/list", nil) + lastNodes = resp.Result + if json.Unmarshal(resp.Result, &listed) == nil { + for _, n := range listed.Nodes { + if n.ID == hostUUID { + return status.Port + } + } + } + } + time.Sleep(300 * time.Millisecond) + } + t.Fatalf("the proxy never listed %s as a routing target; nodes = %s", hostUUID, lastNodes) + return 0 +} diff --git a/services/tests/model_routing_interop_test.go b/services/tests/model_routing_interop_test.go index b6c13dd7..ea26a3e2 100644 --- a/services/tests/model_routing_interop_test.go +++ b/services/tests/model_routing_interop_test.go @@ -64,12 +64,16 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) { type proxyCase struct { name string rpcPrefix string - path string - port int + // engine names which engine the proxy's manual entry represents. The + // OpenAI proxy fronts more than one, so its manual overlay is keyed by + // (engine, node) and the field is required. + engine string + path string + port int } cases := []proxyCase{ - {name: "ollama", rpcPrefix: "proxy", path: "/api/chat", port: ollamaPort}, - {name: "lmstudio", rpcPrefix: "lmstudio-proxy", path: "/v1/chat/completions", port: lmstudioPort}, + {name: "ollama", rpcPrefix: "proxy", engine: "ollama", path: "/api/chat", port: ollamaPort}, + {name: "lmstudio", rpcPrefix: "lmstudio-proxy", engine: "lmstudio", path: "/v1/chat/completions", port: lmstudioPort}, } client := &http.Client{Timeout: 5 * time.Second} t.Cleanup(client.CloseIdleConnections) @@ -99,6 +103,7 @@ func TestStrictModelRoutingAcrossProcesses(t *testing.T) { for _, node := range nodes { params := map[string]any{ "id": node.id, + "engine": tc.engine, "host": "127.0.0.1", "port": node.port, "addresses": []string{"127.0.0.1"}, diff --git a/services/tests/scheduler_interop_test.go b/services/tests/scheduler_interop_test.go index 102f44cf..fb829663 100644 --- a/services/tests/scheduler_interop_test.go +++ b/services/tests/scheduler_interop_test.go @@ -583,7 +583,7 @@ func TestLMStudioProxyIgnoresPriorityNodesAbsentFromDiscovery(t *testing.T) { proxyPort := waitLMStudioProxyReady(t, stdin, msgs, 15*time.Second) writeRawFrame(t, stdin, fmt.Sprintf( - `{"jsonrpc":"2.0","id":90,"method":"lmstudio-proxy:node/add-manual","params":{"id":"real-lm","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["chat-model"]}}`, + `{"jsonrpc":"2.0","id":90,"method":"lmstudio-proxy:node/add-manual","params":{"id":"real-lm","engine":"lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["chat-model"]}}`, realPort, )) if resp := waitForResponse(t, msgs, 5*time.Second); resp.Error != nil { diff --git a/services/tests/workload_identity_interop_test.go b/services/tests/workload_identity_interop_test.go index f3879a74..a0f904e1 100644 --- a/services/tests/workload_identity_interop_test.go +++ b/services/tests/workload_identity_interop_test.go @@ -93,7 +93,7 @@ func TestWorkloadCrossEngineIdentityDistinct(t *testing.T) { waitForResponse(t, msgs, 5*time.Second) writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":51,"method":"proxy:node/select","params":{"id":"fake-ollama"}}`) waitForResponse(t, msgs, 5*time.Second) - writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":52,"method":"lmstudio-proxy:node/add-manual","params":{"id":"fake-lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, lmstudioPort)) + writeRawFrame(t, stdin, fmt.Sprintf(`{"jsonrpc":"2.0","id":52,"method":"lmstudio-proxy:node/add-manual","params":{"id":"fake-lmstudio","engine":"lmstudio","host":"127.0.0.1","port":%d,"addresses":["127.0.0.1"],"models":["crossengine-model"]}}`, lmstudioPort)) waitForResponse(t, msgs, 5*time.Second) writeRawFrame(t, stdin, `{"jsonrpc":"2.0","id":53,"method":"lmstudio-proxy:node/select","params":{"id":"fake-lmstudio"}}`) waitForResponse(t, msgs, 5*time.Second) diff --git a/services/versions.json b/services/versions.json index 29d8c230..1c2be141 100644 --- a/services/versions.json +++ b/services/versions.json @@ -1,20 +1,20 @@ { "$comment": "Single source of truth for all version numbers. See VERSIONING.md for bump rules.", - "product": "0.91.7", - "installer": "0.91.7", + "product": "0.93.0", + "installer": "0.93.0", "components": { - "ollama-proxy": "0.26.2", - "lmstudio-proxy": "0.16.2", - "nvpair-node-info": "0.13.3", - "nvpair-node-scanner": "0.20.3", - "nvpair-manual-nodes": "0.11.1", - "nvpair-workload-manager": "0.13.3", - "nvpair-errors": "0.7.4", + "ollama-proxy": "0.27.0", + "lmstudio-proxy": "1.1.0", + "nvpair-node-info": "0.14.0", + "nvpair-node-scanner": "0.20.5", + "nvpair-manual-nodes": "0.13.0", + "nvpair-workload-manager": "0.13.5", + "nvpair-errors": "0.7.6", "nvpair-node-settings": "1.0.4", - "nvpair-ui-broker": "0.40.2", - "nvpair-engine-manager": "0.17.4", - "nvpair-cluster-manager": "1.1.4", - "nvpair-job-scheduler": "0.4.1", - "nvpair-tui": "0.7.2" + "nvpair-ui-broker": "0.42.0", + "nvpair-engine-manager": "0.19.0", + "nvpair-cluster-manager": "1.1.5", + "nvpair-job-scheduler": "0.6.0", + "nvpair-tui": "0.7.4" } }